On Jun 17, 2009, at 08:43 , Peter Soetens wrote:
> https://www.fmtc.be/bugzilla/orocos/show_bug.cgi?id=673
>
>
> Peter Soetens <peter [..] ...> changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> CC| |peter [..] ...
>
>
>
>
> --- Comment #1 from Peter Soetens <peter [..] ...>
> 2009-06-17 14:43:43 ---
> (In reply to comment #0)
>> Warning ... this is a weird one, and sorry that it is so long.
>
> Thanks for the detailed report !
>
>>
>> When mixing different versions of the same CORBA library (ACE/TAO),
>> we get
>> either CORBA exceptions or lockups when getting methods from remote
>> components.
>> Our system (one deployer and one GUI program) works fine when used
>> on the same
>> machine. The fault appears to be independant of where the name
>> service is
>> actually running. We realise that mixing different versions might
>> cause
>> problems, and we realise that the error may well be in the CORBA
>> library., but
>> given that only certain Orocos calls cause this we wanted to submit
>> this.
>
> Different TAO versions shouldn't be a problem, even different ORBs
> shouldn't
> be.
>
>>
>> Computers involved: Mac with ACE 5.6.6 (compiled from source),
>> Ubuntu Hardy
>> with ACE 5.4.7, and Debian Lenny + Xenomai with ACE 5.6.3.
>>
>> SCENARIO 1: Deployer on Mac and GUI on either Linux computer.
>> Everything works.
>>
>> SCENARIO 2: Deployer on Ubuntu and GUI attempted to run on Mac.
>>
>> GUI lockups with the backtrace below. The GUI has already looked up
>> the remote
>> components it is interested in, connected to the 10+ ports of
>> interest, and is
>> working through a connecting a set of methods. The fault occurs on
>> the first
>> method that has a parameter - we rearranged the code and both 1 and
>> 2 parameter
>> methods cause the lockup. The GUI had already succsesfully
>> connected six (6)
>> void-parameter methods, though each one does output the
>>
>> 88.061 [ ERROR ][CorbaFallBackProtocol] Failing Corba::Any >> creation of type >> void. >>
>> error message within the deployer. So void-parameter methods aren't
>> a problem.
>
> One important aspect that might be involved here is that for methods
> having 1
> or more arguments, the deployer calls your GUI back, to ask it where
> the
> parameters will be stored. It is vastly important that when your GUI
> calls your
> deployer, it does so with a hostname/ip address that can be resolved
> from the
> deployer. If that doesn't work out, you get exactly the behaviour
> you are
> describing. To see how your GUI advertises itself, use this tool on
> the (on
> screen or file) IOR's on the gui side:
>
> tao-catior -f gui.ior
>
> The hostname must be resolvable from your deployer.
We're not using hostnames. We are using IP addresses along with the "-
ORBDottedDecimalAddresses 1" flag to the naming service.
The GUI doesn't drop an IOR file. Taking the IOR that is printed by
the Deployer (upon GUI connection) as the GUI IOR, then both this IOR
and the deployer.ior file produce the same IP address. Actually all
our IOR files show the same IP address.
Naming service running with
Naming_Service -m 0 -ORBDottedDecimalAddresses 1 -ORBListenEndpoints iiop://192.168.1.12:2809 -ORBDaemon
Corba environment variable set to
NameServiceIOR=corbaloc:iiop:192.168.1.12:2809/NameService
Should the taskbrowser leave an IOR file?
Passing "-ORBDottedDecimalAddresses 1" to both Deployer and GUI makes
no difference.
> If this goes well, the second thing to check is if the ORB thread
> hasn't been
> killed. This might happen on Xenomai systems with watchdog enabled.
> You can
> check this by inspecting the output of 'dmesg'. If it reads 'watchdog
> killed...'
> we need to look further into this.
This might affect the Xenomai scenarios, but won't be affecting the
non-RT Linux. I'll check.
> I'm sorry this is blowing up in your face, but it still smells like
> a network
> configuration problem since on-same-host setups work correctly... I
> wouldn't
> blame TAO versions yet.
That would be a nice easy fix!!! It did strike me as odd that
different ACE/TAO versions would cause this issue ... not the way that
Corba is *supposed* to work.
Stephen
[Bug 673] Unable mix different CORBA library versions
On Wed, Jun 17, 2009 at 15:50, S Roderick <kiwi [dot] net [..] ...> wrote:
> On Jun 17, 2009, at 08:43 , Peter Soetens wrote:
>
> https://www.fmtc.be/bugzilla/orocos/show_bug.cgi?id=673
>>
>>
>> Peter Soetens <peter [..] ...> changed:
>>
>> What |Removed |Added
>>
>> ----------------------------------------------------------------------------
>> CC| |peter [..] ...
>>
>>
>>
>>
>> --- Comment #1 from Peter Soetens <peter [..] ...> 2009-06-17
>> 14:43:43 ---
>> (In reply to comment #0)
>>
>>> Warning ... this is a weird one, and sorry that it is so long.
>>>
>>
>> Thanks for the detailed report !
>>
>>
>>> When mixing different versions of the same CORBA library (ACE/TAO), we
>>> get
>>> either CORBA exceptions or lockups when getting methods from remote
>>> components.
>>> Our system (one deployer and one GUI program) works fine when used on the
>>> same
>>> machine. The fault appears to be independant of where the name service is
>>> actually running. We realise that mixing different versions might cause
>>> problems, and we realise that the error may well be in the CORBA
>>> library., but
>>> given that only certain Orocos calls cause this we wanted to submit this.
>>>
>>
>> Different TAO versions shouldn't be a problem, even different ORBs
>> shouldn't
>> be.
>>
>>
>>> Computers involved: Mac with ACE 5.6.6 (compiled from source), Ubuntu
>>> Hardy
>>> with ACE 5.4.7, and Debian Lenny + Xenomai with ACE 5.6.3.
>>>
>>> SCENARIO 1: Deployer on Mac and GUI on either Linux computer. Everything
>>> works.
>>>
>>> SCENARIO 2: Deployer on Ubuntu and GUI attempted to run on Mac.
>>>
>>> GUI lockups with the backtrace below. The GUI has already looked up the
>>> remote
>>> components it is interested in, connected to the 10+ ports of interest,
>>> and is
>>> working through a connecting a set of methods. The fault occurs on the
>>> first
>>> method that has a parameter - we rearranged the code and both 1 and 2
>>> parameter
>>> methods cause the lockup. The GUI had already succsesfully connected six
>>> (6)
>>> void-parameter methods, though each one does output the
>>>
>>> error message within the deployer. So void-parameter methods aren't a
>>> problem.
>>>
>>
>> One important aspect that might be involved here is that for methods
>> having 1
>> or more arguments, the deployer calls your GUI back, to ask it where the
>> parameters will be stored. It is vastly important that when your GUI calls
>> your
>> deployer, it does so with a hostname/ip address that can be resolved from
>> the
>> deployer. If that doesn't work out, you get exactly the behaviour you are
>> describing. To see how your GUI advertises itself, use this tool on the
>> (on
>> screen or file) IOR's on the gui side:
>>
>> tao-catior -f gui.ior
>>
>> The hostname must be resolvable from your deployer.
>>
>
> We're not using hostnames. We are using IP addresses along with the
> "-ORBDottedDecimalAddresses 1" flag to the naming service.
All right, we should put this in a corba faq :-) Just for the record: you're
not going through a firewall/gateway/router do you ?
>
>
> The GUI doesn't drop an IOR file. Taking the IOR that is printed by the
> Deployer (upon GUI connection) as the GUI IOR, then both this IOR and the
> deployer.ior file produce the same IP address. Actually all our IOR files
> show the same IP address.
>
> Naming service running with
>
>
It's not a naming service issue. The naming service is only used to get a
pointer to the TaskContext. Once you have that, everything is done using
object references (IORs), also the callback from deployer to GUI. That's why
the ior of your GUI is so *very* important to be correct, and printing one
and inspecting it manually is a first thing to make sure if it does work on
the same host, but not across hosts. You can in your GUI print the ior by
doing:
The deployer must be able to make sense out of this, also with respect to
ipv4/ipv6.
>
> Corba environment variable set to
>
>
>
> Should the taskbrowser leave an IOR file?
No, since it's purely a client.
Peter
[Bug 673] Unable mix different CORBA library versions
On Jun 17, 2009, at 15:48 , Peter Soetens wrote:
> On Wed, Jun 17, 2009 at 15:50, S Roderick <kiwi [dot] net [..] ...> wrote:
> On Jun 17, 2009, at 08:43 , Peter Soetens wrote:
>
> https://www.fmtc.be/bugzilla/orocos/show_bug.cgi?id=673
>
>
> Peter Soetens <peter [..] ...> changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> CC| |
> peter [..] ...
>
>
>
>
> --- Comment #1 from Peter Soetens <peter [..] ...>
> 2009-06-17 14:43:43 ---
> (In reply to comment #0)
> Warning ... this is a weird one, and sorry that it is so long.
>
> Thanks for the detailed report !
>
>
> When mixing different versions of the same CORBA library (ACE/TAO),
> we get
> either CORBA exceptions or lockups when getting methods from remote
> components.
> Our system (one deployer and one GUI program) works fine when used
> on the same
> machine. The fault appears to be independant of where the name
> service is
> actually running. We realise that mixing different versions might
> cause
> problems, and we realise that the error may well be in the CORBA
> library., but
> given that only certain Orocos calls cause this we wanted to submit
> this.
>
> Different TAO versions shouldn't be a problem, even different ORBs
> shouldn't
> be.
>
>
> Computers involved: Mac with ACE 5.6.6 (compiled from source),
> Ubuntu Hardy
> with ACE 5.4.7, and Debian Lenny + Xenomai with ACE 5.6.3.
>
> SCENARIO 1: Deployer on Mac and GUI on either Linux computer.
> Everything works.
>
> SCENARIO 2: Deployer on Ubuntu and GUI attempted to run on Mac.
>
> GUI lockups with the backtrace below. The GUI has already looked up
> the remote
> components it is interested in, connected to the 10+ ports of
> interest, and is
> working through a connecting a set of methods. The fault occurs on
> the first
> method that has a parameter - we rearranged the code and both 1 and
> 2 parameter
> methods cause the lockup. The GUI had already succsesfully connected
> six (6)
> void-parameter methods, though each one does output the
>
> error message within the deployer. So void-parameter methods aren't
> a problem.
>
> One important aspect that might be involved here is that for methods
> having 1
> or more arguments, the deployer calls your GUI back, to ask it where
> the
> parameters will be stored. It is vastly important that when your GUI
> calls your
> deployer, it does so with a hostname/ip address that can be resolved
> from the
> deployer. If that doesn't work out, you get exactly the behaviour
> you are
> describing. To see how your GUI advertises itself, use this tool on
> the (on
> screen or file) IOR's on the gui side:
>
> tao-catior -f gui.ior
>
> The hostname must be resolvable from your deployer.
>
> We're not using hostnames. We are using IP addresses along with the
> "-ORBDottedDecimalAddresses 1" flag to the naming service.
>
> All right, we should put this in a corba faq :-) Just for the
> record: you're not going through a firewall/gateway/router do you ?
I had the computers all on the same ethernet switch, behind a
commercial firewall router, which is behind many non-commercial grade
firewalls. Hence the IP adrs's. I'm trying to reproduce at home ...
with OmniORB ... and are seeing different behaviour but still not
right. More investigation to do.
> The GUI doesn't drop an IOR file. Taking the IOR that is printed by
> the Deployer (upon GUI connection) as the GUI IOR, then both this
> IOR and the deployer.ior file produce the same IP address. Actually
> all our IOR files show the same IP address.
>
>
> Naming service running with
>
>
>
> It's not a naming service issue. The naming service is only used to
> get a pointer to the TaskContext. Once you have that, everything is
> done using object references (IORs), also the callback from deployer
> to GUI. That's why the ior of your GUI is so *very* important to be
> correct, and printing one and inspecting it manually is a first
> thing to make sure if it does work on the same host, but not across
> hosts. You can in your GUI print the ior by doing:
>
>
>
> The deployer must be able to make sense out of this, also with
> respect to ipv4/ipv6.
What is "guiref.in()"? Can I get this same data out of a
ControlTaskProxy, or from something else within RTT's corba code?
I'm trying to boil it down to a simple test case to reproduce results.
Will forward that if I have success.
> Corba environment variable set to
>
>
>
> Should the taskbrowser leave an IOR file?
>
> No, since it's purely a client.
Hold on, if taskbrowser is "purely a client" and has no IOR, then why
is the GUI not a client? The GUI is *not* running components, but has
simply started up Orocos and created several ports/methods/etc which
it manually connects. Isn't it just a client too?
S
[Bug 673] Unable mix different CORBA library versions
On Thu, Jun 18, 2009 at 02:41, Stephen Roderick<kiwi [dot] net [..] ...> wrote:
> On Jun 17, 2009, at 15:48 , Peter Soetens wrote:
>
>
>
> The deployer must be able to make sense out of this, also with respect to
> ipv4/ipv6.
>
> What is "guiref.in()"? Can I get this same data out of a ControlTaskProxy,
> or from something else within RTT's corba code?
It's indeed a bit more complex than I thought. When you create a remote method
from the proxy class, CORBA objects are created to hold the arguments
of the method invocation. These wait until the method is executed, at
which point in time they are called for their values. This is overly
complex (and the call back is a performance hit) and I'll try to do
away with that system in RTT 2.0. But that doesn't help us now, the
point is that the IORs of these CORBA objects on the gui side must be
checked.
These are unfortunately hidden by the implementation, but with a patch
on ExpressionServer.cpp in orocos-rtt/src/corba, we can print them.
See the attachment.
> I'm trying to boil it down to a simple test case to reproduce results. Will
> forward that if I have success.
The 'errors' about the void failing are not errors. I'll fix this in
the CORBA layer to remove these messages in case of voids.
>>
>> Corba environment variable set to
>>
>>
>>
>> Should the taskbrowser leave an IOR file?
>
> No, since it's purely a client.
>
> Hold on, if taskbrowser is "purely a client" and has no IOR, then why is the
> GUI not a client? The GUI is *not* running components, but has simply
> started up Orocos and created several ports/methods/etc which it manually
> connects. Isn't it just a client too?
I was oversimplifying. Your GUI is a server too for data arguments,
see the comments above.
But it indeed won't print an IOR as long as no TaskContext +
ControlTaskServer have been created on the GUI side (that's actually
an alternative to the patch, to get a print-out of an IOR, just use a
TaskContext a; ControlTaskServer::Create(&a, false); in your gui
code.)
Peter
[Bug 673] Unable mix different CORBA library versions
On Jun 18, 2009, at 04:35 , Peter Soetens wrote:
> On Thu, Jun 18, 2009 at 02:41, Stephen Roderick<kiwi [dot] net [..] ...>
> wrote:
>> On Jun 17, 2009, at 15:48 , Peter Soetens wrote:
>>
>>
>>
>> The deployer must be able to make sense out of this, also with
>> respect to
>> ipv4/ipv6.
>>
>> What is "guiref.in()"? Can I get this same data out of a
>> ControlTaskProxy,
>> or from something else within RTT's corba code?
>
> It's indeed a bit more complex than I thought. When you create a
> remote method
> from the proxy class, CORBA objects are created to hold the arguments
> of the method invocation. These wait until the method is executed, at
> which point in time they are called for their values. This is overly
> complex (and the call back is a performance hit) and I'll try to do
> away with that system in RTT 2.0. But that doesn't help us now, the
> point is that the IORs of these CORBA objects on the gui side must be
> checked.
>
> These are unfortunately hidden by the implementation, but with a patch
> on ExpressionServer.cpp in orocos-rtt/src/corba, we can print them.
> See the attachment.
>
>
>> I'm trying to boil it down to a simple test case to reproduce
>> results. Will
>> forward that if I have success.
>
> The 'errors' about the void failing are not errors. I'll fix this in
> the CORBA layer to remove these messages in case of voids.
>
>>>
>>> Corba environment variable set to
>>>
>>>
>>>
>>> Should the taskbrowser leave an IOR file?
>>
>> No, since it's purely a client.
>>
>> Hold on, if taskbrowser is "purely a client" and has no IOR, then
>> why is the
>> GUI not a client? The GUI is *not* running components, but has simply
>> started up Orocos and created several ports/methods/etc which it
>> manually
>> connects. Isn't it just a client too?
>
> I was oversimplifying. Your GUI is a server too for data arguments,
> see the comments above.
> But it indeed won't print an IOR as long as no TaskContext +
> ControlTaskServer have been created on the GUI side (that's actually
> an alternative to the patch, to get a print-out of an IOR, just use a
> TaskContext a; ControlTaskServer::Create(&a, false); in your gui
> code.)
Interesting new data points ...
Using omniORB on iMac and two Linux VM's (one on iMac and one on Mac
laptop). No firewalls. DNS name server running on iMac. The following
is always running the deployer within Mac laptop Linux VM.
- Using host names for any part of the naming service, with even the
smallest error in the naming service, will cause similar faults to
what I've been seeing. Network misconfiguration == Bad (as you'd
expect).
- OmniORB does not completely honor the NameServiceIOR environment
variable when IP addresses are used.
- Running the GUI on the iMac with only the environment variable set
(so no command line IOR parameters) and it works. The same thing on
the Linux VM on the iMac fails. The ctaskbrowser shows the same
behaviour.
- BUT if you run either the GUI or taskbrowser as such
ctaskbrowser-gnulinux Deployer -- -ORBInitRef
NameService==corbaloc:iiop:192.168.12.132:2809/NameService
it works. For some reason, with IP addresses for the name service you
must use the -ORBInitRef command line parameter to keep everything
happy; the environment variables isn't enough (in general).
I still have to try this on my work setup, which is ACE/TAO anyway,
but this is interesting data to have.
Peter, I've attached the IOR data below for completeness. The data
from the iMac is a mess due to multiple virtual network interfaces.
Also note that the omniORB Linux packages comes with a omniorb.cfg
file, whereas the Macports omniORB does not. Not sure if this makes a
difference.
Goes in the corba FAQ did someone say ...? :-)
Stephen
With the incorrect DNS you get the following, which is failing exactly
at the point that the method-with-parameters is being looked up
Linux VM
iMac VM
[Bug 673] Unable mix different CORBA library versions
On Thu, Jun 18, 2009 at 16:22, Stephen Roderick <kiwi [dot] net [..] ...> wrote:
>
> ctaskbrowser-gnulinux Deployer -- -ORBInitRef NameService==corbaloc:iiop:
> 192.168.12.132:2809/NameService
>
> it works. For some reason, with IP addresses for the name service you must
> use the -ORBInitRef command line parameter to keep everything happy; the
> environment variables isn't enough (in general).
Just a quick hunch: isn't the env variable TAO only ?
I'll need the TAO IORs from your not working setup to check the object
references...
Peter
[Bug 673] Unable mix different CORBA library versions
On Jun 18, 2009, at 10:47 , Peter Soetens wrote:
> On Thu, Jun 18, 2009 at 16:22, Stephen Roderick <kiwi [dot] net [..] ...>
> wrote:
>
> ctaskbrowser-gnulinux Deployer -- -ORBInitRef
> NameService==corbaloc:iiop:192.168.12.132:2809/NameService
>
> it works. For some reason, with IP addresses for the name service
> you must use the -ORBInitRef command line parameter to keep
> everything happy; the environment variables isn't enough (in general).
>
> Just a quick hunch: isn't the env variable TAO only ?
>
> I'll need the TAO IORs from your not working setup to check the
> object references...
Yep!
SOLUTION - ACE/TAO
Set environment variable on both machines
"NameServiceIOR=corbaloc:iiop:192.168.12.132:2809/NameService"
machine 1$ ps auxw | grep Nam
... Naming_Service -m 0 -ORBDottedDecimalAddresses 1 -
ORBListenEndpoints iiop://192.168.1.12:2809 -ORBDaemon
machine1 $ deployer-corba-XXX -s xxx.xml -- -ORBDottedDecimalAddresses 1
machine2 $ ./demogui -ORBDottedDecimalAddresses 1
SOLUTION - OmniORB
No environment variable.
machine 1$ ps auxw | grep Nam
... /usr/bin/omniNames -errlog /var/log/omniorb4-nameserver.log -start
machine1 $ deployer-corba-XXX -s xxx.xml -- -ORBInitRef
NameService=corbaloc:iiop:192.168.12.131:2809/NameService -
ORBDottedDecimalAddresses 1
machine2 $ ./demogui -ORBInitRef NameService=corbaloc:iiop:
192.168.12.131:2809/NameService
NB OmniORB won't accept "-ORBDottedDecimalAddresses 1" thru
ControlTaskProxy::InitOrb(), but will for
ControlTaskServer::InitOrb(). Go figure!?!
I will write a CORBA FAQ page for this. This was insane ...
Peter, you still want the TAO IORs from non-working setup?
HTH
S
[Bug 673] Unable mix different CORBA library versions
On Jun 18, 2009, at 11:28 , Stephen Roderick wrote:
> I will write a CORBA FAQ page for this. This was insane ...
Slight corrections at
http://orocos.org/wiki/rtt/frequently-asked-questions-faq/using-corba
I consider this closed, unless you want to look at the TAO IORs, Peter?
Stephen
[Bug 673] Unable mix different CORBA library versions
On Thu, Jun 18, 2009 at 17:28, Stephen Roderick<kiwi [dot] net [..] ...> wrote:
> On Jun 18, 2009, at 10:47 , Peter Soetens wrote:
>
> On Thu, Jun 18, 2009 at 16:22, Stephen Roderick <kiwi [dot] net [..] ...> wrote:
>>
>> ctaskbrowser-gnulinux Deployer -- -ORBInitRef
>> NameService==corbaloc:iiop:192.168.12.132:2809/NameService
>>
>> it works. For some reason, with IP addresses for the name service you must
>> use the -ORBInitRef command line parameter to keep everything happy; the
>> environment variables isn't enough (in general).
>
> Just a quick hunch: isn't the env variable TAO only ?
>
> I'll need the TAO IORs from your not working setup to check the object
> references...
>
> Yep!
>
> SOLUTION - ACE/TAO
> Set environment variable on both machines
> "NameServiceIOR=corbaloc:iiop:192.168.12.132:2809/NameService"
> machine 1$ ps auxw | grep Nam
> ... Naming_Service -m 0 -ORBDottedDecimalAddresses 1 -ORBListenEndpoints
> iiop://192.168.1.12:2809 -ORBDaemon
> machine1 $ deployer-corba-XXX -s xxx.xml -- -ORBDottedDecimalAddresses 1
> machine2 $ ./demogui -ORBDottedDecimalAddresses 1
>
> SOLUTION - OmniORB
> No environment variable.
> machine 1$ ps auxw | grep Nam
> ... /usr/bin/omniNames -errlog /var/log/omniorb4-nameserver.log -start
> machine1 $ deployer-corba-XXX -s xxx.xml -- -ORBInitRef
> NameService=corbaloc:iiop:192.168.12.131:2809/NameService -ORBDottedDecimalAddresses
> 1
> machine2 $ ./demogui -ORBInitRef
> NameService=corbaloc:iiop:192.168.12.131:2809/NameService
> NB OmniORB won't accept
> "-ORBDottedDecimalAddresses 1" thru ControlTaskProxy::InitOrb(), but will
> for ControlTaskServer::InitOrb(). Go figure!?!
>
> I will write a CORBA FAQ page for this. This was insane ...
Thanks for keeping us posted !
> Peter, you still want the TAO IORs from non-working setup?
No, never mind.
Peter