warn_upon_switch on leaving deployer-xenomai

Submitted by gah on Mon, 2012-04-02 10:50

Orocos-users

Exiting deployer-xenomai after running two simple producer-consumer components (exchanging unsigned ints) dumps a lot of stacktraces to the console, such as this one:

=====

/home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
[0xb77b6400]
/usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
/home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
/home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
/home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p<RTT::Activity>::dispose()+0x18)[0xb76e0fc8]
/home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
/home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
/home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
/home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
/home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]

=====

The rest is on pastebin: http://pastebin.com/HpG1qazj.

Have I done something wrong with compiling orocos_toolchain perhaps? I've noticed that my consumer has increasing mode switches in its MSW column in 'xenomai/stat'. There is no logging / printf / std::cout / new / delete going on, so I am not sure what is causing this.

Any ideas?

thank you,

system info:

Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
GCC 4.4.3-4ubuntu5.1
ROS electric
Xenomai 2.6.0 (dlopen enabled)
orocos_toolchain git checkout
OROCOS_TARGET=xenomai

warn_upon_switch on leaving deployer-xenomai

Submitted by peter on Mon, 2012-04-02 16:12.

2012/4/2 g ah <gaohml [..] ...>:
>
> Exiting deployer-xenomai after running two simple producer-consumer components (exchanging unsigned ints) dumps a lot of stacktraces to the console, such as this one:
>
> =====
>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
> [0xb77b6400]
> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p<RTT::Activity>::dispose()+0x18)[0xb76e0fc8]
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>
> =====
>
> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>
> Have I done something wrong with compiling orocos_toolchain perhaps? I've noticed that my consumer has increasing mode switches in its MSW column in 'xenomai/stat'. There is no logging / printf / std::cout / new / delete going on, so I am not sure what is causing this.
>
> Any ideas?

It shouldn't happen at all since warnings on switch should be enabled
only explicitly. Your cases are 'valid' switches: they happen during
the cleanup of Xenomai and Orocos services, which are indeed not
real-time, and cause a Xenomai switch.

I'm wondering how this mechanism got enabled in your setup. In RTT we
have two functions: rtos_enable_rt_warning() and
rtos_disable_rt_warning() which explicitly turn on and off this
behavior. Maybe you used such a function in configure or start Hook ?
In that case, you enabled it for the main() thread, which is never
real-time in Orocos. Use it in your updateHook or make your
startHook() an 'OwnThread' operation and put the warning code in there
using

this->addOperation("start", &TaskContext::start,this, OwnThread);

in your constructor.

>
>
> thank you,
>
>
>
>
> system info:
>
> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
> GCC 4.4.3-4ubuntu5.1
> ROS electric
> Xenomai 2.6.0 (dlopen enabled)
> orocos_toolchain git checkout
> OROCOS_TARGET=xenomai

Peter

warn_upon_switch on leaving deployer-xenomai

Submitted by gah on Tue, 2012-04-03 10:12.

Peter Soetens wrote:
> 2012/4/2 g ah <gaohml [..] ...>:
>> Exiting deployer-xenomai after running two simple producer-consumer components (exchanging unsigned ints) dumps a lot of stacktraces to the console, such as this one:
>>
>> =====
>>
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>> [0xb77b6400]
>> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p<RTT::Activity>::dispose()+0x18)[0xb76e0fc8]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>>
>> =====
>>
>> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>>
>> Have I done something wrong with compiling orocos_toolchain perhaps? I've noticed that my consumer has increasing mode switches in its MSW column in 'xenomai/stat'. There is no logging / printf / std::cout / new / delete going on, so I am not sure what is causing this.
>>
>> Any ideas?
>
> It shouldn't happen at all since warnings on switch should be enabled
> only explicitly. Your cases are 'valid' switches: they happen during
> the cleanup of Xenomai and Orocos services, which are indeed not
> real-time, and cause a Xenomai switch.
>
> I'm wondering how this mechanism got enabled in your setup. In RTT we
> have two functions: rtos_enable_rt_warning() and
> rtos_disable_rt_warning() which explicitly turn on and off this
> behavior. Maybe you used such a function in configure or start Hook ?
> In that case, you enabled it for the main() thread, which is never
> real-time in Orocos. Use it in your updateHook or make your
> startHook() an 'OwnThread' operation and put the warning code in there
> using
>
> this->addOperation("start", &TaskContext::start,this, OwnThread);
>
> in your constructor.

I have been trying to figure this out for some time now this morning, but am a bit at a loss.

I've added some more stack trace dumping to rtos_{en,dis}able_rt_warning() for xenomai and other functions, as well as dumping of RT_TASK_INFO checking for T_WARNSW.

Afaict T_WARNSW is always 0 (or at least at the places I've checked, which were the orocos functions right before the warn_upon_switch was triggered).

As I have no debug build as of yet, gdb didn't provide much information either.

>> system info:
>>
>> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
>> GCC 4.4.3-4ubuntu5.1
>> ROS electric
>> Xenomai 2.6.0 (dlopen enabled)
>> orocos_toolchain git checkout
>> OROCOS_TARGET=xenomai

Willy Lambert mentioned something about not many people using xenomai 2.6.0 with rtt 2.x: would trying an earlier version help?

warn_upon_switch on leaving deployer-xenomai

Submitted by Charles Lesire-... on Tue, 2012-04-03 10:16.

2012/4/3 g ah <gaohml [..] ...>

>
> Peter Soetens wrote:
> > 2012/4/2 g ah <gaohml [..] ...>:
> >> Exiting deployer-xenomai after running two simple producer-consumer
> components (exchanging unsigned ints) dumps a lot of stacktraces to the
> console, such as this one:
> >>
> >> =====
> >>
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
> >> [0xb77b6400]
> >> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p<RTT::Activity>::dispose()+0x18)[0xb76e0fc8]
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
> >> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
> >>
> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
> >>
> >> =====
> >>
> >> The rest is on pastebin: http://pastebin.com/HpG1qazj.
> >>
> >> Have I done something wrong with compiling orocos_toolchain perhaps?
> I've noticed that my consumer has increasing mode switches in its MSW
> column in 'xenomai/stat'. There is no logging / printf / std::cout / new /
> delete going on, so I am not sure what is causing this.
> >>
> >> Any ideas?
> >
> > It shouldn't happen at all since warnings on switch should be enabled
> > only explicitly. Your cases are 'valid' switches: they happen during
> > the cleanup of Xenomai and Orocos services, which are indeed not
> > real-time, and cause a Xenomai switch.
> >
> > I'm wondering how this mechanism got enabled in your setup. In RTT we
> > have two functions: rtos_enable_rt_warning() and
> > rtos_disable_rt_warning() which explicitly turn on and off this
> > behavior. Maybe you used such a function in configure or start Hook ?
> > In that case, you enabled it for the main() thread, which is never
> > real-time in Orocos. Use it in your updateHook or make your
> > startHook() an 'OwnThread' operation and put the warning code in there
> > using
> >
> > this->addOperation("start", &TaskContext::start,this, OwnThread);
> >
> > in your constructor.
>
> I have been trying to figure this out for some time now this morning, but
> am a bit at a loss.
>
> I've added some more stack trace dumping to rtos_{en,dis}able_rt_warning()
> for xenomai and other functions, as well as dumping of RT_TASK_INFO
> checking for T_WARNSW.
>
> Afaict T_WARNSW is always 0 (or at least at the places I've checked, which
> were the orocos functions right before the warn_upon_switch was triggered).
>
> As I have no debug build as of yet, gdb didn't provide much information
> either.
>
>
> >> system info:
> >>
> >> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
> >> GCC 4.4.3-4ubuntu5.1
> >> ROS electric
> >> Xenomai 2.6.0 (dlopen enabled)
> >> orocos_toolchain git checkout
> >> OROCOS_TARGET=xenomai
>
> Willy Lambert mentioned something about not many people using xenomai
> 2.6.0 with rtt 2.x: would trying an earlier version help?
>
>
I think it may come from some Debug option set to ON in Xenomai itself.
How did you compiled and installed Xenomai? From linux packages?

Charles.

warn_upon_switch on leaving deployer-xenomai

Submitted by gah on Tue, 2012-04-03 10:48.

Charles Lesire-Cabaniols wrote:
> 2012/4/3 g ah
>
>> Peter Soetens wrote:
>>> 2012/4/2 g ah :
>>>> Exiting deployer-xenomai after running two simple producer-consumer
>> components (exchanging unsigned ints) dumps a lot of stacktraces to the
>> console, such as this one:
>>>> =====
>>>>
>>>>
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>>>> [0xb77b6400]
>>>> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>>>>
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p::dispose()+0x18)[0xb76e0fc8]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>>>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>>>>
>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>>>> =====
>>>>
>>>> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>>>>
>>>> Have I done something wrong with compiling orocos_toolchain perhaps?
>> I've noticed that my consumer has increasing mode switches in its MSW
>> column in 'xenomai/stat'. There is no logging / printf / std::cout / new /
>> delete going on, so I am not sure what is causing this.
>>>> Any ideas?
>>> It shouldn't happen at all since warnings on switch should be enabled
>>> only explicitly. Your cases are 'valid' switches: they happen during
>>> the cleanup of Xenomai and Orocos services, which are indeed not
>>> real-time, and cause a Xenomai switch.
>>>
>>> I'm wondering how this mechanism got enabled in your setup. In RTT we
>>> have two functions: rtos_enable_rt_warning() and
>>> rtos_disable_rt_warning() which explicitly turn on and off this
>>> behavior. Maybe you used such a function in configure or start Hook ?
>>> In that case, you enabled it for the main() thread, which is never
>>> real-time in Orocos. Use it in your updateHook or make your
>>> startHook() an 'OwnThread' operation and put the warning code in there
>>> using
>>>
>>> this->addOperation("start", &TaskContext::start,this, OwnThread);
>>>
>>> in your constructor.
>> I have been trying to figure this out for some time now this morning, but
>> am a bit at a loss.
>>
>> I've added some more stack trace dumping to rtos_{en,dis}able_rt_warning()
>> for xenomai and other functions, as well as dumping of RT_TASK_INFO
>> checking for T_WARNSW.
>>
>> Afaict T_WARNSW is always 0 (or at least at the places I've checked, which
>> were the orocos functions right before the warn_upon_switch was triggered).
>>
>> As I have no debug build as of yet, gdb didn't provide much information
>> either.
>>
>>
>>>> system info:
>>>>
>>>> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
>>>> GCC 4.4.3-4ubuntu5.1
>>>> ROS electric
>>>> Xenomai 2.6.0 (dlopen enabled)
>>>> orocos_toolchain git checkout
>>>> OROCOS_TARGET=xenomai
>> Willy Lambert mentioned something about not many people using xenomai
>> 2.6.0 with rtt 2.x: would trying an earlier version help?
>>
>>
> I think it may come from some Debug option set to ON in Xenomai itself.
> How did you compiled and installed Xenomai? From linux packages?

Xenomai was compiled from source tarball, xenomai-2.6.0.tar.bz2 from xenomai.org. I then updated 'debian/rules' as per http://lists.mech.kuleuven.be/pipermail/orocos-users/2011-August/004287.... ("CONFIG_OPTS += --enable-dlopen-skins").

After that, it was basically following http://xenomai.org/index.php/Building_Debian_packages#from_a_tar.bz2_arc....

Kernel configuration was mostly standard as well. Only thing I changed was the rt_heap size (from default to 4MB).

"grep -i xeno_opt_debug .config" shows:

CONFIG_XENO_OPT_DEBUG=y # doesn't do anything by itself
CONFIG_XENO_OPT_DEBUG_NATIVE=y # warns about auto-clean operations
CONFIG_XENO_OPT_DEBUG_POSIX=y # same, but for posix skin resources
CONFIG_XENO_OPT_DEBUG_RTDM_APPL=y # same, but for rtdm skin
CONFIG_XENO_OPT_DEBUG_XNLOCK=y # auto enabled on smp systems

then typical "fakeroot make-kpkg ... kernel_headers kernel_image" and installation.

anything stand out?

warn_upon_switch on leaving deployer-xenomai

Submitted by peter on Tue, 2012-04-03 11:40.

2012/4/3 g ah <gaohml [..] ...>:
>
>
> Charles Lesire-Cabaniols wrote:
>> 2012/4/3 g ah
>>
>>> Peter Soetens wrote:
>>>> 2012/4/2 g ah :
>>>>> Exiting deployer-xenomai after running two simple producer-consumer
>>> components (exchanging unsigned ints) dumps a lot of stacktraces to the
>>> console, such as this one:
>>>>> =====
>>>>>
>>>>>
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>>>>> [0xb77b6400]
>>>>> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>>>>>
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p::dispose()+0x18)[0xb76e0fc8]
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>>>>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>>>>>
>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>>>>> =====
>>>>>
>>>>> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>>>>>
>>>>> Have I done something wrong with compiling orocos_toolchain perhaps?
>>> I've noticed that my consumer has increasing mode switches in its MSW
>>> column in 'xenomai/stat'. There is no logging / printf / std::cout / new /
>>> delete going on, so I am not sure what is causing this.
>>>>> Any ideas?
>>>> It shouldn't happen at all since warnings on switch should be enabled
>>>> only explicitly. Your cases are 'valid' switches: they happen during
>>>> the cleanup of Xenomai and Orocos services, which are indeed not
>>>> real-time, and cause a Xenomai switch.
>>>>
>>>> I'm wondering how this mechanism got enabled in your setup. In RTT we
>>>> have two functions: rtos_enable_rt_warning() and
>>>> rtos_disable_rt_warning() which explicitly turn on and off this
>>>> behavior. Maybe you used such a function in configure or start Hook ?
>>>> In that case, you enabled it for the main() thread, which is never
>>>> real-time in Orocos. Use it in your updateHook or make your
>>>> startHook() an 'OwnThread' operation and put the warning code in there
>>>> using
>>>>
>>>> this->addOperation("start", &TaskContext::start,this, OwnThread);
>>>>
>>>> in your constructor.
>>> I have been trying to figure this out for some time now this morning, but
>>> am a bit at a loss.
>>>
>>> I've added some more stack trace dumping to rtos_{en,dis}able_rt_warning()
>>> for xenomai and other functions, as well as dumping of RT_TASK_INFO
>>> checking for T_WARNSW.
>>>
>>> Afaict T_WARNSW is always 0 (or at least at the places I've checked, which
>>> were the orocos functions right before the warn_upon_switch was triggered).
>>>
>>> As I have no debug build as of yet, gdb didn't provide much information
>>> either.
>>>
>>>
>>>>> system info:
>>>>>
>>>>> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
>>>>> GCC 4.4.3-4ubuntu5.1
>>>>> ROS electric
>>>>> Xenomai 2.6.0 (dlopen enabled)
>>>>> orocos_toolchain git checkout
>>>>> OROCOS_TARGET=xenomai
>>> Willy Lambert mentioned something about not many people using xenomai
>>> 2.6.0 with rtt 2.x: would trying an earlier version help?
>>>
>>>
>> I think it may come from some Debug option set to ON in Xenomai itself.
>> How did you compiled and installed Xenomai? From linux packages?
>
> Xenomai was compiled from source tarball, xenomai-2.6.0.tar.bz2 from xenomai.org. I then updated 'debian/rules' as per http://lists.mech.kuleuven.be/pipermail/orocos-users/2011-August/004287.... ("CONFIG_OPTS += --enable-dlopen-skins").
>
> After that, it was basically following http://xenomai.org/index.php/Building_Debian_packages#from_a_tar.bz2_arc....
>
> Kernel configuration was mostly standard as well. Only thing I changed was the rt_heap size (from default to 4MB).
>
> "grep -i xeno_opt_debug .config" shows:
>
> CONFIG_XENO_OPT_DEBUG=y # doesn't do anything by itself
> CONFIG_XENO_OPT_DEBUG_NATIVE=y # warns about auto-clean operations
> CONFIG_XENO_OPT_DEBUG_POSIX=y # same, but for posix skin resources
> CONFIG_XENO_OPT_DEBUG_RTDM_APPL=y # same, but for rtdm skin
> CONFIG_XENO_OPT_DEBUG_XNLOCK=y # auto enabled on smp systems
>
> then typical "fakeroot make-kpkg ... kernel_headers kernel_image" and installation.
>
> anything stand out?

No. I think it's an Orocos bug. There are two reasons to receive
SIGXCPU: when mlockall is not called or T_WARNSW is passed during
task creation. We definately do the former, but maybe there is a bug
in our recent affinity patches which set this T_WARNSW flag.

I previously noticed that the CPU affinity code for Xenomai is not
working, so this could indicate indeed a problem in that part of the
code. I don't dare to say that it won't be a problem for 2.5.

Peter

warn_upon_switch on leaving deployer-xenomai

Submitted by peter on Tue, 2012-04-03 11:56.

On Tue, Apr 3, 2012 at 1:39 PM, Peter Soetens <peter [..] ...> wrote:
> 2012/4/3 g ah <gaohml [..] ...>:
>>
>>
>> Charles Lesire-Cabaniols wrote:
>>> 2012/4/3 g ah
>>>
>>>> Peter Soetens wrote:
>>>>> 2012/4/2 g ah :
>>>>>> Exiting deployer-xenomai after running two simple producer-consumer
>>>> components (exchanging unsigned ints) dumps a lot of stacktraces to the
>>>> console, such as this one:
>>>>>> =====
>>>>>>
>>>>>>
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>>>>>> [0xb77b6400]
>>>>>> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>>>>>>
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p::dispose()+0x18)[0xb76e0fc8]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>>>>>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>>>>>>
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>>>>>> =====
>>>>>>
>>>>>> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>>>>>>
>>>>>> Have I done something wrong with compiling orocos_toolchain perhaps?
>>>> I've noticed that my consumer has increasing mode switches in its MSW
>>>> column in 'xenomai/stat'. There is no logging / printf / std::cout / new /
>>>> delete going on, so I am not sure what is causing this.
>>>>>> Any ideas?
>>>>> It shouldn't happen at all since warnings on switch should be enabled
>>>>> only explicitly. Your cases are 'valid' switches: they happen during
>>>>> the cleanup of Xenomai and Orocos services, which are indeed not
>>>>> real-time, and cause a Xenomai switch.
>>>>>
>>>>> I'm wondering how this mechanism got enabled in your setup. In RTT we
>>>>> have two functions: rtos_enable_rt_warning() and
>>>>> rtos_disable_rt_warning() which explicitly turn on and off this
>>>>> behavior. Maybe you used such a function in configure or start Hook ?
>>>>> In that case, you enabled it for the main() thread, which is never
>>>>> real-time in Orocos. Use it in your updateHook or make your
>>>>> startHook() an 'OwnThread' operation and put the warning code in there
>>>>> using
>>>>>
>>>>> this->addOperation("start", &TaskContext::start,this, OwnThread);
>>>>>
>>>>> in your constructor.
>>>> I have been trying to figure this out for some time now this morning, but
>>>> am a bit at a loss.
>>>>
>>>> I've added some more stack trace dumping to rtos_{en,dis}able_rt_warning()
>>>> for xenomai and other functions, as well as dumping of RT_TASK_INFO
>>>> checking for T_WARNSW.
>>>>
>>>> Afaict T_WARNSW is always 0 (or at least at the places I've checked, which
>>>> were the orocos functions right before the warn_upon_switch was triggered).
>>>>
>>>> As I have no debug build as of yet, gdb didn't provide much information
>>>> either.
>>>>
>>>>
>>>>>> system info:
>>>>>>
>>>>>> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
>>>>>> GCC 4.4.3-4ubuntu5.1
>>>>>> ROS electric
>>>>>> Xenomai 2.6.0 (dlopen enabled)
>>>>>> orocos_toolchain git checkout
>>>>>> OROCOS_TARGET=xenomai
>>>> Willy Lambert mentioned something about not many people using xenomai
>>>> 2.6.0 with rtt 2.x: would trying an earlier version help?
>>>>
>>>>
>>> I think it may come from some Debug option set to ON in Xenomai itself.
>>> How did you compiled and installed Xenomai? From linux packages?
>>
>> Xenomai was compiled from source tarball, xenomai-2.6.0.tar.bz2 from xenomai.org. I then updated 'debian/rules' as per http://lists.mech.kuleuven.be/pipermail/orocos-users/2011-August/004287.... ("CONFIG_OPTS += --enable-dlopen-skins").
>>
>> After that, it was basically following http://xenomai.org/index.php/Building_Debian_packages#from_a_tar.bz2_arc....
>>
>> Kernel configuration was mostly standard as well. Only thing I changed was the rt_heap size (from default to 4MB).
>>
>> "grep -i xeno_opt_debug .config" shows:
>>
>> CONFIG_XENO_OPT_DEBUG=y # doesn't do anything by itself
>> CONFIG_XENO_OPT_DEBUG_NATIVE=y # warns about auto-clean operations
>> CONFIG_XENO_OPT_DEBUG_POSIX=y # same, but for posix skin resources
>> CONFIG_XENO_OPT_DEBUG_RTDM_APPL=y # same, but for rtdm skin
>> CONFIG_XENO_OPT_DEBUG_XNLOCK=y # auto enabled on smp systems
>>
>> then typical "fakeroot make-kpkg ... kernel_headers kernel_image" and installation.
>>
>> anything stand out?
>
> No. I think it's an Orocos bug. There are two reasons to receive
> SIGXCPU: when mlockall is not called or T_WARNSW is passed during
> task creation. We definately do the former, but maybe there is a bug
> in our recent affinity patches which set this T_WARNSW flag.
>
> I previously noticed that the CPU affinity code for Xenomai is not
> working, so this could indicate indeed a problem in that part of the
> code. I don't dare to say that it won't be a problem for 2.5.

I fixed an issue in the deployer yesterday for Xenomai targets. Did
you check with the latest checkout of git's toolchain-2.5 ?

The relevant patch was:
https://www.gitorious.org/orocos-toolchain/ocl/commit/cf452859869712b8c7...

It's just a long shot, but you never know...

Peter

warn_upon_switch on leaving deployer-xenomai

Submitted by gah on Tue, 2012-04-03 12:36.

Peter Soetens wrote:
> On Tue, Apr 3, 2012 at 1:39 PM, Peter Soetens <peter [..] ...> wrote:
>> 2012/4/3 g ah <gaohml [..] ...>:
>>>
>>> Charles Lesire-Cabaniols wrote:
>>>> 2012/4/3 g ah
>>>>
>>>>> Peter Soetens wrote:
>>>>>> 2012/4/2 g ah :
>>>>>>> Exiting deployer-xenomai after running two simple producer-consumer
>>>>> components (exchanging unsigned ints) dumps a lot of stacktraces to the
>>>>> console, such as this one:
>>>>>>> =====
>>>>>>>
>>>>>>>
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>>>>>>> [0xb77b6400]
>>>>>>> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>>>>>>>
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p::dispose()+0x18)[0xb76e0fc8]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>>>>>>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>>>>>>>
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>>>>>>> =====
>>>>>>>
>>>>>>> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>>>>>>>
>>>>>>> Have I done something wrong with compiling orocos_toolchain perhaps?
>>>>> I've noticed that my consumer has increasing mode switches in its MSW
>>>>> column in 'xenomai/stat'. There is no logging / printf / std::cout / new /
>>>>> delete going on, so I am not sure what is causing this.
>>>>>>> Any ideas?
>>>>>> It shouldn't happen at all since warnings on switch should be enabled
>>>>>> only explicitly. Your cases are 'valid' switches: they happen during
>>>>>> the cleanup of Xenomai and Orocos services, which are indeed not
>>>>>> real-time, and cause a Xenomai switch.
>>>>>>
>>>>>> I'm wondering how this mechanism got enabled in your setup. In RTT we
>>>>>> have two functions: rtos_enable_rt_warning() and
>>>>>> rtos_disable_rt_warning() which explicitly turn on and off this
>>>>>> behavior. Maybe you used such a function in configure or start Hook ?
>>>>>> In that case, you enabled it for the main() thread, which is never
>>>>>> real-time in Orocos. Use it in your updateHook or make your
>>>>>> startHook() an 'OwnThread' operation and put the warning code in there
>>>>>> using
>>>>>>
>>>>>> this->addOperation("start", &TaskContext::start,this, OwnThread);
>>>>>>
>>>>>> in your constructor.
>>>>> I have been trying to figure this out for some time now this morning, but
>>>>> am a bit at a loss.
>>>>>
>>>>> I've added some more stack trace dumping to rtos_{en,dis}able_rt_warning()
>>>>> for xenomai and other functions, as well as dumping of RT_TASK_INFO
>>>>> checking for T_WARNSW.
>>>>>
>>>>> Afaict T_WARNSW is always 0 (or at least at the places I've checked, which
>>>>> were the orocos functions right before the warn_upon_switch was triggered).
>>>>>
>>>>> As I have no debug build as of yet, gdb didn't provide much information
>>>>> either.
>>>>>
>>>>>
>>>>>>> system info:
>>>>>>>
>>>>>>> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
>>>>>>> GCC 4.4.3-4ubuntu5.1
>>>>>>> ROS electric
>>>>>>> Xenomai 2.6.0 (dlopen enabled)
>>>>>>> orocos_toolchain git checkout
>>>>>>> OROCOS_TARGET=xenomai
>>>>> Willy Lambert mentioned something about not many people using xenomai
>>>>> 2.6.0 with rtt 2.x: would trying an earlier version help?
>>>>>
>>>>>
>>>> I think it may come from some Debug option set to ON in Xenomai itself.
>>>> How did you compiled and installed Xenomai? From linux packages?
>>> Xenomai was compiled from source tarball, xenomai-2.6.0.tar.bz2 from xenomai.org. I then updated 'debian/rules' as per http://lists.mech.kuleuven.be/pipermail/orocos-users/2011-August/004287.... ("CONFIG_OPTS += --enable-dlopen-skins").
>>>
>>> After that, it was basically following http://xenomai.org/index.php/Building_Debian_packages#from_a_tar.bz2_arc....
>>>
>>> Kernel configuration was mostly standard as well. Only thing I changed was the rt_heap size (from default to 4MB).
>>>
>>> "grep -i xeno_opt_debug .config" shows:
>>>
>>> CONFIG_XENO_OPT_DEBUG=y # doesn't do anything by itself
>>> CONFIG_XENO_OPT_DEBUG_NATIVE=y # warns about auto-clean operations
>>> CONFIG_XENO_OPT_DEBUG_POSIX=y # same, but for posix skin resources
>>> CONFIG_XENO_OPT_DEBUG_RTDM_APPL=y # same, but for rtdm skin
>>> CONFIG_XENO_OPT_DEBUG_XNLOCK=y # auto enabled on smp systems
>>>
>>> then typical "fakeroot make-kpkg ... kernel_headers kernel_image" and installation.
>>>
>>> anything stand out?
>> No. I think it's an Orocos bug. There are two reasons to receive
>> SIGXCPU: when mlockall is not called or T_WARNSW is passed during
>> task creation. We definately do the former, but maybe there is a bug
>> in our recent affinity patches which set this T_WARNSW flag.
>>
>> I previously noticed that the CPU affinity code for Xenomai is not
>> working, so this could indicate indeed a problem in that part of the
>> code. I don't dare to say that it won't be a problem for 2.5.
>
> I fixed an issue in the deployer yesterday for Xenomai targets. Did
> you check with the latest checkout of git's toolchain-2.5 ?
>
> The relevant patch was:
> https://www.gitorious.org/orocos-toolchain/ocl/commit/cf452859869712b8c7...
>
> It's just a long shot, but you never know...

yes, I've tested using that branch. Same results.

Although I haven't done much, I'll have to postpone trying to locate the cause of this for now. Development can continue on gnulinux.

I can't find any logs, but remember the affinity masks in deployer-xenomai showed ULONG_MAX values for all threads. Not sure if that is of any importance though.

Thanks for now,

PS: on the bright side, I do now have a working ccache + distcc setup (I think). Now to find more builders ..

warn_upon_switch on leaving deployer-xenomai

Submitted by peter on Tue, 2012-04-03 12:44.

2012/4/3 g ah <gaohml [..] ...>:
>
> Peter Soetens wrote:
>> On Tue, Apr 3, 2012 at 1:39 PM, Peter Soetens <peter [..] ...> wrote:
>>> 2012/4/3 g ah <gaohml [..] ...>:
>>>>
>>>> Charles Lesire-Cabaniols wrote:
>>>>> 2012/4/3 g ah
>>>>>
>>>>>> Peter Soetens wrote:
>>>>>>> 2012/4/2 g ah :
>>>>>>>> Exiting deployer-xenomai after running two simple producer-consumer
>>>>>> components (exchanging unsigned ints) dumps a lot of stacktraces to the
>>>>>> console, such as this one:
>>>>>>>> =====
>>>>>>>>
>>>>>>>>
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>>>>>>>> [0xb77b6400]
>>>>>>>> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>>>>>>>>
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p::dispose()+0x18)[0xb76e0fc8]
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>>>>>>>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>>>>>>>>
>>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>>>>>>>> =====
>>>>>>>>
>>>>>>>> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>>>>>>>>
>>>>>>>> Have I done something wrong with compiling orocos_toolchain perhaps?
>>>>>> I've noticed that my consumer has increasing mode switches in its MSW
>>>>>> column in 'xenomai/stat'. There is no logging / printf / std::cout / new /
>>>>>> delete going on, so I am not sure what is causing this.
>>>>>>>> Any ideas?
>>>>>>> It shouldn't happen at all since warnings on switch should be enabled
>>>>>>> only explicitly. Your cases are 'valid' switches: they happen during
>>>>>>> the cleanup of Xenomai and Orocos services, which are indeed not
>>>>>>> real-time, and cause a Xenomai switch.
>>>>>>>
>>>>>>> I'm wondering how this mechanism got enabled in your setup. In RTT we
>>>>>>> have two functions: rtos_enable_rt_warning() and
>>>>>>> rtos_disable_rt_warning() which explicitly turn on and off this
>>>>>>> behavior. Maybe you used such a function in configure or start Hook ?
>>>>>>> In that case, you enabled it for the main() thread, which is never
>>>>>>> real-time in Orocos. Use it in your updateHook or make your
>>>>>>> startHook() an 'OwnThread' operation and put the warning code in there
>>>>>>> using
>>>>>>>
>>>>>>> this->addOperation("start", &TaskContext::start,this, OwnThread);
>>>>>>>
>>>>>>> in your constructor.
>>>>>> I have been trying to figure this out for some time now this morning, but
>>>>>> am a bit at a loss.
>>>>>>
>>>>>> I've added some more stack trace dumping to rtos_{en,dis}able_rt_warning()
>>>>>> for xenomai and other functions, as well as dumping of RT_TASK_INFO
>>>>>> checking for T_WARNSW.
>>>>>>
>>>>>> Afaict T_WARNSW is always 0 (or at least at the places I've checked, which
>>>>>> were the orocos functions right before the warn_upon_switch was triggered).
>>>>>>
>>>>>> As I have no debug build as of yet, gdb didn't provide much information
>>>>>> either.
>>>>>>
>>>>>>
>>>>>>>> system info:
>>>>>>>>
>>>>>>>> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
>>>>>>>> GCC 4.4.3-4ubuntu5.1
>>>>>>>> ROS electric
>>>>>>>> Xenomai 2.6.0 (dlopen enabled)
>>>>>>>> orocos_toolchain git checkout
>>>>>>>> OROCOS_TARGET=xenomai
>>>>>> Willy Lambert mentioned something about not many people using xenomai
>>>>>> 2.6.0 with rtt 2.x: would trying an earlier version help?
>>>>>>
>>>>>>
>>>>> I think it may come from some Debug option set to ON in Xenomai itself.
>>>>> How did you compiled and installed Xenomai? From linux packages?
>>>> Xenomai was compiled from source tarball, xenomai-2.6.0.tar.bz2 from xenomai.org. I then updated 'debian/rules' as per http://lists.mech.kuleuven.be/pipermail/orocos-users/2011-August/004287.... ("CONFIG_OPTS += --enable-dlopen-skins").
>>>>
>>>> After that, it was basically following http://xenomai.org/index.php/Building_Debian_packages#from_a_tar.bz2_arc....
>>>>
>>>> Kernel configuration was mostly standard as well. Only thing I changed was the rt_heap size (from default to 4MB).
>>>>
>>>> "grep -i xeno_opt_debug .config" shows:
>>>>
>>>> CONFIG_XENO_OPT_DEBUG=y # doesn't do anything by itself
>>>> CONFIG_XENO_OPT_DEBUG_NATIVE=y # warns about auto-clean operations
>>>> CONFIG_XENO_OPT_DEBUG_POSIX=y # same, but for posix skin resources
>>>> CONFIG_XENO_OPT_DEBUG_RTDM_APPL=y # same, but for rtdm skin
>>>> CONFIG_XENO_OPT_DEBUG_XNLOCK=y # auto enabled on smp systems
>>>>
>>>> then typical "fakeroot make-kpkg ... kernel_headers kernel_image" and installation.
>>>>
>>>> anything stand out?
>>> No. I think it's an Orocos bug. There are two reasons to receive
>>> SIGXCPU: when mlockall is not called or T_WARNSW is passed during
>>> task creation. We definately do the former, but maybe there is a bug
>>> in our recent affinity patches which set this T_WARNSW flag.
>>>
>>> I previously noticed that the CPU affinity code for Xenomai is not
>>> working, so this could indicate indeed a problem in that part of the
>>> code. I don't dare to say that it won't be a problem for 2.5.
>>
>> I fixed an issue in the deployer yesterday for Xenomai targets. Did
>> you check with the latest checkout of git's toolchain-2.5 ?
>>
>> The relevant patch was:
>> https://www.gitorious.org/orocos-toolchain/ocl/commit/cf452859869712b8c7...
>>
>> It's just a long shot, but you never know...
>
> yes, I've tested using that branch. Same results.
>
> Although I haven't done much, I'll have to postpone trying to locate the cause of this for now. Development can continue on gnulinux.
>
> I can't find any logs, but remember the affinity masks in deployer-xenomai showed ULONG_MAX values for all threads. Not sure if that is of any importance though.

I know. It returns ~0 by default since we can't query Xenomai for
this. We need to cache it in RTT and return that number...

Peter

warn_upon_switch on leaving deployer-xenomai

Submitted by gah on Mon, 2012-04-02 16:20.

See my reply to willy lambert: just starting and quiting the deployer-xenomai already triggers the traces.

As I never touched the deployer source, it is probably an issue on my end (compilation, linking)?

warn_upon_switch on leaving deployer-xenomai

Submitted by willy on Mon, 2012-04-02 12:04.

They are many reasons for context switching in xenomai. For instance
if you have a port with a type that is doing printf/new/.. and many
other in its constructor.
I had a look at logs but I personnaly can't see anything in this hudge
mess. I suggest you provide your simpliest code/deployment and that
you slow your application to have w.u.s at a readable rate so you can
say when they happens (during deployment, at each cycle). You may also
put your app asyncrhonous and trigger your component by hand in the
taskbrowser.

Did you attached an activity to the deployer ? if yes it cannot be real_time.

>
>
>
> system info:
>
> Ubuntu Lucid (on vanilla 2.6.38.8 with adeos 2.11-02)
> GCC 4.4.3-4ubuntu5.1
> ROS electric
> Xenomai 2.6.0 (dlopen enabled)
> orocos_toolchain git checkout
> OROCOS_TARGET=xenomai
>
>
> --
> Orocos-Users mailing list
> Orocos-Users [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-users
>

warn_upon_switch on leaving deployer-xenomai

Submitted by gah on Mon, 2012-04-02 12:52.

> Date: Mon, 2 Apr 2012 14:00:31 +0200
> Subject: Re: [Orocos-users] warn_upon_switch on leaving deployer-xenomai
> From: lambert [dot] willy [..] ...
> To: gaohml [..] ...
> CC: orocos-users [..] ...
>
> 2012/4/2 g ah <gaohml [..] ...>:
> >
> > Exiting deployer-xenomai after running two simple producer-consumer components (exchanging unsigned ints) dumps a lot of stacktraces to the console, such as this one:
> >
> > =====
> >
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
> > [0xb77b6400]
> > /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p<RTT::Activity>::dispose()+0x18)[0xb76e0fc8]
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
> > /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
> > /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
> >
> > =====
> >
> > The rest is on pastebin: http://pastebin.com/HpG1qazj.
> >
> > Have I done something wrong with compiling orocos_toolchain perhaps? I've noticed that my consumer has increasing mode switches in its MSW column in 'xenomai/stat'. There is no logging / printf / std::cout / new / delete going on, so I am not sure what is causing this.
> >
> > Any ideas?
> >
> >
> > thank you,
> >
>
> They are many reasons for context switching in xenomai. For instance
> if you have a port with a type that is doing printf/new/.. and many
> other in its constructor.
> I had a look at logs but I personnaly can't see anything in this hudge
> mess. I suggest you provide your simpliest code/deployment and that
> you slow your application to have w.u.s at a readable rate so you can
> say when they happens (during deployment, at each cycle). You may also
> put your app asyncrhonous and trigger your component by hand in the
> taskbrowser.
>
> Did you attached an activity to the deployer ? if yes it cannot be real_time.

thank you for your comments.

I realise the pastebin link doesn't give a whole lot of information, but I was hoping they might seem familiar to someone.

Some more testing shows that I don't even have to run anything. Just starting the 'deployer-xenomai' and immediately exiting (ctrl+d) results in similar stacktraces being dumped to the console. For every trace, the frame before 'warn_upon_switch' is always 'rt_mutex_release', coming from Thread::~Thread(), ~TaskContext(), ~ExecutionEngine(), ~PluginLoader() and ~Logger(). I've put the trace on pastebin again: http://pastebin.com/z4ay2Mbq.

GDB shows this (gdb `rosstack find orocos_toolchain`/install/bin/deployer-xenomai):

=====

Program received signal SIGXCPU, CPU time limit exceeded.
0xb7a65ea0 in rt_mutex_release () from /usr/lib/libnative.so.3
(gdb) bt
#0 0xb7a65ea0 in rt_mutex_release () from /usr/lib/libnative.so.3
#1 0xb7f3bb59 in rtos_mutex_rec_unlock (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/xenomai/fosi.h:316
#2 RTT::os::MutexRecursive::unlock (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Mutex.hpp:242
#3 ~MutexRecursive (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Mutex.hpp:230
#4 ~Thread (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Thread.cpp:337
#5 0xb7eedb81 in ~Activity (this=0x80ae080, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/Activity.cpp:83
#6 0xb7f0cfc8 in checked_delete<RTT::Activity> (this=0x80a9fd0) at /usr/include/boost/checked_delete.hpp:34
#7 boost::detail::sp_counted_impl_p<RTT::Activity>::dispose (this=0x80a9fd0) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#8 0xb7ee08c8 in boost::detail::sp_counted_base::release (this=0xbfffeca4, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145
#9 ~shared_count (this=0xbfffeca4, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
#10 0xb7f07b30 in ~shared_ptr (this=0xbfffec18, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169
#11 ~TaskContext (this=0xbfffec18, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/TaskContext.cpp:152
#12 0xb7dc5858 in ~TaskBrowser (this=0xbfffec18, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/taskbrowser/TaskBrowser.cpp:687
#13 0x08053352 in main (argc=<value optimised out>, argv=0xbffff0c4) at /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer.cpp:168

=====

Don't know if a minimal working example is still necessary?

warn_upon_switch on leaving deployer-xenomai

Submitted by willy on Mon, 2012-04-02 13:00.

2012/4/2 g ah <gaohml [..] ...>:
>
>> Date: Mon, 2 Apr 2012 14:00:31 +0200
>> Subject: Re: [Orocos-users] warn_upon_switch on leaving deployer-xenomai
>> From: lambert [dot] willy [..] ...
>> To: gaohml [..] ...
>> CC: orocos-users [..] ...
>>
>> 2012/4/2 g ah <gaohml [..] ...>:
>> >
>> > Exiting deployer-xenomai after running two simple producer-consumer components (exchanging unsigned ints) dumps a lot of stacktraces to the console, such as this one:
>> >
>> > =====
>> >
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>> > [0xb77b6400]
>> > /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p<RTT::Activity>::dispose()+0x18)[0xb76e0fc8]
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>> > /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>> > /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>> >
>> > =====
>> >
>> > The rest is on pastebin: http://pastebin.com/HpG1qazj.
>> >
>> > Have I done something wrong with compiling orocos_toolchain perhaps? I've noticed that my consumer has increasing mode switches in its MSW column in 'xenomai/stat'. There is no logging / printf / std::cout / new / delete going on, so I am not sure what is causing this.
>> >
>> > Any ideas?
>> >
>> >
>> > thank you,
>> >
>>
>> They are many reasons for context switching in xenomai. For instance
>> if you have a port with a type that is doing printf/new/.. and many
>> other in its constructor.
>> I had a look at logs but I personnaly can't see anything in this hudge
>> mess. I suggest you provide your simpliest code/deployment and that
>> you slow your application to have w.u.s at a readable rate so you can
>> say when they happens (during deployment, at each cycle). You may also
>> put your app asyncrhonous and trigger your component by hand in the
>> taskbrowser.
>>
>> Did you attached an activity to the deployer ? if yes it cannot be real_time.
>
> thank you for your comments.
>
> I realise the pastebin link doesn't give a whole lot of information, but I was hoping they might seem familiar to someone.
>
> Some more testing shows that I don't even have to run anything. Just starting the 'deployer-xenomai' and immediately exiting (ctrl+d) results in similar stacktraces being dumped to the console. For every trace, the frame before 'warn_upon_switch' is always 'rt_mutex_release', coming from Thread::~Thread(), ~TaskContext(), ~ExecutionEngine(), ~PluginLoader() and ~Logger(). I've put the trace on pastebin again: http://pastebin.com/z4ay2Mbq.
>
> GDB shows this (gdb `rosstack find orocos_toolchain`/install/bin/deployer-xenomai):
>
> =====
>
> Program received signal SIGXCPU, CPU time limit exceeded.
> 0xb7a65ea0 in rt_mutex_release () from /usr/lib/libnative.so.3
> (gdb) bt
> #0 0xb7a65ea0 in rt_mutex_release () from /usr/lib/libnative.so.3
> #1 0xb7f3bb59 in rtos_mutex_rec_unlock (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/xenomai/fosi.h:316
> #2 RTT::os::MutexRecursive::unlock (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Mutex.hpp:242
> #3 ~MutexRecursive (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Mutex.hpp:230
> #4 ~Thread (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Thread.cpp:337
> #5 0xb7eedb81 in ~Activity (this=0x80ae080, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/Activity.cpp:83
> #6 0xb7f0cfc8 in checked_delete<RTT::Activity> (this=0x80a9fd0) at /usr/include/boost/checked_delete.hpp:34
> #7 boost::detail::sp_counted_impl_p<RTT::Activity>::dispose (this=0x80a9fd0) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
> #8 0xb7ee08c8 in boost::detail::sp_counted_base::release (this=0xbfffeca4, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145
> #9 ~shared_count (this=0xbfffeca4, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
> #10 0xb7f07b30 in ~shared_ptr (this=0xbfffec18, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169
> #11 ~TaskContext (this=0xbfffec18, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/TaskContext.cpp:152
> #12 0xb7dc5858 in ~TaskBrowser (this=0xbfffec18, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/taskbrowser/TaskBrowser.cpp:687
> #13 0x08053352 in main (argc=<value optimised out>, argv=0xbffff0c4) at /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer.cpp:168
>
> =====
>
> Don't know if a minimal working example is still necessary?
>

Not yet ^^

Do you run all this as root ?

warn_upon_switch on leaving deployer-xenomai

Submitted by gah on Mon, 2012-04-02 13:16.

Willy Lambert wrote:
> 2012/4/2 g ah <gaohml [..] ...>:
>>> Date: Mon, 2 Apr 2012 14:00:31 +0200
>>> Subject: Re: [Orocos-users] warn_upon_switch on leaving deployer-xenomai
>>> From: lambert [dot] willy [..] ...
>>> To: gaohml [..] ...
>>> CC: orocos-users [..] ...
>>>
>>> 2012/4/2 g ah <gaohml [..] ...>:
>>>> Exiting deployer-xenomai after running two simple producer-consumer components (exchanging unsigned ints) dumps a lot of stacktraces to the console, such as this one:
>>>>
>>>> =====
>>>>
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>>>> [0xb77b6400]
>>>> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p<RTT::Activity>::dispose()+0x18)[0xb76e0fc8]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>>>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>>>>
>>>> =====
>>>>
>>>> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>>>>
>>>> Have I done something wrong with compiling orocos_toolchain perhaps? I've noticed that my consumer has increasing mode switches in its MSW column in 'xenomai/stat'. There is no logging / printf / std::cout / new / delete going on, so I am not sure what is causing this.
>>>>
>>>> Any ideas?
>>>>
>>>>
>>>> thank you,
>>>>
>>> They are many reasons for context switching in xenomai. For instance
>>> if you have a port with a type that is doing printf/new/.. and many
>>> other in its constructor.
>>> I had a look at logs but I personnaly can't see anything in this hudge
>>> mess. I suggest you provide your simpliest code/deployment and that
>>> you slow your application to have w.u.s at a readable rate so you can
>>> say when they happens (during deployment, at each cycle). You may also
>>> put your app asyncrhonous and trigger your component by hand in the
>>> taskbrowser.
>>>
>>> Did you attached an activity to the deployer ? if yes it cannot be real_time.
>> thank you for your comments.
>>
>> I realise the pastebin link doesn't give a whole lot of information, but I was hoping they might seem familiar to someone.
>>
>> Some more testing shows that I don't even have to run anything. Just starting the 'deployer-xenomai' and immediately exiting (ctrl+d) results in similar stacktraces being dumped to the console. For every trace, the frame before 'warn_upon_switch' is always 'rt_mutex_release', coming from Thread::~Thread(), ~TaskContext(), ~ExecutionEngine(), ~PluginLoader() and ~Logger(). I've put the trace on pastebin again: http://pastebin.com/z4ay2Mbq.
>>
>> GDB shows this (gdb `rosstack find orocos_toolchain`/install/bin/deployer-xenomai):
>>
>> =====
>>
>> Program received signal SIGXCPU, CPU time limit exceeded.
>> 0xb7a65ea0 in rt_mutex_release () from /usr/lib/libnative.so.3
>> (gdb) bt
>> #0 0xb7a65ea0 in rt_mutex_release () from /usr/lib/libnative.so.3
>> #1 0xb7f3bb59 in rtos_mutex_rec_unlock (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/xenomai/fosi.h:316
>> #2 RTT::os::MutexRecursive::unlock (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Mutex.hpp:242
>> #3 ~MutexRecursive (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Mutex.hpp:230
>> #4 ~Thread (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Thread.cpp:337
>> #5 0xb7eedb81 in ~Activity (this=0x80ae080, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/Activity.cpp:83
>> #6 0xb7f0cfc8 in checked_delete<RTT::Activity> (this=0x80a9fd0) at /usr/include/boost/checked_delete.hpp:34
>> #7 boost::detail::sp_counted_impl_p<RTT::Activity>::dispose (this=0x80a9fd0) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
>> #8 0xb7ee08c8 in boost::detail::sp_counted_base::release (this=0xbfffeca4, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145
>> #9 ~shared_count (this=0xbfffeca4, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
>> #10 0xb7f07b30 in ~shared_ptr (this=0xbfffec18, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169
>> #11 ~TaskContext (this=0xbfffec18, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/TaskContext.cpp:152
>> #12 0xb7dc5858 in ~TaskBrowser (this=0xbfffec18, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/taskbrowser/TaskBrowser.cpp:687
>> #13 0x08053352 in main (argc=<value optimised out>, argv=0xbffff0c4) at /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer.cpp:168
>>
>> =====
>>
>> Don't know if a minimal working example is still necessary?
>>
>
> Not yet ^^
>
> Do you run all this as root ?

No, a regular user. But the user is a member of the 'xenomai' group, so this should work according to the udev scripts in '/etc/udev/rules.d/' (rtheap.rules/rtpipe.rules: chmod 0660, group=xenomai).

I will try to run as root later and report back.

warn_upon_switch on leaving deployer-xenomai

Submitted by gah on Mon, 2012-04-02 14:08.

g ah wrote:
> Willy Lambert wrote:
>> 2012/4/2 g ah <gaohml [..] ...>:
>>>> Date: Mon, 2 Apr 2012 14:00:31 +0200
>>>> Subject: Re: [Orocos-users] warn_upon_switch on leaving deployer-xenomai
>>>> From: lambert [dot] willy [..] ...
>>>> To: gaohml [..] ...
>>>> CC: orocos-users [..] ...
>>>>
>>>> 2012/4/2 g ah <gaohml [..] ...>:
>>>>> Exiting deployer-xenomai after running two simple producer-consumer components (exchanging unsigned ints) dumps a lot of stacktraces to the console, such as this one:
>>>>>
>>>>> =====
>>>>>
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(warn_upon_switch+0x33)[0xb7718053]
>>>>> [0xb77b6400]
>>>>> /usr/lib/libnative.so.3(rt_mutex_release+0xc0)[0xb7239ea0]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::os::Thread::~Thread()+0x149)[0xb770fb59]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::Activity::~Activity()+0x41)[0xb76c1b81]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::sp_counted_impl_p<RTT::Activity>::dispose()+0x18)[0xb76e0fc8]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(boost::detail::shared_count::~shared_count()+0x38)[0xb76b48c8]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/install/lib/liborocos-rtt-xenomai.so.2.5(RTT::TaskContext::~TaskContext()+0x100)[0xb76dbb30]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/build/taskbrowser/liborocos-ocl-taskbrowser-xenomai.so.2.5.0(OCL::TaskBrowser::~TaskBrowser()+0x1b8)[0xb7599858]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai(main+0x352)[0x8053352]
>>>>> /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xb6aedbd6]
>>>>> /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer-xenomai[0x8052f31]
>>>>>
>>>>> =====
>>>>>
>>>>> The rest is on pastebin: http://pastebin.com/HpG1qazj.
>>>>>
>>>>> Have I done something wrong with compiling orocos_toolchain perhaps? I've noticed that my consumer has increasing mode switches in its MSW column in 'xenomai/stat'. There is no logging / printf / std::cout / new / delete going on, so I am not sure what is causing this.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>>
>>>>> thank you,
>>>>>
>>>> They are many reasons for context switching in xenomai. For instance
>>>> if you have a port with a type that is doing printf/new/.. and many
>>>> other in its constructor.
>>>> I had a look at logs but I personnaly can't see anything in this hudge
>>>> mess. I suggest you provide your simpliest code/deployment and that
>>>> you slow your application to have w.u.s at a readable rate so you can
>>>> say when they happens (during deployment, at each cycle). You may also
>>>> put your app asyncrhonous and trigger your component by hand in the
>>>> taskbrowser.
>>>>
>>>> Did you attached an activity to the deployer ? if yes it cannot be real_time.
>>> thank you for your comments.
>>>
>>> I realise the pastebin link doesn't give a whole lot of information, but I was hoping they might seem familiar to someone.
>>>
>>> Some more testing shows that I don't even have to run anything. Just starting the 'deployer-xenomai' and immediately exiting (ctrl+d) results in similar stacktraces being dumped to the console. For every trace, the frame before 'warn_upon_switch' is always 'rt_mutex_release', coming from Thread::~Thread(), ~TaskContext(), ~ExecutionEngine(), ~PluginLoader() and ~Logger(). I've put the trace on pastebin again: http://pastebin.com/z4ay2Mbq.
>>>
>>> GDB shows this (gdb `rosstack find orocos_toolchain`/install/bin/deployer-xenomai):
>>>
>>> =====
>>>
>>> Program received signal SIGXCPU, CPU time limit exceeded.
>>> 0xb7a65ea0 in rt_mutex_release () from /usr/lib/libnative.so.3
>>> (gdb) bt
>>> #0 0xb7a65ea0 in rt_mutex_release () from /usr/lib/libnative.so.3
>>> #1 0xb7f3bb59 in rtos_mutex_rec_unlock (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/xenomai/fosi.h:316
>>> #2 RTT::os::MutexRecursive::unlock (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Mutex.hpp:242
>>> #3 ~MutexRecursive (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Mutex.hpp:230
>>> #4 ~Thread (this=0x6f, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/os/Thread.cpp:337
>>> #5 0xb7eedb81 in ~Activity (this=0x80ae080, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/Activity.cpp:83
>>> #6 0xb7f0cfc8 in checked_delete<RTT::Activity> (this=0x80a9fd0) at /usr/include/boost/checked_delete.hpp:34
>>> #7 boost::detail::sp_counted_impl_p<RTT::Activity>::dispose (this=0x80a9fd0) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
>>> #8 0xb7ee08c8 in boost::detail::sp_counted_base::release (this=0xbfffeca4, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145
>>> #9 ~shared_count (this=0xbfffeca4, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
>>> #10 0xb7f07b30 in ~shared_ptr (this=0xbfffec18, __in_chrg=<value optimised out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:169
>>> #11 ~TaskContext (this=0xbfffec18, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/rtt/rtt/TaskContext.cpp:152
>>> #12 0xb7dc5858 in ~TaskBrowser (this=0xbfffec18, __in_chrg=<value optimised out>) at /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/taskbrowser/TaskBrowser.cpp:687
>>> #13 0x08053352 in main (argc=<value optimised out>, argv=0xbffff0c4) at /home/gah/ros/oro-src-stacks/orocos_toolchain/ocl/bin/deployer.cpp:168
>>>
>>> =====
>>>
>>> Don't know if a minimal working example is still necessary?
>>>
>> Not yet ^^
>>
>> Do you run all this as root ?
>
> No, a regular user. But the user is a member of the 'xenomai' group, so this should work according to the udev scripts in '/etc/udev/rules.d/' (rtheap.rules/rtpipe.rules: chmod 0660, group=xenomai).
>
> I will try to run as root later and report back.

no difference whether I run with root or not.

Same traces.