Thank you Klass, I will give a try!

2009/10/30 Klaas Gadeyne <klaas [dot] gadeyne [..] ...>

> On Fri, Oct 30, 2009 at 4:19 PM, Philippe Hamelin
> <philippe [dot] hamelin [..] ...> wrote:
> > You are suggesting a good precision Klass. First, let me describe my
> > problem. I currently have an Orocos application with about 20 components.
> I
> > have a master component with a periodic activity which calls the other 19
> > slave activities at 500Hz (gnulinux ORO_SHED_RT). Actually, this is
> taking
> > about 3% CPU so there is no major computational effort. When I push up
> the
> > frequency to 1 kHz this suddenly lead the application to 100% CPU
> > utilization. I guess the problem comes from the gnulinux scheduler
> (debian
> > lenny vanilla kernel) which is not able to deal with this setup. So this
> > problem has lead me to look forward for an other scheduler which may help
> me
> > to get down to a lower CPU utilization.
>
> I don't know if your guess is correct, but a smaller (and faster to
> check) step would maybe be to check with
> <http://www.pengutronix.de/software/linux-rt/debian_en.html>
>
> > Currently, we use custom device drivers that directly write to I/O. We
> need
> > a solution which is easy to maintain by any common programmer which is
> not
> > specialized in real time linux. Also, the solution should be stable
> because
> > this is to be used in an industrial product.
>
> _If_ your device drivers are using plain linux systems calls, neither
> xenomai/rtai are going to help you unless you rewrite them (e.g. using
> RTDM). And to me, that means you'll need a programmer which is aware
> of (possible) real-time issues.
>
> Klaas
>

Hello,
after some investigations I found interesting things.

I measured the execution time of all slave activities :

void MasterComponent::updateHook()
{
    referenceTime = TimeService::Instance()->getTicks();
 
    getPeer("slaveA")->getActivity()->execute();
    getPeer("slaveB")->getActivity()->execute();
    getPeer("slaveC")->getActivity()->execute();
 
    processTime = TimeService::Instance()->secondsSince(referenceTime);
}

and I found that I'm using 0.9ms while my periodic activity has a 1ms
period. So, I'm effectively using most of the CPU.
Then, I wanted to know which component is taking so much CPU. I first tested
component "SlaveA" :

void MasterComponent::updateHook()
{
    referenceTime = TimeService::Instance()->getTicks();
 
    getPeer("slaveA")->getActivity()->execute();
 
    processTime = TimeService::Instance()->secondsSince(referenceTime);
 
    getPeer("slaveB")->getActivity()->execute();
    getPeer("slaveC")->getActivity()->execute();
 
}

I found that the process time of slaveA was 0.15ms. The comopnent "slaveA"
just have some code in his updateHook (no FSM, no programs...). I removed
all the code in the updateHook() of "slaveA" and then I measured again the
process time of the single line :

    getPeer("slaveA")->getActivity()->execute();

I found that just calling the (empty) slaveActivity takes about 0.1ms. My
system has in reality about 10 slaves activities which are called by the
masterComponent, so TotalTime = 10 * 0.1ms = 1ms. It means that it takes
100% CPU @ 1kHz just to call empty components.

Do you think that this overhead comes from Orocos or from the "context
switching" time of Linux? Does each slave activity has his own thread or the
slave activities is executed in the calling thread? If the slave activity is
called inside the master thread I don't see where this switching time could
come from?

Also, I tested the RT kernel from
http://www.pengutronix.de/software/linux-rt/debian_en.html and I had similar
results. I'm using a Celeron M 1GHz with 512Mb DDR.

Philippe

2009/10/30 Peter Soetens <peter [..] ...>

> On Fri, Oct 30, 2009 at 16:19, Philippe Hamelin
> <philippe [dot] hamelin [..] ...> wrote:
> > You are suggesting a good precision Klass. First, let me describe my
> > problem. I currently have an Orocos application with about 20 components.
> I
> > have a master component with a periodic activity which calls the other 19
> > slave activities at 500Hz (gnulinux ORO_SHED_RT). Actually, this is
> taking
> > about 3% CPU so there is no major computational effort. When I push up
> the
> > frequency to 1 kHz this suddenly lead the application to 100% CPU
> > utilization. I guess the problem comes from the gnulinux scheduler
> (debian
> > lenny vanilla kernel) which is not able to deal with this setup. So this
> > problem has lead me to look forward for an other scheduler which may help
> me
> > to get down to a lower CPU utilization.
>
> top is notorious for reporting wrong CPU utilization numbers when
> using SCHED_FIFO. The numbers are complete nonsense and are indeed
> typically 100%.
>
> If you care, you'll need a better way of measuring, for example, by
> using the (dark and hidden) Orocos Thread Scope.
>
> Peter
>

I know that the CPU utilization given by standard linux tools aren't
reliable. My application don't really take 100% CPU because there is only a
small amount of calculation. However, I know that the CPU is overloaded
because the system is anormaly slow for all other process during executing
of my orocos application. I'm using a Celeron M 1 GHz with 512Mb DDR so I
think it's faster enough for the small amount of calculation I do. I tested
the same application (and same OS) on a Core 2 Duo and it runs fine with a
CPU utilization of about 2-3% at 1 kHz. So I don't understand how my
application can go from 3% to 100% of CPU utilization only by doubling the
sampling rate. There seems to be a choke point somewhere in the scheduling
process. I already had that kind of problem in QNX and I resolved the
problem by changing the scheduler time period from 1ms to 100us.

2009/10/30 Peter Soetens <peter [..] ...>

> On Fri, Oct 30, 2009 at 16:19, Philippe Hamelin
> <philippe [dot] hamelin [..] ...> wrote:
> > You are suggesting a good precision Klass. First, let me describe my
> > problem. I currently have an Orocos application with about 20 components.
> I
> > have a master component with a periodic activity which calls the other 19
> > slave activities at 500Hz (gnulinux ORO_SHED_RT). Actually, this is
> taking
> > about 3% CPU so there is no major computational effort. When I push up
> the
> > frequency to 1 kHz this suddenly lead the application to 100% CPU
> > utilization. I guess the problem comes from the gnulinux scheduler
> (debian
> > lenny vanilla kernel) which is not able to deal with this setup. So this
> > problem has lead me to look forward for an other scheduler which may help
> me
> > to get down to a lower CPU utilization.
>
> top is notorious for reporting wrong CPU utilization numbers when
> using SCHED_FIFO. The numbers are complete nonsense and are indeed
> typically 100%.
>
> If you care, you'll need a better way of measuring, for example, by
> using the (dark and hidden) Orocos Thread Scope.
>
> Peter
>