We have a need to synchronize a suite of components, controlled by a single Coordination component, to a hardware signal. But that signal isn't available during initialization and shutdown (which takes many seconds), and may block or dissappear at certain times. During these periods, we still want the Coordinator to be able to run periodically. So I think we need a custom Activity that is periodic by default, but otherwise synchronized to the hardware signal when available. Obviously the Coordinator component and/or the activity will have to deal appropriately with the timeout on the hardware signal.
Has anyone done anything like this before? Does anyone have any alternative suggestions?
NB we're using RTT v1, but if there's applicable RTT v2 code, we're happy to backport
Cheers
S
Two-headed activities
On Fri, Jan 06, 2012 at 01:42:38PM +0100, S Roderick wrote:
> We have a need to synchronize a suite of components, controlled by a
> single Coordination component, to a hardware signal. But that signal
> isn't available during initialization and shutdown (which takes many
> seconds), and may block or dissappear at certain times. During these
> periods, we still want the Coordinator to be able to run
> periodically. So I think we need a custom Activity that is periodic
> by default, but otherwise synchronized to the hardware signal when
> available. Obviously the Coordinator component and/or the activity
> will have to deal appropriately with the timeout on the hardware
> signal.
>
> Has anyone done anything like this before? Does anyone have any
Not exactly, but similar.
> alternative suggestions?
The event driven scenario: Have a periodic TimerComponent trigger your
event driven Coordinator at your desired "base" frequency. Then simply
connect your event generating hardware component (or any other!).
I like this one somewhat better, because it doesn't require a new
primitive (the custom Activity). One the other hand for ultra-low
latencies the latter might do slightly better; YMMV.
> NB we're using RTT v1, but if there's applicable RTT v2 code, we're happy to backport
Best regards
Markus
Two-headed activities
On Jan 6, 2012, at 09:08 , Markus Klotzbuecher wrote:
> On Fri, Jan 06, 2012 at 01:42:38PM +0100, S Roderick wrote:
>> We have a need to synchronize a suite of components, controlled by a
>> single Coordination component, to a hardware signal. But that signal
>> isn't available during initialization and shutdown (which takes many
>> seconds), and may block or dissappear at certain times. During these
>> periods, we still want the Coordinator to be able to run
>> periodically. So I think we need a custom Activity that is periodic
>> by default, but otherwise synchronized to the hardware signal when
>> available. Obviously the Coordinator component and/or the activity
>> will have to deal appropriately with the timeout on the hardware
>> signal.
>>
>> Has anyone done anything like this before? Does anyone have any
>
> Not exactly, but similar.
>
>> alternative suggestions?
>
> The event driven scenario: Have a periodic TimerComponent trigger your
> event driven Coordinator at your desired "base" frequency. Then simply
> connect your event generating hardware component (or any other!).
>
> I like this one somewhat better, because it doesn't require a new
> primitive (the custom Activity). One the other hand for ultra-low
> latencies the latter might do slightly better; YMMV.
Your idea does have merit, however, you hit the nail on the head as we need very low latency. But out of curiosity, how do you imagine connecting the "event generating hardware component"? Particularly as that item isn't an Orocos component.
Cheers
S
Two-headed activities
On Fri, Jan 06, 2012 at 07:38:19PM +0100, Stephen Roderick wrote:
> On Jan 6, 2012, at 09:08 , Markus Klotzbuecher wrote:
>
> > On Fri, Jan 06, 2012 at 01:42:38PM +0100, S Roderick wrote:
> >> We have a need to synchronize a suite of components, controlled by a
> >> single Coordination component, to a hardware signal. But that signal
> >> isn't available during initialization and shutdown (which takes many
> >> seconds), and may block or dissappear at certain times. During these
> >> periods, we still want the Coordinator to be able to run
> >> periodically. So I think we need a custom Activity that is periodic
> >> by default, but otherwise synchronized to the hardware signal when
> >> available. Obviously the Coordinator component and/or the activity
> >> will have to deal appropriately with the timeout on the hardware
> >> signal.
> >>
> >> Has anyone done anything like this before? Does anyone have any
> >
> > Not exactly, but similar.
> >
> >> alternative suggestions?
> >
> > The event driven scenario: Have a periodic TimerComponent trigger your
> > event driven Coordinator at your desired "base" frequency. Then simply
> > connect your event generating hardware component (or any other!).
> >
> > I like this one somewhat better, because it doesn't require a new
> > primitive (the custom Activity). One the other hand for ultra-low
> > latencies the latter might do slightly better; YMMV.
>
> Your idea does have merit, however, you hit the nail on the head as
> we need very low latency. But out of curiosity, how do you imagine
Ok. But note that RTT is indeed quite efficient at communicating event
messages. On my machine using C++ components the wakeup latency with
this approach was few tens of microseconds ([1], Section 4.2).
> connecting the "event generating hardware component"? Particularly
> as that item isn't an Orocos component.
You will have a software (driver) component too, no? That one defines
a port to which it writes a data-sample according to the hardware
signal. Then you only need to connect that output port to an event
port of your coordinator.
Markus
[1] https://lwn.net/images/conf/rtlws-2011/paper.05.html
Two-headed activities
On Jan 7, 2012, at 04:15 , Markus Klotzbuecher wrote:
> On Fri, Jan 06, 2012 at 07:38:19PM +0100, Stephen Roderick wrote:
>> On Jan 6, 2012, at 09:08 , Markus Klotzbuecher wrote:
>>
>>> On Fri, Jan 06, 2012 at 01:42:38PM +0100, S Roderick wrote:
>>>> We have a need to synchronize a suite of components, controlled by a
>>>> single Coordination component, to a hardware signal. But that signal
>>>> isn't available during initialization and shutdown (which takes many
>>>> seconds), and may block or dissappear at certain times. During these
>>>> periods, we still want the Coordinator to be able to run
>>>> periodically. So I think we need a custom Activity that is periodic
>>>> by default, but otherwise synchronized to the hardware signal when
>>>> available. Obviously the Coordinator component and/or the activity
>>>> will have to deal appropriately with the timeout on the hardware
>>>> signal.
>>>>
>>>> Has anyone done anything like this before? Does anyone have any
>>>
>>> Not exactly, but similar.
>>>
>>>> alternative suggestions?
>>>
>>> The event driven scenario: Have a periodic TimerComponent trigger your
>>> event driven Coordinator at your desired "base" frequency. Then simply
>>> connect your event generating hardware component (or any other!).
>>>
>>> I like this one somewhat better, because it doesn't require a new
>>> primitive (the custom Activity). One the other hand for ultra-low
>>> latencies the latter might do slightly better; YMMV.
>>
>> Your idea does have merit, however, you hit the nail on the head as
>> we need very low latency. But out of curiosity, how do you imagine
>
> Ok. But note that RTT is indeed quite efficient at communicating event
> messages. On my machine using C++ components the wakeup latency with
> this approach was few tens of microseconds ([1], Section 4.2).
>
>> connecting the "event generating hardware component"? Particularly
>> as that item isn't an Orocos component.
>
> You will have a software (driver) component too, no? That one defines
> a port to which it writes a data-sample according to the hardware
> signal. Then you only need to connect that output port to an event
> port of your coordinator.
>
> Markus
We have ended up doing this a little differently, but I wanted to communicate the result, along with a few interesting things learned along the way.
Our hardware driver is in a non-Orocos controlled thread, and writes to a pipe when the hardware updates. This is the signal we want to sync on. This signal is not available until the hardware is fully initialized (takes several seconds), and is not available if certain hardware faults occur.
Our coordinator is simply a state machine, now run by a FileDescriptorActivity (FDA). The FDA timeout capability is used to drive this component when the hardware signal is not available. We had to add a method call to the run() blocks for each state of interest, to process the FDA results (error, timeout, isUpdated, etc) and react accordingly.
Interesting observations (these pertain to RTT v1)
- emitting an event to the coordinator causes additional step() executions to occur (i.e update hook executions, and state machine cycles). This can overload CPUs ... which is what happened to us (we went from the intended 500 Hz, to almost 4 kHz). This will also occur for commands (search for trigger() in EventProcessor and CommandProcessor implementations). The user implemention must deal with this.
- it might be possible that the work fd set in FDA can be used when not initialized, if an isUpdated() call occurs before loop(). This would not be the typical call sequence, but ...
- I believe that there is a possible race condition in the FDA implementation, that can theoretically cause fd activity to be missed. The timing for this to occur is very fine though. You have to watch() a new fd exactly when an existing fd activity occurs. The loop within a loop in loop() that processes the trigger-update-sets command can set do_trigger to false, causing the fd activity of that cycle to be ignored.
I'll send our backport of FDA to v1, and associated OCL changes, to the ML next week. We've also changed it to support microsecond level timeouts (fully backwards compatible), have added test cases, and will modify the constructor to support CPU affinity (as with the other activity implementations).
HTH
S
Two-headed activities
On Jan 13, 2012, at 10:35 , Stephen Roderick wrote:
> On Jan 7, 2012, at 04:15 , Markus Klotzbuecher wrote:
>
>> On Fri, Jan 06, 2012 at 07:38:19PM +0100, Stephen Roderick wrote:
>>> On Jan 6, 2012, at 09:08 , Markus Klotzbuecher wrote:
>>>
>>>> On Fri, Jan 06, 2012 at 01:42:38PM +0100, S Roderick wrote:
>>>>> We have a need to synchronize a suite of components, controlled by a
>>>>> single Coordination component, to a hardware signal. But that signal
>>>>> isn't available during initialization and shutdown (which takes many
>>>>> seconds), and may block or dissappear at certain times. During these
>>>>> periods, we still want the Coordinator to be able to run
>>>>> periodically. So I think we need a custom Activity that is periodic
>>>>> by default, but otherwise synchronized to the hardware signal when
>>>>> available. Obviously the Coordinator component and/or the activity
>>>>> will have to deal appropriately with the timeout on the hardware
>>>>> signal.
>>>>>
>>>>> Has anyone done anything like this before? Does anyone have any
>>>>
>>>> Not exactly, but similar.
>>>>
>>>>> alternative suggestions?
>>>>
>>>> The event driven scenario: Have a periodic TimerComponent trigger your
>>>> event driven Coordinator at your desired "base" frequency. Then simply
>>>> connect your event generating hardware component (or any other!).
>>>>
>>>> I like this one somewhat better, because it doesn't require a new
>>>> primitive (the custom Activity). One the other hand for ultra-low
>>>> latencies the latter might do slightly better; YMMV.
>>>
>>> Your idea does have merit, however, you hit the nail on the head as
>>> we need very low latency. But out of curiosity, how do you imagine
>>
>> Ok. But note that RTT is indeed quite efficient at communicating event
>> messages. On my machine using C++ components the wakeup latency with
>> this approach was few tens of microseconds ([1], Section 4.2).
>>
>>> connecting the "event generating hardware component"? Particularly
>>> as that item isn't an Orocos component.
>>
>> You will have a software (driver) component too, no? That one defines
>> a port to which it writes a data-sample according to the hardware
>> signal. Then you only need to connect that output port to an event
>> port of your coordinator.
>>
>> Markus
>
> We have ended up doing this a little differently, but I wanted to communicate the result, along with a few interesting things learned along the way.
>
> Our hardware driver is in a non-Orocos controlled thread, and writes to a pipe when the hardware updates. This is the signal we want to sync on. This signal is not available until the hardware is fully initialized (takes several seconds), and is not available if certain hardware faults occur.
>
> Our coordinator is simply a state machine, now run by a FileDescriptorActivity (FDA). The FDA timeout capability is used to drive this component when the hardware signal is not available. We had to add a method call to the run() blocks for each state of interest, to process the FDA results (error, timeout, isUpdated, etc) and react accordingly.
>
> Interesting observations (these pertain to RTT v1)
>
> - emitting an event to the coordinator causes additional step() executions to occur (i.e update hook executions, and state machine cycles). This can overload CPUs ... which is what happened to us (we went from the intended 500 Hz, to almost 4 kHz). This will also occur for commands (search for trigger() in EventProcessor and CommandProcessor implementations). The user implemention must deal with this.
>
> - it might be possible that the work fd set in FDA can be used when not initialized, if an isUpdated() call occurs before loop(). This would not be the typical call sequence, but ...
>
> - I believe that there is a possible race condition in the FDA implementation, that can theoretically cause fd activity to be missed. The timing for this to occur is very fine though. You have to watch() a new fd exactly when an existing fd activity occurs. The loop within a loop in loop() that processes the trigger-update-sets command can set do_trigger to false, causing the fd activity of that cycle to be ignored.
And one thing I missed
- for cases such as ours, where the file descriptor being watched is periodic in nature, I think we should allow the FDA to be configured with an _intended_ period. This allows the component, or components called via slave activities, to use the intended period (e.g. getPeriod() ) in computations. So a Computational component used within a Periodic activity in one deployment, or with a FD activity that is based on a periodic signal in a different deployemtn, requires no additional configuration or changes.
Two-headed activities
On Fri, 6 Jan 2012, Stephen Roderick wrote:
> On Jan 6, 2012, at 09:08 , Markus Klotzbuecher wrote:
>
>> On Fri, Jan 06, 2012 at 01:42:38PM +0100, S Roderick wrote:
>>> We have a need to synchronize a suite of components, controlled by a
>>> single Coordination component, to a hardware signal. But that signal
>>> isn't available during initialization and shutdown (which takes many
>>> seconds), and may block or dissappear at certain times. During these
>>> periods, we still want the Coordinator to be able to run
>>> periodically. So I think we need a custom Activity that is periodic
>>> by default, but otherwise synchronized to the hardware signal when
>>> available. Obviously the Coordinator component and/or the activity
>>> will have to deal appropriately with the timeout on the hardware
>>> signal.
>>>
>>> Has anyone done anything like this before? Does anyone have any
>>
>> Not exactly, but similar.
>>
>>> alternative suggestions?
>>
>> The event driven scenario: Have a periodic TimerComponent trigger your
>> event driven Coordinator at your desired "base" frequency. Then simply
>> connect your event generating hardware component (or any other!).
>>
>> I like this one somewhat better, because it doesn't require a new
>> primitive (the custom Activity). One the other hand for ultra-low
>> latencies the latter might do slightly better; YMMV.
>
> Your idea does have merit, however, you hit the nail on the head as we
> need very low latency.
Latency is most of the time determined by the _platform_ your system runs
on, not on its software design. The latter can _screw up_ your latency, if
done wrong, read: has livelocks, but otherwise _any_ good software design
can be _deployed_ in an as low as physically possible way! The latter
requires skills at the platform level, of course. More concretely: "events"
(which are behind the design that Markus suggests) can be communicated as
efficiently as your hardware allows, provided you have either the right
tools (which do not yet exist, I think, although there are things like
d-bus, but not at the realtime level) or the right skills to do this
optimization manually.
> But out of curiosity, how do you imagine connecting the "event generating
> hardware component"? Particularly as that item isn't an Orocos component.
You have to encapsulate that hardware inside an RTT component, and let it
work in the RTT network via events!
Note that it is not required that every component is a TaskContext, or gets
one single thread or process at deployment! This is _the_ major flaw in the
current RTT implementations: the seemingly inevitable connection between
the concept of a software component, the RTT TaskContext primitive, and an
Activity; these are really three different things: the first on is a
software design primitive dealing with optimal decoupling; the second one
is a deployment design thing, where several components are deployed in the
same "framework container"; the last one is a deployment design thing,
where the runtime platform properties are being taken into account.
Herman
Two-headed activities
On 01/06/2012 01:42 PM, S Roderick wrote:
> We have a need to synchronize a suite of components, controlled by a single Coordination component, to a hardware signal. But that signal isn't available during initialization and shutdown (which takes many seconds), and may block or dissappear at certain times. During these periods, we still want the Coordinator to be able to run periodically. So I think we need a custom Activity that is periodic by default, but otherwise synchronized to the hardware signal when available. Obviously the Coordinator component and/or the activity will have to deal appropriately with the timeout on the hardware signal.
>
> Has anyone done anything like this before? Does anyone have any alternative suggestions?
>
> NB we're using RTT v1, but if there's applicable RTT v2 code, we're happy to backport
Depends. The FD activity does all of that:
* wait for data on some fds
* have a timeout (which can fake periodicity)
* allow explicit triggering
But that's only for file descriptors ...
Two-headed activities
On Jan 6, 2012, at 08:05 , Sylvain Joyeux wrote:
> On 01/06/2012 01:42 PM, S Roderick wrote:
>> We have a need to synchronize a suite of components, controlled by a single Coordination component, to a hardware signal. But that signal isn't available during initialization and shutdown (which takes many seconds), and may block or dissappear at certain times. During these periods, we still want the Coordinator to be able to run periodically. So I think we need a custom Activity that is periodic by default, but otherwise synchronized to the hardware signal when available. Obviously the Coordinator component and/or the activity will have to deal appropriately with the timeout on the hardware signal.
>>
>> Has anyone done anything like this before? Does anyone have any alternative suggestions?
>>
>> NB we're using RTT v1, but if there's applicable RTT v2 code, we're happy to backport
> Depends. The FD activity does all of that:
>
> * wait for data on some fds
> * have a timeout (which can fake periodicity)
> * allow explicit triggering
>
> But that's only for file descriptors ...
That was on my list to go look at. We are going to have to use some kind of IPC to bring the hardware signal out of the driver, so an FD is a possibility. You select() on the FD's w/ timeout?
Anything you can think of in the FD implementation that would be difficult to backport to v1?
Cheers
S
Two-headed activities
> You select() on the FD's w/ timeout?
Yes.
> Anything you can think of in the FD implementation that would be difficult to backport to v1?
Not out of the top of my head. The FD activity was part of v1
originally, so you may want to make a diff between the old and new
version and see what changed.
For a guide on how to use the activity:
http://rock-robotics.org/documentation/orogen/triggering/fd.html
Sylvain