A frequently popping up issue users have is that they see 'old' data in their
buffer because the new samples got rejected since it's full. There are two
solutions to this:
1. Change the behavior of the BUFFER type connection policy to be circular,
drop the old behavior.
2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
(so then we have DATA, BUFFER, CIRCULAR)
Votes ?
Peter
Introducing circular buffers in connections...
On Tue, Jul 12, 2011 at 05:47:33PM +0200, Peter Soetens wrote:
> A frequently popping up issue users have is that they see 'old' data in their
> buffer because the new samples got rejected since it's full. There are two
> solutions to this:
>
> 1. Change the behavior of the BUFFER type connection policy to be circular,
> drop the old behavior.
>
> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
> (so then we have DATA, BUFFER, CIRCULAR)
2!
Additionally, for both BUFFER and CIRCULAR I would like to have a
counter added to the port that holds the number of dropped/overrun
samples. These counters could be monitored by a global connection
monitor.
Markus
PS: Raised an event on overflow would be an alternative, but through
which port should that be done? That would require each Port have a
(meta-) event port? :-) ?!
Introducing circular buffers in connections...
On 07/13/2011 08:16 AM, Markus Klotzbuecher wrote:
> Additionally, for both BUFFER and CIRCULAR I would like to have a
> counter added to the port that holds the number of dropped/overrun
> samples. These counters could be monitored by a global connection
> monitor.
And DATA !
Funnily, I had a discussion with one of my colleagues about it yesterday.
A solution he proposed was to add a counter per channel. Each time a
sample is written, the current value of the counter is passed along with
the data element. This way, when one pulls a value out of the channel,
he knows how many samples have been dropped since its last read. The
interesting point being that it would be independent from the transport
used (for Orocos-designed transports, the ROS transport would not have
this information)
Introducing circular buffers in connections...
On Wed, 13 Jul 2011, Sylvain Joyeux wrote:
> On 07/13/2011 08:16 AM, Markus Klotzbuecher wrote:
>> Additionally, for both BUFFER and CIRCULAR I would like to have a
>> counter added to the port that holds the number of dropped/overrun
>> samples. These counters could be monitored by a global connection
>> monitor.
> And DATA !
>
> Funnily, I had a discussion with one of my colleagues about it yesterday.
>
> A solution he proposed was to add a counter per channel. Each time a
> sample is written, the current value of the counter is passed along with
> the data element. This way, when one pulls a value out of the channel,
> he knows how many samples have been dropped since its last read.
These are fine solutions, for some applications. I would _not_ put those in
the RTT framework, but let the application developers add this overhead
when they need it. They are the ones that have to spend the extra effort to
_deal_ with this information anyway. Hence, making it easy to add such
"application-dependent tags" to communication transport is a "Good Thing",
but putting one or more of them in RTT is "Not a Good Thing". Featuritis!
In other words, I interpret these suggestions (and many that circulate
these days on the mailinglist) as very valuable inputs to _tooling_ around
RTT, but certainly not as "feature requests" to integrate in RTT. If we
give in (and, in retrospect, "we" did already that too many times in the
past) RTT grows too large and too ad hoc (read, with only partial feature
support in some sub-domains). I prefer a lean and mean RTT, with a healthy
and varied tooling ecosystem around it; those tools should not be "RTT
dependent", by the way, just "RTT compatible"...
> The
> interesting point being that it would be independent from the transport
> used (for Orocos-designed transports, the ROS transport would not have
> this information)
Hence, it should be something that is to be done outside of RTT.
Introducing circular buffers in connections...
On 07/13/2011 08:16 AM, Markus Klotzbuecher wrote:
> On Tue, Jul 12, 2011 at 05:47:33PM +0200, Peter Soetens wrote:
>> A frequently popping up issue users have is that they see 'old' data in their
>> buffer because the new samples got rejected since it's full. There are two
>> solutions to this:
>>
>> 1. Change the behavior of the BUFFER type connection policy to be circular,
>> drop the old behavior.
>>
>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
>> (so then we have DATA, BUFFER, CIRCULAR)
>
> 2!
>
> Additionally, for both BUFFER and CIRCULAR I would like to have a
> counter added to the port that holds the number of dropped/overrun
> samples. These counters could be monitored by a global connection
> monitor.
+1
Sylvain
Introducing circular buffers in connections...
On Wed, 13 Jul 2011, Markus Klotzbuecher wrote:
> On Tue, Jul 12, 2011 at 05:47:33PM +0200, Peter Soetens wrote:
>> A frequently popping up issue users have is that they see 'old' data in their
>> buffer because the new samples got rejected since it's full. There are two
>> solutions to this:
>>
>> 1. Change the behavior of the BUFFER type connection policy to be circular,
>> drop the old behavior.
>>
>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
>> (so then we have DATA, BUFFER, CIRCULAR)
>
> 2!
>
> Additionally, for both BUFFER and CIRCULAR I would like to have a
> counter added to the port that holds the number of dropped/overrun
> samples. These counters could be monitored by a global connection
> monitor.
Overkill! If your buffer overflows, your application has a severe
problem, and counting will not help, at all. If you want to debug your
application, you should make your components log their state, and try to
find the _cause_ of the problem, instead of trying to analyse the _symptom_
which is the overrun.
> Markus
>
> PS: Raised an event on overflow would be an alternative, but through
> which port should that be done? That would require each Port have a
> (meta-) event port? :-) ?!
You see... adding all these Communication protocols into RTT is a "nice"
example of featuritis.
Herman
Introducing circular buffers in connections...
On 07/13/2011 08:47 AM, Herman Bruyninckx wrote:
> On Wed, 13 Jul 2011, Markus Klotzbuecher wrote:
>
>> On Tue, Jul 12, 2011 at 05:47:33PM +0200, Peter Soetens wrote:
>>> A frequently popping up issue users have is that they see 'old' data in their
>>> buffer because the new samples got rejected since it's full. There are two
>>> solutions to this:
>>>
>>> 1. Change the behavior of the BUFFER type connection policy to be circular,
>>> drop the old behavior.
>>>
>>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
>>> (so then we have DATA, BUFFER, CIRCULAR)
>>
>> 2!
>>
>> Additionally, for both BUFFER and CIRCULAR I would like to have a
>> counter added to the port that holds the number of dropped/overrun
>> samples. These counters could be monitored by a global connection
>> monitor.
>
> Overkill! If your buffer overflows, your application has a severe
> problem, and counting will not help, at all. If you want to debug your
> application, you should make your components log their state, and try to
> find the _cause_ of the problem, instead of trying to analyse the _symptom_
> which is the overrun.
Counting helps as it allows to diagnose the problem. The first step
towards fixing problems is finding them, and for that *any* data is
important.
I do log component states (and much more fine-grained states than what
plain RTT provides), but that's not enough in this case. There are quite
a few things involved in the dataflow (MQ, CORBA, general scheduling,
...) that only diagnostics / coordination should have to know about.
This counter is a first step towards providing the much needed information.
Introducing circular buffers in connections...
On Wed, 13 Jul 2011, Sylvain Joyeux wrote:
> On 07/13/2011 08:47 AM, Herman Bruyninckx wrote:
>> On Wed, 13 Jul 2011, Markus Klotzbuecher wrote:
>>
>>> On Tue, Jul 12, 2011 at 05:47:33PM +0200, Peter Soetens wrote:
>>>> A frequently popping up issue users have is that they see 'old' data in their
>>>> buffer because the new samples got rejected since it's full. There are two
>>>> solutions to this:
>>>>
>>>> 1. Change the behavior of the BUFFER type connection policy to be circular,
>>>> drop the old behavior.
>>>>
>>>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
>>>> (so then we have DATA, BUFFER, CIRCULAR)
>>>
>>> 2!
>>>
>>> Additionally, for both BUFFER and CIRCULAR I would like to have a
>>> counter added to the port that holds the number of dropped/overrun
>>> samples. These counters could be monitored by a global connection
>>> monitor.
>>
>> Overkill! If your buffer overflows, your application has a severe
>> problem, and counting will not help, at all. If you want to debug your
>> application, you should make your components log their state, and try to
>> find the _cause_ of the problem, instead of trying to analyse the _symptom_
>> which is the overrun.
> Counting helps as it allows to diagnose the problem. The first step
> towards fixing problems is finding them, and for that *any* data is
> important.
That data should come from the components that _generate_ the data, because
they have the context in which to interpret these numbers! Again, putting
the burden on the communication code is not fair, not useful, and not
"clean".
> I do log component states (and much more fine-grained states than what
> plain RTT provides), but that's not enough in this case. There are quite
> a few things involved in the dataflow (MQ, CORBA, general scheduling,
> ...) that only diagnostics / coordination should have to know about.
> This counter is a first step towards providing the much needed information.
I don't agree:
- the problems of buffer overflow are _not_ the responsibility of the
communication component, but of the data generating/consuming components.
- instrumenting the communication components is a "Good Thing" but only to
diagnose the problems _caused_ by the communication, independently of the
generating/consuming components.
- if your application wants to be able to trace the fate of data messages,
(and I agree that that is a good thing to do, very often) it's your
application that should add explicit semantic tags to its communication
messages, instead of letting the "framework" do a soft and semantic-less
version of it. Don't expect the framework to help the application for
things only the application really solve.
The "BRICS Component Model", with its 5C decoupling helps me _a lot_ to
assign responsibilities to the rights players in an application. And it
becomes so obviously clear that many of the "common practices" in the
domain are not at all "best practices", _because_ there is no such thing as
"5C awareness"...
> Sylvain Joyeux (Dr.Ing.)
Herman
> Space & Security Robotics
>
> !!! Achtung, neue Telefonnummer!!!
>
> Standort Bremen:
> DFKI GmbH
> Robotics Innovation Center
> Robert-Hooke-Straße 5
> 28359 Bremen, Germany
>
> Phone: +49 (0)421 178-454136
> Fax: +49 (0)421 218-454150
> E-Mail: robotik [..] ...
>
> Weitere Informationen: http://www.dfki.de/robotik
> -----------------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Straße 122, D-67663 Kaiserslautern
> Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
> (Vorsitzender) Dr. Walter Olthoff
> Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
> Amtsgericht Kaiserslautern, HRB 2313
> Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
> USt-Id.Nr.: DE 148646973
> Steuernummer: 19/673/0060/3
> -----------------------------------------------------------------------
>
--
K.U.Leuven, Mechanical Eng., Mechatronics & Robotics Research Group
<http://people.mech.kuleuven.be/~bruyninc> Tel: +32 16 328056
EURON Coordinator (European Robotics Research Network) <http://www.euron.org>
Open Realtime Control Services <http://www.orocos.org>
Associate Editor JOSER <http://www.joser.org>, IJRR <http://www.ijrr.org>
Introducing circular buffers in connections...
On Jul 13, 2011, at 07:31 , Herman Bruyninckx wrote:
> On Wed, 13 Jul 2011, Sylvain Joyeux wrote:
>
>> On 07/13/2011 08:47 AM, Herman Bruyninckx wrote:
>>> On Wed, 13 Jul 2011, Markus Klotzbuecher wrote:
>>>
>>>> On Tue, Jul 12, 2011 at 05:47:33PM +0200, Peter Soetens wrote:
>>>>> A frequently popping up issue users have is that they see 'old' data in their
>>>>> buffer because the new samples got rejected since it's full. There are two
>>>>> solutions to this:
>>>>>
>>>>> 1. Change the behavior of the BUFFER type connection policy to be circular,
>>>>> drop the old behavior.
>>>>>
>>>>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
>>>>> (so then we have DATA, BUFFER, CIRCULAR)
>>>>
>>>> 2!
>>>>
>>>> Additionally, for both BUFFER and CIRCULAR I would like to have a
>>>> counter added to the port that holds the number of dropped/overrun
>>>> samples. These counters could be monitored by a global connection
>>>> monitor.
>>>
>>> Overkill! If your buffer overflows, your application has a severe
>>> problem, and counting will not help, at all. If you want to debug your
>>> application, you should make your components log their state, and try to
>>> find the _cause_ of the problem, instead of trying to analyse the _symptom_
>>> which is the overrun.
>> Counting helps as it allows to diagnose the problem. The first step
>> towards fixing problems is finding them, and for that *any* data is
>> important.
>
> That data should come from the components that _generate_ the data, because
> they have the context in which to interpret these numbers! Again, putting
> the burden on the communication code is not fair, not useful, and not
> "clean".
You both have points here. Sometimes you need to instrument everything to track down a hard to find bug ...
>> I do log component states (and much more fine-grained states than what
>> plain RTT provides), but that's not enough in this case. There are quite
>> a few things involved in the dataflow (MQ, CORBA, general scheduling,
>> ...) that only diagnostics / coordination should have to know about.
>> This counter is a first step towards providing the much needed information.
>
> I don't agree:
> - the problems of buffer overflow are _not_ the responsibility of the
> communication component, but of the data generating/consuming components.
> - instrumenting the communication components is a "Good Thing" but only to
> diagnose the problems _caused_ by the communication, independently of the
> generating/consuming components.
> - if your application wants to be able to trace the fate of data messages,
> (and I agree that that is a good thing to do, very often) it's your
> application that should add explicit semantic tags to its communication
> messages, instead of letting the "framework" do a soft and semantic-less
> version of it. Don't expect the framework to help the application for
> things only the application really solve.
I tend to agree with Herman here [to some degree :-) ]. Leave some of the suggested changes outside RTT, and just keep the basics in RTT. That makes RTT smaller, lighter, and easier to test correctness.
>From Peter's original comments it sounds like adding a flag to deal with CIRCULAR vs BUFFER is a very small task, and adds almost nothing to RTT's complexity. We have to ensure that we communicate well, the differences between the two.
S
Introducing circular buffers in connections...
On Wed, 13 Jul 2011, S Roderick wrote:
> On Jul 13, 2011, at 07:31 , Herman Bruyninckx wrote:
>
>> On Wed, 13 Jul 2011, Sylvain Joyeux wrote:
>>
>>> On 07/13/2011 08:47 AM, Herman Bruyninckx wrote:
>>>> On Wed, 13 Jul 2011, Markus Klotzbuecher wrote:
>>>>
>>>>> On Tue, Jul 12, 2011 at 05:47:33PM +0200, Peter Soetens wrote:
>>>>>> A frequently popping up issue users have is that they see 'old' data in their
>>>>>> buffer because the new samples got rejected since it's full. There are two
>>>>>> solutions to this:
>>>>>>
>>>>>> 1. Change the behavior of the BUFFER type connection policy to be circular,
>>>>>> drop the old behavior.
>>>>>>
>>>>>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
>>>>>> (so then we have DATA, BUFFER, CIRCULAR)
>>>>>
>>>>> 2!
>>>>>
>>>>> Additionally, for both BUFFER and CIRCULAR I would like to have a
>>>>> counter added to the port that holds the number of dropped/overrun
>>>>> samples. These counters could be monitored by a global connection
>>>>> monitor.
>>>>
>>>> Overkill! If your buffer overflows, your application has a severe
>>>> problem, and counting will not help, at all. If you want to debug your
>>>> application, you should make your components log their state, and try to
>>>> find the _cause_ of the problem, instead of trying to analyse the _symptom_
>>>> which is the overrun.
>>> Counting helps as it allows to diagnose the problem. The first step
>>> towards fixing problems is finding them, and for that *any* data is
>>> important.
>>
>> That data should come from the components that _generate_ the data, because
>> they have the context in which to interpret these numbers! Again, putting
>> the burden on the communication code is not fair, not useful, and not
>> "clean".
>
> You both have points here. Sometimes you need to instrument everything to track down a hard to find bug ...
I fully agree with the need for _instrumentation_. But that should be a
configurable _add on_ and not baked-in as a "policy". "Separation of
concerns", "composability" and "minimality of features" are my motto in
this discussion.
>>> I do log component states (and much more fine-grained states than what
>>> plain RTT provides), but that's not enough in this case. There are quite
>>> a few things involved in the dataflow (MQ, CORBA, general scheduling,
>>> ...) that only diagnostics / coordination should have to know about.
>>> This counter is a first step towards providing the much needed information.
>>
>> I don't agree:
>> - the problems of buffer overflow are _not_ the responsibility of the
>> communication component, but of the data generating/consuming components.
>> - instrumenting the communication components is a "Good Thing" but only to
>> diagnose the problems _caused_ by the communication, independently of the
>> generating/consuming components.
>> - if your application wants to be able to trace the fate of data messages,
>> (and I agree that that is a good thing to do, very often) it's your
>> application that should add explicit semantic tags to its communication
>> messages, instead of letting the "framework" do a soft and semantic-less
>> version of it. Don't expect the framework to help the application for
>> things only the application really solve.
>
> I tend to agree with Herman here [to some degree :-) ].
I don't expect anyone to agree to the full extent. That level can only be
reached after _many_ years of maturity :-) Nobody has ever succeeded, by
the way. (Maybe, just maybe, that means _I_ have a problem, and not
"everybody" :-))
> Leave some of the
> suggested changes outside RTT, and just keep the basics in RTT. That
> makes RTT smaller, lighter, and easier to test correctness.
The core of the discussion is: "what are those basics?..."
> From Peter's original comments it sounds like adding a flag to deal with
> CIRCULAR vs BUFFER is a very small task, and adds almost nothing to RTT's
> complexity. We have to ensure that we communicate well, the differences
> between the two.
Incremental cruft remains cruft, whether it costs effort or not...
<http://en.wikipedia.org/wiki/Cruft>
> S
Herman
Introducing circular buffers in connections...
On Tue, 12 Jul 2011, Peter Soetens wrote:
> A frequently popping up issue users have is that they see 'old' data in their
> buffer because the new samples got rejected since it's full. There are two
> solutions to this:
>
> 1. Change the behavior of the BUFFER type connection policy to be circular,
> drop the old behavior.
>
> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
> (so then we have DATA, BUFFER, CIRCULAR)
>
> Votes ?
I am against adding hardcoded Communication policies... If a buffer fills
up, the _application_ has to take the action that fits at that moment, so
it should react to an event signalling the buffer condition. Trying to
"solve" it "once and for all" behind the back of the application is not a
good thing.
If you go the suggested way, you could (should?) add LIFO behaviour to the
FIFO (current BUFFER) and CIRCULAR options?
> Peter
Herman
Introducing circular buffers in connections...
On 13/07/11 14:20, Herman Bruyninckx wrote:
> On Tue, 12 Jul 2011, Peter Soetens wrote:
>
>> A frequently popping up issue users have is that they see 'old' data in their
>> buffer because the new samples got rejected since it's full. There are two
>> solutions to this:
>>
>> 1. Change the behavior of the BUFFER type connection policy to be circular,
>> drop the old behavior.
>>
>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
>> (so then we have DATA, BUFFER, CIRCULAR)
Shouldn't this sort of policy be a property of the receiving port, not
the connection?
>>
>> Votes ?
>
> I am against adding hardcoded Communication policies... If a buffer fills
> up, the _application_ has to take the action that fits at that moment, so
> it should react to an event signalling the buffer condition. Trying to
> "solve" it "once and for all" behind the back of the application is not a
> good thing.
This is how we do it in OpenRTM, for better or worse. The writer gets
notified when one of its receivers has an overflow, and the receiver
gets notified when it has an overflow. I think most users currently
ignore those callbacks, though.
Geoff
>
> If you go the suggested way, you could (should?) add LIFO behaviour to the
> FIFO (current BUFFER) and CIRCULAR options?
>
>> Peter
>
> Herman
Introducing circular buffers in connections...
On Thu, 14 Jul 2011, Geoffrey Biggs wrote:
> On 13/07/11 14:20, Herman Bruyninckx wrote:
>> On Tue, 12 Jul 2011, Peter Soetens wrote:
>>
>>> A frequently popping up issue users have is that they see 'old' data in their
>>> buffer because the new samples got rejected since it's full. There are two
>>> solutions to this:
>>>
>>> 1. Change the behavior of the BUFFER type connection policy to be circular,
>>> drop the old behavior.
>>>
>>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
>>> (so then we have DATA, BUFFER, CIRCULAR)
>
> Shouldn't this sort of policy be a property of the receiving port, not
> the connection?
If you make it a policy of the Port, than you implicitly add another
"COmmunication" component. Which could be what you want.
>>> Votes ?
>>
>> I am against adding hardcoded Communication policies... If a buffer fills
>> up, the _application_ has to take the action that fits at that moment, so
>> it should react to an event signalling the buffer condition. Trying to
>> "solve" it "once and for all" behind the back of the application is not a
>> good thing.
>
> This is how we do it in OpenRTM, for better or worse. The writer gets
> notified when one of its receivers has an overflow, and the receiver
> gets notified when it has an overflow. I think most users currently
> ignore those callbacks, though.
>
> Geoff
>
>>
>> If you go the suggested way, you could (should?) add LIFO behaviour to the
>> FIFO (current BUFFER) and CIRCULAR options?
>>
>>> Peter
Herman
Introducing circular buffers in connections...
On 15/07/11 17:12, Herman Bruyninckx wrote:
> On Thu, 14 Jul 2011, Geoffrey Biggs wrote:
>
>> On 13/07/11 14:20, Herman Bruyninckx wrote:
>>> On Tue, 12 Jul 2011, Peter Soetens wrote:
>>>
>>>> A frequently popping up issue users have is that they see 'old' data
>>>> in their
>>>> buffer because the new samples got rejected since it's full. There
>>>> are two
>>>> solutions to this:
>>>>
>>>> 1. Change the behavior of the BUFFER type connection policy to be
>>>> circular,
>>>> drop the old behavior.
>>>>
>>>> 2. Add a CIRCULAR type connection policy which is like BUFFER, but
>>>> circular.
>>>> (so then we have DATA, BUFFER, CIRCULAR)
>>
>> Shouldn't this sort of policy be a property of the receiving port, not
>> the connection?
>
> If you make it a policy of the Port, than you implicitly add another
> "COmmunication" component. Which could be what you want.
I guess it depends on where your buffers are, because ultimately, that's
where the policy belongs.
Geoff
>
>>>> Votes ?
>>>
>>> I am against adding hardcoded Communication policies... If a buffer
>>> fills
>>> up, the _application_ has to take the action that fits at that
>>> moment, so
>>> it should react to an event signalling the buffer condition. Trying to
>>> "solve" it "once and for all" behind the back of the application is
>>> not a
>>> good thing.
>>
>> This is how we do it in OpenRTM, for better or worse. The writer gets
>> notified when one of its receivers has an overflow, and the receiver
>> gets notified when it has an overflow. I think most users currently
>> ignore those callbacks, though.
>>
>> Geoff
>>
>>>
>>> If you go the suggested way, you could (should?) add LIFO behaviour
>>> to the
>>> FIFO (current BUFFER) and CIRCULAR options?
>>>
>>>> Peter
>
> Herman
>
>
Introducing circular buffers in connections...
On Jul 12, 2011, at 11:47 , Peter Soetens wrote:
> A frequently popping up issue users have is that they see 'old' data in their
> buffer because the new samples got rejected since it's full. There are two
> solutions to this:
>
> 1. Change the behavior of the BUFFER type connection policy to be circular,
> drop the old behavior.
>
> 2. Add a CIRCULAR type connection policy which is like BUFFER, but circular.
> (so then we have DATA, BUFFER, CIRCULAR)
How much extra work and extra code between the two options? Option 1 breaks existing applications ... one would argue that that should only be done with good justification.
S
Introducing circular buffers in connections...
On Tuesday 12 July 2011 18:26:56 S Roderick wrote:
> On Jul 12, 2011, at 11:47 , Peter Soetens wrote:
> > A frequently popping up issue users have is that they see 'old' data in
> > their buffer because the new samples got rejected since it's full. There
> > are two solutions to this:
> >
> > 1. Change the behavior of the BUFFER type connection policy to be
> > circular, drop the old behavior.
> >
> > 2. Add a CIRCULAR type connection policy which is like BUFFER, but
> > circular. (so then we have DATA, BUFFER, CIRCULAR)
>
> How much extra work and extra code between the two options? Option 1 breaks
> existing applications ... one would argue that that should only be done
> with good justification. S
The changes of both options are trivial.
Basically, instead of returning 'false' on full, we pop one element and then
push it anyway. This can be done in a loop in case its a Multi-Writer buffer
(to work out races).
Peter
Introducing circular buffers in connections...
2011/7/12 Peter Soetens <peter [..] ...>
> On Tuesday 12 July 2011 18:26:56 S Roderick wrote:
> > On Jul 12, 2011, at 11:47 , Peter Soetens wrote:
> > > A frequently popping up issue users have is that they see 'old' data in
> > > their buffer because the new samples got rejected since it's full.
> There
> > > are two solutions to this:
> > >
> > > 1. Change the behavior of the BUFFER type connection policy to be
> > > circular, drop the old behavior.
> > >
> > > 2. Add a CIRCULAR type connection policy which is like BUFFER, but
> > > circular. (so then we have DATA, BUFFER, CIRCULAR)
>
+1
> >
> > How much extra work and extra code between the two options? Option 1
> breaks
> > existing applications ... one would argue that that should only be done
> > with good justification. S
>
> The changes of both options are trivial.
>
> Basically, instead of returning 'false' on full, we pop one element and
> then
> push it anyway. This can be done in a loop in case its a Multi-Writer
> buffer
> (to work out races).
>
> Peter
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>
Introducing circular buffers in connections...
On 07/12/2011 05:47 PM, Peter Soetens wrote:
> A frequently popping up issue users have is that they see 'old' data in their
> buffer because the new samples got rejected since it's full. There are two
> solutions to this:
>
> 1. Change the behavior of the BUFFER type connection policy to be circular,
> drop the old behavior.
+1 for (1), make DATA an alias for BUFFER of size 1 and reduce the code
size in the process. (2) would increase it (right ?)
If you are changing the policy interface, I would be a happy man if you
could introduce BUFFER_NO_OLDDATA which does not store the last read
sample (i.e. have only NoData/NewData)
Introducing circular buffers in connections...
On Tuesday 12 July 2011 17:51:20 Sylvain Joyeux wrote:
> On 07/12/2011 05:47 PM, Peter Soetens wrote:
> > A frequently popping up issue users have is that they see 'old' data in
> > their buffer because the new samples got rejected since it's full. There
> > are two solutions to this:
> >
> > 1. Change the behavior of the BUFFER type connection policy to be
> > circular, drop the old behavior.
>
> +1 for (1), make DATA an alias for BUFFER of size 1 and reduce the code
> size in the process.
DATA is more efficient than (circular) BUFFER iirc, but I'd need tests to know
how much more efficient... It will certainly reduce code size if only one class
of buffer exists (next to locked/lock-free/unsync).
> (2) would increase it (right ?)
Not much. I would keep a flag in each buffer that says if its circular or not.
The flag controls 'return false' or 'pop&push'. It will add some bytes, but far
less then adding a new template class...
>
> If you are changing the policy interface, I would be a happy man if you
> could introduce BUFFER_NO_OLDDATA which does not store the last read
> sample (i.e. have only NoData/NewData)
Yes. Maybe we should have solved it like this from the start and avoid the
'olddata' flag in read()... ? I noticed people don't understand why they need
this flag...
Peter
Introducing circular buffers in connections...
On 07/12/2011 06:52 PM, Peter Soetens wrote:
> On Tuesday 12 July 2011 17:51:20 Sylvain Joyeux wrote:
>> On 07/12/2011 05:47 PM, Peter Soetens wrote:
>>> A frequently popping up issue users have is that they see 'old' data in
>>> their buffer because the new samples got rejected since it's full. There
>>> are two solutions to this:
>>>
>>> 1. Change the behavior of the BUFFER type connection policy to be
>>> circular, drop the old behavior.
>>
>> +1 for (1), make DATA an alias for BUFFER of size 1 and reduce the code
>> size in the process.
>
> DATA is more efficient than (circular) BUFFER iirc, but I'd need tests to know
> how much more efficient... It will certainly reduce code size if only one class
> of buffer exists (next to locked/lock-free/unsync).
By more efficient, you mean speed or size ?
>> (2) would increase it (right ?)
>
> Not much. I would keep a flag in each buffer that says if its circular or not.
> The flag controls 'return false' or 'pop&push'. It will add some bytes, but far
> less then adding a new template class...
Fair enough. We should probably have both then ...
>> If you are changing the policy interface, I would be a happy man if you
>> could introduce BUFFER_NO_OLDDATA which does not store the last read
>> sample (i.e. have only NoData/NewData)
>
> Yes. Maybe we should have solved it like this from the start and avoid the
> 'olddata' flag in read()... ? I noticed people don't understand why they need
> this flag...
Mmmmm... At the time, it felt like it was two different use cases. The
false flag tells the RTT that it does not need the data /at this point
in the code/ while, in the other, the system deployment forbids to have
old data at all.
Moreover, in the first case you can get an OldData (because the code
*knows* that it gets notified without samples). In the second case, you
will *have* to return NoData as no samples have been copied *and* the
component has no way to know that.
AFAIK, they don't "need" this flag. They can use it if they want to, but
not using it is a perfectly sane default.
Introducing circular buffers in connections...
On Tue, Jul 12, 2011 at 7:14 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:
> On 07/12/2011 06:52 PM, Peter Soetens wrote:
>>
>> On Tuesday 12 July 2011 17:51:20 Sylvain Joyeux wrote:
>>>
>>> On 07/12/2011 05:47 PM, Peter Soetens wrote:
>>>>
>>>> A frequently popping up issue users have is that they see 'old' data in
>>>> their buffer because the new samples got rejected since it's full. There
>>>> are two solutions to this:
>>>>
>>>> 1. Change the behavior of the BUFFER type connection policy to be
>>>> circular, drop the old behavior.
>>>
>>> +1 for (1), make DATA an alias for BUFFER of size 1 and reduce the code
>>> size in the process.
>>
>> DATA is more efficient than (circular) BUFFER iirc, but I'd need tests to
>> know
>> how much more efficient... It will certainly reduce code size if only one
>> class
>> of buffer exists (next to locked/lock-free/unsync).
>
> By more efficient, you mean speed or size ?
Both. The DataObjectLockFree in data flow connections has been
optimized for single writer, single reader and is very compact code.
The buffer variant is two template classes to start with and both are
larger than DataObjectLockFree.
>
>>> (2) would increase it (right ?)
>>
>> Not much. I would keep a flag in each buffer that says if its circular or
>> not.
>> The flag controls 'return false' or 'pop&push'. It will add some bytes,
>> but far
>> less then adding a new template class...
>
> Fair enough. We should probably have both then ...
OK.
>
>>> If you are changing the policy interface, I would be a happy man if you
>>> could introduce BUFFER_NO_OLDDATA which does not store the last read
>>> sample (i.e. have only NoData/NewData)
>>
>> Yes. Maybe we should have solved it like this from the start and avoid the
>> 'olddata' flag in read()... ? I noticed people don't understand why they
>> need
>> this flag...
>
> Mmmmm... At the time, it felt like it was two different use cases. The false
> flag tells the RTT that it does not need the data /at this point in the
> code/ while, in the other, the system deployment forbids to have old data at
> all.
>
> Moreover, in the first case you can get an OldData (because the code *knows*
> that it gets notified without samples). In the second case, you will *have*
> to return NoData as no samples have been copied *and* the component has no
> way to know that.
You're right. I'm wondering now if OldData should fill in at all ever. The input
port api forces you to provide a variable to receive the data. If you want to
keep a received sample, use a member variable, otherwise, put the
variable in the local scope...
Also, the buffer channels need to keep an extra copy of the data, just for this
case...
I don't regret the OldData / NoData / NewData interface, I do regret
that OldData
modifies the arg of read()...
Peter
--
Orocos-Dev mailing list
Orocos-Dev [..] ...
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
Introducing circular buffers in connections...
On 07/12/2011 10:12 PM, Peter Soetens wrote:
> Also, the buffer channels need to keep an extra copy of the data, just for this
> case...
Which is my problem: it breaks the purpose of the ReadOnlyPointer since
the dataflow itself saves one sample.
> I don't regret the OldData / NoData / NewData interface, I do regret
> that OldData
> modifies the arg of read()...
+1 on this analysis. I did not realize it before you said it.
--
Sylvain Joyeux (Dr.Ing.)
Space & Security Robotics
!!! Achtung, neue Telefonnummer!!!
Standort Bremen:
DFKI GmbH
Robotics Innovation Center
Robert-Hooke-Straße 5
28359 Bremen, Germany
Phone: +49 (0)421 178-454136
Fax: +49 (0)421 218-454150
E-Mail: robotik [..] ...
Weitere Informationen: http://www.dfki.de/robotik
-----------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Straße 122, D-67663 Kaiserslautern
Geschaeftsfuehrung: Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster
(Vorsitzender) Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats: Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
Sitz der Gesellschaft: Kaiserslautern (HRB 2313)
USt-Id.Nr.: DE 148646973
Steuernummer: 19/673/0060/3
-----------------------------------------------------------------------
--
Orocos-Dev mailing list
Orocos-Dev [..] ...
http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev