On Tue, Aug 25, 2009 at 12:10, Markus
Klotzbuecher<markus [dot] klotzbuecher [..] ...> wrote:
> On Tue, Aug 25, 2009 at 09:38:49AM +0200, Herman Bruyninckx wrote:
>> On Mon, 24 Aug 2009, Peter Soetens wrote:
>>
>> > On Mon, Aug 24, 2009 at 14:22, Markus
>> > Klotzbuecher<markus [dot] klotzbuecher [..] ...> wrote:
>> >> On Mon, Aug 24, 2009 at 01:21:13PM +0200, S Roderick wrote:
>> >>> On Aug 24, 2009, at 05:33 , Markus Klotzbuecher wrote:
>> >>>
>> >>>> On Mon, Aug 24, 2009 at 11:19:58AM +0200, Markus Klotzbuecher wrote:
>> >>>>> On Sun, Aug 23, 2009 at 03:13:19PM +0200, Peter Soetens wrote:
>> >>>>>
>> >>>>>>> Now, it is actually possible to have a MO/SI model, by allowing
>> >>>>>>> an input port
>> >>>>>>> to have multiple incoming channels, and having InputPort::read
>> >>>>>>> round-robin on
>> >>>>>>> those channels. As an added nicety, one can listen to the "new
>> >>>>>>> data" event and
>> >>>>>>> access only the port for which we have an indication that new
>> >>>>>>> data can be
>> >>>>>>> available. Implementing that would require very little added
>> >>>>>>> code, since the
>> >>>>>>> management of multiple channels is already present in OutputPort.
>> >>>>>>
>> >>>>>> Well, a 'polling' + keeping a pointer to the last read channel, such
>> >>>>>> that we try that one always first, and only start polling if that
>> >>>>>> channel turned out 'empty'. This empty detection is problematic for
>> >>>>>> shared data connections (opposed to buffered),  because once they
>> >>>>>> are
>> >>>>>> written, they always show a valid value. We might need to add that
>> >>>>>> once an input port is read, the sample is consumed, and a next read
>> >>>>>> will return false (no new data).
>> >>>>>
>> >>>>> Is there really a usecase for multiple incoming but unbuffered
>> >>>>> connections? It seems to me that the result would be quite arbitrary.
>> >>>>
>> >>>> Of course there is. If you think at a more broader scope there could
>> >>>> be a coordination component controlling the individual components such
>> >>>> that the results are not arbitrary at all.
>> >>>>
>> >>>> In fact this is a good example of explicit vs. implicit coordination.
>> >>>
>> >>> This is _exactly_ the situation we have in our projects. Multiple
>> >>> components with unbuffered output connections, to a single input
>> >>> connection on another component. A coordination component ensures that
>> >>> only one of the input components is running at a time, but they are
>> >>> all connected.
>> >>>
>> >>> Here, we want the latest data value available. No more, no less.
>> >>
>> >> And if you used bufferports of capacity 1 instead of dataports? This
>> >> would have the additional benefit of replacing polling by event driven
>> >> behavior. I must admit I still find this sampling of dataport values a
>> >> bit odd.
>> >
>> > First of all, there is no such thing as data ports or buffer ports (in
>> > RTT 2.0). There are only connection policies, and there are currently
>> > two major ones: 'shared data' and 'buffered'. They allow the
>> > application builder to choose which type of data exchange is the most
>> > efficient between components A and B. For example, in a completely
>> > synchronous data flow system, there is no use in installing buffers
>> > between components, just 'sharing data' is sufficient. There are other
>> > factors in play as well. As we discussed this earlier, we should write
>> > this down in the wiki such that application builders can make an
>> > informed choice.
>>
>> Thanks for this summary. I badly needed it :-)
>>
>> > In principle, as Herman pointed out, the algorithm in the component
>> > should be completely independent of a connection policy. That's also
>> > why he opposes the default policy settings of the input port. It
>> > 'taints' a clean separation of the 4 C's. Imho, convenience sometimes
>> > overrules clean design. This is especially so for small applications
>> > where 'specifying something in one place' overrides 'separation of
>> > concerns'. In practice this means that a user prefers 1 file of 40
>> > lines than 4 files of 10 lines, while on the other hand perfers 10
>> > files of 4.000 lines over 1 file of 40.000 lines of code. We have to
>> > accommodate both cases.
>> >
>> > The latter aside, the algorithm vs connection policy independence is
>> > the fundamental guideline for designing the input-port and output-port
>> > interface in C++. That's why we need to think thourougly over the
>> > semantics of read() and write(). We decided that write() was sent and
>> > forget. For read, it now returns a bool with the following 'truth'
>> > table:
>> >
>> > {{{
>> > Table of return values or read( Sample ) (True/False):
>> > status\policy    |  data  |  buffered |
>> > not connected  |          |              |
>> > or never written |    F    |      F      |
>> > connected,       |          |              |
>> > written in past   |   T     |      F      |
>> > connected, and |          |              |
>> > new data          |    T    |       T      |
>> > }}}
>> >
>> > There are in fact three states and only two return values. I believe
>> > that for uniformity and algorithm independence, we might benefit from
>> > a three state return value as well. As such, the algorithm or
>> > component logic can decide: what to do if the ports are not connected
>> > (flag error, which is caught by the coordination layer); what to do if
>> > some port did not receive new data (but others might have); what to do
>> > if a new sample has arrived. These decisions are independent of the
>> > connection policy. If on the other hand, only two return values are
>> > provided (like in the table), the algorithm might have to make
>> > assumptions on the type of connection policy because the middle case
>> > has different return values for data or buffered.
>> >
>> > So I'm basically agreeing with Markus here: to the algorithm, the
>> > ports only see samples ( as if everything is buffered) and the choice
>> > of data vs buffered is 'just' an optimization strategy when setting up
>> > the intercomponent connections. But there still remain three cases
>> > independent of the policy: 'not connected', 'connected but no new
>> > data' and 'connected and new data'.
>>
>> For the reader the first two cases do not make a difference at all! The
>> result is that the component wanted data that is not there, and it cannot
>> do anything about it (i.e,, it can (should!) do nothing about the
>> connection problem, or about the data-not-being-there problem). Hence, I
>> see no need to reflect this status difference in the read() return value.
>
> I don't agree generally that data-not-being-there should be treated
> identically as a connection error. This might be true for certain
> components which rely on data always being available, but might not be
> true for others which check for data but continue doing something else
> if none is available. Example state machine, the UML event queue is
> modeled as a port with a buffered policy. If event in queue then
> transition, else continue executing "do" behavior.

I agree with Markus again: one state is an error, the two others are
data/no data.

>
>> And why is the write() case so different? It also should have a return
>> value that indicates whether the write was successful or not. (But it
>> should not have to know the _cause_ of the failure, similarly to the read()
>> call.)
>
> One problem I see is defining what successful means. Does is mean the
> data was locally stored in the output queue or that is was put in the
> recipient queue or read by the recipient? I suppose these could be QoS
> policies of the connection. But when does a writer need to know if the
> write was successful or not? One situation could be the case where it
> is absolutely necessary that a particular datum is delivered. The
> sequence of coordination (e.g. for a disconnected cable) could be as
> follows:
>
> 1) component writes value to port, but gets an error. The same error
>   is propagated to the coordinator
>
> 2) coordinator stops writer and reader, removes connection and creates
>   backup connection through some other physical connection
>
> 3) coordinator restarts writer and reader, writer retries.
>
> However this coordination would require knowledge on the writer side
> (namely to retry), so computation and coordination is slightly
> coupled. The only way to avoid this I can see would be to have a local
> "reliable transmission" helper component which takes care of the above
> sequence, without complicating the actual computation. Or is this
> example to contrived?

It is. The component can't know the sample 'must' be delivered. That
is a system property. So logically, the component can only push stuff
into its output port and assume that the system will take action if
there are no receivers but there should have been.

Peter

On Tuesday 25 August 2009 12:10:39 Markus Klotzbuecher wrote:
> On Tue, Aug 25, 2009 at 09:38:49AM +0200, Herman Bruyninckx wrote:
> > On Mon, 24 Aug 2009, Peter Soetens wrote:
> > > On Mon, Aug 24, 2009 at 14:22, Markus
> > >
> > > Klotzbuecher<markus [dot] klotzbuecher [..] ...> wrote:
> > >> On Mon, Aug 24, 2009 at 01:21:13PM +0200, S Roderick wrote:
> > >>> On Aug 24, 2009, at 05:33 , Markus Klotzbuecher wrote:
> > >>>> On Mon, Aug 24, 2009 at 11:19:58AM +0200, Markus Klotzbuecher wrote:
> > >>>>> On Sun, Aug 23, 2009 at 03:13:19PM +0200, Peter Soetens wrote:
> > >>>>>>> Now, it is actually possible to have a MO/SI model, by allowing
> > >>>>>>> an input port
> > >>>>>>> to have multiple incoming channels, and having InputPort::read
> > >>>>>>> round-robin on
> > >>>>>>> those channels. As an added nicety, one can listen to the "new
> > >>>>>>> data" event and
> > >>>>>>> access only the port for which we have an indication that new
> > >>>>>>> data can be
> > >>>>>>> available. Implementing that would require very little added
> > >>>>>>> code, since the
> > >>>>>>> management of multiple channels is already present in OutputPort.
> > >>>>>>
> > >>>>>> Well, a 'polling' + keeping a pointer to the last read channel,
> > >>>>>> such that we try that one always first, and only start polling if
> > >>>>>> that channel turned out 'empty'. This empty detection is
> > >>>>>> problematic for shared data connections (opposed to buffered),
> > >>>>>> because once they are
> > >>>>>> written, they always show a valid value. We might need to add that
> > >>>>>> once an input port is read, the sample is consumed, and a next
> > >>>>>> read will return false (no new data).
> > >>>>>
> > >>>>> Is there really a usecase for multiple incoming but unbuffered
> > >>>>> connections? It seems to me that the result would be quite
> > >>>>> arbitrary.
> > >>>>
> > >>>> Of course there is. If you think at a more broader scope there could
> > >>>> be a coordination component controlling the individual components
> > >>>> such that the results are not arbitrary at all.
> > >>>>
> > >>>> In fact this is a good example of explicit vs. implicit
> > >>>> coordination.
> > >>>
> > >>> This is _exactly_ the situation we have in our projects. Multiple
> > >>> components with unbuffered output connections, to a single input
> > >>> connection on another component. A coordination component ensures
> > >>> that only one of the input components is running at a time, but they
> > >>> are all connected.
> > >>>
> > >>> Here, we want the latest data value available. No more, no less.
> > >>
> > >> And if you used bufferports of capacity 1 instead of dataports? This
> > >> would have the additional benefit of replacing polling by event driven
> > >> behavior. I must admit I still find this sampling of dataport values a
> > >> bit odd.
> > >
> > > First of all, there is no such thing as data ports or buffer ports (in
> > > RTT 2.0). There are only connection policies, and there are currently
> > > two major ones: 'shared data' and 'buffered'. They allow the
> > > application builder to choose which type of data exchange is the most
> > > efficient between components A and B. For example, in a completely
> > > synchronous data flow system, there is no use in installing buffers
> > > between components, just 'sharing data' is sufficient. There are other
> > > factors in play as well. As we discussed this earlier, we should write
> > > this down in the wiki such that application builders can make an
> > > informed choice.
> >
> > Thanks for this summary. I badly needed it :-)
> >
> > > In principle, as Herman pointed out, the algorithm in the component
> > > should be completely independent of a connection policy. That's also
> > > why he opposes the default policy settings of the input port. It
> > > 'taints' a clean separation of the 4 C's. Imho, convenience sometimes
> > > overrules clean design. This is especially so for small applications
> > > where 'specifying something in one place' overrides 'separation of
> > > concerns'. In practice this means that a user prefers 1 file of 40
> > > lines than 4 files of 10 lines, while on the other hand perfers 10
> > > files of 4.000 lines over 1 file of 40.000 lines of code. We have to
> > > accommodate both cases.
> > >
> > > The latter aside, the algorithm vs connection policy independence is
> > > the fundamental guideline for designing the input-port and output-port
> > > interface in C++. That's why we need to think thourougly over the
> > > semantics of read() and write(). We decided that write() was sent and
> > > forget. For read, it now returns a bool with the following 'truth'
> > > table:
> > >
> > > {{{
> > > Table of return values or read( Sample ) (True/False):
> > > status\policy | data | buffered |
> > > not connected | | |
> > > or never written | F | F |
> > > connected, | | |
> > > written in past | T | F |
> > > connected, and | | |
> > > new data | T | T |
> > > }}}
> > >
> > > There are in fact three states and only two return values. I believe
> > > that for uniformity and algorithm independence, we might benefit from
> > > a three state return value as well. As such, the algorithm or
> > > component logic can decide: what to do if the ports are not connected
> > > (flag error, which is caught by the coordination layer); what to do if
> > > some port did not receive new data (but others might have); what to do
> > > if a new sample has arrived. These decisions are independent of the
> > > connection policy. If on the other hand, only two return values are
> > > provided (like in the table), the algorithm might have to make
> > > assumptions on the type of connection policy because the middle case
> > > has different return values for data or buffered.
> > >
> > > So I'm basically agreeing with Markus here: to the algorithm, the
> > > ports only see samples ( as if everything is buffered) and the choice
> > > of data vs buffered is 'just' an optimization strategy when setting up
> > > the intercomponent connections. But there still remain three cases
> > > independent of the policy: 'not connected', 'connected but no new
> > > data' and 'connected and new data'.
> >
> > For the reader the first two cases do not make a difference at all! The
> > result is that the component wanted data that is not there, and it cannot
> > do anything about it (i.e,, it can (should!) do nothing about the
> > connection problem, or about the data-not-being-there problem). Hence, I
> > see no need to reflect this status difference in the read() return value.
>
> I don't agree generally that data-not-being-there should be treated
> identically as a connection error. This might be true for certain
> components which rely on data always being available, but might not be
> true for others which check for data but continue doing something else
> if none is available. Example state machine, the UML event queue is
> modeled as a port with a buffered policy. If event in queue then
> transition, else continue executing "do" behavior.
>
> > And why is the write() case so different? It also should have a return
> > value that indicates whether the write was successful or not. (But it
> > should not have to know the _cause_ of the failure, similarly to the
> > read() call.)
>
> One problem I see is defining what successful means. Does is mean the
> data was locally stored in the output queue or that is was put in the
> recipient queue or read by the recipient? I suppose these could be QoS
> policies of the connection. But when does a writer need to know if the
> write was successful or not? One situation could be the case where it
> is absolutely necessary that a particular datum is delivered. The
> sequence of coordination (e.g. for a disconnected cable) could be as
> follows:
>
> 1) component writes value to port, but gets an error. The same error
> is propagated to the coordinator
>
> 2) coordinator stops writer and reader, removes connection and creates
> backup connection through some other physical connection
>
> 3) coordinator restarts writer and reader, writer retries.
>
> However this coordination would require knowledge on the writer side
> (namely to retry), so computation and coordination is slightly
> coupled. The only way to avoid this I can see would be to have a local
> "reliable transmission" helper component which takes care of the above
> sequence, without complicating the actual computation. Or is this
> example to contrived?

As you wrote it yourself, the overall fault-tolerance is achieved by a
coordinator component, not by the writer himself. Hence, the writer does not
have to know if the write was successful or not. I.e. it is achieved by a
component that does not have access to the return value of write() ...

In my opinion, what one needs is an introspection interface for the transport.
The most important feature of that interface is that the individual components
should not have to know about it, only the supervision components should.

Now, on the return value of read(). I think there is two visions about the
data connections:
- the "distributed state" vision. When you write to a data connection, you
just update part of the state with the latest estimate. The point being that,
by reading the connection, you always get the latest state estimate.
- the "data item" vision. In that case, data connections are just like buffer
connections of size 1. I actually don't see the "optimization" claim of Peter
-- I don't see how a data connection is an optimization versus a buffer of size
1.

In my opinion, there is room for both buffer and data connections as they are
now. The only issue with data connections is that they are completely
unsuitable for polling (as there is always data on them). I actually feel like
the tri-state return value is the best, as it unifies data and buffers (and,
later on, ring buffers if they are implemented). If a component only cares
about data availability, then it just tests the return value as true/false.
Otherwise, it can have the information available to him *regardless of the
type of connection it is reading from*.