> Some classic producer/consumer examples
> - a camera captures images and sends them on to a processing pipeline (that
> is all data object connections). A separate connection stores the camera
> images in a buffer, which are dumped to disk.
> - data is read (non-real-time) from a file and pushed into a buffer for
> processing (real-time)
> - data samples coming from a remote source (non-real-time) is buffered on
> arrival into the deployer for later processing (real-time).

Asked the wrong question ... thanks for answering it perfectly ;-)

Anyways, what I wanted is a case where:
- an input port is connected to a bunch of output ports *using
buffers* (I got the problem with data policies)
- the overall behavior is governed by maintaining the overall
incoming sample order (i.e. that if a sample of output port O_1
arrives before a sample of output port O_2, that order needs to be
reflected on read())

I may be missing it, but the use-cases above don't cover that, do they ?

> > Agreed, but this choice is currently bundled with having a buffer per
> > channel or one per input port.
>
> I don't follow, Sylvain. Bundled in the existing implementation or the
> proposed changes?

In the proposed change (current v2 does not have the option of having
a single buffer per input port). Until now, it is clear to me that
this bundling is both Johannes and Peter's intent (see Peter's email
above). It does not seem to be yours, though.

> > If you put the buffer per-port, you end up with one policy per port.
> > Port-driven triggering (event ports) are governed by the data object
> > (i.e. per-port), data object size and type becomes per-port as well.
> > What's left in the connection policy is the info necessary to reach
> > the other side (i.e. the topic name in ROS), which means that it is
> > not a policy anymore but a "remote endpoint"

> From your comment above "Port-driven triggering (event ports) …"

Badly ordered bits of sentences. What I meant is:

> If you put the buffer per-port, you end up with one policy per port. Data object size and type becomes per-port. If using port-driven triggering (event ports), the triggering is governed by the data object and is therefore also becoming per-port.

> Looking at Section 5.1 of Johannes' google doc, something looks wrong or I'm
> remembering previous discussions incorrectly. I was expecting push/pull vs
> shared/private to be truly orthogonal, but the push/private figure says
> differently. Johannes? Peter?

They already clarified that for me. Basically, "private" does not
necessarily means "per channel". On a push connection, "private" for
them means "per-port".

> Again, my intent was to add more v1-like semantics to the system, as well as
> have more control on where latency occurred. Not to replace any aspect of
> the v2 semantics.

I got that from your previous email already, but it seems that it's
not Johannes/Peter's intent

Sylvain

2015-09-28 14:30 GMT-03:00 S Roderick <kiwi [dot] net [..] ...>:
> On Sep 28, 2015, at 10:28 AM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:
>
> I agree with much at the above, and we have similar problems that we've
>
> solved in similar ways. But there are also huge parts of our system that,
>
> while "data driven", aren't at all interested in time. The system itself is
>
> synchronized and time is (largely) irrelevant - everything operates on a
>
> per-cycle basis. In the long run, we time stamp only a tiny portion of the
>
> data in our system due to this. Equally valid, but different.
>
>
> I don't dispute that you probably have a very good point, you usually
> do, but could you provide me with a concrete example of such a setup
> that uses buffers ? It would really help me grasp the situation.
>
>
> Some classic producer/consumer examples
> - a camera captures images and sends them on to a processing pipeline (that
> is all data object connections). A separate connection stores the camera
> images in a buffer, which are dumped to disk.
> - data is read (non-real-time) from a file and pushed into a buffer for
> processing (real-time)
> - data samples coming from a remote source (non-real-time) is buffered on
> arrival into the deployer for later processing (real-time).
>
> Sometimes we want to control whether the network access (and hence
>
> delay/non-real-time part) is on the writer or reader side. That should be an
>
> application-level choice.
>
>
> Agreed, but this choice is currently bundled with having a buffer per
> channel or one per input port.
>
>
> I don't follow, Sylvain. Bundled in the existing implementation or the
> proposed changes?
>
> If you put the buffer per-port, you end up with one policy per port.
>
> Port-driven triggering (event ports) are governed by the data object
>
> (i.e. per-port), data object size and type becomes per-port as well.
>
> What's left in the connection policy is the info necessary to reach
>
> the other side (i.e. the topic name in ROS), which means that it is
>
> not a policy anymore but a "remote endpoint"
>
>
>
> What if you don't use event ports?
>
>
> How is that relevant ? Are we misunderstanding each other here ?
>
>
> From your comment above "Port-driven triggering (event ports) …"
>
> As I understand the proposal so far, we have: either you pull (hit
> network on read, one buffer per channel) or push (not hit network on
> read, have a one buffer per port). The middle ground of
> one-buffer-per-channel and not hitting network is not covered, and
> that's what *greatly* bothers me.
>
>
> Our original intent was that pull/push would be independent of
> shared/private (if you mean that by "one buffer per channel" and "one buffer
> per port").
>
> I know you guys probably have a good case here, I am trying very hard
> for you to not break mine
>
>
> I'm with you on this, Sylvain!
>
> Note the "selectively remove" above - we still want to be able to use
>
> v2-flow-status type setups within our system.
>
>
> My understanding of the proposal is that the v2 semantic are not
> available anymore. I'm not pushing against what you need, I want you
> to not break what I need.
>
>
> I'm personally against changing the default, even though it would help me.
> This kind of pervasive breakage isn't good for the community ...
>
> - what are going to be the options of network/no-network and
> placement of buffer ?
>
> - how can these options be or not be used with a single output /
> single input ? Are they mutually exclusive ? Can they be used together
> on the same input port ? On the same output port ?
>
>
> Looking at Section 5.1 of Johannes' google doc, something looks wrong or I'm
> remembering previous discussions incorrectly. I was expecting push/pull vs
> shared/private to be truly orthogonal, but the push/private figure says
> differently. Johannes? Peter?
>
> In particular, from Section 5.2:
>
> Other than currently in v2, in the push case (pull=false) there will be only
> one buffer per input port, that is shared among all connections. The
> connected output ports “push” their data through the channel and fill the
> input buffer (or overwrite the data object), but every input port has its
> own buffer or data object and can read independently (Fig. 4)
>
> Why doesn't every input port have a connection object per associated output
> port? Why isn't the push/private figure the same as the pull/private figure,
> except that the connection object is on the reader side (and for each input
> port, you'd have one connection object per output port)? Or is this to try
> and fix the non-determinstic behavior when an InputPort reads from multiple
> connection objects?
>
> We have several hundred local connections in our deployers. We have maybe
>
> less than 10 remote connections.
>
>
> Again, how is this relevant ? If the policy is specified per-port, you
> won't have any "kind-of-sane default" to configure the CORBA
> dispatchers with.
>
> Buffers vs data connections definitely change the flavor of things. The vast
>
> majority of our connections are data connections, not buffers.
>
>
> I *know* which leads to this long discussion, because I am on the
> completely other end of things: we mostly use buffered connections.
>
>
> Which is good IMHO, because it means that you and I effectively span the
> spectrum of system approaches. So if we can find a way to satisfy both our
> system approaches, then likely we'll cater for everyone else as well.
>
> I think that we're getting off topic as a whole.
>
>
> I don't think so. So far, my concerns with the proposal are still
> there: by getting closer to v1 semantics, the proposal breaks v2
> semantics in a way that is fundamental to me. Peter's post actually
> leads me to think that this breakage and known and desired, which is
> why I am spending so much time trying to get my point across, while I
> already have way too much on my plate.
>
>
> Again, my intent was to add more v1-like semantics to the system, as well as
> have more control on where latency occurred. Not to replace any aspect of
> the v2 semantics.
>
> a) push vs pull, or specifying whether the data/buffer object is on the
>
> reader or writer side. This is very useful for system designers dealing with
>
> significant time delay.
>
> b) private vs shared, whether a connection is per input/output port pair or
>
> whether the connection is shared between multiple input/output ports.
>
>
> So far, "private" is defined one buffer *per input port* (change in
> semantic) not *per input/output port pair* (current v2 semantic)
>
>
> I think we should postpone the "changing the defaults" discussion. For a
> v2.9 type release that could include the proposed changes, having the
> default not change would be of most benefit to the community I think. In the
> future, if/when we decide that a different set of defaults is more
> appropriate, we could phase that into future releases in a way that
> minimizes disruption to the community.
>
> Cheers
> Stephen

On Mon, 28 Sep 2015, Johannes Meyer wrote:

> Hi,
> thanks for all your input and ideas and I agree with Herman that the "one-framework-for-everything" approach of RTT
> is probably not how you would start a new project nowadays.

Some inputs from a discussion on the ROS mailinglist which is very much
related to this RTT-centric thread:
<http://lists.ros.org/pipermail/ros-users/2015-September/069663.html>
<http://lists.ros.org/pipermail/ros-users/2015-September/069664.html>

These are the critical remarks of two experienced people towards the
"one size fits all" approach of communication in ROS2.0; at the same time,
they also provide concrete use cases and trade-offs that are relevant for
the envisaged update of comunication in RTT...

Herman

PS I am of course more than a bit biased: both messages reflect also my
personal view on this issue of "communication middleware", and are not just
random samples from that ROS thread.

PS2 Geoff Biggs is another RTT-knowledgeable person who _did_ intervene in
that ROS thread, as well as in this RTT thread. Are there others on this
list who, like Geoff and myself, are following both threads? Because,
together, we might "mine" both threads to extract very interesting inputs
from potential users, with widely varying expectations and contexts...

> But I would like to support Stephen's call to focus this
> discussion on the technical aspects of how to improve the existing implementation in as small as possible
> ("incremental") steps that add support for some new use cases that are not covered yet (but have been partially
> covered by v1).

> Maintaining backwards compatibility is definitely the way to go and we can conclude that the new default semantics
> (especially per-connection vs. per-input buffers) have to be the same as in previous v2 releases. I hope you find it
> acceptable that a new version would certainly be not ABI-compatible and breaks some internal API calls that are
> typically not used directly by a user. This might break more advanced things like custom transport implementations.
> I fear that these kind of changes cannot be avoided completely, especially for the ConnFactory and ChannelElement
> classes.
>
> We should not limit the use cases of RTT on the core API level, like enforcing per-input storage objects for data
> connections and per-connection storage for buffer connections, but only add new use cases. It is more a question of
> having good documentation and choosing reasonable defaults that cover as many applications as possible, but at the
> end the application designer has to have the freedom to deploy the application with whatever connection semantics it
> demands for.
>
> In the following I try to wrap up and comment on some of the concerns that came up during the discussion so far, and
> suggest compromise solutions which have been at least partially discussed with Peter:
>
> 1) There is definitely a need for per-input port buffers AND for per-connection buffers - INDEPENDENT of whether the
> data/buffer object is installed on the reader and writer side.
>
> Indeed, according to the original proposal the questions of at what side of a remote connection the data object will
> be installed and whether there will be a per-input or a per-connection buffer cannot specified separately. Peter and
> I could not imagine why you would want separate buffers if the data is already available locally anyways (in the
> push case) and we still think that at least for data connections the current per-connection implementation can be
> considered as broken. Even with readNewest() there is no guarantee that you get the latest sample written. On the
> other hand, Sylvain correctly stated that there are no guarantees anyway as soon as remote connections (or any
> non-realtime transport) are considered and the only true solution is proper timestamping and synchronization. But I
> think we agree that for most deployments with only local connections the assumption that the latest sample is the
> last written sample holds and that timestamping is beyond the scope of RTT and can be left to user code or more
> advanced tools like the Rock stream aligner.
>
> Even with the proposed changes (let's call them v3 semantics as a working title) the connection policy is still
> primarily describing the connection, even if *some* settings limit the set of allowed policies for other
> connections, making *some* other settings de-facto a per-port setting. This would be primarily true for input ports
> and not at all for output ports, unless it has a connection of the new shared type. But it is not an option to
> associate the connection policy *only* with a port because not *all* possible connections imply per-port buffers.
> Specifying a connection policy on a writer/reader pair base as it is now and let certain invalid connection attempts
> fail is still the most general approach.
>
> Proposal: In order to not break existing applications, we introduce a new ConnPolicy field "read_policy" that
> toggles between per-input and per-connection buffers, instead of implying that by the existing "pull" flag for
> remote connections. The read_policy would be an enum with two possible values for now, "SingleInputBuffer" and
> "PreferCurrentConnection". The reason why we propose an enum here is that you could also think of other policies on
> how to select the next channel to read from in the per-connection case, e.g. RoundRobin, PreferLastSignaled, etc.
> The current v2 default would correspond to PreferCurrentConnection, as the channel that was last read with new data
> is preferred by the input port and only then it polls all other connections in the order they have been made.
> We have not thought this idea to the end yet and the names of the constants are still under discussion. We do not
> want to call the input buffers "shared", because the read policy is independent of the new concept of a "shared
> connection" where multiple input ports read from the same buffer. Or someone can come up with a better name for this
> type of connection...
>
>
> 2) Should the new per-input port buffers become the new default or not (SingleInputBuffer read policy)?
>
> With that respect the conclusion is that backwards compatibility is more important than the new "features" and the
> default parameters for the current ConnPolicy class should not be touched. However, there are different ways to
> tackle this:
>
> 2.1 Make it a compile-time option.
>
> 2.2 Only add *new* fields to the ConnPolicy class without changing the default parameter values in the constructor
> and static methods, and a new class ConnPolicy2 or ConnPolicy3 that inherits from ConnPolicy and might have other
> default parameters or another C++ API (I am thinking of a named-parameter based API) and can also be used for
> scripting. Internally, the connection factory would work with const references to the base class only.
>
> 2.3 We add a DefaultConnPolicy attribute to the GlobalsRepository. This application-wide (per process) policy
> instance will be copied to newly constructed ConnPolicy instances and the constructors and static methods only
> overwrite the respective parameters.
>
> It might be obsoleted by the above proposal, but in the meantime I added a compile-time option
> RTT_V2_COMPATIBILITY_MODE to my working branch (see ef99f2b), that, if set, reverts to the current v2 per-connection
> model, but lets RTT print meaningful warning messages in the case a new connection would be invalid with v3
> semantics because of conflicts with other connections to the same port. Such a flag could smooth the transition
> phase and help application designers to find the minimal set of connections that require connection policy changes.
>
> There is one small exception from the backwards-compatibility rule, that I think is worth discussing:  The static
> method
>
> static ConnPolicy data(int lock_policy = LOCK_FREE, bool init_connection = true, bool pull = false);
>
> initializes the init_connection field to true by default, which was kind of surprising me, because it means that
> every new connection to an output port with this policy will overwrite a sample previously written by another port.
> Is this what we really want as a default? It is also not consistent with the constructor called as
>
> ConnPolicy(DATA)
>
> which would initialize the init_connection flag to false. The static method data(...) is only used in the
> OutputPortInterface::createDataConnection(...) method from within RTT and OCL itself.
>
> By the way, the init_connection flag is a good example where the API offers some degree of freedom to the
> application builder that semantically does not make sense at all for certain combinations, like in combination with
> a buffer connection. It is still not forbidden to set the flag, but it might be worth a warning log message if a
> connection is created like this.
>
>
> 3) How to add the "shared" concept within the current API, or to use a "new" API?
>
> This point was brought up by Janosch, who asked for "a more explicit interface for shared connections", with the
> main argument that it is fundamentally different from a normal connection and a "connect port A to B with policy C"
> kind of API does not fit well to that concept. That's true. We considered to use the createStream(...) API first,
> with a name_id string that identifies the shared buffer to connect to. This would actually fit better to the shared
> connection model, but it still enforces the same buffer type policies for all connections. We withdraw that idea
> because the existing API is more general and disallowing certain policy combinations is required anyway for private
> push connections with a single input buffer. At least in this case it cannot break existing applications. I do not
> think that there is a need for a completely new API only for shared connections.
>
> My current implementation in the updated-dataflow-semantics branch is as follows, and new/other ideas are always
> welcome:
>
> - Shared connection objects (the shared data object or buffered) are added to a process-wide registry. Every shared
> connection is identified by a unique string, either set explictly in the name_id field of the ConnPolicy or assigned
> automatically, in which case the name of the connection is returned in exactly this field (name_id is and always was
> declared as mutable).
> - For new connections with the name_id set, the shared connection is looked up in the registry, or created as new.
> As a consequence, if the same ConnPolicy instance is used to make a second connection, even if name_id was not set
> explicitly before, you will always connect to the same connection object.
> - For new connections with no name_id set, the port objects provide an API to return a pointer to an existing shared
> connection if they are already connected to an existing one, and only the non-connected port will be added. So ports
> can be connected in any order to each other, as long as there is always one of the two ports already connected.
> Otherwise the name_id has to be set or a new connection instance would be created. Perhaps this one could be dropped
> because it it too implicit and does not work very well for remote connections...
> - If an existing connection was found, the buffer policies (type, size and locking policy) have to match.
> - All ports will only store a single pointer to the shared connection instance in their connection list (in
> internal::ConnectionManager) and cannot know directly which other ports are connected.
> - For remote input ports with a shared connection, only a proxy is created locally and can be looked up in the
> shared connection repository. Other ports will only connect to either the  proxy or the real shared connection
> instance, but never build a new remote channel once the proxy has been created for the first remote connection
> (works in both directions).
> - All connect and disconnect calls to shared connections are logged with log level Debug to the RTT logger, so at
> least there is a manual way to check whether the resulting connections have been made as expected. Graphical tools
> like the rtt_dot_service could be patched later to visualize shared connections correctly.
>
>
> 4) Whether to encode the result of a write() in existing enums or not?
>
> Janosch asked for introducing a separate enum for the return value of write(...) calls, in order to not break or
> trigger compiler warnings for existing code that already uses the FlowStatus enum in a switch statement. Another
> minor issue with a single enum is the default value of FlowStatus attributes, which can be only meaningful for one
> use case.
>
> That's also fine for me and I will update the document and implementation accordingly (add a new enum WriteStatus).
> Note that the evaluation in boolean context is counter-intuitive in this case, other than for FlowStatus, because
> the value WriteSuccess (=0) value evaluates to false and all error cases (WriteFailure and NotConnected) evaluate to
> true.
>
>
> I hope I did not forget anything. This mail already became much longer than expected. I am sorry.
>
> I also did some performance measurements of read and write calls last week in terms of absolute time, CPU time,
> number of assignments and so on of the current, the new and even the v1 data flow implementations in different
> scenarios. I plan to come up with some results during this week. The tests revealed some bugs and performance
> caveats, and some of them are already present since the early days of RTT v2, e.g. that data samples might be lost
> (never read as NewData) because the read flag is not lock-protected or stored within the lock-free data structure. I
> am preparing a pull request to the current master branch, but it will also break the API of class
> base::DataObjectInterface.
>
> Best regards,
> Johannes
>
>
>
> On Mon, Sep 28, 2015 at 3:36 PM, S Roderick <kiwi [dot] net [..] ...> wrote:
>
> Johannes' proposed changes are focussed on
> a) push vs pull, or specifying whether the data/buffer object is on the reader or writer side. This is very
> useful for system designers dealing with significant time delay. 
> b) private vs shared, whether a connection is per input/output port pair or whether the connection is shared
> between multiple input/output ports.
> c) mandatory, primarily intended to support knowing whether writes to a buffer succeeded (so that traditional
> producer/consumer relationships have some knowledge about the buffer state)
>
> It appears to me that most people are worried about changing the defaults, more than they're worried about the
> actual proposed changes. Actual technical concerns with the proposed changes appear to be
>
> 1) whether to encode the result of a write() in existing enums or not
> 2) how to add the "shared" concept within the current API, or to use a "new" API
> 3) changing the RTT defaults, which then affect an application's semantics.
>
> Did I miss any technical issues?
>
>
> My huge personal caveat to all of this is that I do *not* think we should change the current system semantics
> in introducing this change. Too many Orocos changes have broken backwards compatibility and forced system
> designers and implementors to change their system in response. I don't think we should force that on users
> without an incredibly good reason. Otherwise it breeds frustration in the community.
>
> If there is some way to introduce the proposed changes to the system, but to not change the behavior of
> current systems, then I think that is a win for all. There does seem to be some general agreement that there
> are valid use cases that the current port implementation handles poorly, and that this proposed approach does
> fix some of those. The proposed approach also improves the situation for systems that don't want or need to
> use the flow status approach. Both are equally valid approaches - IMHO it would be beneficial to the community
> as a whole if RTT supported both approaches.
>
> Cheers
> Stephen
>
>
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>
>
>
>

Hi Johannes, thanks for the long explanation.

On the implementation side, this whole discussion revived an idea I
have had a while ago, and is pretty much in line with the discussion
Herman started.

Basically, one could get rid of (a.k.a. "deprecate" of course) the
connection API on the ports themselves, and replace it by a separate
object (let's call it "ConnectionManager") which would be in charge of
managing all these buffers and create all these connections and
implement the . In its simplest form, each process would have one of
these objects.

The basic operations it would have support are:
- create/remove storage object (data object / buffer object)
- connect output port to storage object
- connect storage object to input port

A "pull" connection would involve the ConnectionManager of the output
port. A "push" connection the ConnectionManager of the receiving side.
How one gets shared or private (or mixed) connections is obvious.
Flags would be split between the two halves (which is good !). For
instance, 'init' only affects the output->buffer side.

In principle -- but that's a principle that even I take with a big
grain of salt given that I did not seriously touch that stuff for
years -- the current underlying API (creation of the two channel
halves) would be reused. In other words, only where the connections
are created would change, not how they are created.

On top of that, it's trivial to create higher-level, easier-to-use
APIs, and also implement the current connection API so as to not break
existing code.

It would open the path to a lot of interesting things:
- finally have a decent way to monitor the data objects (i.e. buffer
fill rates, "data loss", data counters, that kind of thing). It would
also make obvious how one could replace the RTT "base connection
manager" by something else without touching the components. It would
spell how wonderfully relevant RTT still is to those who don't see it
;-)
- if we add the possibility of inter-ConnectionManager connections
would allow us to move the keepSample functionalities of OutputPort
out of it, which would also make the whole init functionality a lot
more transparent. It would allow to expose (and monitor ...) the CORBA
dispatchers.

Now, I do understand that it might sound "blue sky" and that you might
not have the time for that ... just sayin' ;-)

Sylvain

2015-09-28 16:52 GMT-03:00 Johannes Meyer <johannes [..] ...>:
> Hi,
>
> thanks for all your input and ideas and I agree with Herman that the
> "one-framework-for-everything" approach of RTT is probably not how you would
> start a new project nowadays. But I would like to support Stephen's call to
> focus this discussion on the technical aspects of how to improve the
> existing implementation in as small as possible ("incremental") steps that
> add support for some new use cases that are not covered yet (but have been
> partially covered by v1).
>
> Maintaining backwards compatibility is definitely the way to go and we can
> conclude that the new default semantics (especially per-connection vs.
> per-input buffers) have to be the same as in previous v2 releases. I hope
> you find it acceptable that a new version would certainly be not
> ABI-compatible and breaks some internal API calls that are typically not
> used directly by a user. This might break more advanced things like custom
> transport implementations. I fear that these kind of changes cannot be
> avoided completely, especially for the ConnFactory and ChannelElement
> classes.
>
> We should not limit the use cases of RTT on the core API level, like
> enforcing per-input storage objects for data connections and per-connection
> storage for buffer connections, but only add new use cases. It is more a
> question of having good documentation and choosing reasonable defaults that
> cover as many applications as possible, but at the end the application
> designer has to have the freedom to deploy the application with whatever
> connection semantics it demands for.
>
> In the following I try to wrap up and comment on some of the concerns that
> came up during the discussion so far, and suggest compromise solutions which
> have been at least partially discussed with Peter:
>
> 1) There is definitely a need for per-input port buffers AND for
> per-connection buffers - INDEPENDENT of whether the data/buffer object is
> installed on the reader and writer side.
>
> Indeed, according to the original proposal the questions of at what side of
> a remote connection the data object will be installed and whether there will
> be a per-input or a per-connection buffer cannot specified separately. Peter
> and I could not imagine why you would want separate buffers if the data is
> already available locally anyways (in the push case) and we still think that
> at least for data connections the current per-connection implementation can
> be considered as broken. Even with readNewest() there is no guarantee that
> you get the latest sample written. On the other hand, Sylvain correctly
> stated that there are no guarantees anyway as soon as remote connections (or
> any non-realtime transport) are considered and the only true solution is
> proper timestamping and synchronization. But I think we agree that for most
> deployments with only local connections the assumption that the latest
> sample is the last written sample holds and that timestamping is beyond the
> scope of RTT and can be left to user code or more advanced tools like the
> Rock stream aligner.
>
> Even with the proposed changes (let's call them v3 semantics as a working
> title) the connection policy is still primarily describing the connection,
> even if *some* settings limit the set of allowed policies for other
> connections, making *some* other settings de-facto a per-port setting. This
> would be primarily true for input ports and not at all for output ports,
> unless it has a connection of the new shared type. But it is not an option
> to associate the connection policy *only* with a port because not *all*
> possible connections imply per-port buffers. Specifying a connection policy
> on a writer/reader pair base as it is now and let certain invalid connection
> attempts fail is still the most general approach.
>
> Proposal: In order to not break existing applications, we introduce a new
> ConnPolicy field "read_policy" that toggles between per-input and
> per-connection buffers, instead of implying that by the existing "pull" flag
> for remote connections. The read_policy would be an enum with two possible
> values for now, "SingleInputBuffer" and "PreferCurrentConnection". The
> reason why we propose an enum here is that you could also think of other
> policies on how to select the next channel to read from in the
> per-connection case, e.g. RoundRobin, PreferLastSignaled, etc. The current
> v2 default would correspond to PreferCurrentConnection, as the channel that
> was last read with new data is preferred by the input port and only then it
> polls all other connections in the order they have been made. We have not
> thought this idea to the end yet and the names of the constants are still
> under discussion. We do not want to call the input buffers "shared", because
> the read policy is independent of the new concept of a "shared connection"
> where multiple input ports read from the same buffer. Or someone can come up
> with a better name for this type of connection...
>
>
> 2) Should the new per-input port buffers become the new default or not
> (SingleInputBuffer read policy)?
>
> With that respect the conclusion is that backwards compatibility is more
> important than the new "features" and the default parameters for the current
> ConnPolicy class should not be touched. However, there are different ways to
> tackle this:
>
> 2.1 Make it a compile-time option.
>
> 2.2 Only add *new* fields to the ConnPolicy class without changing the
> default parameter values in the constructor and static methods, and a new
> class ConnPolicy2 or ConnPolicy3 that inherits from ConnPolicy and might
> have other default parameters or another C++ API (I am thinking of a
> named-parameter based API) and can also be used for scripting. Internally,
> the connection factory would work with const references to the base class
> only.
>
> 2.3 We add a DefaultConnPolicy attribute to the GlobalsRepository. This
> application-wide (per process) policy instance will be copied to newly
> constructed ConnPolicy instances and the constructors and static methods
> only overwrite the respective parameters.
>
> It might be obsoleted by the above proposal, but in the meantime I added a
> compile-time option RTT_V2_COMPATIBILITY_MODE to my working branch (see
> ef99f2b), that, if set, reverts to the current v2 per-connection model, but
> lets RTT print meaningful warning messages in the case a new connection
> would be invalid with v3 semantics because of conflicts with other
> connections to the same port. Such a flag could smooth the transition phase
> and help application designers to find the minimal set of connections that
> require connection policy changes.
>
> There is one small exception from the backwards-compatibility rule, that I
> think is worth discussing: The static method
>
> static ConnPolicy data(int lock_policy = LOCK_FREE, bool init_connection =
> true, bool pull = false);
>
> initializes the init_connection field to true by default, which was kind of
> surprising me, because it means that every new connection to an output port
> with this policy will overwrite a sample previously written by another port.
> Is this what we really want as a default? It is also not consistent with the
> constructor called as
>
> ConnPolicy(DATA)
>
> which would initialize the init_connection flag to false. The static method
> data(...) is only used in the OutputPortInterface::createDataConnection(...)
> method from within RTT and OCL itself.
>
> By the way, the init_connection flag is a good example where the API offers
> some degree of freedom to the application builder that semantically does not
> make sense at all for certain combinations, like in combination with a
> buffer connection. It is still not forbidden to set the flag, but it might
> be worth a warning log message if a connection is created like this.
>
>
> 3) How to add the "shared" concept within the current API, or to use a "new"
> API?
>
> This point was brought up by Janosch, who asked for "a more explicit
> interface for shared connections", with the main argument that it is
> fundamentally different from a normal connection and a "connect port A to B
> with policy C" kind of API does not fit well to that concept. That's true.
> We considered to use the createStream(...) API first, with a name_id string
> that identifies the shared buffer to connect to. This would actually fit
> better to the shared connection model, but it still enforces the same buffer
> type policies for all connections. We withdraw that idea because the
> existing API is more general and disallowing certain policy combinations is
> required anyway for private push connections with a single input buffer. At
> least in this case it cannot break existing applications. I do not think
> that there is a need for a completely new API only for shared connections.
>
> My current implementation in the updated-dataflow-semantics branch is as
> follows, and new/other ideas are always welcome:
>
> - Shared connection objects (the shared data object or buffered) are added
> to a process-wide registry. Every shared connection is identified by a
> unique string, either set explictly in the name_id field of the ConnPolicy
> or assigned automatically, in which case the name of the connection is
> returned in exactly this field (name_id is and always was declared as
> mutable).
> - For new connections with the name_id set, the shared connection is looked
> up in the registry, or created as new. As a consequence, if the same
> ConnPolicy instance is used to make a second connection, even if name_id was
> not set explicitly before, you will always connect to the same connection
> object.
> - For new connections with no name_id set, the port objects provide an API
> to return a pointer to an existing shared connection if they are already
> connected to an existing one, and only the non-connected port will be added.
> So ports can be connected in any order to each other, as long as there is
> always one of the two ports already connected. Otherwise the name_id has to
> be set or a new connection instance would be created. Perhaps this one could
> be dropped because it it too implicit and does not work very well for remote
> connections...
> - If an existing connection was found, the buffer policies (type, size and
> locking policy) have to match.
> - All ports will only store a single pointer to the shared connection
> instance in their connection list (in internal::ConnectionManager) and
> cannot know directly which other ports are connected.
> - For remote input ports with a shared connection, only a proxy is created
> locally and can be looked up in the shared connection repository. Other
> ports will only connect to either the proxy or the real shared connection
> instance, but never build a new remote channel once the proxy has been
> created for the first remote connection (works in both directions).
> - All connect and disconnect calls to shared connections are logged with log
> level Debug to the RTT logger, so at least there is a manual way to check
> whether the resulting connections have been made as expected. Graphical
> tools like the rtt_dot_service could be patched later to visualize shared
> connections correctly.
>
>
> 4) Whether to encode the result of a write() in existing enums or not?
>
> Janosch asked for introducing a separate enum for the return value of
> write(...) calls, in order to not break or trigger compiler warnings for
> existing code that already uses the FlowStatus enum in a switch statement.
> Another minor issue with a single enum is the default value of FlowStatus
> attributes, which can be only meaningful for one use case.
>
> That's also fine for me and I will update the document and implementation
> accordingly (add a new enum WriteStatus). Note that the evaluation in
> boolean context is counter-intuitive in this case, other than for
> FlowStatus, because the value WriteSuccess (=0) value evaluates to false and
> all error cases (WriteFailure and NotConnected) evaluate to true.
>
>
> I hope I did not forget anything. This mail already became much longer than
> expected. I am sorry.
>
> I also did some performance measurements of read and write calls last week
> in terms of absolute time, CPU time, number of assignments and so on of the
> current, the new and even the v1 data flow implementations in different
> scenarios. I plan to come up with some results during this week. The tests
> revealed some bugs and performance caveats, and some of them are already
> present since the early days of RTT v2, e.g. that data samples might be lost
> (never read as NewData) because the read flag is not lock-protected or
> stored within the lock-free data structure. I am preparing a pull request to
> the current master branch, but it will also break the API of class
> base::DataObjectInterface.
>
> Best regards,
> Johannes
>
>
>
> On Mon, Sep 28, 2015 at 3:36 PM, S Roderick <kiwi [dot] net [..] ...> wrote:
>>
>>
>> Johannes' proposed changes are focussed on
>> a) push vs pull, or specifying whether the data/buffer object is on the
>> reader or writer side. This is very useful for system designers dealing with
>> significant time delay.
>> b) private vs shared, whether a connection is per input/output port pair
>> or whether the connection is shared between multiple input/output ports.
>> c) mandatory, primarily intended to support knowing whether writes to a
>> buffer succeeded (so that traditional producer/consumer relationships have
>> some knowledge about the buffer state)
>>
>> It appears to me that most people are worried about changing the defaults,
>> more than they're worried about the actual proposed changes. Actual
>> technical concerns with the proposed changes appear to be
>>
>> 1) whether to encode the result of a write() in existing enums or not
>> 2) how to add the "shared" concept within the current API, or to use a
>> "new" API
>> 3) changing the RTT defaults, which then affect an application's
>> semantics.
>>
>> Did I miss any technical issues?
>>
>>
>> My huge personal caveat to all of this is that I do *not* think we should
>> change the current system semantics in introducing this change. Too many
>> Orocos changes have broken backwards compatibility and forced system
>> designers and implementors to change their system in response. I don't think
>> we should force that on users without an incredibly good reason. Otherwise
>> it breeds frustration in the community.
>>
>> If there is some way to introduce the proposed changes to the system, but
>> to not change the behavior of current systems, then I think that is a win
>> for all. There does seem to be some general agreement that there are valid
>> use cases that the current port implementation handles poorly, and that this
>> proposed approach does fix some of those. The proposed approach also
>> improves the situation for systems that don't want or need to use the flow
>> status approach. Both are equally valid approaches - IMHO it would be
>> beneficial to the community as a whole if RTT supported both approaches.
>>
>> Cheers
>> Stephen
>>
>>
>>
>> --
>> Orocos-Dev mailing list
>> Orocos-Dev [..] ...
>> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>>
>

> For private connections the buffer (or buffers in pull mode) belongs to a
> single input port, which does not mean that it cannot be filled by multiple
> writers (in push mode).

> For shared connections multiple input ports read from the same buffer. So
> the "shared" here is meant from the viewpoint of the input port ("I have my
> own input buffer." vs. "I have to share my input buffer with other
> readers."), not for output ports. Output ports may always serve multiple
> connections/channels except in the shared pull mode ("I have a new sample
> available and whoever polls me first will get it.").

Ah. Got the difference now... However, my comment applies on the
"single buffer per input port" model: the first connection (or
whatever it is ...) will define what type of data object should be
used and what size it has. That should definitely be a conscious
system designer choice.

Sylvain

On Sep 28, 2015, at 03:44 PM, Sylvain Joyeux <sylvain [dot] joyeux [..] ...> wrote:

Some classic producer/consumer examples
- a camera captures images and sends them on to a processing pipeline (that
is all data object connections). A separate connection stores the camera
images in a buffer, which are dumped to disk.
- data is read (non-real-time) from a file and pushed into a buffer for
processing (real-time)
- data samples coming from a remote source (non-real-time) is buffered on
arrival into the deployer for later processing (real-time).

Asked the wrong question ... thanks for answering it perfectly ;-)

Anyways, what I wanted is a case where:
- an input port is connected to a bunch of output ports *using
buffers* (I got the problem with data policies)
- the overall behavior is governed by maintaining the overall
incoming sample order (i.e. that if a sample of output port O_1
arrives before a sample of output port O_2, that order needs to be
reflected on read())
 
I think that all of the buffer uses in our system are classic single producer/single consumer connections.  I can't think of any N:M set of connections using buffers.

I may be missing it, but the use-cases above don't cover that, do they ?

> Agreed, but this choice is currently bundled with having a buffer per
> channel or one per input port.

I don't follow, Sylvain. Bundled in the existing implementation or the
proposed changes?

In the proposed change (current v2 does not have the option of having
a single buffer per input port). Until now, it is clear to me that
this bundling is both Johannes and Peter's intent (see Peter's email
above). It does not seem to be yours, though.
 
Well, yes and no. I'm not going to throw Johannes and Peter under the bus. A lot of this work is at our urging, and we've been involved with formulating and evaluating possible solutions. They have put together a solution that they believe solves some of the issues that we, and others, have bought up. They have also come to this proposed solution by actually implementing (some of) it and running into quite a few corner cases, etc.

> If you put the buffer per-port, you end up with one policy per port.
> Port-driven triggering (event ports) are governed by the data object
> (i.e. per-port), data object size and type becomes per-port as well.
> What's left in the connection policy is the info necessary to reach
> the other side (i.e. the topic name in ROS), which means that it is
> not a policy anymore but a "remote endpoint"

From your comment above "Port-driven triggering (event ports) …"

Badly ordered bits of sentences. What I meant is:

If you put the buffer per-port, you end up with one policy per port. Data object size and type becomes per-port. If using port-driven triggering (event ports), the triggering is governed by the data object and is therefore also becoming per-port.

Looking at Section 5.1 of Johannes' google doc, something looks wrong or I'm
remembering previous discussions incorrectly. I was expecting push/pull vs
shared/private to be truly orthogonal, but the push/private figure says
differently. Johannes? Peter?

They already clarified that for me. Basically, "private" does not
necessarily means "per channel". On a push connection, "private" for
them means "per-port".
 
In my head, private means per "connection" (I think that is your "channel" also). The proposed solution may not have that for good reasons, which I'll let them state.

Again, my intent was to add more v1-like semantics to the system, as well as
have more control on where latency occurred. Not to replace any aspect of
the v2 semantics.

I got that from your previous email already, but it seems that it's
not Johannes/Peter's intent
 
Again, they're trying to fix multiple issues in one go (and all the issues are pretty inter-related). They're also dealing with some pretty horrific corner cases, and they're doing the best they can.

HTH
S

On 27/9/15, 14:10, "orocos-dev-bounces [..] ... on behalf of
Herman Bruyninckx" <orocos-dev-bounces [..] ... on behalf of
herman [dot] bruyninckx [..] ...> wrote:
>
>Thanks for this nice effort. I think it is really time to consider a next
>generation of Orocos RTT...
>
>If you allow me to bring in my two cents... It's possible that you won't
>like the
>message, or find it off-topic, but I think I should share it with you
>anyway...
>
>The best document I have read about communication that connects
>high-performance with flexibility and configurability is this one:
>
><https://engineering.linkedin.com/distributed-systems/log-what-every-softw
>are-engineer-should-know-about-real-time-datas-unifying>
>
>It cleanly separates the communication of "events" from (i) what they
>mean,
>(ii) who is interested in them, and (iii) with what policy they are
>consumed.
>
>In our current research at KU Leuven, this is going to be _the_ basis for
>all interaction between asynchronous activities.
>The other basis will be the separation between (i) functional requirements
>of the application, (ii) "roles and responsibilities" of "components",
>(iii) software components to implement both, and (iv) deployment of these
>components on your hardware and OS platforms.
>
>All of the above are not present in Orocos RTT, which I consider design
>flaws.
>So solving these problems is definitely worthwhile. But, and this is a big
>but, my view on solving them (which is currently being transformed into
>a first set of models and code and tools) implies that we do not need
>Orocos RTT at all anymore. The reasons why we started RTT more than
>fifteen
>years ago have not disappeared, but have all been solved in more
>"mainstream"
>software projects than Orocos. Fifteen years ago, what we where after was
>a
>"node-port-connector" modelling-and-coding interface, where nodes could be
>deployed in realtime activities on an operating system. It was the time of
>single core CPUs, with only proprietary solutions to fast networking.
>Those
>days have gone, together with the need to use RTT as a software solution.
>
>More in particular, the "communication" aspect (which is the core of the
>document you distributed) is a lot better understood and supported in
>mainstream than 15 years ago. Taking the model of the event log and the
>corresponding log processing around it means that all the policies that
>your document wants to introduce as "API" methods can (should...) better
>be
>realised by configurations of the event log mechanism in your _application
>architecture_; there is no need to provide APIs in "middleware",
>"frameworks" or "libraries".
>
>_The_ core functionalities of RTT that still survive after 15 years are:
>- its introspection capabilities;
>- the lock-free data buffer.
>The latter is just a special case ("configuration") of the event log
>mechanism, that provides all other policies that you mention (and more!)
>just through other configurations.
>The introspection is a form of "model based" implementation, which
>considers models as first-class citizens in the software; this is great,
>and should be extended, starting with modelling all the relevant event log
>policies explicity. (KU Leuven is working on that; it takes a lot of
>efforts...) Lock-free data buffers are the core of event log processing,
>and are currently better known under the new buzzword "immutable data".
>
>The other "good-intuition-that-turned-out-to-be-badly-implemented" of
>fifteen years ago is that of "stored procedures" (known under various
>names, such as "microservices", proxy computations, etc.). RTT provides a
>very complex and ambiguous API to let one component execute code for
>another component. Again, that functionality should not be provided by an
>"RPC API", but by models and tools of _deployment_.
>
>The added value of Orocos as a project could shift from providing code to
>providing professional quality documentation and tutorials for "best
>practices" and "patterns", some specific realtime-optimized
>implementations, or patches to the large amount of existing libraries that
>support alul of the above.
>
>It is up to the Orocos community to decide whether to go that direction or
>not. I would be glad to cooperate in that directoion, because otherwise we
>will be forced to start a "competing" project.
>
>Note that this message does not contain threats to "fork" or whatever, but
>just my insights after working with Orocos RTT for about a decade, and
>having Orocos RTT work against the architectural requirements in our
>robotics projects for the subsequent five years...
>
>I frankly think it is time to reinvent the project. Big time.
>
>The easiest way is to _deprecate_ things, to make the _core_ things
>simpler
>for users, throwing out more and more code and start using other already
>existing software (ZeroMQ, libevent, the lock-free buffer library,
>RethinkDB,...) instead of trying to squeeze in their functionalities
>within
>NIH APIs and implementations. In the end, RTT will be empty, and the
>development efforts can then finally shift to give Sylvain a much bigger
>hand with ROCK :-)
>(Just make sure to refactor ROCK in order to decouple the _models_ from
>the
> Ruby and RTT _implementations_ :-))
>
>Best regards,
>
>Herman Bruyninckx

There are some potentially interesting ideas in there, Herman. But either
your email is uncharacteristically incoherent or I’m really jet lagged.
Could you explain more clearly what you currently think is the ideal? For
clarity, we can skip the comments Orocos, since we’ve already established
that you think it needs to change.

Geoff

2015-09-27 14:10 GMT+02:00 Herman Bruyninckx <
Herman [dot] Bruyninckx [..] ...>:

> On Mon, 21 Sep 2015, Johannes Meyer wrote:
>
> Dear Orocos community,
>> during the last few weeks we have been working on the concept and
>> implementation of a major redesign of the Orocos
>> RTT dataflow, that we would like to share and ask for your valuable
>> feedback. There was already a previous
>> discussion on the mailing list about the topic earlier this year [1], and
>> the following proposal tries to address
>> all the issues mentioned there.
>>
>> With the version 2 of the Orocos toolchain some years ago, the previous
>> data flow architecture, which was based on a
>> single data object and buffer shared between all ports participating at
>> the same connection, was replaced by a
>> connection architecture based on data flow, where each pair of output and
>> input ports (or streams) is connected
>> through an individual channel, each having its own data object or buffer.
>> While there was a well substantiated
>> motivation for this update and it solved some of the shortcomings in RTT
>> version 1 [2], it also introduced some new,
>> sometimes hidden, problems and broke other use cases. One example is an
>> input port with multiple connections, where
>> due to the ambiguity in which channel to read from the result is
>> sometimes unexpected to the user, who expects that
>> each write overwrites previous samples in case of simple data connections.
>>
>> We assembled a document (see attachment, or [3]) which introduces the
>> version 1 and version 2 data flow semantics
>> for those who are not aware of the details, three example use cases and
>> presents the new dataflow architecture we
>> have in mind, which ideally works for a superset of use cases that are
>> supported by RTT version 1 and 2.
>>
>> We do not want to go into details here, so only a short summary for those
>> who do not want to read the full document:
>> We propose to add two more boolean flags to the ConnPolicy struct that
>> describes the type of connection between two
>> ports: "shared" and "mandatory". Furthermore the existing "pull" flag,
>> which was only meaningful for remote
>> connections, now has a big influence for local connections.
>>
>> "push" vs. "pull"
>>
>>
> Thanks for this nice effort. I think it is really time to consider a next
> generation of Orocos RTT...
>
> If you allow me to bring in my two cents... It's possible that you won't
> like the
> message, or find it off-topic, but I think I should share it with you
> anyway...
>
> The best document I have read about communication that connects
> high-performance with flexibility and configurability is this one:
> <
> https://engineering.linkedin.com/distributed-systems/log-what-every-soft...
> >
>
> It cleanly separates the communication of "events" from (i) what they mean,
> (ii) who is interested in them, and (iii) with what policy they are
> consumed.
>
> In our current research at KU Leuven, this is going to be _the_ basis for
> all interaction between asynchronous activities.
> The other basis will be the separation between (i) functional requirements
> of the application, (ii) "roles and responsibilities" of "components",
> (iii) software components to implement both, and (iv) deployment of these
> components on your hardware and OS platforms.
>
> All of the above are not present in Orocos RTT, which I consider design
> flaws.
> So solving these problems is definitely worthwhile. But, and this is a big
> but, my view on solving them (which is currently being transformed into
> a first set of models and code and tools) implies that we do not need
> Orocos RTT at all anymore. The reasons why we started RTT more than fifteen
> years ago have not disappeared, but have all been solved in more
> "mainstream"
> software projects than Orocos. Fifteen years ago, what we where after was a
> "node-port-connector" modelling-and-coding interface, where nodes could be
> deployed in realtime activities on an operating system. It was the time of
> single core CPUs, with only proprietary solutions to fast networking. Those
> days have gone, together with the need to use RTT as a software solution.
>
> More in particular, the "communication" aspect (which is the core of the
> document you distributed) is a lot better understood and supported in
> mainstream than 15 years ago. Taking the model of the event log and the
> corresponding log processing around it means that all the policies that
> your document wants to introduce as "API" methods can (should...) better be
> realised by configurations of the event log mechanism in your _application
> architecture_; there is no need to provide APIs in "middleware",
> "frameworks" or "libraries".
>
> _The_ core functionalities of RTT that still survive after 15 years are:
> - its introspection capabilities;
> - the lock-free data buffer.
>

+1, this is my case.

> The latter is just a special case ("configuration") of the event log
> mechanism, that provides all other policies that you mention (and more!)
> just through other configurations. The introspection is a form of "model
> based" implementation, which
> considers models as first-class citizens in the software; this is great,
> and should be extended, starting with modelling all the relevant event log
> policies explicity. (KU Leuven is working on that; it takes a lot of
> efforts...) Lock-free data buffers are the core of event log processing,
> and are currently better known under the new buzzword "immutable data".
>
> The other "good-intuition-that-turned-out-to-be-badly-implemented" of
> fifteen years ago is that of "stored procedures" (known under various
> names, such as "microservices", proxy computations, etc.). RTT provides a
> very complex and ambiguous API to let one component execute code for
> another component. Again, that functionality should not be provided by an
> "RPC API", but by models and tools of _deployment_.
>
> The added value of Orocos as a project could shift from providing code to
> providing professional quality documentation and tutorials for "best
> practices" and "patterns", some specific realtime-optimized
> implementations, or patches to the large amount of existing libraries that
> support alul of the above.
>
> It is up to the Orocos community to decide whether to go that direction or
> not. I would be glad to cooperate in that directoion, because otherwise we
> will be forced to start a "competing" project.
>
> Note that this message does not contain threats to "fork" or whatever, but
> just my insights after working with Orocos RTT for about a decade, and
> having Orocos RTT work against the architectural requirements in our
> robotics projects for the subsequent five years...
>
> I frankly think it is time to reinvent the project. Big time.
>
> The easiest way is to _deprecate_ things, to make the _core_ things simpler
> for users, throwing out more and more code and start using other already
> existing software (ZeroMQ, libevent, the lock-free buffer library,
> RethinkDB,...) instead of trying to squeeze in their functionalities within
> NIH APIs and implementations. In the end, RTT will be empty, and the
> development efforts can then finally shift to give Sylvain a much bigger
> hand with ROCK :-)
> (Just make sure to refactor ROCK in order to decouple the _models_ from the
> Ruby and RTT _implementations_ :-))
>
> Best regards,
>
> Herman Bruyninckx

I have never been an active part of the project (testing and using apart),
so I can not comment all this, but :
_ thanks Hermann for sharing
_ today, if I had to choose a solution for my professionnal project, I
would not choose Orocos(Herman has well explained why) and the miss would
be the lock free communication and introspection tools.
_ I would also migrate from pure Orocos to Rock if my application wasn't
considered as "frozen".

I'm not sure it's the good post to chat on that.

>
>
> Like it is already the case for remote connections via CORBA, the "pull"
>> flag will decide whether the connection's
>> data object or buffer is installed at the writer's side (pull) or the
>> reader's side (push) of the connection. The
>> first case is equivalent to the current version 2 connection model and
>> readers have to select the connection/output
>> port to read from, while for push connections there will be only one
>> single buffer per input port. We think that the
>> push connection should be the default connection type for local ports,
>> but this might break existing applications
>> which rely on having one buffer per connection now.
>>
>> "private" vs. "shared"
>>
>> The shared flag reintroduces the shared data connection model that was
>> standard in RTT version 1. For shared
>> connections multiple input ports read from the same data object or
>> consume samples from the same buffer instance, so
>> their read operations can influence each other. Shared connections are
>> especially useful for producer/consumer
>> applications with multiple consumers.
>>
>> "mandatory"
>>
>> The introduction of the "mandatory" flag was necessary to decide on the
>> return value of the write() operation on an
>> output port. In RTT version 2 it was not possible anymore to get a
>> feedback of whether a write operation was
>> successful or failed, e.g. because of a full buffer or broken remote
>> connection. Only connections with the mandatory
>> flag set would be considered for the overall result of write()
>> operations, while failures of non-mandatory
>> connections will be ignored.
>>
>>
>> For those who want to take a look at the current state of the
>> implementation, this is the link to the working branch
>> on GitHub:
>> Tree:
>> https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics
>> Diff to master:
>> https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflo...
>>
>> The implementation is not fully working yet for non-standard transports
>> (like Corba and mqueue), but especially the
>> updated test ports_test.cpp already shows what would change from a user
>> perspective.
>>
>> Obviously, such a major refactoring cannot be done without changes in the
>> public API. However, we tried to keep the
>> required changes in user code as minimal as possible.
>> The new dataflow semantics are targeted for an upcoming version 2.9
>> release (or even name it 3.0 to communicate this
>> potentially breaking change?).
>>
>> We are planning to also come up with some comparative performance
>> measurements during this week, as people have
>> asked for that before [1].
>>
>>
>> Best regards,
>> Johannes and Peter
>>
>> [1]
>> http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation
>> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow
>> [3]
>> https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4Kaotp...
>>
>>
>>
> --
> Orocos-Dev mailing list
> Orocos-Dev [..] ...
> http://lists.mech.kuleuven.be/mailman/listinfo/orocos-dev
>
>

> On Sep 22, 2015, at 09:32, Janosch Machowinski <Janosch [dot] Machowinski [..] ...> wrote:
>
> Hi,
> here are my remarks:
> First of all, I like the idea of the input buffer (push case). But I also see some
> problems with it. The restriction on the same buffer size and type in the
> connection policy will be annoying (In my mind I can see the runtime errors pop up).
> It will be hard to track this in bigger systems. Do I get it correctly, that you only
> want to preserve arrival order and runtime here ?

I’ll have to disagree here. We have several hundred connections in our systems, and we track the policy type amongst all.

> If you are already changing the Connection implementation, I would recommend, to
> put a new thread in every connection in remote case. We experienced the issue, that
> bad Wifi connections, slowed our systems down, as it would hang on the write call.

Not my first choice. I agree with Sylvain - this should be part of the transport layer.

> For the Feedback on the write I would recommend to create a second enum, and not
> to use FlowStatus. Sometimes I use switch/case to react to the read FlowStatus. If you
> Introduce new members, this will result in compile warnings, because of unhandled
> entrie.

Yep, backwards compatibility is a concern. :-(

> For the creation of shared connections, I would recommend an new API. The create call
> should contain all involved members. This spares us the headache of tracking buffer sizes
> and types around in the code. Also this will simplify the code logic, as we don't need the lookup.
> e.g.
> out.connectShared(b.in, c.in, d.in, policy) ;

In the above are “b”, “c”, and “d”, all ports? If so, what about cases were not all port connections are made at the same time (e.g. XML deployments)?

Cheers
S

> Greetings
> Janosch
>
>
>
> Am 21.09.2015 um 15:15 schrieb Johannes Meyer:
>> Dear Orocos community,
>>
>> during the last few weeks we have been working on the concept and implementation of a major redesign of the Orocos RTT dataflow, that we would like to share and ask for your valuable feedback. There was already a previous discussion on the mailing list about the topic earlier this year [1], and the following proposal tries to address all the issues mentioned there.
>>
>> With the version 2 of the Orocos toolchain some years ago, the previous data flow architecture, which was based on a single data object and buffer shared between all ports participating at the same connection, was replaced by a connection architecture based on data flow, where each pair of output and input ports (or streams) is connected through an individual channel, each having its own data object or buffer. While there was a well substantiated motivation for this update and it solved some of the shortcomings in RTT version 1 [2], it also introduced some new, sometimes hidden, problems and broke other use cases. One example is an input port with multiple connections, where due to the ambiguity in which channel to read from the result is sometimes unexpected to the user, who expects that each write overwrites previous samples in case of simple data connections.
>>
>> We assembled a document (see attachment, or [3]) which introduces the version 1 and version 2 data flow semantics for those who are not aware of the details, three example use cases and presents the new dataflow architecture we have in mind, which ideally works for a superset of use cases that are supported by RTT version 1 and 2.
>>
>> We do not want to go into details here, so only a short summary for those who do not want to read the full document:
>> We propose to add two more boolean flags to the ConnPolicy struct that describes the type of connection between two ports: "shared" and "mandatory". Furthermore the existing "pull" flag, which was only meaningful for remote connections, now has a big influence for local connections.
>>
>> "push" vs. "pull"
>>
>> Like it is already the case for remote connections via CORBA, the "pull" flag will decide whether the connection's data object or buffer is installed at the writer's side (pull) or the reader's side (push) of the connection. The first case is equivalent to the current version 2 connection model and readers have to select the connection/output port to read from, while for push connections there will be only one single buffer per input port. We think that the push connection should be the default connection type for local ports, but this might break existing applications which rely on having one buffer per connection now.
>>
>> "private" vs. "shared"
>>
>> The shared flag reintroduces the shared data connection model that was standard in RTT version 1. For shared connections multiple input ports read from the same data object or consume samples from the same buffer instance, so their read operations can influence each other. Shared connections are especially useful for producer/consumer applications with multiple consumers.
>>
>> "mandatory"
>>
>> The introduction of the "mandatory" flag was necessary to decide on the return value of the write() operation on an output port. In RTT version 2 it was not possible anymore to get a feedback of whether a write operation was successful or failed, e.g. because of a full buffer or broken remote connection. Only connections with the mandatory flag set would be considered for the overall result of write() operations, while failures of non-mandatory connections will be ignored.
>>
>>
>> For those who want to take a look at the current state of the implementation, this is the link to the working branch on GitHub:
>> Tree: https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics <https://github.com/orocos-toolchain/rtt/tree/updated-dataflow-semantics>
>> Diff to master: https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflo... <https://github.com/orocos-toolchain/rtt/compare/master...updated-dataflow-semantics>
>>
>> The implementation is not fully working yet for non-standard transports (like Corba and mqueue), but especially the updated test ports_test.cpp already shows what would change from a user perspective.
>>
>> Obviously, such a major refactoring cannot be done without changes in the public API. However, we tried to keep the required changes in user code as minimal as possible.
>> The new dataflow semantics are targeted for an upcoming version 2.9 release (or even name it 3.0 to communicate this potentially breaking change?).
>>
>> We are planning to also come up with some comparative performance measurements during this week, as people have asked for that before [1].
>>
>>
>> Best regards,
>> Johannes and Peter
>>
>>
>>
>> [1] http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation <http://www.orocos.org/forum/rtt/rtt-dev/limitations-v2-port-implementation>
>> [2] http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow <http://www.orocos.org/wiki/rtt/rtt-2.0/dataflow>
>> [3] https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4Kaotp... <https://docs.google.com/document/d/1zDnPPz4SiCVvfEFxYFUZBcVbXKj33o4KaotpCtXM4E0/pub>
>>
>>
>>
>
>

Hi,

On Tue, Sep 22, 2015 at 3:32 PM, Janosch Machowinski <
Janosch [dot] Machowinski [..] ...> wrote:

> Hi,
> here are my remarks:
> First of all, I like the idea of the input buffer (push case). But I also
> see some
> problems with it. The restriction on the same buffer size and type in the
> connection policy will be annoying (In my mind I can see the runtime
> errors pop up).
> It will be hard to track this in bigger systems. Do I get it correctly,
> that you only
> want to preserve arrival order and runtime here ?
>

No, we think the current implementation with one buffer per connection
could really be considered as a bug, at least for certain use cases.
Consider two output ports W1 and W2 which are both connected to the same
input port R with a data connection. What I would expect is:

W1.write("foo")
W2.write("bar")
assert(R.read(x) == NewData && x == "bar")
assert(R.read(x) == OldData && x == "bar")

so each writer overwrites the data sample from previous writes (think of a
robot command). But what happens in RTT v2 is:

W1.write("foo")
W2.write("bar")
assert(R.read(x) == NewData && x == "foo")
assert(R.read(x) == NewData && x == "bar")

and I cannot even be sure if I first read "foo" or "bar" because it depends
on what connection was read last. The first result would be the new default
case (push), while the second can still be achieved by setting pull to
true, which had no effect until now for local connections.

Certainly there is a significant chance that the new restrictions break
existing deployments and that's why we would only release it with a new
minor, if not even major version number. But to fix this serious drawback
of the current dataflow implementation it is certainly worth the effort.

On Tue, Sep 22, 2015 at 4:17 PM, Janosch Machowinski <
Janosch [dot] Machowinski [..] ...> wrote:

> Am 22.09.2015 um 16:09 schrieb Sylvain Joyeux:
> >> If you are already changing the Connection implementation, I would
> >> recommend, to put a new thread in every connection in remote case. We
> experienced the
> >> issue, that bad Wifi connections, slowed our systems down, as it would
> hang on the write
> >> call.
> > This should IMO be done by the middleware ... in the CORBA case we
> > should have a way to spawn separate corba dispatchers (e.g. based on
> > the policie's name field) to isolate the domains. Moreover, there's
> > already been work done to use CORBA's "oneway" call to avoid this
> > problem completely (but I don't know what's the status of that)
> >
> > Sylvain
> This is basically a design decision, thread all remote connection, to be
> sure
> that the do not block you task, or rely on the connection to be properly
> implemented.

I agree with Sylvain here: It is the transport's responsibility to not
block on write or read operations and if the underlying implementation
cannot guarantee that, to use single worker thread, thread pool or one
thread per connection. If this would be enforced by the RTT connection
factory already, it would be impossible to use real-time capable transports
like Xenomai message queues.

We indeed also have an updated CORBA transport ready which introduces
oneway calls wherever possible and might solve some of the problems you
observed. It could be considered as stable and I could prepare another pull
request, but of course there would be plenty of conflicts with the
updated-dataflow-semantics branch, so I would prefer to postpone this until
this one is merged.

>
> For the Feedback on the write I would recommend to create a second enum,
> and not
> to use FlowStatus. Sometimes I use switch/case to react to the read
> FlowStatus. If you
> Introduce new members, this will result in compile warnings, because of
> unhandled
> entrie.
>

You are right and I first added a new enum-like class WriteStatus, but it
would also require a separate TypeInfo, to be added to the default typekit,
a corresponding struct in the CORBA IDL and so on, so at the end it seemed
easier to just add some constants to FlowStatus. The reason to have a class
instead of a simple enum is that the implicit conversion to bool could be
implemented more meaningful, more specifically that WriteSuccess evaluates
to true and all other constants to false.

>
> For the creation of shared connections, I would recommend an new API. The
> create call
> should contain all involved members. This spares us the headache of
> tracking buffer sizes
> and types around in the code. Also this will simplify the code logic, as
> we don't need the lookup.
> e.g.
> out.connectShared(b.in, c.in, d.in, policy) ;
>

A new API instead of adding the shared flag to the ConnPolicy? Or only as a
thin wrapper which implies that the shared flag is set?
It should be noted that it is not the new concept of shared connections
that adds constraints, but especially the per-input port buffers implied by
the default pull = false setting, even for private connections. What you
probably mean is to introduce a separate API for all new dataflow models
and make sure that the existing API calls behave the same as before.

A separate API would at least require additional updates in the deployer in
order to make it accessible from deployment scripts or XML configurations.

If people really feel a need for that, a possible intermediate solution
would be a preprocessor macro and/or cmake variable that enables a
compatibility mode which restores the current v2.x connection model and
basically enforces the pull connection type for local connections and
installs a per-connection buffer at the reader's side for remote push
connections. In this mode, RTT could at least print some deprecation
warnings to the Orocos log if a connection attempt would have failed
without the macro to be defined. Then at least you can postpone necessary
changes in your application until there is more time to do it and still use
other new features of an upcoming RTT release.

> Greetings
> Janosch
>

Best regards
Johannes