RTT 2.0 scripting improvements & semantic changes

Submitted by peter on Thu, 2010-05-06 14:04

Orocos-users

Dear scripting users,

I'm reworking the type system in order to easily integrate with code
generators for composite types and sequences of types. I have made good
progress, existing typekits won't suffer that much (yet).

As a side effect, scripting got a bit leaner and meaner. Both the 'set' and
'do' prefixes are now optional. 'var' is still mandatory. So you can write:

var int a = 0, b = 0;
a = b = 10;

You couldn't do that in 1.x, where assignment was a 'command' (returning a
bool). In 2.x assignment is just returning the left most argument as an
expression. This means assignment can no longer fail in 2.x. In 1.x it failed
if container size (array, string,...) was not sufficient. In 2.x, the container
will expand, it's your job to check the sizes first, like in:

var array a(4), b(10); // pre-reserved sizes
 
b = foo(); // foo returns less than 10 elements
 
if ( b.size <= a.capacity)
   a = b; // real-time
else
   a = b; // not real-time, 'a' expands and may malloc.

Since assignment can no longer fail and returns the left most element, the
following code does something very unexpected:

var bool b = true;
 
b = false; // ouch ! (solution: use try b = false :( )

It will put your program in the Error state. Since the left most variable is
returned, it returns false and the program script execution detects this.
Compare with:

comp.start();
comp.start(); // ouch ! (solution: use try comp.start() :( )

Where the second line will certainly return false, and again, put the script
into the error state. This was mainly invented to detect failing commands, but
commands don't exist anymore in 2.0...

In the light of our error states discussion, I was wondering if:
1. We could remove this 'error state' behaviour when a function / expression
returns false and just ignore user code's return values.
2. and we only go into the program script's Error state if user code throws in
that script, sounds familiar, no ?

The "try foo.bar() catch {...}" script blocks that detected *one* function
returning false could be:
1. Interpreted as: "if !foo.bar() then {...}"
2. OR: be removed from the syntax altogether...

What do the script users think of all this ?

Peter

RTT 2.0 scripting improvements & semantic changes

Submitted by Sylvain Joyeux on Thu, 2010-05-06 16:56.

Peter Soetens wrote:
> Dear scripting users,
>
> I'm reworking the type system in order to easily integrate with code
> generators for composite types and sequences of types. I have made good
> progress, existing typekits won't suffer that much (yet).
>
Since I'm one of the only one (that I know) which actually *do that*,
could you give me a bit of a hint on your plans. I had no problems
whatsoever with the RTT 1.x typekits for type generation, so I am a bit
left to wonder what needs to be "fixed".

RTT 2.0 scripting improvements & semantic changes

Submitted by Sylvain Joyeux on Thu, 2010-05-06 16:56.

RTT 2.0 scripting improvements & semantic changes

Submitted by peter on Thu, 2010-05-06 19:12.

On Thursday 06 May 2010 18:53:35 Sylvain Joyeux wrote:
> Peter Soetens wrote:
> > Dear scripting users,
> >
> > I'm reworking the type system in order to easily integrate with code
> > generators for composite types and sequences of types. I have made good
> > progress, existing typekits won't suffer that much (yet).
>
> Since I'm one of the only one (that I know) which actually *do that*,
> could you give me a bit of a hint on your plans. I had no problems
> whatsoever with the RTT 1.x typekits for type generation, so I am a bit
> left to wonder what needs to be "fixed".

:-) The biggest change is how RTT is told how a type looks like, ie is it a
sequence of stuff (a container), or is it stuff made out of other stuff (a
struct) etc. In 1.x you needed to: a) write compose/decompose property
functions to tell the property system how structs are composed, b) add all
kinds of operators (including the horrific 'dot' operator) to read or
manipulate structs c) do similar things like a) for each transport (since
transports marshal/demarshal), d) write terrible template specialisations in
case you have something as comples as 'an array of things'. People who had to
write all that must have been wondering if there wasn't an easier way.

The easier way is by using the 'boost::serialization' concept of describing
structs with a free function like this:

struct Mystruct {
   int a,b,c;
   double stamps[10];
}
template<class Archive>
void serialize(Archive& a, Mystruct& m) {
   a & BOOST_NVP("a",m.a);
   a & BOOST_NVP("b",m.b);
   a & BOOST_NVP("c",m.c);
   a & BOOST_NVP("stamps", boost::make_array(m.stamps,10) );
}

This is all you need to do property composition/decomposition, reading/writing
a,b,c,... in scripts, automatically creating constructors consisting of the
struct members, printing the contents of the struct and transporting it with
transports requiring reasonable marshalling (CORBA is needs more...). And
maybe some more.

To summarize, for any user type T, it's serialize(T) + StructTypeInfo<T>, and
that's all you need. Now guess what, it's damn easy to generate the
serialize<T> function + it's Orocos independent.

Next to that, I'm looking into automatic typekit/plugin loading too, anyone
else may contribute that too if you were thinking about it...

Now for the CORBA case, not that much changes. What you *could* do is generate
this serialize function for structs generated from CORBA IDL and use these
directly in your components (hard depend on CORBA). If not, then you need to
stick to the current IDL + C struct + serialize() case, which orogen solves
nicely.

I think the changes for orogen will be 'minimal' and until now, all changes
went into TemplateTypeInfo + specialisations. I only see less user code. For
example, the VectorTemplateComposition that Ruben and Tinne wrote will be
replaced by a much thinner SequenceTypeInfo.hpp specialisation that does the
same, and more.

Operators (+,-,/,... but not [] ) and conversion functions (int->float,...),
will still need to be added 'manually' (or generated by a smarter generator)
though, since we have no way of guessing what might be available/possible.

As told earlier, I'm also looking into code size and compilation speeds and
try to optimize that too.

Peter

RTT 2.0 scripting improvements & semantic changes

Submitted by peter on Thu, 2010-05-06 19:12.

The easier way is by using the 'boost::serialization' concept of describing
structs with a free function like this:

struct Mystruct {
   int a,b,c;
   double stamps[10];
}
template<class Archive>
void serialize(Archive& a, Mystruct& m) {
   a & BOOST_NVP("a",m.a);
   a & BOOST_NVP("b",m.b);
   a & BOOST_NVP("c",m.c);
   a & BOOST_NVP("stamps", boost::make_array(m.stamps,10) );
}

To summarize, for any user type T, it's serialize(T) + StructTypeInfo<T>, and
that's all you need. Now guess what, it's damn easy to generate the
serialize<T> function + it's Orocos independent.

Next to that, I'm looking into automatic typekit/plugin loading too, anyone
else may contribute that too if you were thinking about it...

As told earlier, I'm also looking into code size and compilation speeds and
try to optimize that too.

Peter

RTT 2.0 scripting improvements & semantic changes

Submitted by Sylvain Joyeux on Fri, 2010-05-07 09:44.

Peter Soetens wrote:
> On Thursday 06 May 2010 18:53:35 Sylvain Joyeux wrote:
>
>> Peter Soetens wrote:
>>
>>> Dear scripting users,
>>>
>>> I'm reworking the type system in order to easily integrate with code
>>> generators for composite types and sequences of types. I have made good
>>> progress, existing typekits won't suffer that much (yet).
>>>
>> Since I'm one of the only one (that I know) which actually *do that*,
>> could you give me a bit of a hint on your plans. I had no problems
>> whatsoever with the RTT 1.x typekits for type generation, so I am a bit
>> left to wonder what needs to be "fixed".
>>
>
> :-) The biggest change is how RTT is told how a type looks like, ie is it a
> sequence of stuff (a container), or is it stuff made out of other stuff (a
> struct) etc. In 1.x you needed to: a) write compose/decompose property
> functions to tell the property system how structs are composed, b) add all
> kinds of operators (including the horrific 'dot' operator) to read or
> manipulate structs c) do similar things like a) for each transport (since
> transports marshal/demarshal), d) write terrible template specialisations in
> case you have something as comples as 'an array of things'. People who had to
> write all that must have been wondering if there wasn't an easier way.
>
I guess it is horrible for human writing typekits. Code generators don't
care so much about coding ;-)
> The easier way is by using the 'boost::serialization' concept of describing
> structs with a free function like this:
>
>

> struct Mystruct {
>    int a,b,c;
>    double stamps[10];
> }
> template<class Archive>
> void serialize(Archive& a, Mystruct& m) {
>    a & BOOST_NVP("a",m.a);
>    a & BOOST_NVP("b",m.b);
>    a & BOOST_NVP("c",m.c);
>    a & BOOST_NVP("stamps", boost::make_array(m.stamps,10) );
> }
>

>
> This is all you need to do property composition/decomposition, reading/writing
> a,b,c,... in scripts, automatically creating constructors consisting of the
> struct members, printing the contents of the struct and transporting it with
> transports requiring reasonable marshalling (CORBA is needs more...). And
> maybe some more.
>
> To summarize, for any user type T, it's serialize(T) + StructTypeInfo<T>, and
> that's all you need. Now guess what, it's damn easy to generate the
> serialize<T> function + it's Orocos independent.
>
Does "transports requiring reasonable marshalling" means "transports
marshalling using boost::serialize" ? Or is it broader than that ?

Sounds nice in general. How does that handle vectors (and vectors of
vectors and this kind of things) by the way ?

Another optimization that oroGen has for CORBA (and Typelib has in
general) is the detection of types that do not need marshalling at all
(i.e. the whole memory is either memcopied or write()'d to IO). Does the
boost serializations method does that too ?

RTT 2.0 scripting improvements & semantic changes

Submitted by Sylvain Joyeux on Fri, 2010-05-07 09:44.

> struct Mystruct {
>    int a,b,c;
>    double stamps[10];
> }
> template<class Archive>
> void serialize(Archive& a, Mystruct& m) {
>    a & BOOST_NVP("a",m.a);
>    a & BOOST_NVP("b",m.b);
>    a & BOOST_NVP("c",m.c);
>    a & BOOST_NVP("stamps", boost::make_array(m.stamps,10) );
> }
>

Sounds nice in general. How does that handle vectors (and vectors of
vectors and this kind of things) by the way ?

RTT 2.0 scripting improvements & semantic changes

Submitted by peter on Fri, 2010-05-07 10:04.

On Friday 07 May 2010 11:39:51 Sylvain Joyeux wrote:
> Peter Soetens wrote:
> > On Thursday 06 May 2010 18:53:35 Sylvain Joyeux wrote:
> >> Peter Soetens wrote:
> >>> Dear scripting users,
> >>>
> >>> I'm reworking the type system in order to easily integrate with code
> >>> generators for composite types and sequences of types. I have made good
> >>> progress, existing typekits won't suffer that much (yet).
> >>
> >> Since I'm one of the only one (that I know) which actually *do that*,
> >> could you give me a bit of a hint on your plans. I had no problems
> >> whatsoever with the RTT 1.x typekits for type generation, so I am a bit
> >> left to wonder what needs to be "fixed".
> >>
> > :-) The biggest change is how RTT is told how a type looks like, ie is it
> > : a
> >
> > sequence of stuff (a container), or is it stuff made out of other stuff
> > (a struct) etc. In 1.x you needed to: a) write compose/decompose property
> > functions to tell the property system how structs are composed, b) add
> > all kinds of operators (including the horrific 'dot' operator) to read or
> > manipulate structs c) do similar things like a) for each transport (since
> > transports marshal/demarshal), d) write terrible template specialisations
> > in case you have something as comples as 'an array of things'. People who
> > had to write all that must have been wondering if there wasn't an easier
> > way.
>
> I guess it is horrible for human writing typekits. Code generators don't
> care so much about coding ;-)
>
> > The easier way is by using the 'boost::serialization' concept of
> > describing structs with a free function like this:
> >
> >

> > struct Mystruct {
> >    int a,b,c;
> >    double stamps[10];
> > }
> > template<class Archive>
> > void serialize(Archive& a, Mystruct& m) {
> >    a & BOOST_NVP("a",m.a);
> >    a & BOOST_NVP("b",m.b);
> >    a & BOOST_NVP("c",m.c);
> >    a & BOOST_NVP("stamps", boost::make_array(m.stamps,10) );
> > }
> >

> >
> > This is all you need to do property composition/decomposition,
> > reading/writing a,b,c,... in scripts, automatically creating constructors
> > consisting of the struct members, printing the contents of the struct and
> > transporting it with transports requiring reasonable marshalling (CORBA
> > is needs more...). And maybe some more.
> >
> > To summarize, for any user type T, it's serialize(T) + StructTypeInfo<T>,
> > and that's all you need. Now guess what, it's damn easy to generate the
> > serialize<T> function + it's Orocos independent.
>
> Does "transports requiring reasonable marshalling" means "transports
> marshalling using boost::serialize" ? Or is it broader than that ?

I knew I was too vague :-). With reasonable, I mean that the data type on the
wire is the same as in the process. For CORBA this is not true since we have
two representations. For POSIX mqueue transport, it's true, when using the ROS
transport, it might be true too (component depends on ros) or not true when we
do the CORBA mapping trick. Instead of 'reasonable' I should have written
'primitive', meaning, 'transports that leave marshalling to the user'.

>
> Sounds nice in general. How does that handle vectors (and vectors of
> vectors and this kind of things) by the way ?

In the right way. Since serialize() may be templated, you can describe how a
vector<T> is composed, and then, T may be a vector<U> in turn, and so on. It
allows very powerful type *contents* analysis at compile time at the price
of... compile time :-)

>
> Another optimization that oroGen has for CORBA (and Typelib has in
> general) is the detection of types that do not need marshalling at all
> (i.e. the whole memory is either memcopied or write()'d to IO). Does the
> boost serializations method does that too ?

Yes. They can detect 'binary' blobs when make_array like above is used, but
you can also explicitly say so in your serialize() method (for example, you
have a void*) see [1]. The mqueue transport uses this optimization, also when
serializing a std::vector. But I never tested vectors of vectors in this
respect... I wonder. This depends on the capabilities of the 'Archive', not on
the serialize() function.

Peter

[1] http://www.boost.org/doc/libs/1_42_0/libs/serialization/doc/wrappers.html

RTT 2.0 scripting improvements & semantic changes

Submitted by peter on Fri, 2010-05-07 10:04.

> > struct Mystruct {
> >    int a,b,c;
> >    double stamps[10];
> > }
> > template<class Archive>
> > void serialize(Archive& a, Mystruct& m) {
> >    a & BOOST_NVP("a",m.a);
> >    a & BOOST_NVP("b",m.b);
> >    a & BOOST_NVP("c",m.c);
> >    a & BOOST_NVP("stamps", boost::make_array(m.stamps,10) );
> > }
> >

>
> Sounds nice in general. How does that handle vectors (and vectors of
> vectors and this kind of things) by the way ?

Peter

[1] http://www.boost.org/doc/libs/1_42_0/libs/serialization/doc/wrappers.html

RTT 2.0 scripting improvements & semantic changes

Submitted by snrkiwi on Thu, 2010-05-06 18:32.

On May 6, 2010, at 12:53 , Sylvain Joyeux wrote:

> Peter Soetens wrote:
>> Dear scripting users,
>>
>> I'm reworking the type system in order to easily integrate with code
>> generators for composite types and sequences of types. I have made good
>> progress, existing typekits won't suffer that much (yet).
>>
> Since I'm one of the only one (that I know) which actually *do that*,
> could you give me a bit of a hint on your plans. I had no problems
> whatsoever with the RTT 1.x typekits for type generation, so I am a bit
> left to wonder what needs to be "fixed".

They're horrific to write manually in v1 ... we are definitely looking forward to changes to ease creation/generation of such tooklits in the future.

But I don't want Sylvain suffering significantly for the changes, either ...
S

RTT 2.0 scripting improvements & semantic changes

Submitted by snrkiwi on Thu, 2010-05-06 17:28.

On May 6, 2010, at 12:53 , Sylvain Joyeux wrote:

They're horrific to write manually in v1 ... we are definitely looking forward to changes to ease creation/generation of such tooklits in the future.

But I don't want Sylvain suffering significantly for the changes, either ...
S