Hi Peter,

On Mon, Nov 24, 2008 at 08:44:53AM +0100, Peter Soetens wrote:
> > What do you like about rt-program scripts?
>
> That they comply with real-time execution. At a price...
>
> > What should be different?
>
> Efficiency, memory management and possibly the handling of commands.
> What is written below assumes that we don't change the current way scripts are
> parsed. So I may be staring in a hole here...

No, I think it's all open...

> * Every expression is an object. That eats memory. I wonder how other
> interpreters do that...

I think most programs are stored as some type of objects in tree
structures. Is the amount of memory used by these objects really
significant?

> * Because of the pre-allocation of memory, no stack based calculation, like
> recursive function calls, is possible. This is because every variable is
> allocated on the heap using a 'ValueDataSource". We could create a 'script
> stack' (allocated on the heap) for each program script and map out our
> variables in that memory space. With the current TLSF, we could even let the
> stack grow...

But recursive function calls in scripts don't necessarily have to map
to recursion in C++, do they?

> * As said before, commands add huge complexity, both in syntax (until {...})
> and in execution (dispatching, reseting, checking conditions etc.) On the
> other hand they support synchronised execution (script waits until command
> completes).

Hmm, and what are your plans? Is it acceptable and worth while
complexity or might you remove it in the future?

> > What's missing?
>
> As you said, a real interpreter which allows to dynamically add variables,
> write programs etc. This is more of a front-end issue as the current
> implementation already allows such use.

Ok.

> > I think the scripting infrastructure is incredibly important, as it
> > considerably augements the abstraction layer and allows people to
> > build systems (using existing components) without having to touch any
> > C++ code.
> >
> > Having a powerful RT scripting language also helps to distinguish
> > which parts of an OROCOS applications should be implemented as scripts
> > or C++ respectivly. I'll dare propose a best practice: only the basic
> > mechanism of a component should be implemented in C++, all policy
> > (=the glue that ties together the elements of a component) should be
> > scripted.
>
> That depends on your requirements. What's possible in scripts should always be
> possible/easy to do in C++ as well..

Well, I think it depends for whom. And with scripting you also win in
terms of reliability. Even if a script fails it won't ever crash the
system.

> > - Simplification of the parsing infrastructure.
>
> Knock yourself out !

:-) That should be a warning...

> > - Possibility to use arbitrary types from other OROCOS libraries.
>
> We do ! Did you miss the 'Orocos Plugins' manual for adding custom types ?

Ups, I did indeed miss that manual. That's it!

> > - profiling: people often reject scripting for (usually bogus!)
>
> Not in our case. If people ask me, I warn them that script execution is
> typically factor 50 slower than C++ code.

Ok, that makes profiling even more important. I don't think a factor
of 50 is particularly slow though, and thus shouldn't be a problem if
used as intended. You don't seem too convinced that scripting is
actually very useful, am I wrong?

> > performance considerations. A simple profiling mechanism to find
> > bottlenecks in scripts could avoid lots of premature optimization.
> >
> > - require a program to explicitly be declared real-time or
> > non-realtime. Complain if a real-time program causes any
> > allocations.
>
> That's adding complexity. We could just warn if it is not real-time.

Yes, agreed.

> >
> > - Allow real-time memory allocation (and thus garbage collection)? The
> > former could be quite easily achieved with the TLSF[2] allocator, the
> > latter seems much harder.
>
> As Herman suggested, we need to be careful with writing a mature scripting
> language, re-inventing just another language is a huge undertaken. Our

Yes, I absolutely agree. I'm not proposing to write something new! I
think we have the following options:

1) stay with the current philsophy of preallocation.
2) embed an existing hard-real time, lightweight scripting language.
3) embed an existing lightweight scripting language that can be
operated in realtime.

1) is a very reasonable choice and has served well so far. 2) would be
nice, but unfortunately does not really exist and is non-trivial to
do. What possibly could work though is 3) that is to define a
"real-time subset" of language elements of scripting language X which
can function in the same manner as OROCOS scripting does now, namely
simply avoiding memory allocation and garbage collection altogether.

> scripting language went a bit out of control in that respect... On the other
> hand, I did take a look at lua, I don't oppose it, but it's not real-time and
> I don't know how to get commands in there.For some people, a non-real-time lua
> compatible scripting interface to Orocos would still be a major feature.
> Adding a TLSF malloc to lua scripts could even improve that.

Yes, agreed here too. Lua added incremental garbage collection
recently so it will be soft real-time capable at least. One possible
solution could be to admit that there's no silver bullet and go with
two scripting languages...?

Thanks for your feedback!

Best regards
Markus

On Thu, Nov 20, 2008 at 04:58:54PM -0000, philippe [dot] hamelin [..] ... wrote:

> I agree with Herman, a GUI based state machine designer would be
> great! How about the same thing for script? It would be great to
> have the apportunity to browse in the components properties/methods
> to set-up a script. Currently, we have to look at the header file or
> run the TaskBrowser to find out the interfaces.

Yes, that would be useful indeed. A first step could be a validation
tool that checks a script that shall run on a given component in a
certain deployment scenario and verifies it uses only valid OROCOS
types.

Best regards
Markus

On Thu, Nov 20, 2008 at 03:58:14PM +0100, Herman Bruyninckx wrote:
> On Thu, 20 Nov 2008, Markus Klotzbücher wrote:
>
>> I've been looking into OROCOS' real-time program scripts for some time
>> and am wondering what others think.
> [...]
>
> One possibly relevant suggestion could be (you notice how careful I am not
> to try to make a strong statement? :-)) to look into scripting languages

I have noticed :-)

> that have already a much wider support than the Orocos scripting, and that
> have properties that come close to what we like.
>
> Lua could be such
> an example. It is rather "embedded" and its semantics are extensible...
> I quote from the webpage: "In general, Lua strives to provide flexible
> meta-features that can be extended as needed, rather than supply a
> feature-set specific to one programming paradigm. As a result, the base
> language is light — in fact, the full reference interpreter is only about
> 150kB compiled — and easily adaptable to a broad range of applications."
>
> There are FOSS implementations available; the one on
> comes with an MIT license, which is compatible to the Orocos licenses.

Exactly! I think this is a very good example for what we want. The
language should principally serve as a vehicle for implementing our
own internal domain specific language.

An other family of languages with similar properties and that are
traditionally used as extension languages are the Lisp variants such
as Scheme. One I looked at is tinyscheme
, which is used as an
embedded scripting language in various projects. It's only around 70k
and comes under a BSD licence.

The difficulty of course is having these features _and_ hard real-time
behaviour.

>
> _And_ it is developed and maintained by an institute that has even more
> impressive religious ties and backings than our own Katholieke
> Universiteit: the Pontifical Catholic University of Rio de Janeiro :-))))
> So, this can't be wrong! :-)
>

:-) Applying this measure should help to drastically reduce the amount
of software for consideration!

Best regards
Markus

On Thu, Nov 20, 2008 at 01:20:05PM +0100, Herman Bruyninckx wrote:
> On Thu, 20 Nov 2008, Markus Klotzbücher wrote:
>
>> I've been looking into OROCOS' real-time program scripts for some time
>> and am wondering what others think. With respect to the state machine
>> redesign the real-time program scripts are essential. Large parts of
>> each state machine such as actions and expressions will be executed /
>> evaluated by the program interpreter.
>>
>> As the upcoming changes to the state machines will undoubtly cause
>> changes to the program script infrastructure, it might be a good time
>> to make some updates. So, same questions as for the state machines:
>>
>> What do you like about rt-program scripts?
>
> 1. That they are _scripts_.
> 2. That they are transformed into something that could(!) be used in a
> _realtime_ context.
>
>> What should be different?
> The semantic richness and unambiguity are rather limited, so I expect some
> extensions in that direction. However, it is a dangerous road to go
> (feature bloat, realtime safeness, ...) so we should be careful.

Yes, agreed! I think what's important here is that we need a language
that is powerful not in terms of available extensions/libraries, but
in its metaprogramming capabilities, e.g. the possibility to mold the
language to our needs.

>> What's missing?
> 1. UML compliance (in order to solve the semantic ambiguities, to some
> extent).
> Hierarchical state machines come to mind. But also UML "activitiy
> diagrams", which, in my simple opinion, are state machines "distributed"
> over multiple components...

Hmm, not sure I understand this, in what sense are activity diagrams
less "component-internal" than statemachines?

> 2. GUI programming of state machines, including "executable model"
> infrastructure (= automatic code generation and simulation on a virtual
> Orocos runtime...?)
> The GUI has (at least) three complementary levels:
> 2.1 GUI that helps to build textual scripts, but makes sure one can only
> build (syntactically) correct scripts.
> 2.2 GUI that builds graphical state machines, and translates them to
> Orocos scripts. (Or directly to the C++...?)
> 2.3 GUI that interfaces the script building with script execution.

I understand we need GUIs, but what I really see here is parsers,
compilers, tools for transformation, etc. I think that's the hard
part. Once we have those tools in place, building GUIs on top is not
that hard anymore (of course building *good* GUIs is hard enough
itself). Maybe I'm too much of a Unix guy here?

In 2.2. you write "translating state machines to scripts". That's an
interesting remark, because state machines _really_ are program
scripts with some execution logic built around. So an alternative way
(and a good one, if you ask me) to rework the state machines could be:

1) make the scripting language more expressive
2) implement state machines in the scripting language
3) if necessary, optimize for performance (-> implement parts in c++, etc.)

> 3. "Best practices": a set of good templates to start from (good =
> documented, proven in running applications, gotten from other projects,
> etc.)

Yes.

>> I think the scripting infrastructure is incredibly important, as it
>> considerably augements the abstraction layer and allows people to
>> build systems (using existing components) without having to touch any
>> C++ code.
> I fully agree. It is also a way (_the_ way?) to integrate orocos
> development the "Model-Driven Engineering" paradigm (cf Eclipse,
> TopCased.org, etc.)

Yes, agreed. What I think is important with respect to MBE is not to
"skip" too many layers, e.g. generating C++ from GUI models will be
hard to get right. Using scripting as an intermediate layer will make
things a lot easier.

>> Having a powerful RT scripting language also helps to distinguish
>> which parts of an OROCOS applications should be implemented as scripts
>> or C++ respectivly. I'll dare propose a best practice: only the basic
>> mechanism of a component should be implemented in C++, all policy
>> (=the glue that ties together the elements of a component) should be
>> scripted.
> That would be nice. But there are various sorts of glue, and some of them
> have been (or are being) standardized, and supported by larger-scale tools,
> such as Eclipse, MDE, etc.
>
>> Here are some (2.0) ideas:
>>
>> - interactivity: what I really miss is the ability to to develop
>> OROCOS programs in an interactive way using a REPL[1], like with
>> Python or Lisp.
> This is (probably) a similar feature as what I called "executable scripts"
> above...
>
>> - Simplification of the parsing infrastructure.
> What do you mean exactly: parser code refactoring? simplification of how to
> use the parser? ...

The first, with the goal to achieve the latter. Well I'm still looking
into this, but it seems the spirit framework is reaching it's limits
for a task of this complexity. A parser generated from a model could
simplify matters such as extending the language etc.

>> - Possibility to use arbitrary types from other OROCOS libraries.
> Yes! Maybe there are limits to this: how to make sure that types are used
> in a runtime context where the objects behind the types really exist and
> are accessible (in a safe, realtime way)...
>
>> - profiling: people often reject scripting for (usually bogus!)
>> performance considerations. A simple profiling mechanism to find
>> bottlenecks in scripts could avoid lots of premature optimization.
> Yes! But is this a "scripting feature", or an RTT infrastructural feature?

I think it would be a scripting feature, depending on the
meta-programing features of the scripting language it could even be a
script itself, which...

> To what extent can you do this in a "platform-independent" way? What is the

...would guarantee platform independency.

> link with deployment? (Different deployments will have very different
> performances in different parts of the code...)

Good question!

>> - require a program to explicitly be declared real-time or
>> non-realtime. Complain if a real-time program causes any
>> allocations.
> Could be considered to be part of "profiling", where some profiles are
> relevant for real time, others are not.
>
>> - Allow real-time memory allocation (and thus garbage collection)? The
>> former could be quite easily achieved with the TLSF[2] allocator, the
>> latter seems much harder.
> The latter is the core of the realtime Java developments, that Lund
> University is working on very hard :-)

:-)

Best regards
Markus