On Thu, 30 Sep 2010, Peter Soetens wrote:

> On Thursday 30 September 2010 14:24:28 Klaas Gadeyne wrote:
>> On Thu, Sep 30, 2010 at 1:45 PM, Peter Soetens
> <peter [..] ...>wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to
>>> give your feedback. These features will be developed after 2.1 is
>>> released.
>>>
>>> Most users are not happy with the XML format that specifies which
>>> components
>>> are loaded and connected in an application. Especially the flexibility
>>> and the
>>> fixed ordering of configuration/setup steps disturbs even modest
>>> applications.
>>> The scripting interface on the other hand provides a more controlled,
>>> readable
>>> and flexible environment to specify how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps.
>>
>> Aren't you going to fast here? It seems to me that the problems you are
>> sketching above merely saying that the cpf.dtd does not offer enough
>> semantics to express a typical application setup. I don't understand why
>> you would have to throw xml away in favour of scripting?
>
> I'm not throwing it away (cfr reply to Stephen). But users should be aware
> that with the current situation, scripting instead of the XML format will save
> them time. The idea is to remove the rough edges of the scripting approach,
> while leaving the xml approach as-is, and not trying to add more features to
> xml to aleviate it's short-commings.
>
> Even so, the deployer is not offering a supervision framework like DFKI's
> Roby[1], and probably never will be. In this regard, the deployer won't
> accumulate features and should work as a slave/library to a more intelligent
> supervision system.

Good!!!! Don't put efforts in Orocos when external frameworks are (could
be...) doing much better. Only spend the effort there where realtime
performance matters...

Herman

On Thu, 30 Sep 2010, Peter Soetens wrote:

> On Thursday 30 September 2010 14:24:28 Klaas Gadeyne wrote:
>> On Thu, Sep 30, 2010 at 1:45 PM, Peter Soetens
> <peter [..] ...>wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to
>>> give your feedback. These features will be developed after 2.1 is
>>> released.
>>>
>>> Most users are not happy with the XML format that specifies which
>>> components
>>> are loaded and connected in an application. Especially the flexibility
>>> and the
>>> fixed ordering of configuration/setup steps disturbs even modest
>>> applications.
>>> The scripting interface on the other hand provides a more controlled,
>>> readable
>>> and flexible environment to specify how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps.
>>
>> Aren't you going to fast here? It seems to me that the problems you are
>> sketching above merely saying that the cpf.dtd does not offer enough
>> semantics to express a typical application setup. I don't understand why
>> you would have to throw xml away in favour of scripting?
>
> I'm not throwing it away (cfr reply to Stephen). But users should be aware
> that with the current situation, scripting instead of the XML format will save
> them time. The idea is to remove the rough edges of the scripting approach,
> while leaving the xml approach as-is, and not trying to add more features to
> xml to aleviate it's short-commings.
>
> Even so, the deployer is not offering a supervision framework like DFKI's
> Roby[1], and probably never will be. In this regard, the deployer won't
> accumulate features and should work as a slave/library to a more intelligent
> supervision system.

Good!!!! Don't put efforts in Orocos when external frameworks are (could
be...) doing much better. Only spend the effort there where realtime
performance matters...

Herman

On Thu, 30 Sep 2010, Peter Soetens wrote:

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to give your feedback. These features will be developed after 2.1
>>> is released.
>>
>> Add the ability to deploy components in groups, so that group A is deployed
>> completely, followed by group B, ..., and then group B is un-deployed
>> completely, and finally group A is un-deployed completely. This is
>> necessary for things like logging, which currently hacks this by using the
>> site-file approach. This is really just sequencing application startup and
>> shutdown.
>
> You *can* already do this using scripting and a state machine.That's my whole
> point. No XML / DeploymentComponent extensions are necessary for this.

But there is something more: in many deployment scenarios you _need_ more
than a static description of how components have to be interconnected, and
the deployment, start-up and shut-down sequences require _behaviour_, as
provided by state machines.

>>> Most users are not happy with the XML format that specifies which
>>> components are loaded and connected in an application. Especially the
>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>> even modest applications. The scripting interface on the other hand
>>> provides a more controlled, readable and flexible environment to specify
>>> how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps. For example: importing
>>> component libraries or plugins and instantiating components. Everything
>>> after that should be done in a script.
>>
>> What exactly are you talking about changing here? In particular, what are
>> you talking about removing? I would really like it if nothing that
>> currently works was removed - we already have a high enough barrier
>> getting from v1 to v2 due to "removed features" that we use. It would be
>> nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
>>
>> The only issue we have with XML is that large-scale deployments are tricky
>> due to difficulty in configuring the deployment. The recent discussion on
>> use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.
>

This is the same discussion as we have within BRICS, about how to best
separate the structural and behavioural parts, more specifically of
deployment int his case. To support behaviour, one needs FSM functionality.
But both structural and behavioural parts can be described in XML form. The
concrete RTT problem is that we have no XML tags/DTD yet for FSMs.

It's a valid and timely question to ask now: what makes more sense,
extending our RTT-specific XML/scripting or adopting an existing deployment
"language" (such as OSGi, which uses a more readable file format than XML,
but that's nothing more than "just" a user-interfacing detail).

>>> For this to work as painless as possible, we need to relook at the
>>> scripting API of the DeploymentComponent and how XML and scripting
>>> integrate better. These features would be 'primary' for the XML:
>>>
>>> * Loading plugins, component libraries etc. -- 'Import'
>>> * Instantiating components -- component struct
>>> * Loading a script (which contains programs andor state machines) --
>>> 'LoadScript'
>>> * Starting a specific program or state machine 'StartScript'.
>>
>> Can you please list here what existing features you intend to remove from
>> the XML approach?
>
> None.
>
Indeed: it serves the structural parts already well. It's the behavioural
part that has to be introduced. With new RTT-XML, or with RTT-scripting, or
with an external deployment framework. I am in favour of the latter, if a
almost perfect one can be found.

>>> (the old XML features would remain as-is but are not recommended)
>>
>> So are you not removing XML features, but deprecating them with an intent
>> to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'.

"The right direction" = more strict decoupling between the 4Cs...!

> I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).
>
>>
>>> This would result in this deployment file which 90% of the applications
>>> uses:

>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>> <properties>
>>> <!-- set import paths -->
>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>>
>>> <!-- Load startup script in Deployer -->
>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>>
>>>  <!-- The name of a file: -->
>>>  <simple name="LoadScript"
>>>  type="string"><value>startup.ops</value></simple> <!-- The name of a
>>>  program or state machine: -->
>>>  <simple name="StartScript"
>>>  type="string"><value>BarProgram</value></simple>
>>>
>>> </struct>
>>> </properties>
>>> 

>>> var int counter = 0; // counter is added to the attributes of the
>>> scripting service (global in this service).
>>>
>>> // foo program becomes a new service in the component
>>> program foo() {
>>>
>>>   var int foo_counter =1;
>>>   while (true) {
>>>
>>>      foo_counter = foo_counter + 1
>>>      yield
>>>
>>>  }
>>>
>>> }
>>>
>>> // after foo has been parsed, it is available:
>>> foo.foo_counter = 0
>>> foo.start()
>>> 

Can you specify in detail why it is bad practice? (Good to have for our
BRICS project! :-))

>>> - connectTwoPorts: does not take a connection policy as argument
>>>
>>> -> would go too and be replaced by:
>>> + connect("comp1.portA","comp2.portB",ConnPolicy)
>>>
>>> -> connects two ports (possibly in services) with a policy
>>>
>>> + createStream("comp1.portC", ConnPolicy)
>>>
>>> -> setup a stream from a port (in a service) with a policy
>>>
>>> Any other suggestions or objections welcome.

These suggestions follow the _procedural_ approach (= you say which actions
have to be taken); the traditional alternative is the _declarative_
approach (= you say which deployment constraints have to be satisfied, and
there is a 'deployment run-time' available in the system that creates a
concrete set of deployment actions from the declarative information.

Declarative approaches have one big advantage: they are the most
composable! Their big disadvantages are: (i) need for a runtime, and (ii)
concrete sequence of actions can become difficult to predict. I have no
clear suggestion about which trade-off to choose here. Principally
speaking, I prefer the declarative approach, because of its composability.

>> The above is not very clear in what changes you intend for user
>> applications. Can you give a more concrete example (pseudo code, not
>> details) showing a v1 XML, and what you intend the v2 version to look
>> like. For instance, the program script you show above looks specific to
>> just one component. How is that helping or hindering large scale or
>> complex deployments?
>
> I need some time to work that out, but I understand the question...
>
>> Can you also list the specific problems in XML that you are trying to
>> address. All I see above is "XML bad, scripting better".

> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions

That's an example of a "runtime" requirement. Very valid use case!

> - Group start/stop/configure components using a custom state machine
Same remark.

> - connect ports/services that were dynamically created during configureHook()
Same remark.

> - conditionally execute deployment steps
Same remark.

> - go step-by-step through a deployment process (using pause()/step() api)
Same remark.

> - Specify easily connection policies etc.
This is the use case of separating the _Configuration_ of the Coordination,
from the _Computation_ of the Coordination to be done at runtime.
Very valid use case.

> - have a very readable/sequential deployment scenario
"readability" is a not so objective metric...

> - have deployment react to events, changing situations etc.
Again the runtime requirement remark.

Overall, I very much support these suggestions. But let's first look at
other frameworks where this has already been done successfully. I could ask
one of the BRICS PhD students to try to look into this matter...

>>> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
>>> too. I expect 2.2.0 to be around by the end of october. There is no
>>> schedule yet for a 1.14.
>>
>> Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.
>
> Peter

Herman

On Thu, 30 Sep 2010, Peter Soetens wrote:

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to give your feedback. These features will be developed after 2.1
>>> is released.
>>
>> Add the ability to deploy components in groups, so that group A is deployed
>> completely, followed by group B, ..., and then group B is un-deployed
>> completely, and finally group A is un-deployed completely. This is
>> necessary for things like logging, which currently hacks this by using the
>> site-file approach. This is really just sequencing application startup and
>> shutdown.
>
> You *can* already do this using scripting and a state machine.That's my whole
> point. No XML / DeploymentComponent extensions are necessary for this.

But there is something more: in many deployment scenarios you _need_ more
than a static description of how components have to be interconnected, and
the deployment, start-up and shut-down sequences require _behaviour_, as
provided by state machines.

>>> Most users are not happy with the XML format that specifies which
>>> components are loaded and connected in an application. Especially the
>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>> even modest applications. The scripting interface on the other hand
>>> provides a more controlled, readable and flexible environment to specify
>>> how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps. For example: importing
>>> component libraries or plugins and instantiating components. Everything
>>> after that should be done in a script.
>>
>> What exactly are you talking about changing here? In particular, what are
>> you talking about removing? I would really like it if nothing that
>> currently works was removed - we already have a high enough barrier
>> getting from v1 to v2 due to "removed features" that we use. It would be
>> nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
>>
>> The only issue we have with XML is that large-scale deployments are tricky
>> due to difficulty in configuring the deployment. The recent discussion on
>> use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.
>

This is the same discussion as we have within BRICS, about how to best
separate the structural and behavioural parts, more specifically of
deployment int his case. To support behaviour, one needs FSM functionality.
But both structural and behavioural parts can be described in XML form. The
concrete RTT problem is that we have no XML tags/DTD yet for FSMs.

It's a valid and timely question to ask now: what makes more sense,
extending our RTT-specific XML/scripting or adopting an existing deployment
"language" (such as OSGi, which uses a more readable file format than XML,
but that's nothing more than "just" a user-interfacing detail).

>>> For this to work as painless as possible, we need to relook at the
>>> scripting API of the DeploymentComponent and how XML and scripting
>>> integrate better. These features would be 'primary' for the XML:
>>>
>>> * Loading plugins, component libraries etc. -- 'Import'
>>> * Instantiating components -- component struct
>>> * Loading a script (which contains programs andor state machines) --
>>> 'LoadScript'
>>> * Starting a specific program or state machine 'StartScript'.
>>
>> Can you please list here what existing features you intend to remove from
>> the XML approach?
>
> None.
>
Indeed: it serves the structural parts already well. It's the behavioural
part that has to be introduced. With new RTT-XML, or with RTT-scripting, or
with an external deployment framework. I am in favour of the latter, if a
almost perfect one can be found.

>>> (the old XML features would remain as-is but are not recommended)
>>
>> So are you not removing XML features, but deprecating them with an intent
>> to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'.

"The right direction" = more strict decoupling between the 4Cs...!

> I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).
>
>>
>>> This would result in this deployment file which 90% of the applications
>>> uses:

>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>> <properties>
>>> <!-- set import paths -->
>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>>
>>> <!-- Load startup script in Deployer -->
>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>>
>>>  <!-- The name of a file: -->
>>>  <simple name="LoadScript"
>>>  type="string"><value>startup.ops</value></simple> <!-- The name of a
>>>  program or state machine: -->
>>>  <simple name="StartScript"
>>>  type="string"><value>BarProgram</value></simple>
>>>
>>> </struct>
>>> </properties>
>>> 

>>> var int counter = 0; // counter is added to the attributes of the
>>> scripting service (global in this service).
>>>
>>> // foo program becomes a new service in the component
>>> program foo() {
>>>
>>>   var int foo_counter =1;
>>>   while (true) {
>>>
>>>      foo_counter = foo_counter + 1
>>>      yield
>>>
>>>  }
>>>
>>> }
>>>
>>> // after foo has been parsed, it is available:
>>> foo.foo_counter = 0
>>> foo.start()
>>> 

Can you specify in detail why it is bad practice? (Good to have for our
BRICS project! :-))

>>> - connectTwoPorts: does not take a connection policy as argument
>>>
>>> -> would go too and be replaced by:
>>> + connect("comp1.portA","comp2.portB",ConnPolicy)
>>>
>>> -> connects two ports (possibly in services) with a policy
>>>
>>> + createStream("comp1.portC", ConnPolicy)
>>>
>>> -> setup a stream from a port (in a service) with a policy
>>>
>>> Any other suggestions or objections welcome.

These suggestions follow the _procedural_ approach (= you say which actions
have to be taken); the traditional alternative is the _declarative_
approach (= you say which deployment constraints have to be satisfied, and
there is a 'deployment run-time' available in the system that creates a
concrete set of deployment actions from the declarative information.

Declarative approaches have one big advantage: they are the most
composable! Their big disadvantages are: (i) need for a runtime, and (ii)
concrete sequence of actions can become difficult to predict. I have no
clear suggestion about which trade-off to choose here. Principally
speaking, I prefer the declarative approach, because of its composability.

>> The above is not very clear in what changes you intend for user
>> applications. Can you give a more concrete example (pseudo code, not
>> details) showing a v1 XML, and what you intend the v2 version to look
>> like. For instance, the program script you show above looks specific to
>> just one component. How is that helping or hindering large scale or
>> complex deployments?
>
> I need some time to work that out, but I understand the question...
>
>> Can you also list the specific problems in XML that you are trying to
>> address. All I see above is "XML bad, scripting better".

> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions

That's an example of a "runtime" requirement. Very valid use case!

> - Group start/stop/configure components using a custom state machine
Same remark.

> - connect ports/services that were dynamically created during configureHook()
Same remark.

> - conditionally execute deployment steps
Same remark.

> - go step-by-step through a deployment process (using pause()/step() api)
Same remark.

> - Specify easily connection policies etc.
This is the use case of separating the _Configuration_ of the Coordination,
from the _Computation_ of the Coordination to be done at runtime.
Very valid use case.

> - have a very readable/sequential deployment scenario
"readability" is a not so objective metric...

> - have deployment react to events, changing situations etc.
Again the runtime requirement remark.

Overall, I very much support these suggestions. But let's first look at
other frameworks where this has already been done successfully. I could ask
one of the BRICS PhD students to try to look into this matter...

>>> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
>>> too. I expect 2.2.0 to be around by the end of october. There is no
>>> schedule yet for a 1.14.
>>
>> Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.
>
> Peter

Herman

If I can from my little Orocos experience, XML is not designed to be
read/edit by humans. Even seperated in small parts it's boring to
write/edit/maintain. XML whould be appropriate if a GUI-tool was available
to create the deployment files.

My personnal point of view :
script deployment is more close to your [human] way of thinking when your
are thinking about deployment, whereas XML needs "mind reorganisations"
which is tiring and leads to more bugs.

2010/9/30 Peter Soetens <peter [..] ...>

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> > On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> > > We're looking at improving deployment of components, and now's a good
> > > time to give your feedback. These features will be developed after 2.1
> > > is released.
> >
> > Add the ability to deploy components in groups, so that group A is
> deployed
> > completely, followed by group B, ..., and then group B is un-deployed
> > completely, and finally group A is un-deployed completely. This is
> > necessary for things like logging, which currently hacks this by using
> the
> > site-file approach. This is really just sequencing application startup
> and
> > shutdown.
>
> You *can* already do this using scripting and a state machine.That's my
> whole
> point. No XML / DeploymentComponent extensions are necessary for this.
>
> >
> > > Most users are not happy with the XML format that specifies which
> > > components are loaded and connected in an application. Especially the
> > > flexibility and the fixed ordering of configuration/setup steps
> disturbs
> > > even modest applications. The scripting interface on the other hand
> > > provides a more controlled, readable and flexible environment to
> specify
> > > how an application must be setup.
> > >
> > > Therefor, the 'best practice' in deployment will emphasis doing things
> in
> > > scripts and leave XML for basic setup steps. For example: importing
> > > component libraries or plugins and instantiating components. Everything
> > > after that should be done in a script.
> >
> > What exactly are you talking about changing here? In particular, what are
> > you talking about removing? I would really like it if nothing that
> > currently works was removed - we already have a high enough barrier
> > getting from v1 to v2 due to "removed features" that we use. It would be
> > nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
> >
> > The only issue we have with XML is that large-scale deployments are
> tricky
> > due to difficulty in configuring the deployment. The recent discussion on
> > use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.
>
> >
> > > For this to work as painless as possible, we need to relook at the
> > > scripting API of the DeploymentComponent and how XML and scripting
> > > integrate better. These features would be 'primary' for the XML:
> > >
> > > * Loading plugins, component libraries etc. -- 'Import'
> > > * Instantiating components -- component struct
> > > * Loading a script (which contains programs andor state machines) --
> > > 'LoadScript'
> > > * Starting a specific program or state machine 'StartScript'.
> >
> > Can you please list here what existing features you intend to remove from
> > the XML approach?
>
> None.
>
> >
> > > (the old XML features would remain as-is but are not recommended)
> >
> > So are you not removing XML features, but deprecating them with an intent
> > to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'. I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).
>
> >
> > > This would result in this deployment file which 90% of the applications
> > > uses:

> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <!DOCTYPE properties SYSTEM "cpf.dtd">
> > > <properties>
> > > <!-- set import paths -->
> > > <simple name="Import"
> type="string"><value>/usr/local/lib/orocos</simple>
> > >
> > > <!-- Load startup script in Deployer -->
> > > <struct name="Deployer" type="OCL::DeploymentComponent">
> > >
> > >  <!-- The name of a file: -->
> > >  <simple name="LoadScript"
> > >  type="string"><value>startup.ops</value></simple> <!-- The name of a
> > >  program or state machine: -->
> > >  <simple name="StartScript"
> > >  type="string"><value>BarProgram</value></simple>
> > >
> > > </struct>
> > > </properties>
> > > 

> > > var int counter = 0; // counter is added to the attributes of the
> > > scripting service (global in this service).
> > >
> > > // foo program becomes a new service in the component
> > > program foo() {
> > >
> > >   var int foo_counter =1;
> > >   while (true) {
> > >
> > >      foo_counter = foo_counter + 1
> > >      yield
> > >
> > >  }
> > >
> > > }
> > >
> > > // after foo has been parsed, it is available:
> > > foo.foo_counter = 0
> > > foo.start()
> > > 

If I can from my little Orocos experience, XML is not designed to be
read/edit by humans. Even seperated in small parts it's boring to
write/edit/maintain. XML whould be appropriate if a GUI-tool was available
to create the deployment files.

My personnal point of view :
script deployment is more close to your [human] way of thinking when your
are thinking about deployment, whereas XML needs "mind reorganisations"
which is tiring and leads to more bugs.

2010/9/30 Peter Soetens <peter [..] ...>

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
> > On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
> > > We're looking at improving deployment of components, and now's a good
> > > time to give your feedback. These features will be developed after 2.1
> > > is released.
> >
> > Add the ability to deploy components in groups, so that group A is
> deployed
> > completely, followed by group B, ..., and then group B is un-deployed
> > completely, and finally group A is un-deployed completely. This is
> > necessary for things like logging, which currently hacks this by using
> the
> > site-file approach. This is really just sequencing application startup
> and
> > shutdown.
>
> You *can* already do this using scripting and a state machine.That's my
> whole
> point. No XML / DeploymentComponent extensions are necessary for this.
>
> >
> > > Most users are not happy with the XML format that specifies which
> > > components are loaded and connected in an application. Especially the
> > > flexibility and the fixed ordering of configuration/setup steps
> disturbs
> > > even modest applications. The scripting interface on the other hand
> > > provides a more controlled, readable and flexible environment to
> specify
> > > how an application must be setup.
> > >
> > > Therefor, the 'best practice' in deployment will emphasis doing things
> in
> > > scripts and leave XML for basic setup steps. For example: importing
> > > component libraries or plugins and instantiating components. Everything
> > > after that should be done in a script.
> >
> > What exactly are you talking about changing here? In particular, what are
> > you talking about removing? I would really like it if nothing that
> > currently works was removed - we already have a high enough barrier
> > getting from v1 to v2 due to "removed features" that we use. It would be
> > nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
> >
> > The only issue we have with XML is that large-scale deployments are
> tricky
> > due to difficulty in configuring the deployment. The recent discussion on
> > use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.
>
> >
> > > For this to work as painless as possible, we need to relook at the
> > > scripting API of the DeploymentComponent and how XML and scripting
> > > integrate better. These features would be 'primary' for the XML:
> > >
> > > * Loading plugins, component libraries etc. -- 'Import'
> > > * Instantiating components -- component struct
> > > * Loading a script (which contains programs andor state machines) --
> > > 'LoadScript'
> > > * Starting a specific program or state machine 'StartScript'.
> >
> > Can you please list here what existing features you intend to remove from
> > the XML approach?
>
> None.
>
> >
> > > (the old XML features would remain as-is but are not recommended)
> >
> > So are you not removing XML features, but deprecating them with an intent
> > to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'. I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).
>
> >
> > > This would result in this deployment file which 90% of the applications
> > > uses:

> > > <?xml version="1.0" encoding="UTF-8"?>
> > > <!DOCTYPE properties SYSTEM "cpf.dtd">
> > > <properties>
> > > <!-- set import paths -->
> > > <simple name="Import"
> type="string"><value>/usr/local/lib/orocos</simple>
> > >
> > > <!-- Load startup script in Deployer -->
> > > <struct name="Deployer" type="OCL::DeploymentComponent">
> > >
> > >  <!-- The name of a file: -->
> > >  <simple name="LoadScript"
> > >  type="string"><value>startup.ops</value></simple> <!-- The name of a
> > >  program or state machine: -->
> > >  <simple name="StartScript"
> > >  type="string"><value>BarProgram</value></simple>
> > >
> > > </struct>
> > > </properties>
> > > 

> > > var int counter = 0; // counter is added to the attributes of the
> > > scripting service (global in this service).
> > >
> > > // foo program becomes a new service in the component
> > > program foo() {
> > >
> > >   var int foo_counter =1;
> > >   while (true) {
> > >
> > >      foo_counter = foo_counter + 1
> > >      yield
> > >
> > >  }
> > >
> > > }
> > >
> > > // after foo has been parsed, it is available:
> > > foo.foo_counter = 0
> > > foo.start()
> > > 

On Sep 30, 2010, at 08:27 , Peter Soetens wrote:

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to give your feedback. These features will be developed after 2.1
>>> is released.
>>
>> Add the ability to deploy components in groups, so that group A is deployed
>> completely, followed by group B, ..., and then group B is un-deployed
>> completely, and finally group A is un-deployed completely. This is
>> necessary for things like logging, which currently hacks this by using the
>> site-file approach. This is really just sequencing application startup and
>> shutdown.
>
> You *can* already do this using scripting and a state machine.That's my whole
> point. No XML / DeploymentComponent extensions are necessary for this.

Got an example?

>>> Most users are not happy with the XML format that specifies which
>>> components are loaded and connected in an application. Especially the
>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>> even modest applications. The scripting interface on the other hand
>>> provides a more controlled, readable and flexible environment to specify
>>> how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps. For example: importing
>>> component libraries or plugins and instantiating components. Everything
>>> after that should be done in a script.
>>
>> What exactly are you talking about changing here? In particular, what are
>> you talking about removing? I would really like it if nothing that
>> currently works was removed - we already have a high enough barrier
>> getting from v1 to v2 due to "removed features" that we use. It would be
>> nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
>>
>> The only issue we have with XML is that large-scale deployments are tricky
>> due to difficulty in configuring the deployment. The recent discussion on
>> use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.

Can you give an example here. Besides master/slave ordering, we've not hit any issues like this, so I'd like to understand the problem better.

>>>
>>> For this to work as painless as possible, we need to relook at the
>>> scripting API of the DeploymentComponent and how XML and scripting
>>> integrate better. These features would be 'primary' for the XML:
>>>
>>> * Loading plugins, component libraries etc. -- 'Import'
>>> * Instantiating components -- component struct
>>> * Loading a script (which contains programs andor state machines) --
>>> 'LoadScript'
>>> * Starting a specific program or state machine 'StartScript'.
>>
>> Can you please list here what existing features you intend to remove from
>> the XML approach?
>
> None.
>
>>
>>> (the old XML features would remain as-is but are not recommended)
>>
>> So are you not removing XML features, but deprecating them with an intent
>> to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'. I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).

Read = cutting out any new features from XML users, forcing us to change to this combined XML/scripting approach? I'm ok with this, we all have to evolve, but I would really like some serious discussion on this. We've already gotten badly bitten by v2, I'd like to see such issues coming more clearly.

>>>
>>> This would result in this deployment file which 90% of the applications
>>> uses:

>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>> <properties>
>>> <!-- set import paths -->
>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>> 
>>> <!-- Load startup script in Deployer -->
>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>> 
>>> <!-- The name of a file: -->
>>> <simple name="LoadScript"
>>> type="string"><value>startup.ops</value></simple> <!-- The name of a
>>> program or state machine: -->
>>> <simple name="StartScript"
>>> type="string"><value>BarProgram</value></simple>
>>> 
>>> </struct>
>>> </properties>
>>> 

>>> var int counter = 0; // counter is added to the attributes of the
>>> scripting service (global in this service).
>>> 
>>> // foo program becomes a new service in the component
>>> program foo() {
>>> 
>>>  var int foo_counter =1;
>>>  while (true) {
>>> 
>>>     foo_counter = foo_counter + 1
>>>     yield
>>> 
>>> }
>>> 
>>> }
>>> 
>>> // after foo has been parsed, it is available:
>>> foo.foo_counter = 0
>>> foo.start()
>>> 

Ack

>> Can you also list the specific problems in XML that you are trying to
>> address. All I see above is "XML bad, scripting better".
>
> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions

Ack

> - Group start/stop/configure components using a custom state machine

We already do this in SM. What are you getting at?

> - connect ports/services that were dynamically created during configureHook()

Ack

> - conditionally execute deployment steps

Ack

> - go step-by-step through a deployment process (using pause()/step() api)

Debugging, ok

> - Specify easily connection policies etc.

Instead of adding this to XML?

> - have a very readable/sequential deployment scenario

XML isn't?

> - have deployment react to events, changing situations etc.

Can you give more of an example of these two. Not sure what you are really getting at.

>>>
>>> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
>>> too. I expect 2.2.0 to be around by the end of october. There is no
>>> schedule yet for a 1.14.
>>
>> Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.

Please do. I would like to have some input ... I'm sure others in the community do also.

I understand your desire to change this, and definitely agree with some of them, but without a complete example it is very hard to see what issues we are trying to solve. Orocos is flexible enough to provide many different ways of doing the same thing, and not all of us hit the same issues as you (and hence don't necessarily understand why you want to change something).
S

On Sep 30, 2010, at 08:27 , Peter Soetens wrote:

> On Thursday 30 September 2010 14:09:34 S Roderick wrote:
>> On Sep 30, 2010, at 07:45 , Peter Soetens wrote:
>>> We're looking at improving deployment of components, and now's a good
>>> time to give your feedback. These features will be developed after 2.1
>>> is released.
>>
>> Add the ability to deploy components in groups, so that group A is deployed
>> completely, followed by group B, ..., and then group B is un-deployed
>> completely, and finally group A is un-deployed completely. This is
>> necessary for things like logging, which currently hacks this by using the
>> site-file approach. This is really just sequencing application startup and
>> shutdown.
>
> You *can* already do this using scripting and a state machine.That's my whole
> point. No XML / DeploymentComponent extensions are necessary for this.

Got an example?

>>> Most users are not happy with the XML format that specifies which
>>> components are loaded and connected in an application. Especially the
>>> flexibility and the fixed ordering of configuration/setup steps disturbs
>>> even modest applications. The scripting interface on the other hand
>>> provides a more controlled, readable and flexible environment to specify
>>> how an application must be setup.
>>>
>>> Therefor, the 'best practice' in deployment will emphasis doing things in
>>> scripts and leave XML for basic setup steps. For example: importing
>>> component libraries or plugins and instantiating components. Everything
>>> after that should be done in a script.
>>
>> What exactly are you talking about changing here? In particular, what are
>> you talking about removing? I would really like it if nothing that
>> currently works was removed - we already have a high enough barrier
>> getting from v1 to v2 due to "removed features" that we use. It would be
>> nice if this barrier didn't grow further.
>
> I'm not planning to remove any of them.
>
>>
>> The only issue we have with XML is that large-scale deployments are tricky
>> due to difficulty in configuring the deployment. The recent discussion on
>> use of the XML standard ENTITY element may solve much of that for us.
>
> That's true, but it still won't solve ordering problems.

Can you give an example here. Besides master/slave ordering, we've not hit any issues like this, so I'd like to understand the problem better.

>>>
>>> For this to work as painless as possible, we need to relook at the
>>> scripting API of the DeploymentComponent and how XML and scripting
>>> integrate better. These features would be 'primary' for the XML:
>>>
>>> * Loading plugins, component libraries etc. -- 'Import'
>>> * Instantiating components -- component struct
>>> * Loading a script (which contains programs andor state machines) --
>>> 'LoadScript'
>>> * Starting a specific program or state machine 'StartScript'.
>>
>> Can you please list here what existing features you intend to remove from
>> the XML approach?
>
> None.
>
>>
>>> (the old XML features would remain as-is but are not recommended)
>>
>> So are you not removing XML features, but deprecating them with an intent
>> to not maintain in the future?
>
> Well, 'deprecating with the intent to point users in the right direction in
> the future'. I'm also saying this: 'not adding any more features to the XML
> format (ie freezing it).

Read = cutting out any new features from XML users, forcing us to change to this combined XML/scripting approach? I'm ok with this, we all have to evolve, but I would really like some serious discussion on this. We've already gotten badly bitten by v2, I'd like to see such issues coming more clearly.

>>>
>>> This would result in this deployment file which 90% of the applications
>>> uses:

>>> <?xml version="1.0" encoding="UTF-8"?>
>>> <!DOCTYPE properties SYSTEM "cpf.dtd">
>>> <properties>
>>> <!-- set import paths -->
>>> <simple name="Import" type="string"><value>/usr/local/lib/orocos</simple>
>>> 
>>> <!-- Load startup script in Deployer -->
>>> <struct name="Deployer" type="OCL::DeploymentComponent">
>>> 
>>> <!-- The name of a file: -->
>>> <simple name="LoadScript"
>>> type="string"><value>startup.ops</value></simple> <!-- The name of a
>>> program or state machine: -->
>>> <simple name="StartScript"
>>> type="string"><value>BarProgram</value></simple>
>>> 
>>> </struct>
>>> </properties>
>>> 

>>> var int counter = 0; // counter is added to the attributes of the
>>> scripting service (global in this service).
>>> 
>>> // foo program becomes a new service in the component
>>> program foo() {
>>> 
>>>  var int foo_counter =1;
>>>  while (true) {
>>> 
>>>     foo_counter = foo_counter + 1
>>>     yield
>>> 
>>> }
>>> 
>>> }
>>> 
>>> // after foo has been parsed, it is available:
>>> foo.foo_counter = 0
>>> foo.start()
>>> 

Ack

>> Can you also list the specific problems in XML that you are trying to
>> address. All I see above is "XML bad, scripting better".
>
> What we can do in scripts and not (easily) in xml during deployment:
> - Use data/properties in other components to make deployment decisions

Ack

> - Group start/stop/configure components using a custom state machine

We already do this in SM. What are you getting at?

> - connect ports/services that were dynamically created during configureHook()

Ack

> - conditionally execute deployment steps

Ack

> - go step-by-step through a deployment process (using pause()/step() api)

Debugging, ok

> - Specify easily connection policies etc.

Instead of adding this to XML?

> - have a very readable/sequential deployment scenario

XML isn't?

> - have deployment react to events, changing situations etc.

Can you give more of an example of these two. Not sure what you are really getting at.

>>>
>>> I'll release 2.0.2 today, 2.1 early next week and 1.12 in the coming days
>>> too. I expect 2.2.0 to be around by the end of october. There is no
>>> schedule yet for a 1.14.
>>
>> Do we have roadmap somewhere for what is going in these future releases?
>
> We can prepare one on the wiki.

Please do. I would like to have some input ... I'm sure others in the community do also.

I understand your desire to change this, and definitely agree with some of them, but without a complete example it is very hard to see what issues we are trying to solve. Orocos is flexible enough to provide many different ways of doing the same thing, and not all of us hit the same issues as you (and hence don't necessarily understand why you want to change something).
S