I've created a little table with both Peter and my proposal (attached to
this mail). It represents the possible transitions, the hooks that get
called and when each transition/hook gets called.

For the record, this idea of "separating the application state machine
from the component lifetime state machine" still does not make any sense
to me. Both state machines are obviously tied together (a motor
controller cannot be in CONTROLLING_MOTORS state while the component is
in FatalError).

The main differences between the two are:
- I add a UnrecoverableError state which is Peter's FatalError state.
FatalError, in my case, is recoverable, it only requires a restart of
the component.
- unhandled exceptions always end up terminating the component's
execution. I have no clue how a component could continue running (i.e.
providing functionality) after it raised an unexpected/unhandled
exception. The only sane thing to do *by default* is to try and stop
everything you can, and see if you can restart properly.

I also added, in my proposal, what to do with unhandled exceptions in
configureHook() and startHook(). I'm really less sure about those. It
just happens that, in Roby [supervision layer], I tried different things
and terminating tasks when the "start" command fails ended up being the
safest.

Sylvain

On Mon, May 3, 2010 at 12:31 PM, Peter Soetens <peter [..] ...> wrote:
> To all lurking on this thread, could we have a 'voting' about this ?
>
> In case of doubt, I follow the user's opinion, but since we only have Sylvain
> and me arguing, I wonder how much user there is going on here actually :-)

I've probably given up since I noticed the thread had more than 100
SLOT (T for text) and not a single FSM diagram ;-)

> Summary:
>
> * The RTT 1.x component states are mixing component lifecycle and application
> states. For example, configureHook() sets up method calls using 'getPeer()'  or
> checks if input (read) ports are connected. It could *also* configure a device
> or so, but with limited flexibility, since the thread of the component was not
> running yet. So some configuration could be necessary in updateHook(), for
> example, if you were talking to a device bus. On the other hand, the component
> has some clear 'application' error states, like RunTimeError, which a
> component will only enter if user code instructs it to do so.
>
> I wanted to change this in 2.0 to a reduced life cycle, where the only states
> that a component has are independent of application states. For example,
> RunTimeError would mean that an exception was leaked in updateHook(). If
> errorHook() leaked an exception, the component would enter the FatalError
> state, which is unrecoverable (hence 'fatal'). My main motivation for this was
> to define what happens when user code throws exceptions. I wanted to have a
> similar scheme as was happening with program scripts: it's not because one
> program is in error, that the whole component should stop. One change
> contributing to this filosophy is that the thread of a component now always
> runs.
>
> I should have known by looking at historical evidence, but with this change, I
> stepped on Sylvain's turf.
>
> He proposes (correct me if I'm wrong) to keep close to the current application
> states, at least the RunTimeError state for user errors and use FatalError
> (=stop+cleanup) if user code throws (in any place). If transition to fatal
> fails, an unrecoverable-worse-than-fatal state is entered. His reasoning is
> that supervision needs info of application health for every component, and
> that this belongs in the interface every component.
>
> Pleaes read full details below or in the thread.

No, thx :-)

> I don't think the water is that deep between us, most developers use
> configureHook/errorHook already for application states so I see the point, and
> no one is complainging... We should mainly focus on user's ease of
> programming/minimal coding effort, but what do the others think ?

Regarding the mix of application/framework states: we recently had an
issue with an application which needs to be examined in detail (I
expect Steven to post on this during the course of this week), but
_seems_ at first sight (I know this is dangerous, I haven't even
looked at the code :-) to be caused by the following.

For those in need of a diagram: See
<http://www.orocos.org/stable/documentation/rtt/v1.10.x/doc-xml/orocos-components-manual.html#id3110846>
You can create a component and instruct it to start in preoperational.
However, it looked like you can call the destructor on that same
component if it's in stopped state, and it won't "force" it back to
preop first (hence executing cleanupHook). Ofcourse, you can force
this transition in your destructor yourself, but I wonder if this was
intentionally "left blank" (usecase?).

Best regards,

Klaas [..] ...

On May 3, 2010, at 06:31 , Peter Soetens wrote:

> To all lurking on this thread, could we have a 'voting' about this ?

1) Could someone draw up a state chart of what you guys propose? The long discussion below is hell of a fragmented ...

> In case of doubt, I follow the user's opinion, but since we only have Sylvain
> and me arguing, I wonder how much user there is going on here actually :-)
>
> Summary:
>
> * The RTT 1.x component states are mixing component lifecycle and application
> states. For example, configureHook() sets up method calls using 'getPeer()' or
> checks if input (read) ports are connected. It could *also* configure a device
> or so, but with limited flexibility, since the thread of the component was not
> running yet. So some configuration could be necessary in updateHook(), for
> example, if you were talking to a device bus. On the other hand, the component
> has some clear 'application' error states, like RunTimeError, which a
> component will only enter if user code instructs it to do so.
>
> I wanted to change this in 2.0 to a reduced life cycle, where the only states
> that a component has are independent of application states. For example,
> RunTimeError would mean that an exception was leaked in updateHook(). If
> errorHook() leaked an exception, the component would enter the FatalError
> state, which is unrecoverable (hence 'fatal'). My main motivation for this was
> to define what happens when user code throws exceptions. I wanted to have a
> similar scheme as was happening with program scripts: it's not because one
> program is in error, that the whole component should stop. One change
> contributing to this filosophy is that the thread of a component now always
> runs.

In my 2.5 years using Orocos, I've never used either an errorHook() nor the RunTimeError-related stuff. I can see a couple of cases where I should have ...

> I should have known by looking at historical evidence, but with this change, I
> stepped on Sylvain's turf.
>
> He proposes (correct me if I'm wrong) to keep close to the current application
> states, at least the RunTimeError state for user errors and use FatalError
> (=stop+cleanup) if user code throws (in any place). If transition to fatal
> fails, an unrecoverable-worse-than-fatal state is entered. His reasoning is
> that supervision needs info of application health for every component, and
> that this belongs in the interface every component.

It _looks_ like one or both of you are proposing

exception/error in updateHook() = transition to RunTimeError state = call errorHook()
exception/error in errorHook() = transition to FatalError = call stopHook() then cleanupHook().

And RunTimeError is a running state, so scripts and state machines are still running (if they aren't the cause of the throw), right? Does errorHook() get called as updateHook() would have (ie periodically if in a PeriodicActivity)?

How do you get out of errorHook()?

IMHO the state after fatalError() should be STOPPED, not PRE_OPERATIONAL. The component is done.

What happens if you exception/error in stopHook() or cleanupHook() that was called as part of a FatalError? Personally, I would think to catch it in FatalError and don't continue doing anything. I don't see the point of yet another worse-than-fatal-error state.

Sylvain, we do use scripts some, particularly where the corresponding C++ code is too long, but they do have their issues that limit their usefulness.

We do some potentially blocking driver setup in configure() also.

Peter, what is the difference with v2 w.r.t. the thread being active during configureHook(), or being able to send commands prior to start()?

I don't like the idea of a SupervisionInterface. Yet another thing to learn, and I see little benefit.
S

>
> Pleaes read full details below or in the thread.
>
> I don't think the water is that deep between us, most developers use
> configureHook/errorHook already for application states so I see the point, and
> no one is complainging... We should mainly focus on user's ease of
> programming/minimal coding effort, but what do the others think ?
>
> Peter
>
> On Wednesday 21 April 2010 14:31:17 Sylvain Joyeux wrote:
>> Peter Soetens wrote:
>>> On Wednesday 21 April 2010 12:37:09 Sylvain Joyeux wrote:
>>>> Peter Soetens wrote:
>>>>>>>> As far as I saw on the RTT2 ExecutionEngine code, exception handling
>>>>>>>> is as follows (Peter: tell me if I'm wrong):
>>>>>>>>
>>>>>>>> * an uncaught exception in updateHook() transitions to
>>>>>>>> RUNTIME_ERROR * an uncaught exception in errorHook() transitions to
>>>>>>>> FATAL_ERROR
>>>>>>>
>>>>>>> Correct. Fatal error (should) leads to stopHook() + cleanupHook() and
>>>>>>> then wait for component cleanup/removal.
>>>>>>>
>>>>>>>> This assumes that errorHook() is able to handle unspecified errors.
>>>>>>>> This seems to broad for my POV. How we design components is that the
>>>>>>>> transition to fatal() is basically a stop() + some sort of tentative
>>>>>>>> cleanup. runtime_error is used as a runtime state categorization
>>>>>>>> (i.e. a way to "regroup" internal states), but its interpretation as
>>>>>>>> an actual error is situation dependent.
>>>>>>>
>>>>>>> It is. errorHook() is reserved for an RTT/Component specific error
>>>>>>> state, not for application error states. So you would need to write
>>>>>>> your application specific error states in updateHook().
>>>>>>
>>>>>> This completely negates the usefulness of the taskcontext state
>>>>>> machine.
>>>>>
>>>>> ... for application specific state machines. From a component life
>>>>> cycle view, these states are still necessary. I think we did it the
>>>>> wrong way in 1.x, coupling component lifecycle states with application
>>>>> states. They may overlap, and we can't/won't prevent that, but they
>>>>> don't have to overlap.
>>>>
>>>> OK, then define a default application state machine (and see the number
>>>> of state explode). Not having a default application state machine
>>>> defined negates completely the use of
>>>
>>> You have to see this in the light of the state machines in the scripting.
>>> These are by definition application specific state machines. So this made
>>> us realize that there is a difference between the lifecycle of a
>>> component (hooks) and of an application (states in scripts).
>>>
>>>>>>>> For instance, our motor controllers go into runtime_error when the
>>>>>>>> motors can't be driven (because of hardware protection mechanisms
>>>>>>>> for instance), but the electronics still *reads* the encoder + motor
>>>>>>>> data. I.e. they are read-only when in runtime error and can be used
>>>>>>>> if only reading is needed. fatal_error would be entered if we are
>>>>>>>> not able to talk to the electronics anymore.
>>>>>>>
>>>>>>> These are all application states and should not be implemented in the
>>>>>>> component lifecycle states.
>>>>>>
>>>>>> I don't follow you in this split between application and lifecycle
>>>>>> states. The component goes into various states because of the
>>>>>> application.
>>>>>
>>>>> Not from the viewpoint of the RTT or deployer. For example,
>>>>> configureHook is to check if input ports are connected or if required
>>>>> services are available. This is independent of a component requiring
>>>>> additional configuration of parameters.
>>>>
>>>> There is definitely a mismatch between our uses of states. We use
>>>> configureHook() to verify that the component can run. This means:
>>>> checking if devices are there, if properties are set to sane values and
>>>> so on. In effect, for device drivers, configureHook() is the place
>>>> where the device gets accessed and configured.
>>>
>>> I agree here. RTT 2.x actually supports this better, since you can also
>>> send 'commands' (in 2.x: send method calls) to a component before it is
>>> started. This means that if configuration does some blocking/asynchronous
>>> work or depends on a script to complete, this can all be done before
>>> start().
>>>
>>>>>>>> In a way, runtime_error is not very useful in this case ...
>>>>>>>>
>>>>>>>> To get back to the point: I think that runtime_error should be used
>>>>>>>> when the component is still able to provide a limited functionality,
>>>>>>>> fatalError being used when the component does not provide any
>>>>>>>> functionality anymore. Thus, the default exception handling of
>>>>>>>> updateHook() should IMO transition to FATAL_ERROR: I don't see how a
>>>>>>>> component can *know* what it is doing when an uncaught exception has
>>>>>>>> been raised by updateHook().
>>>>>>>>
>>>>>>>> Thoughts ?
>>>>>>>
>>>>>>> I think you identified a painpoint when trying to apply the component
>>>>>>> error states to application error states. It will never work. The
>>>>>>> idea for run time error is that any code in updateHook might throw,
>>>>>>> even if the user is unaware of this (during development for example).
>>>>>>> In critical components, you can put safe state code in errorHook(),
>>>>>>> for example, writing data to ports (which will never throw). If you
>>>>>>> did a
>>>>>>>
>>>>>>> bad job there, you go to fatal error. So all cases are covered. We
>>>>>>> don't want to go to fatal error immediately because this is an
>>>>>>> unrecoverable state, meaning, RTT judged that it can no longer
>>>>>>> execute that *instance* of a component.
>>>>>>>
>>>>>>> All the other stuff goes into updateHook and you need to define your
>>>>>>> own application states in there, using own operations or attributes
>>>>>>> or so.
>>>>>>>
>>>>>>> Makes sense ?
>>>>>>
>>>>>> I don't think it does ...
>>>>>>
>>>>>> First of all, most components will have nothing in errorHook(). Thus,
>>>>>> you will have a still-running component that has failed in a way that
>>>>>> was not predicted by the designer.
>>>>>
>>>>> We could install a default action in errorHook() ourselves.
>>>>
>>>> What could you meanginfully do in errorHook() that is completely generic
>>>> and will handle the underlying problem (updateHook() threw an exception,
>>>> the component is not functional anymore).
>>>
>>> Yeah, I wasn't making sense here.
>>>
>>>>>> From a more conceptual point of view, I don't think that a component
>>>>>> should be allowed to run even if it had an unexpected exception. An
>>>>>> exception means "the internal state of the component is unspecified as
>>>>>> of now". The only thing that could make sense is to try to "emergency
>>>>>> stop" it, which -- I though -- is what fatal error is there for.
>>>>>
>>>>> Agreed, but C++ exceptions do not cause an unspecified state. They
>>>>> unwind the stack and cleanup resources by calling destructors. It's not
>>>>> the same like a segfault. On the other hand, if your updateHook() has
>>>>> the scenario port1.write (exception) port2.write, only the first write
>>>>> will succeed, leading to a non consistent output. So yes, a leaked
>>>>> exception is maybe more 'grave' than it is considered now.
>>>>
>>>> I don't agree there. An *uncaught* exception means that the component
>>>> designer was not *expecting* this particular error. If the code is
>>>> well-written (which is very unlikely), ressources will be freed and so
>>>> on, but from a global logic point of view, the application will *not*
>>>> know where it was (hey, otherwise it would have caught this exception).
>>>
>>> So actually we agree, in the end, the application does not know. So this
>>> is a problematic state it ends up in.
>>
>> Yes, so it makes no sense to remain in a running state (which
>> RUNTIME_ERROR is).
>>
>>>>> Realize that the scripts can also leak exceptions, because they call
>>>>> user functions too. We also need to define a state when this happens, I
>>>>> don't want to go into 'unrecoverable error' when this happens. For a
>>>>> script, this would just cause the 'E'rror status of that script, while
>>>>> other scripts/updateHook keep running. That's also why runtime error
>>>>> only relates to an exception in updateHook(), while in runtime error,
>>>>> the scripts etc keep on executing (unless they reach the Error status
>>>>> too).
>>>>>
>>>>>> Second, the way you define fatal error states makes no sense to me.
>>>>>> The whole point of having a component model a-la RTT is that it should
>>>>>> be able to go back to a defined state (in my POV, through the
>>>>>> fatalError() cycle).
>>>>>>
>>>>>>
>>>>>>
>>>>>> I.e. a "completely unrecoverable" error should only be a diagnostics
>>>>>> estimation, for instance triggered because a component that went to an
>>>>>> unspecified fatalError() (fatal-error-that-we-don't-know) refused to
>>>>>> reconfigure and/or restart), thus showing that the component does not
>>>>>> know how to recover.
>>>>>
>>>>> There is another reason: what if the RTT figures out that it can no
>>>>> longer execute the component ? Fatal by definition means unrecoverable,
>>>>> so let's keep it with these semantics. There is no way to recover from
>>>>> the fatal error state in the current implementation. So fatal means:
>>>>> unload/kill me please.
>>>>>
>>>>> What you describe sounds to me like a run-time error (recoverable), you
>>>>> can still recover from it.
>>>>
>>>> Again, we have a different understanding of the state machine. Yes, it
>>>> can recover from it, but that will require a stop()/configure()/start()
>>>> cycle. How the state machine is interpreted by our supervision is:
>>>>
>>>> configure: the component verifies that everything it needs to be
>>>> functional is there. This means: checking property values, port
>>>> connections, accessing external processes/hardware when applicable. The
>>>> goal of that step is to make start() as simple as possible, and have it
>>>> most likely return true (i.e. have the longest and most likely to fail
>>>> steps that can be done in advance in configure()).
>>>
>>> OK.
>>>
>>>> start: start the component functionality. I.e. turn on data
>>>> acquisition for a driver (for instance).
>>>
>>> OK.
>>>
>>>> runtime_error: the component still provides a somewhat limited
>>>> functionality. The actual semantic of this is very application
>>>> dependent. In practice, we use orogen to specialize it into sub-states.
>>>
>>> Please define which triggers cause a transition to this state and from
>>> this state away + which hooks are called.
>>
>> Here's the thing: I'm not using scripts, and I completely do not intend
>> to use them. I do see their possible usefulness, it is just that I did
>> not (yet) encounter a situation where they were needed.
>>
>> So: triggers
>> * any application-defined situation which means that the component
>> provides a limited functionality.
>> * hooks: errorHook(), in the same situations than updateHook() (i.e.
>> activity triggers)
>> * getting out of there: component specific and component-decided
>>
>>>> stop: the component stops functioning, either because it reached its
>>>> stated goal (case for a planner), or because it has been requested
>>>
>>> OK.
>>>
>>>> fatal: the component cannot provide its stated functionality anymore,
>>>> and therefore stopped.
>>>
>>> so stopHook() is called automatically ? As above, define the triggers +
>>> possible next states ?
>>
>> Triggers: internal component diagnostics which detected a situation
>> representing a loss of functionality.
>> Possible next states: STOPPED or PRE_OPERATIONAL (depending on whether
>> the component needs a configure step).
>>
>>>> It should try to clean up as much as possible so
>>>> that a configure()/start() cycle has a change to recover from the
>>>> problem. In the same way than for runtime error, orogen specializes it
>>>> into substates.
>>>>
>>>> This state machine allowed us to keep the updateHook() simple (since it
>>>> does not have to deal with initialization/recovery/...), and has most of
>>>> the information needed for supervision.
>>>>
>>>>> Maybe we should change it then to these semantics:
>>>>>
>>>>> Fatal error: can be entered from any state, triggered by in RTT code or
>>>>> error recovery code. Causes stopHook()->cleanupHook() in transition (if
>>>>> necessary). Only step left is delete component.
>>>>>
>>>>> Runtime error: triggered by exceptions in updateHook() or by user in
>>>>> updateHook()/script.
>>>>
>>>> There is a funny thing: on the one hand you say "raised exceptions
>>>> should leave the application in a well-defined state" and "if an
>>>> exception is raised in errorHook()" we can't recover ever, we actually
>>>> need to destroy everything". This seems contradictory to me.
>>>
>>> But makes sense to me: if your error recovery throws, it went really bad,
>>> it means your last resort to pull things right did not succeed. There
>>> *is* no way out, this *is* fatal, literally as in 'terminal'. No
>>> transition succeeds.
>>
>> Here is my proposal:
>> * RUNTIME_ERROR remains an application state. The component announces
>> that it has limited functionality due to something non-nominal happening.
>> * unexpected exceptions in running states (RUNNING and RUNTIME_ERROR)
>> transition to fatal. This calls a fatalHook() which -- by default --
>> calls stopHook() and cleanupHook(). The component can also transition to
>> fatal to announce that something non-nominal happened that makes the
>> component's service not available.
>> * if fatalHook() and/or stopHook() raise, then we go into the
>> "unrecoverable fault" (we can't even go into FATAL ...)
>>
>>>> As to the interpretation of "fatal": it depends on the point of view.
>>>> From the point of view of the supervision, the "fatal" I described
>>>> above *is* fatal as the component does not provide the service it should
>>>> provide, and that happened because of something non-nominal.
>>>
>>> There must be a posibility to resolve these constraints we both have:
>>>
>>> 1. Define a transition/state when an exception is thrown in updateHook().
>>> The last thing we want to do is call it again, the component is possibly
>>> in a 'messy' state and it may throw 'ad infinitum'.
>>>
>>> 2. Define a transition/state when error recovery from point 1 failed as
>>> well.
>>>
>>> 3. Define a transition/state when the RTT can no longer execute a
>>> component. This might be the same as #2.
>>>
>>> From the RTT point of view, these are the things I *need* to define,
>>> without even caring for application-level supervision. Supervision is a
>>> fundamental part of every application (ie handle faults the component can
>>> not solve by itself) so I am not against in adding support for that in
>>> the TaskContext, on the other hand, you/me are biased and I wonder if
>>> it's not better to stick to the minimal.
>>
>> Yes, but in my opinion a basic application state machine *is* part of
>> the minimal.
>>
>>> A possible clean solution I see here is to define a supervision interface
>>> that defines these extra states that your supervision software requires.
>>> So your component inherits TaskContext + SuperviseInterface. where the
>>> latter sets up a 'supervise' provided interface with the methods/states
>>> you require.
>>>
>>> The supervisor component/user can than query each component if it has
>>> this interface and proceed from there if it has.
>>>
>>> This 'extendability' is actually one of the major issues I wanted to
>>> solve in 2.x. The component itself has only a minimal life cycle
>>> interface and the rest is set into 'plugins'/'interfaces'.
>>
>> While I see why you want that (you are the RTT-as-a-universal-framework
>> guy), I do see a lot of practical issues. The biggest issue being that
>> you will start to completely fragment what components can run on what
>> tools and make the whole "RTT ecosystem" (for lack of a better name) a
>> huge mess.
>>
>> We're having that discussion *because* I want to avoid this. I could
>> live on with Roby and oroGen: they already provide all the tools I need
>> to "work around" the state machine you define to get what I want. We're
>> having that discussion because I think it would be a very bad idea.
>>
>> So, yes, being able to extend is important. Now, I feel that the RTT
>> *must* provide a basic standard, supervise-able, interface to ALL RTT
>> components. And -- more importantly -- should make the component
>> developer aware that this interface is important.
>>