[Please CC the list]

Begin forwarded message:

> From: Milan Vukov <milan [dot] vukov [..] ...>
> Subject: Re: [Orocos-users] Reporter limitations? Benchmarks? Alternatives?
> Date: June 8, 2012 4:18:13 AM EDT
> To: Stephen Roderick <kiwi [dot] net [..] ...>
> Reply-To: milan [dot] vukov [..] ...
>
> Hello,
>
> On Wed, Jun 6, 2012 at 1:31 PM, Stephen Roderick <kiwi [dot] net [..] ...> wrote:
>> On Jun 6, 2012, at 07:21 , Milan Vukov wrote:
>>
>>> Hello,
>>>
>>> On Wed, Jun 6, 2012 at 12:15 AM, S Roderick <kiwi [dot] net [..] ...> wrote:
>>>> On Jun 5, 2012, at 17:58 , Milan Vukov wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am interested to know if someone has performed benchmarks and/or
>>>>> some test on reporter component? I heard that the OCL reporter is not
>>>>> "hard" real-time, and that it can influence the real-time data flow.
>>>>> So, how much of this is true / not true? What about logging "large"
>>>>> amounts of data in real-time, cca 500 doubles @ 50-100 ms, or possibly
>>>>> more?
>>>>
>>>> I'm fairly certain that the existing reporter is _not_ hard realtime.
>>>>
>>>> We use the OCL real-time logging, with custom reporters that produce text strings for logging. It works very well, though it takes a little more boiler-plate code to setup than the existing OCL reporting. Our implementation is hard real-time, and we use it to log ~3kb per cycle at 500 Hz (~1.5 MB/s). With careful tuning of buffer sizes this works very well.
>>>>
>>>> We're also doing this on top of logging 400kb images at 20 Hz ... and this is all with off the shelf computers with HDDs (not SSDs).
>>>
>>> Before all, thank you very much for the quick reply! :) Can you give
>>> some indications (hints?) how can I setup real-time logging using OCL
>>> real-time logging?
>>
>> http://www.orocos.org/wiki/rtt/examples-and-tutorials/using-real-time-lo...
>>
>>>>>
>>>>> On the other hand, is there around any reporter / logging component
>>>>> that does not influence the real-time data flow? Or maybe a logger
>>>>> that really dumps bytes in a binary file (not ASCII).
>>>>
>>>> Binary vs ASCII isn't the primary issue. That mostly just affects disk bandwidth. The issue is sampling in real-time and getting those samples out to a non-real-time item that then logs the data.
>>>
>>> I agree that the issue is how to extract samples from real-time flow.
>>> After that, "one" can put extracted data into one e.g. circular FIFO,
>>> and later some component (running in a separate, non-real-time thread)
>>> should get data from FIFO and save it to disk -- I assume we have
>>> enough of RAM memory for caching. If one is able to allocate a large
>>> buffer in RAM (>= 100 MB?), I think the disk bandwidth should not be
>>> an issue. Moreover, I really do not care how long it can take for data
>>> to be saved -- 0.1 s or 2s. I just need it saved at some point of
>>> time.
>>
>> Ah yes, you do care how long it takes to save the data. This becomes a factor in how big your buffers/FIFOs need to be. The longer it takes to flush to disk, the bigger your buffers need to be.
>
> I am completely aware of this. If we works on x86 desktop computer(s)
> with enough RAM, this is not maybe the bottleneck.
>
>>
>>> From my point of view, saving the data in ASCII format is...
>>> introducing unnecessary overhead. I think of it as an eye-candy. :)
>>
>> Eye-candy, maybe. But it works well and is completely hard real-time.
>
> OK, agree :)
>
>>
>>> Reading this link:
>>>
>>> http://www.orocos.org/forum/orocos/orocos-users/realtime-logging
>>>
>>> I found about the Netcdf reporter. Are there any instructions how to
>>> use it in OROCOS?
>>
>> Not that I know of.
>>
>>> Related to "non-real-time" reporter in OROCOS -- is there a way to
>>> change the number of samples in the buffer? If I remember correctly,
>>> by default it is something like 10(?) samples and I have not seen an
>>> option to change this...
>>
>> The problem with Orocos buffers is that their size is set upon construction. There were bugs in v1 (at least) that prevented changing the size of an already connected buffer (or something similar). We end up compiling in the required size. :-(
>>
>>>>> Thanx :) BTW, I have not experienced any problems during my work with
>>>>> OROCOS and reporter component so far, but my current application would
>>>>> possibly require dumping "larger" amounts of data -- so I asked all
>>>>> the questions above :)
>>>>
>>>> Have you analysed any diagnostics to do with real-time to prove you aren't having problems? Do you show any major faults during operations?
>>>
>>> Hm... not exactly. To be honest, I do not know how can I run some
>>> "true" diagnostics. We put some "time-stamping" in our components, so
>>> that we can analyze the results off-line. Do you have any other
>>> suggestion?
>>
>> Then with all respect, you don't _know_ that you're actually running in real time. Put at least some min/average/max across both the work time of your components, _and_ across the cycle time. We do that for all components, and we always log this data at the end of each session. We also note the number of major and minor faults, in the process and within each thread of interest, and log those. Major faults are bad. Minor faults you can (sometimes) live with.
>
> I am aware of this also, and via this e-mails I wanted to find out how
> to improve logging in our application. My general conclusion is that
> reporter is not so efficient, so people build custom solutions -- so,
> back to the lab :)

I wouldn't say that the reporters are ineffecient, no. I think that they are very good for their intended purpose. They simply don't cope with hard real time AFAIK. And because of that, people build their own solutions. The reason we use text logging is that it builds on top of the real-time logging (OCL/logging) that we already use throughout our application. A binary format has plenty of advantages (cf netCDF), however, it is really nice having just one system that does logging and reporting.

Having said that, given the sheer volume of what we log now, we are looking at possibly using netCDF on the backend, for disk and socket access. We will continue to use the OCL/logging for all our application tracing and diagnostics.

HTH
S