Hello,
we currently have a vision application that mostly use the OpenCV library.
We use a proprietary frame grabber, but the images are stored in IplImage
structs. We would like to port our application to Orocos. We will have many
components that need access to the image data. The last thing we want to do
is to copy the entire image on each port of the components. Based on the
current IplImage port implementation (http://www.orocos.org/node/1105), does
the raw image is copied on each port, or only the imageData pointer address
is copied?
Thank you!
Philippe Hamelin
Newbie question about OpenCV IplImage in Orocos
On Fri, 1 May 2009, Philippe Hamelin wrote:
> we currently have a vision application that mostly use the OpenCV library.
> We use a proprietary frame grabber, but the images are stored in IplImage
> structs. We would like to port our application to Orocos. We will have many
> components that need access to the image data. The last thing we want to do
> is to copy the entire image on each port of the components. Based on the
> current IplImage port implementation (http://www.orocos.org/node/1105), does
> the raw image is copied on each port, or only the imageData pointer address
> is copied?
>
I hope it is _not_ copying the pointer... Because this would lead
to data inconsistencies right away: you have no guarantee that the pointer
you received still points to a valid image, or even to one that another
client is not working on at this moment...
In my opinion, there are two correct architectures:
- to copy the image to each DataPort that wants it (in a LockFree manner
preferably), so that you can be sure that no data inconsistency occurs.
- to provide an ImageProcessing component, that offers the services of the
image processing that the other components would like to do.
Herman
Newbie question about OpenCV IplImage in Orocos
This is quite tricky, indeed ...
If you pass the image pointer, you loose ALL the advantages of
thread-safety or Orocos.
Copying the whole image is a HUGE overhead for complex vision systems.
The solution is not easy at all, actually.
You can pass the pointer __ONLY__ if all your clients are read-only.
If any of this clients needs to modify the original image they need to
do a copy, because otherwise they are corrupting the original one.
Anyway, passing the pointer would work only in single-process
deployment. if you deploy the components with CORBA, you need a copy
of the whole image in any component!
What we have been doing in our team (where "1 component = 1 process"
is the rule) is to pass the pointer of the image with the shared
memory mechanism. Unfortunately, Orocos doesn't support such
mechanism.
> - to provide an ImageProcessing component, that offers the services of the
> image processing that the other components would like to do.
>
How this ImageProcessing component would be? I imagine that it
provides methods only... in that case, why aren't we just linking the
vision library?
Davide
Newbie question about OpenCV IplImage in Orocos
On Fri, 1 May 2009, Davide Faconti wrote:
> This is quite tricky, indeed ...
> If you pass the image pointer, you loose ALL the advantages of
> thread-safety or Orocos.
Yes...
> Copying the whole image is a HUGE overhead for complex vision systems.
That's not true in practice on many hardwares! Especially not compared to
the computational effort that even the simplest vision processing tasks
require... And especially if you use the LockFree support of RTT...
> The solution is not easy at all, actually.
> You can pass the pointer __ONLY__ if all your clients are read-only.
> If any of this clients needs to modify the original image they need to
> do a copy, because otherwise they are corrupting the original one.
Don't forget the _source_ of the image: that is _never_ "read only", so it
should use a different image location for any new image anyway.
> Anyway, passing the pointer would work only in single-process
> deployment. if you deploy the components with CORBA, you need a copy
> of the whole image in any component!
CORBA, or any other inter-process communication middleware, for that
matter.
> What we have been doing in our team (where "1 component = 1 process"
> is the rule) is to pass the pointer of the image with the shared
> memory mechanism. Unfortunately, Orocos doesn't support such
> mechanism.
Of course not: it's _by definition_ outside of the scope of
_component_-based development... But you can do it of course, using the
bare OS primitives.
>> - to provide an ImageProcessing component, that offers the services of the
>> image processing that the other components would like to do.
>
> How this ImageProcessing component would be? I imagine that it
> provides methods only... in that case, why aren't we just linking the
> vision library?
>
Indeed...
Herman
Newbie question about OpenCV IplImage in Orocos
.
>
>> Copying the whole image is a HUGE overhead for complex vision systems.
>
> That's not true in practice on many hardwares! Especially not compared to
> the computational effort that even the simplest vision processing tasks
> require... And especially if you use the LockFree support of RTT...
>
Ok, I have exaggerated with the word "huge", but once i have
programmed a vision algorithm on a XScale with 256 Mb of RAM, and I
have still nightmares about it :)
On desktop computer you can basically do what you want: you have
processing power and RAM. problems come in embedded applications.
>
> Don't forget the _source_ of the image: that is _never_ "read only", so it
> should use a different image location for any new image anyway.
>
Absolutely true. Usually, for this reason, the Component which read
the image should use double buffered images. It is a quite common
approach.
>> Anyway, passing the pointer would work only in single-process
>> deployment. if you deploy the components with CORBA, you need a copy
>> of the whole image in any component!
>
> CORBA, or any other inter-process communication middleware, for that
> matter.
Of course. I said "CORBA" only because it is the middleware of Orocos.
Newbie question about OpenCV IplImage in Orocos
On Fri, 1 May 2009, Davide Faconti wrote:
[...]
>>> Anyway, passing the pointer would work only in single-process
>>> deployment. if you deploy the components with CORBA, you need a copy
>>> of the whole image in any component!
>>
>> CORBA, or any other inter-process communication middleware, for that
>> matter.
>
> Of course. I said "CORBA" only because it is the middleware of Orocos.
>
Please, don't say that! :-)
CORBA is the only one (more or less) supported, but we definitely want to
keep Orocos independent of lock-ins to any particular middleware!
Herman
Newbie question about OpenCV IplImage in Orocos
2009/5/1 Herman Bruyninckx <Herman [dot] Bruyninckx [..] ...>
> On Fri, 1 May 2009, Philippe Hamelin wrote:
>
> > we currently have a vision application that mostly use the OpenCV
> library.
> > We use a proprietary frame grabber, but the images are stored in IplImage
> > structs. We would like to port our application to Orocos. We will have
> many
> > components that need access to the image data. The last thing we want to
> do
> > is to copy the entire image on each port of the components. Based on the
> > current IplImage port implementation (http://www.orocos.org/node/1105),
> does
> > the raw image is copied on each port, or only the imageData pointer
> address
> > is copied?
> >
> I hope it is _not_ copying the pointer... Because this would lead
> to data inconsistencies right away: you have no guarantee that the pointer
> you received still points to a valid image, or even to one that another
> client is not working on at this moment...
>
> In my opinion, there are two correct architectures:
> - to copy the image to each DataPort that wants it (in a LockFree manner
> preferably), so that you can be sure that no data inconsistency occurs.
> - to provide an ImageProcessing component, that offers the services of the
> image processing that the other components would like to do.
>
Copying large images on multiple ports will lead to a large overhead in
memory and CPU utilization, so I think this is not an alternative for an
optimized vision software. I don't think that passing the image pointer is a
bad idea when using SlaveActivity to trigger the different image processing
steps. A _master_ process could be in charge of triggering the image
grabbing and deleting when all processes have been iterated. On the other
hand, I will consider the second proposed option.
Philippe Hamelin
Newbie question about OpenCV IplImage in Orocos
On Fri, 1 May 2009, Philippe Hamelin wrote:
> 2009/5/1 Herman Bruyninckx <Herman [dot] Bruyninckx [..] ...>
> On Fri, 1 May 2009, Philippe Hamelin wrote:
>
> > we currently have a vision application that mostly use the OpenCV
> library.
> > We use a proprietary frame grabber, but the images are stored in
> IplImage
> > structs. We would like to port our application to Orocos. We will
> have many
> > components that need access to the image data. The last thing we
> want to do
> > is to copy the entire image on each port of the components. Based on
> the
> > current IplImage port implementation
> (http://www.orocos.org/node/1105), does
> > the raw image is copied on each port, or only the imageData pointer
> address
> > is copied?
> >
> I hope it is _not_ copying the pointer... Because this would lead
> to data inconsistencies right away: you have no guarantee that the
> pointer
> you received still points to a valid image, or even to one that
> another
> client is not working on at this moment...
>
> In my opinion, there are two correct architectures:
> - to copy the image to each DataPort that wants it (in a LockFree
> manner
> preferably), so that you can be sure that no data inconsistency
> occurs.
> - to provide an ImageProcessing component, that offers the services of
> the
> image processing that the other components would like to do.
>
> Copying large images on multiple ports will lead to a large overhead in
> memory and CPU utilization, so I think this is not an alternative for an
> optimized vision software.
Well, it's up to you whether you want to optimize image processing on a
corrupted image... :-) Optimization in this case only makes sense when you
_know_ in advance what the scheduling of your different "image clients"
will be wrt the "image server"...
> I don't think that passing the image pointer is a
> bad idea when using SlaveActivity to trigger the different image processing
> steps. A _master_ process could be in charge of triggering the image
> grabbing and deleting when all processes have been iterated. On the other
> hand, I will consider the second proposed option.
Your suggestion is along the lines of what I say in my answer above, _if_
you know in advance that the SlaveActivities will be finished before the
next image comes in, and _if_ your master is doing a strict serialization
of the SlaveActivities. (And under these conditions, you might as well do
all image processing via serial method calls in the image server...).
Herman