Memory leak in LocalOperationCaller using rt_allocator

Submitted by phamelin on Tue, 2011-07-19 15:28

RTT-dev

Hello,

I think I found a memory leak when calling remote operation through
OperationCaller. It seems that the memory allocated by LocalOperationCaller
::cloneRT() is never deallocated, even if the destructor of the SendHandle
is called. Indeed, the allocate rt memory size is always growing. I attached
a source file to reproduce the problem.

Thank you,

Philippe

Attachment	Size
rt_mem_leak.cpp	3.45 KB

Memory leak in LocalOperationCaller using rt_allocator

Submitted by peter on Tue, 2011-07-19 23:28.

On Tue, Jul 19, 2011 at 5:24 PM, Philippe Hamelin
<philippe [dot] hamelin [..] ...> wrote:
> Hello,
> I think I found a memory leak when calling remote operation through
> OperationCaller. It seems that the memory allocated by LocalOperationCaller
> ::cloneRT() is never deallocated, even if the destructor of the SendHandle
> is called. Indeed, the allocate rt memory size is always growing. I attached
> a source file to reproduce the problem.

I don't have the time to look at this right now, but could you at
least scope the test case such that the TC is destructed before you
print out the statistics and free the rt_heep ?

Since you used asynchronous operations, it's possible that the
operation caller object still lives although the sendhandle is cleaned
up. It will be cleaned up if the TC no longer holds a reference to it.

Another possible cause is that the operations queue (see Queue) did
not call the destructor of the objects it holds yet.

Peter

Memory leak in LocalOperationCaller using rt_allocator

Submitted by phamelin on Mon, 2011-07-25 13:16.

2011/7/19 Peter Soetens <peter [..] ...>

> On Tue, Jul 19, 2011 at 5:24 PM, Philippe Hamelin
> <philippe [dot] hamelin [..] ...> wrote:
> > Hello,
> > I think I found a memory leak when calling remote operation through
> > OperationCaller. It seems that the memory allocated by
> LocalOperationCaller
> > ::cloneRT() is never deallocated, even if the destructor of the
> SendHandle
> > is called. Indeed, the allocate rt memory size is always growing. I
> attached
> > a source file to reproduce the problem.
>
> I don't have the time to look at this right now, but could you at
> least scope the test case such that the TC is destructed before you
> print out the statistics and free the rt_heep ?
>
> Since you used asynchronous operations, it's possible that the
> operation caller object still lives although the sendhandle is cleaned
> up. It will be cleaned up if the TC no longer holds a reference to it.
>
> Another possible cause is that the operations queue (see Queue) did
> not call the destructor of the objects it holds yet.
>
>
>
I was able to reproduce the leak is a slightly different way. I scoped both
the TC and the TaskContextServer cleanups together and this lead to some
memory still allocated during the rt-mem destruction, even if the TC were
destroyed. I added a second test case in the attached archive to reproduce
the problem. In the second test case, I also directly used the operation
instead of using the require/provide mechanism.

Philippe

Memory leak in LocalOperationCaller using rt_allocator

Submitted by phamelin on Mon, 2011-07-25 13:08.

2011/7/19 Peter Soetens <peter [..] ...>

I am

Orogen default Corba setting

Submitted by Nico Hochgeschwender on Mon, 2011-07-25 15:20.

Hello all,

We tried to install the latest version of the Orocos toolchain
according to the instructions on the "Quick Start page". We disabled
the Corba transport.

After everything builds successfully, we created components with
orocreate-pkg. The resulting code fails to build, because it searches
for Corba bindings. We solved the problem by modifying orogen and
typegen scripts, which seem to have default settings for Corba
transport. See below. Maybe this is intended?

DEFAULT_TRANSPORTS = %w{corba typelib}
 
verbosity = 0
extended_states = true
component = Orocos::Generation::Project.new
 
transports = []
no_transports = false
 
parser = OptionParser.new do |opt|
     opt.banner = "Usage: orogen [options] <template file>"
 
     opt.on("--[no-]corba", "enables/disables corba support  
(deprecated, use --transports instead)") do |flag|
         component.enable_transports('corba')
     end

Nico & Azamat

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

Orogen default Corba setting

Submitted by peter on Mon, 2011-07-25 21:16.

On Mon, Jul 25, 2011 at 5:09 PM, Nico Hochgeschwender
<nico [dot] hochgeschwender [..] ...> wrote:
> Hello all,
>
> We tried to install the latest version of the Orocos toolchain
> according to the instructions on the "Quick Start page". We disabled
> the Corba transport.
>
> After everything builds successfully, we created components with
> orocreate-pkg. The resulting code fails to build, because it searches
> for Corba bindings. We solved the problem by modifying orogen and
> typegen scripts, which seem to have default settings for Corba
> transport. See below. Maybe this is intended?

Yes orogen assumes corba is enabled. You could modify the cmake files
in order to let orogen generate the corba code but only compile it if
RTT is compiled with corba too.

Did you say that you had a patch for this ? It would be very welcome.

Peter

Memory leak in LocalOperationCaller using rt_allocator

Submitted by phamelin on Wed, 2011-07-20 12:52.

2011/7/19 Peter Soetens <peter [..] ...>

$ ./rtt-corba-mem-leak-gnulinux
Real-time memory: 3616 bytes free of 10000 allocated.
[...]
Currently used rt-mem bytes = 0
Invoke count = 0, currently used rt-mem bytes = 208
[...]
Invoke count = 9, currently used rt-mem bytes = 2080
Used rt-mem bytes after SendHandle cleanup = 2080
Used rt-mem bytes after TC cleanup = 2080
Used rt-mem bytes after Corba server cleanup = 0
TLSF bytes allocated=10000 overhead=6384 max-used=8464 currently-used=6384
still-allocated=0

I attached the updated test case with a working CMakeLists.

As we can see, In this test case the used memory pool grows of 208 bytes per
send(). With the default rt-mem pool size of 20k I am usually getting a
std::bad_alloc exception after 20-40 send().

Philippe

Memory leak in LocalOperationCaller using rt_allocator

Submitted by snrkiwi on Wed, 2011-07-27 17:44.

On Jul 20, 2011, at 08:51 , Philippe Hamelin wrote:

> 2011/7/19 Peter Soetens <peter [..] ...>
> On Tue, Jul 19, 2011 at 5:24 PM, Philippe Hamelin
> <philippe [dot] hamelin [..] ...> wrote:
> > Hello,
> > I think I found a memory leak when calling remote operation through
> > OperationCaller. It seems that the memory allocated by LocalOperationCaller
> > ::cloneRT() is never deallocated, even if the destructor of the SendHandle
> > is called. Indeed, the allocate rt memory size is always growing. I attached
> > a source file to reproduce the problem.
>
> I don't have the time to look at this right now, but could you at
> least scope the test case such that the TC is destructed before you
> print out the statistics and free the rt_heep ?
>
>
> I added the scoping and it seems that the leaks come from the Corba server. Indeed, here is the stripped output:
>
> $ ./rtt-corba-mem-leak-gnulinux
> Real-time memory: 3616 bytes free of 10000 allocated.
> [...]
> Currently used rt-mem bytes = 0
> Invoke count = 0, currently used rt-mem bytes = 208
> [...]
> Invoke count = 9, currently used rt-mem bytes = 2080
> Used rt-mem bytes after SendHandle cleanup = 2080
> Used rt-mem bytes after TC cleanup = 2080
> Used rt-mem bytes after Corba server cleanup = 0
> TLSF bytes allocated=10000 overhead=6384 max-used=8464 currently-used=6384 still-allocated=0
>
> I attached the updated test case with a working CMakeLists.
>
> As we can see, In this test case the used memory pool grows of 208 bytes per send(). With the default rt-mem pool size of 20k I am usually getting a std::bad_alloc exception after 20-40 send().

Looks like you're using a 10k RT memory pool above ...?

FYI we've found that 20k is nowhere near enough for most situations. Especially as a third of it dissappears in overhead on 64-bit systems.
S

Memory leak in LocalOperationCaller using rt_allocator

Submitted by phamelin on Wed, 2011-07-27 17:48.

Le mercredi 27 juillet 2011, S Roderick <kiwi [dot] net [..] ...> a écrit :
> On Jul 20, 2011, at 08:51 , Philippe Hamelin wrote:
>
> 2011/7/19 Peter Soetens <peter [..] ...>
>>
>> On Tue, Jul 19, 2011 at 5:24 PM, Philippe Hamelin
>> <philippe [dot] hamelin [..] ...> wrote:
>> > Hello,
>> > I think I found a memory leak when calling remote operation through
>> > OperationCaller. It seems that the memory allocated by
LocalOperationCaller
>> > ::cloneRT() is never deallocated, even if the destructor of the
SendHandle
>> > is called. Indeed, the allocate rt memory size is always growing. I
attached
>> > a source file to reproduce the problem.
>>
>> I don't have the time to look at this right now, but could you at
>> least scope the test case such that the TC is destructed before you
>> print out the statistics and free the rt_heep ?
>>
>
> I added the scoping and it seems that the leaks come from the Corba
server. Indeed, here is the stripped output:
> $ ./rtt-corba-mem-leak-gnulinux
> Real-time memory: 3616 bytes free of 10000 allocated.
> [...]
> Currently used rt-mem bytes = 0
> Invoke count = 0, currently used rt-mem bytes = 208
> [...]
> Invoke count = 9, currently used rt-mem bytes = 2080
> Used rt-mem bytes after SendHandle cleanup = 2080
> Used rt-mem bytes after TC cleanup = 2080
> Used rt-mem bytes after Corba server cleanup = 0
> TLSF bytes allocated=10000 overhead=6384 max-used=8464 currently-used=6384
still-allocated=0
> I attached the updated test case with a working CMakeLists.
> As we can see, In this test case the used memory pool grows of 208 bytes
per send(). With the default rt-mem pool size of 20k I am usually getting a
std::bad_alloc exception after 20-40 send().
>
> Looks like you're using a 10k RT memory pool above ...?
> FYI we've found that 20k is nowhere near enough for most situations.
Especially as a third of it dissappears in overhead on 64-bit systems.

I used 10k only to shorten the test. Increasing the rt memory pool size just
increase the time before getting a std::bad_alloc exception when using
remote operations.

Philippe