[SR-Users] Memory Leak on DB Errors?

[Daniel-Constantin Mierla]

Hi Klaus,

over the weekend I looked a bit at snmpstats module. These allocated 
chunks are for exporting location records. Are you pulling them over 
snmp? At the first sight, there should be a free of the memory when the 
records are consumed.

The fact is that they are not pulled from usrloc module at the time of 
the request over snmp, but cached in snmp when registration happens. 
Practically, it is a partial clone of usrloc commands, which is not the 
best solution IMO, but I am not the developer. For the moment, I added a 
parameter to control whether the location records should be cached by 
snmpstats module or not (if not, they cannot be exported), to fix this 
issue. If you actually pull the location records over snmp, let me know.

I could not test, but if you can give a try (maybe you have a testbed 
for 3.2 with snmpstats) and see if the memory is steady with 
export_registrar set to 0 (which is default):

http://kamailio.org/docs/modules/devel/modules_k/snmpstats.html#id2539456

Cheers,
Daniel

On 10/6/11 6:03 PM, Daniel-Constantin Mierla wrote:
> Hello,
>
> seem the leak is in snmpstats, I see lot of allocations like:
>
> ALERT: qm_status:    37599. N  address=0xf30cdf74 frag=0xf30cdf5c 
> size=20 used=1
> ALERT: qm_status:           alloc'd from snmpstats: 
> interprocess_buffer.c: handleContactCallbacks(143)
> ALERT: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, 
> abcdefed
> ALERT: qm_status:    37600. N  address=0xf30cdfb8 frag=0xf30cdfa0 
> size=16 used=1
> ALERT: qm_status:           alloc'd from snmpstats: utilities.c: 
> convertStrToCharString(62)
> ALERT: qm_status:          start check=f0f0f0f0, end check= c0c0c0c0, 
> abcdefed
>
> There are some from usrloc, but very likely they are ok, because they 
> are persistent in shm for long time, unless snmpstats asks for some 
> clones of the structures from usrloc and forgets to free them (i see 
> one allocation is from handleContactCallbacks).
>
> No time to look in the sources, but this is a lead to follow if you 
> want to investigate further.
>
> In general, fr a memleak you have to look at allocated chunks that are 
> done from same place in the code and there are many of them. The 
> decide whether it is something that should be there for long time 
> (like usrloc records) or they should be freed quicker comparing with 
> the number of allocations.
>
> Pkg log looks very clean, allocations only from startup time (maybe is 
> the main process).
>
> Cheers,
> Daniel
>
> On 10/6/11 5:31 PM, Klaus Darilion wrote:
>> Indeed, DBG_QM_MALLOC is defined. So I have set memlog=1 and dumped 
>> mem_info with:
>> sercmd cfg.set_now_int core mem_dump_pkg 13286
>> sercmd cfg.set_now_int core mem_dump_shm 13286
>>
>> The dumps were done after ~1h uptime. I can not offload the traffic 
>> and wait until transactions are freed, thus the logs are quite huge 
>> (~15MByte)
>>
>> http://pernau.at/kd/memlog.zip
>>
>> I have no idea for what I should look for - any hints how to analyze 
>> the mem_dump?
>>
>> Thanks
>> Klaus
>>
>>
>> On 06.10.2011 13:07, Daniel-Constantin Mierla wrote:
>>> Hello,
>>>
>>> On 10/5/11 11:18 AM, Klaus Darilion wrote:
>>>>
>>>>
>>>> On 04.10.2011 14:03, Daniel-Constantin Mierla wrote:
>>>>> Hello,
>>>>>
>>>>> On 10/4/11 12:27 PM, Klaus Darilion wrote:
>>>>>> Meanwhile the server was restarted and the DB problems were 
>>>>>> fixed. As
>>>>>> it is a production server I can not reproduce anymore.
>>>>>
>>>>> So, once it started it didn't recovered, continued always with that
>>>>> error? How much of shm did you configure?
>>>>>
>>>>> You can try to attach from time to time to one process (can be 
>>>>> even the
>>>>> main one to avoid blocking a sip worker) and walk through the shm
>>>>> allocated chunks, in order to see if there are some unexpected
>>>>> repetitions of allocation from same place in sources.
>>>>>
>>>>> I posted the gdb script for walking through pkg at some point, the
>>>>> difference will be to start from the head of shm list (i.e., starting
>>>>> with shm_block->first_frag instead of mem_block->first_frag):
>>>>>
>>>>> http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory#walking_through_pkg_with_gdb 
>>>>>
>>>>>
>>>>
>>>> Hi Daniel!
>>>>
>>>> After reading this wiki page I came to the conclusion that for further
>>>> debugging I have to recompile Kamailio (using DBG_QM_MALLOC memory
>>>> manager instead of F_MALLOC). With the default memory manager it is
>>>> not possible to debug the problem. Is it correct?
>>> in 3.1 malloc debug was left on (with the goal of catching buffer
>>> overflows quickly after several years of development of no using this
>>> flag in production), so unless you switched if off, you should get the
>>> reports. you can check in the output of kamailio -V
>>>
>>> Cheers,
>>> Daniel
>>>
>

-- 
Daniel-Constantin Mierla -- http://www.asipto.com
Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kat
http://linkedin.com/in/miconda -- http://twitter.com/miconda