[SR-Users] Memory leak in 3.3.4

David Cunningham dcunningham at voisonics.com
Wed Jul 24 04:24:31 CEST 2013


Hello,

Thank you very much for the email. In reply:

1. The system ran out of memory. Linux's oom-killer killed Kamailio.

2. You're right, DEBUG_MEMORY is a local configuration setting. If defined
it sets memdbg to -2, and memlog to -2. The debug setting is -1.

3. We'll try setting mem_summary=12, thanks.

4. We'll try setting asynchronous syslog, thanks.

5.  Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?

6. We'll try setting mem_join=1, thanks.



On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda at gmail.com> wrote:

> Hello,
>
> first, to clarify, is the system memory or kamailio's pkg/shm memory
> running out? If the operating system runs out of memory, then should be a
> leak in a library, because kamailio modules uses only from a pre-allocated
> chunk, not going over it.
>
>
> On 7/23/13 7:33 AM, David Cunningham wrote:
>
>> Hello,
>>
>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using more
>> and more memory. Over a couple of weeks it will run out of system memory.
>>
>> We tried to enable memory debugging doing the following, but it resulted
>> in Kamailio not responding to any SIP packets. Would anyone have advice
>> please on how to debug the situation?
>>
>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>> 2. In kamailio.cfg add the line:
>> #!define DEBUG_MEMORY 1
>>
> do you set something special in config when DEBUG_MEMORY is 1? It is not
> by default there, so I assume you added some rules based on this
> pre-processor directive.
>
> For memory troubleshooting, set memlog to a value lower than debug
> parameter in config file and try with mem_summary=12 for a more compact
> output. See more about these parameters in the wiki:
>
> - http://www.kamailio.org/wiki/**cookbooks/3.3.x/core#memlog<http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog>
>
> Run kamailio for a while in normal conditions, then restart it to get the
> memory usage summaries. There should be indication if there is some leak,
> by seeing memory chunks allocated many times from a function used at
> runtime. You can send the memory summary for a process here, we can look at
> it.
>
>
>
>> While this was running and Kamailio didn't respond to packets, it logged
>> lots of lines like this:
>>
>
> Do you have syslog to be configured in asynchronous mode? See the notes
> from:
>
> - http://www.kamailio.org/wiki/**tutorials/3.2.x/syslog<http://www.kamailio.org/wiki/tutorials/3.2.x/syslog>
>
> The memdbg is less than debug value, that means printing few log messages
> for each memory operation. You can make memdbg higher and rely on memlog
> for memory summaries, otherwise will be lot of log messages related to
> memory.
>
>
>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
>> (size=128) on 1 -th hit
>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
>> (size=128) on 1 -th hit
>>
> addstr() is a function used only for parsing configuration file, as long
> as you can still see them, the configuration file parsing was not finish.
> addstr() is not a source of leaks because it is not used at runtime.
>
> If you have large config file, then you can get close to the limits of the
> private memory, which is set to 4MB. You can increase its value using -M
> parameter (e.g., start kamailio with -M 8 to set it to use 8MB of memory).
>
> Over the time, the private memory can get used due to fragmentation, you
> can set the mem_join parameter in config file to avoid it (works when
> compiled with MEMDBG=1).
>
> To monitor usage of internal pkg memory, then you can use sercmd with
> pkg.stats command:
>
> http://kamailio.org/docs/**modules/3.3.x/modules_k/kex.**html#idp16972640<http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640>
>
> Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
>
> When you see significant increase of the memory usage, then you can
> restart to get the summaries.
>
> You should run these commands after start, just to see the initial usage
> of memory.
>
> Cheers,
> Daniel
>
> --
> Daniel-Constantin Mierla - http://www.asipto.com
> http://twitter.com/#!/miconda - http://www.linkedin.com/in/**miconda<http://www.linkedin.com/in/miconda>
>
>
> ______________________________**_________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/**cgi-bin/mailman/listinfo/sr-**users<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>



-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20130724/238cb4cb/attachment-0001.html>


More information about the sr-users mailing list