[SR-Users] Memory leak in 3.3.4

[Daniel-Constantin Mierla]

Hello,

first, to clarify, is the system memory or kamailio's pkg/shm memory 
running out? If the operating system runs out of memory, then should be 
a leak in a library, because kamailio modules uses only from a 
pre-allocated chunk, not going over it.

On 7/23/13 7:33 AM, David Cunningham wrote:
> Hello,
>
> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using 
> more and more memory. Over a couple of weeks it will run out of system 
> memory.
>
> We tried to enable memory debugging doing the following, but it 
> resulted in Kamailio not responding to any SIP packets. Would anyone 
> have advice please on how to debug the situation?
>
> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
> 2. In kamailio.cfg add the line:
> #!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY is 1? It is not 
by default there, so I assume you added some rules based on this 
pre-processor directive.

For memory troubleshooting, set memlog to a value lower than debug 
parameter in config file and try with mem_summary=12 for a more compact 
output. See more about these parameters in the wiki:

- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog

Run kamailio for a while in normal conditions, then restart it to get 
the memory usage summaries. There should be indication if there is some 
leak, by seeing memory chunks allocated many times from a function used 
at runtime. You can send the memory summary for a process here, we can 
look at it.

>
> While this was running and Kamailio didn't respond to packets, it 
> logged lots of lines like this:

Do you have syslog to be configured in asynchronous mode? See the notes 
from:

- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

The memdbg is less than debug value, that means printing few log 
messages for each memory operation. You can make memdbg higher and rely 
on memlog for memory summaries, otherwise will be lot of log messages 
related to memory.
>
> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: 
> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: 
> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900 
> (size=128) on 1 -th hit
> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: 
> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: 
> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0 
> (size=128) on 1 -th hit
addstr() is a function used only for parsing configuration file, as long 
as you can still see them, the configuration file parsing was not 
finish. addstr() is not a source of leaks because it is not used at runtime.

If you have large config file, then you can get close to the limits of 
the private memory, which is set to 4MB. You can increase its value 
using -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB 
of memory).

Over the time, the private memory can get used due to fragmentation, you 
can set the mem_join parameter in config file to avoid it (works when 
compiled with MEMDBG=1).

To monitor usage of internal pkg memory, then you can use sercmd with 
pkg.stats command:

http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640

Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'

When you see significant increase of the memory usage, then you can 
restart to get the summaries.

You should run these commands after start, just to see the initial usage 
of memory.

Cheers,
Daniel

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda