[SR-Users] Memory leak in 3.3.4

Daniel-Constantin Mierla miconda at gmail.com
Wed Jul 24 20:55:06 CEST 2013


Hello,

I would say that perl_exec() is the one with the highest chances to be 
the reason for the leak. Next is line would be db_mysql module, if liked 
with some custom mysql client library, although even in this case will 
be unlikely.

Back to perl, the module itself does not call any malloc, so it might be 
the embedding Perl API that is not used properly in the module.

Can you use some testbed, set children=1 and run kamailio under 
valgrind, then do some calls and see if it detects the source of the leak?

I'm not using the perl module, I will try to check it whenever I get a 
chance in the next days. What version of perl do you have installed?

Cheers,
Daniel

On 7/24/13 10:31 AM, David Cunningham wrote:
> Hello,
>
> We don't do any kamctl commands at all. We do have various modules 
> loaded, as follows. The primary functions we use Kamailio for are 
> phone registrations through usrloc, and routing calls to Asterisk 
> through logic contained in Perl via perl_exec().
> Thanks for all your advice so far!
>
> loadmodule "tm.so"
> loadmodule "tmx.so"
> loadmodule "usrloc.so"
> loadmodule "auth.so"
> loadmodule "auth_db.so"
> loadmodule "ctl.so"
> loadmodule "db_mysql.so"
> loadmodule "kex.so"
> loadmodule "maxfwd.so"
> loadmodule "mi_fifo.so"
> loadmodule "mi_rpc.so"
> loadmodule "nathelper.so"
> loadmodule "perl.so"
> loadmodule "pv.so"
> loadmodule "registrar.so"
> loadmodule "rr.so"
> loadmodule "sanity.so"
> loadmodule "siputils.so"
> loadmodule "sl.so"
> loadmodule "textops.so"
> loadmodule "xlog.so"
>
>
> On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda at gmail.com 
> <mailto:miconda at gmail.com>> wrote:
>
>     Hello,
>
>
>     On 7/24/13 4:24 AM, David Cunningham wrote:
>>     Hello,
>>
>>     Thank you very much for the email. In reply:
>>
>>     1. The system ran out of memory. Linux's oom-killer killed Kamailio.
>     then all the instructions I gave are useless, they are for
>     debugging kamailio's internal memory manager, which handles pkg
>     and shm mallocs.
>
>     The chances to be from kamailio itself are very low now. Do you do
>     lot of mi commands (e.g., kamctl ...)? The mi api uses system
>     malloc, but the rest of code should use internal memory manager
>     which does not go beyond the limits set with -m and -M, thus not
>     causing an OS memory exhaustion.
>
>     Can you list what modules are you loading? At some point it was a
>     leak in libssl, in case you use tls a lot. But could be another
>     external library...
>
>     Cheers,
>     Daniel
>
>
>>
>>     2. You're right, DEBUG_MEMORY is a local configuration setting.
>>     If defined it sets memdbg to -2, and memlog to -2. The debug
>>     setting is -1.
>>
>>     3. We'll try setting mem_summary=12, thanks.
>>
>>     4. We'll try setting asynchronous syslog, thanks.
>>
>>     5.  Our configuration totals 338 lines, or approx 8.5kb. Is that
>>     a lot?
>>
>>     6. We'll try setting mem_join=1, thanks.
>>
>>
>>
>>     On 23 July 2013 16:53, Daniel-Constantin Mierla
>>     <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>>         Hello,
>>
>>         first, to clarify, is the system memory or kamailio's pkg/shm
>>         memory running out? If the operating system runs out of
>>         memory, then should be a leak in a library, because kamailio
>>         modules uses only from a pre-allocated chunk, not going over it.
>>
>>
>>         On 7/23/13 7:33 AM, David Cunningham wrote:
>>
>>             Hello,
>>
>>             We're running a Kamailio 3.3.4 system, and Kamailio is
>>             slowly using more and more memory. Over a couple of weeks
>>             it will run out of system memory.
>>
>>             We tried to enable memory debugging doing the following,
>>             but it resulted in Kamailio not responding to any SIP
>>             packets. Would anyone have advice please on how to debug
>>             the situation?
>>
>>             1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>>             2. In kamailio.cfg add the line:
>>             #!define DEBUG_MEMORY 1
>>
>>         do you set something special in config when DEBUG_MEMORY is
>>         1? It is not by default there, so I assume you added some
>>         rules based on this pre-processor directive.
>>
>>         For memory troubleshooting, set memlog to a value lower than
>>         debug parameter in config file and try with mem_summary=12
>>         for a more compact output. See more about these parameters in
>>         the wiki:
>>
>>         - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>
>>         Run kamailio for a while in normal conditions, then restart
>>         it to get the memory usage summaries. There should be
>>         indication if there is some leak, by seeing memory chunks
>>         allocated many times from a function used at runtime. You can
>>         send the memory summary for a process here, we can look at it.
>>
>>
>>
>>             While this was running and Kamailio didn't respond to
>>             packets, it logged lots of lines like this:
>>
>>
>>         Do you have syslog to be configured in asynchronous mode? See
>>         the notes from:
>>
>>         - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>
>>         The memdbg is less than debug value, that means printing few
>>         log messages for each memory operation. You can make memdbg
>>         higher and rely on memlog for memory summaries, otherwise
>>         will be lot of log messages related to memory.
>>
>>
>>             Jul 22 21:32:22 hostname kamailio: : <core>
>>             [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called
>>             from <core>: cfg.lex: addstr(1438)
>>             Jul 22 21:32:22 hostname kamailio: : <core>
>>             [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
>>             address 0x40048918 frag. 0x40048900 (size=128) on 1 -th hit
>>             Jul 22 21:32:22 hostname kamailio: : <core>
>>             [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called
>>             from <core>: cfg.lex: addstr(1438)
>>             Jul 22 21:32:22 hostname kamailio: : <core>
>>             [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
>>             address 0x400489c8 frag. 0x400489b0 (size=128) on 1 -th hit
>>
>>         addstr() is a function used only for parsing configuration
>>         file, as long as you can still see them, the configuration
>>         file parsing was not finish. addstr() is not a source of
>>         leaks because it is not used at runtime.
>>
>>         If you have large config file, then you can get close to the
>>         limits of the private memory, which is set to 4MB. You can
>>         increase its value using -M parameter (e.g., start kamailio
>>         with -M 8 to set it to use 8MB of memory).
>>
>>         Over the time, the private memory can get used due to
>>         fragmentation, you can set the mem_join parameter in config
>>         file to avoid it (works when compiled with MEMDBG=1).
>>
>>         To monitor usage of internal pkg memory, then you can use
>>         sercmd with pkg.stats command:
>>
>>         http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>
>>         Shared memory stats are printed by 'kamctl fifo
>>         get_statistics shmem:'
>>
>>         When you see significant increase of the memory usage, then
>>         you can restart to get the summaries.
>>
>>         You should run these commands after start, just to see the
>>         initial usage of memory.
>>
>>         Cheers,
>>         Daniel
>>
>>         -- 
>>         Daniel-Constantin Mierla - http://www.asipto.com
>>         http://twitter.com/#!/miconda
>>         <http://twitter.com/#%21/miconda> -
>>         http://www.linkedin.com/in/miconda
>>
>>
>>         _______________________________________________
>>         SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
>>         mailing list
>>         sr-users at lists.sip-router.org
>>         <mailto:sr-users at lists.sip-router.org>
>>         http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>>
>>
>>     -- 
>>     David Cunningham, Voisonics
>>     http://voisonics.com/
>>     USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>     UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
>>     Australia: +61 (0) 2 8063 9019
>>     <tel:%2B61%20%280%29%202%208063%209019>
>
>     -- 
>     Daniel-Constantin Mierla -http://www.asipto.com
>     http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
>
>
>
>
> -- 
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20130724/c1ce5666/attachment-0001.html>


More information about the sr-users mailing list