[SR-Users] Memory leak in 3.3.4

Daniel-Constantin Mierla miconda at gmail.com
Wed Jul 31 12:54:46 CEST 2013


Hello,

revising the patch I noticed I was moving the initialization of the 
variable after pushing it to perl environment (from the perl docs, the 
variable should have been initialized after initializing the environment 
-- what I tried to do with previous patch).

There is a new smaller patch to be added:

http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain;h=3935fedf23f3bf2b6675182193cef6af3bbd903a

Practically, the line XPUSHs(m); has to be moved after the line with 
SvREADONLY_on(SvRV(m));

Cheers,
Daniel

On 7/31/13 8:26 AM, David Cunningham wrote:
> Hi Daniel,
>
> We tried that patch, but Kamailio logged lots of errors like the 
> following. The undefined value in question is $m, which should be the 
> SIP message. Would you have any advice? Thanks.
>
> Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl 
> [openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an 
> undefined value at Foo.pm line 247.#012
>
>
> On 25 July 2013 17:11, David Cunningham <dcunningham at voisonics.com 
> <mailto:dcunningham at voisonics.com>> wrote:
>
>     Hi Daniel,
>
>     I'll suggest that to the customer. Thank you!
>
>
>
>     On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda at gmail.com
>     <mailto:miconda at gmail.com>> wrote:
>
>         Hello,
>
>         can you try the attached patch? It's the same patch, just for
>         two versions, one is for 3.3.x and the other for devel version
>
>         It initializes the SIP message variable that is passed to perl
>         after creating the temporary environment, so it is actually
>         destroyed by the perl embedded interpreter.
>
>         Cheers,
>         Daniel
>
>
>         On 7/25/13 1:29 AM, David Cunningham wrote:
>>         Hi Daniel,
>>
>>         The system is running Perl 5.8.8 on Red Hat Enterprise Linux
>>         Server release 5.4. If I remember right programs running
>>         under Valgrind can have issues, so I'm not sure if the
>>         customer will want to do that. Ideally we'd do it on a test
>>         system, but I'm not sure if we have any RHEL available.
>>         I'll see what we can do. Thanks again.
>>
>>
>>         On 25 July 2013 04:55, Daniel-Constantin Mierla
>>         <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>>             Hello,
>>
>>             I would say that perl_exec() is the one with the highest
>>             chances to be the reason for the leak. Next is line would
>>             be db_mysql module, if liked with some custom mysql
>>             client library, although even in this case will be unlikely.
>>
>>             Back to perl, the module itself does not call any malloc,
>>             so it might be the embedding Perl API that is not used
>>             properly in the module.
>>
>>             Can you use some testbed, set children=1 and run kamailio
>>             under valgrind, then do some calls and see if it detects
>>             the source of the leak?
>>
>>             I'm not using the perl module, I will try to check it
>>             whenever I get a chance in the next days. What version of
>>             perl do you have installed?
>>
>>             Cheers,
>>             Daniel
>>
>>
>>             On 7/24/13 10:31 AM, David Cunningham wrote:
>>>             Hello,
>>>
>>>             We don't do any kamctl commands at all. We do have
>>>             various modules loaded, as follows. The primary
>>>             functions we use Kamailio for are phone registrations
>>>             through usrloc, and routing calls to Asterisk through
>>>             logic contained in Perl via perl_exec().
>>>             Thanks for all your advice so far!
>>>
>>>             loadmodule "tm.so"
>>>             loadmodule "tmx.so"
>>>             loadmodule "usrloc.so"
>>>             loadmodule "auth.so"
>>>             loadmodule "auth_db.so"
>>>             loadmodule "ctl.so"
>>>             loadmodule "db_mysql.so"
>>>             loadmodule "kex.so"
>>>             loadmodule "maxfwd.so"
>>>             loadmodule "mi_fifo.so"
>>>             loadmodule "mi_rpc.so"
>>>             loadmodule "nathelper.so"
>>>             loadmodule "perl.so"
>>>             loadmodule "pv.so"
>>>             loadmodule "registrar.so"
>>>             loadmodule "rr.so"
>>>             loadmodule "sanity.so"
>>>             loadmodule "siputils.so"
>>>             loadmodule "sl.so"
>>>             loadmodule "textops.so"
>>>             loadmodule "xlog.so"
>>>
>>>
>>>             On 24 July 2013 16:33, Daniel-Constantin Mierla
>>>             <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>>                 Hello,
>>>
>>>
>>>                 On 7/24/13 4:24 AM, David Cunningham wrote:
>>>>                 Hello,
>>>>
>>>>                 Thank you very much for the email. In reply:
>>>>
>>>>                 1. The system ran out of memory. Linux's oom-killer
>>>>                 killed Kamailio.
>>>                 then all the instructions I gave are useless, they
>>>                 are for debugging kamailio's internal memory
>>>                 manager, which handles pkg and shm mallocs.
>>>
>>>                 The chances to be from kamailio itself are very low
>>>                 now. Do you do lot of mi commands (e.g., kamctl
>>>                 ...)? The mi api uses system malloc, but the rest of
>>>                 code should use internal memory manager which does
>>>                 not go beyond the limits set with -m and -M, thus
>>>                 not causing an OS memory exhaustion.
>>>
>>>                 Can you list what modules are you loading? At some
>>>                 point it was a leak in libssl, in case you use tls a
>>>                 lot. But could be another external library...
>>>
>>>                 Cheers,
>>>                 Daniel
>>>
>>>
>>>>
>>>>                 2. You're right, DEBUG_MEMORY is a local
>>>>                 configuration setting. If defined it sets memdbg to
>>>>                 -2, and memlog to -2. The debug setting is -1.
>>>>
>>>>                 3. We'll try setting mem_summary=12, thanks.
>>>>
>>>>                 4. We'll try setting asynchronous syslog, thanks.
>>>>
>>>>                 5.  Our configuration totals 338 lines, or approx
>>>>                 8.5kb. Is that a lot?
>>>>
>>>>                 6. We'll try setting mem_join=1, thanks.
>>>>
>>>>
>>>>
>>>>                 On 23 July 2013 16:53, Daniel-Constantin Mierla
>>>>                 <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>>
>>>>                     Hello,
>>>>
>>>>                     first, to clarify, is the system memory or
>>>>                     kamailio's pkg/shm memory running out? If the
>>>>                     operating system runs out of memory, then
>>>>                     should be a leak in a library, because kamailio
>>>>                     modules uses only from a pre-allocated chunk,
>>>>                     not going over it.
>>>>
>>>>
>>>>                     On 7/23/13 7:33 AM, David Cunningham wrote:
>>>>
>>>>                         Hello,
>>>>
>>>>                         We're running a Kamailio 3.3.4 system, and
>>>>                         Kamailio is slowly using more and more
>>>>                         memory. Over a couple of weeks it will run
>>>>                         out of system memory.
>>>>
>>>>                         We tried to enable memory debugging doing
>>>>                         the following, but it resulted in Kamailio
>>>>                         not responding to any SIP packets. Would
>>>>                         anyone have advice please on how to debug
>>>>                         the situation?
>>>>
>>>>                         1. In Makefile.defs set MEMDBG to 1 and
>>>>                         recompile Kamailio.
>>>>                         2. In kamailio.cfg add the line:
>>>>                         #!define DEBUG_MEMORY 1
>>>>
>>>>                     do you set something special in config when
>>>>                     DEBUG_MEMORY is 1? It is not by default there,
>>>>                     so I assume you added some rules based on this
>>>>                     pre-processor directive.
>>>>
>>>>                     For memory troubleshooting, set memlog to a
>>>>                     value lower than debug parameter in config file
>>>>                     and try with mem_summary=12 for a more compact
>>>>                     output. See more about these parameters in the
>>>>                     wiki:
>>>>
>>>>                     -
>>>>                     http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>>>
>>>>                     Run kamailio for a while in normal conditions,
>>>>                     then restart it to get the memory usage
>>>>                     summaries. There should be indication if there
>>>>                     is some leak, by seeing memory chunks allocated
>>>>                     many times from a function used at runtime. You
>>>>                     can send the memory summary for a process here,
>>>>                     we can look at it.
>>>>
>>>>
>>>>
>>>>                         While this was running and Kamailio didn't
>>>>                         respond to packets, it logged lots of lines
>>>>                         like this:
>>>>
>>>>
>>>>                     Do you have syslog to be configured in
>>>>                     asynchronous mode? See the notes from:
>>>>
>>>>                     -
>>>>                     http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>>>
>>>>                     The memdbg is less than debug value, that means
>>>>                     printing few log messages for each memory
>>>>                     operation. You can make memdbg higher and rely
>>>>                     on memlog for memory summaries, otherwise will
>>>>                     be lot of log messages related to memory.
>>>>
>>>>
>>>>                         Jul 22 21:32:22 hostname kamailio: : <core>
>>>>                         [mem/q_malloc.c:369]: qm_malloc(0x4000e008,
>>>>                         128) called from <core>: cfg.lex: addstr(1438)
>>>>                         Jul 22 21:32:22 hostname kamailio: : <core>
>>>>                         [mem/q_malloc.c:413]: qm_malloc(0x4000e008,
>>>>                         128) returns address 0x40048918 frag.
>>>>                         0x40048900 (size=128) on 1 -th hit
>>>>                         Jul 22 21:32:22 hostname kamailio: : <core>
>>>>                         [mem/q_malloc.c:369]: qm_malloc(0x4000e008,
>>>>                         128) called from <core>: cfg.lex: addstr(1438)
>>>>                         Jul 22 21:32:22 hostname kamailio: : <core>
>>>>                         [mem/q_malloc.c:413]: qm_malloc(0x4000e008,
>>>>                         128) returns address 0x400489c8 frag.
>>>>                         0x400489b0 (size=128) on 1 -th hit
>>>>
>>>>                     addstr() is a function used only for parsing
>>>>                     configuration file, as long as you can still
>>>>                     see them, the configuration file parsing was
>>>>                     not finish. addstr() is not a source of leaks
>>>>                     because it is not used at runtime.
>>>>
>>>>                     If you have large config file, then you can get
>>>>                     close to the limits of the private memory,
>>>>                     which is set to 4MB. You can increase its value
>>>>                     using -M parameter (e.g., start kamailio with
>>>>                     -M 8 to set it to use 8MB of memory).
>>>>
>>>>                     Over the time, the private memory can get used
>>>>                     due to fragmentation, you can set the mem_join
>>>>                     parameter in config file to avoid it (works
>>>>                     when compiled with MEMDBG=1).
>>>>
>>>>                     To monitor usage of internal pkg memory, then
>>>>                     you can use sercmd with pkg.stats command:
>>>>
>>>>                     http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>>>
>>>>                     Shared memory stats are printed by 'kamctl fifo
>>>>                     get_statistics shmem:'
>>>>
>>>>                     When you see significant increase of the memory
>>>>                     usage, then you can restart to get the summaries.
>>>>
>>>>                     You should run these commands after start, just
>>>>                     to see the initial usage of memory.
>>>>
>>>>                     Cheers,
>>>>                     Daniel
>>>>
>>>>                     -- 
>>>>                     Daniel-Constantin Mierla - http://www.asipto.com
>>>>                     http://twitter.com/#!/miconda
>>>>                     <http://twitter.com/#%21/miconda> -
>>>>                     http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>>                     _______________________________________________
>>>>                     SIP Express Router (SER) and Kamailio (OpenSER)
>>>>                     - sr-users mailing list
>>>>                     sr-users at lists.sip-router.org
>>>>                     <mailto:sr-users at lists.sip-router.org>
>>>>                     http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>>
>>>>
>>>>                 -- 
>>>>                 David Cunningham, Voisonics
>>>>                 http://voisonics.com/
>>>>                 USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>>>                 UK: +44 (0) 20 3298 1642
>>>>                 <tel:%2B44%20%280%29%2020%203298%201642>
>>>>                 Australia: +61 (0) 2 8063 9019
>>>>                 <tel:%2B61%20%280%29%202%208063%209019>
>>>
>>>                 -- 
>>>                 Daniel-Constantin Mierla -http://www.asipto.com
>>>                 http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
>>>
>>>
>>>
>>>
>>>             -- 
>>>             David Cunningham, Voisonics
>>>             http://voisonics.com/
>>>             USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>>             UK: +44 (0) 20 3298 1642
>>>             <tel:%2B44%20%280%29%2020%203298%201642>
>>>             Australia: +61 (0) 2 8063 9019
>>>             <tel:%2B61%20%280%29%202%208063%209019>
>>
>>             -- 
>>             Daniel-Constantin Mierla -http://www.asipto.com
>>             http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
>>
>>
>>
>>
>>         -- 
>>         David Cunningham, Voisonics
>>         http://voisonics.com/
>>         USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>         UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
>>         Australia: +61 (0) 2 8063 9019
>>         <tel:%2B61%20%280%29%202%208063%209019>
>
>         -- 
>         Daniel-Constantin Mierla -http://www.asipto.com
>         http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
>
>
>
>
>     -- 
>     David Cunningham, Voisonics
>     http://voisonics.com/
>     USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>     UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
>     Australia: +61 (0) 2 8063 9019 <tel:%2B61%20%280%29%202%208063%209019>
>
>
>
>
> -- 
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
> UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
> Australia: +61 (0) 2 8063 9019 <tel:%2B61%20%280%29%202%208063%209019>

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20130731/aa40b789/attachment-0001.html>


More information about the sr-users mailing list