[SR-Users] Memory leak in 3.3.4
David Cunningham
dcunningham at voisonics.com
Fri Aug 2 03:34:29 CEST 2013
Hi Daniel,
Thank you, we will try that.
On 31 July 2013 21:54, Daniel-Constantin Mierla <miconda at gmail.com> wrote:
> Hello,
>
> I pushed the commit to branch 3.3 now -- forgot about it.
>
> The code you pasted below becomes (after the two patches, thus is the
> final version that should be used):
>
>
> ENTER; /* everything created after here */
> SAVETMPS; /* ...is a temporary variable. */
> PUSHMARK(SP); /* remember the stack pointer */
>
> m = sv_newmortal();
> sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
> SvREADONLY_on(SvRV(m));
>
> XPUSHs(m); /* Our reference to the stack... */
>
> if (mystr)
> XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
>
>
> Cheers,
> Daniel
>
>
> On 7/31/13 1:48 PM, David Cunningham wrote:
>
> Hi Daniel,
>
> In 3.3.x the code is arranged a little differently and I'd like to make
> sure we get it right. Could you please advise?
>
>
> m = sv_newmortal();
> sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
> SvREADONLY_on(SvRV(m));
>
>
> ENTER; /* everything created after here */
> SAVETMPS; /* ...is a temporary variable. */
> PUSHMARK(SP); /* remember the stack pointer */
> XPUSHs(m); /* Our reference to the stack... */
>
> if (mystr)
> XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
> /* Our string to the stack... */
>
>
>
> On 31 July 2013 21:44, David Cunningham <dcunningham at voisonics.com> wrote:
>
>> Hi Daniel,
>>
>> Thank you, we will try that.
>>
>>
>>
>> On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>
>>> Hello,
>>>
>>> revising the patch I noticed I was moving the initialization of the
>>> variable after pushing it to perl environment (from the perl docs, the
>>> variable should have been initialized after initializing the environment --
>>> what I tried to do with previous patch).
>>>
>>> There is a new smaller patch to be added:
>>>
>>>
>>> http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain;h=3935fedf23f3bf2b6675182193cef6af3bbd903a
>>>
>>> Practically, the line XPUSHs(m); has to be moved after the line with
>>> SvREADONLY_on(SvRV(m));
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 7/31/13 8:26 AM, David Cunningham wrote:
>>>
>>> Hi Daniel,
>>>
>>> We tried that patch, but Kamailio logged lots of errors like the
>>> following. The undefined value in question is $m, which should be the SIP
>>> message. Would you have any advice? Thanks.
>>>
>>> Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
>>> [openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an
>>> undefined value at Foo.pm line 247.#012
>>>
>>>
>>> On 25 July 2013 17:11, David Cunningham <dcunningham at voisonics.com>wrote:
>>>
>>>> Hi Daniel,
>>>>
>>>> I'll suggest that to the customer. Thank you!
>>>>
>>>>
>>>>
>>>> On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> can you try the attached patch? It's the same patch, just for two
>>>>> versions, one is for 3.3.x and the other for devel version
>>>>>
>>>>> It initializes the SIP message variable that is passed to perl after
>>>>> creating the temporary environment, so it is actually destroyed by the perl
>>>>> embedded interpreter.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>>> On 7/25/13 1:29 AM, David Cunningham wrote:
>>>>>
>>>>> Hi Daniel,
>>>>>
>>>>> The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
>>>>> release 5.4. If I remember right programs running under Valgrind can have
>>>>> issues, so I'm not sure if the customer will want to do that. Ideally we'd
>>>>> do it on a test system, but I'm not sure if we have any RHEL available.
>>>>> I'll see what we can do. Thanks again.
>>>>>
>>>>>
>>>>> On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I would say that perl_exec() is the one with the highest chances to
>>>>>> be the reason for the leak. Next is line would be db_mysql module, if liked
>>>>>> with some custom mysql client library, although even in this case will be
>>>>>> unlikely.
>>>>>>
>>>>>> Back to perl, the module itself does not call any malloc, so it might
>>>>>> be the embedding Perl API that is not used properly in the module.
>>>>>>
>>>>>> Can you use some testbed, set children=1 and run kamailio under
>>>>>> valgrind, then do some calls and see if it detects the source of the leak?
>>>>>>
>>>>>> I'm not using the perl module, I will try to check it whenever I get
>>>>>> a chance in the next days. What version of perl do you have installed?
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 7/24/13 10:31 AM, David Cunningham wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> We don't do any kamctl commands at all. We do have various modules
>>>>>> loaded, as follows. The primary functions we use Kamailio for are phone
>>>>>> registrations through usrloc, and routing calls to Asterisk through logic
>>>>>> contained in Perl via perl_exec().
>>>>>> Thanks for all your advice so far!
>>>>>>
>>>>>> loadmodule "tm.so"
>>>>>> loadmodule "tmx.so"
>>>>>> loadmodule "usrloc.so"
>>>>>> loadmodule "auth.so"
>>>>>> loadmodule "auth_db.so"
>>>>>> loadmodule "ctl.so"
>>>>>> loadmodule "db_mysql.so"
>>>>>> loadmodule "kex.so"
>>>>>> loadmodule "maxfwd.so"
>>>>>> loadmodule "mi_fifo.so"
>>>>>> loadmodule "mi_rpc.so"
>>>>>> loadmodule "nathelper.so"
>>>>>> loadmodule "perl.so"
>>>>>> loadmodule "pv.so"
>>>>>> loadmodule "registrar.so"
>>>>>> loadmodule "rr.so"
>>>>>> loadmodule "sanity.so"
>>>>>> loadmodule "siputils.so"
>>>>>> loadmodule "sl.so"
>>>>>> loadmodule "textops.so"
>>>>>> loadmodule "xlog.so"
>>>>>>
>>>>>>
>>>>>> On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>>
>>>>>>> On 7/24/13 4:24 AM, David Cunningham wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Thank you very much for the email. In reply:
>>>>>>>
>>>>>>> 1. The system ran out of memory. Linux's oom-killer killed Kamailio.
>>>>>>>
>>>>>>> then all the instructions I gave are useless, they are for debugging
>>>>>>> kamailio's internal memory manager, which handles pkg and shm mallocs.
>>>>>>>
>>>>>>> The chances to be from kamailio itself are very low now. Do you do
>>>>>>> lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but
>>>>>>> the rest of code should use internal memory manager which does not go
>>>>>>> beyond the limits set with -m and -M, thus not causing an OS memory
>>>>>>> exhaustion.
>>>>>>>
>>>>>>> Can you list what modules are you loading? At some point it was a
>>>>>>> leak in libssl, in case you use tls a lot. But could be another external
>>>>>>> library...
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2. You're right, DEBUG_MEMORY is a local configuration setting. If
>>>>>>> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
>>>>>>>
>>>>>>> 3. We'll try setting mem_summary=12, thanks.
>>>>>>>
>>>>>>> 4. We'll try setting asynchronous syslog, thanks.
>>>>>>>
>>>>>>> 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a
>>>>>>> lot?
>>>>>>>
>>>>>>> 6. We'll try setting mem_join=1, thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> first, to clarify, is the system memory or kamailio's pkg/shm
>>>>>>>> memory running out? If the operating system runs out of memory, then should
>>>>>>>> be a leak in a library, because kamailio modules uses only from a
>>>>>>>> pre-allocated chunk, not going over it.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 7/23/13 7:33 AM, David Cunningham wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly
>>>>>>>>> using more and more memory. Over a couple of weeks it will run out of
>>>>>>>>> system memory.
>>>>>>>>>
>>>>>>>>> We tried to enable memory debugging doing the following, but it
>>>>>>>>> resulted in Kamailio not responding to any SIP packets. Would anyone have
>>>>>>>>> advice please on how to debug the situation?
>>>>>>>>>
>>>>>>>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>>>>>>>>> 2. In kamailio.cfg add the line:
>>>>>>>>> #!define DEBUG_MEMORY 1
>>>>>>>>>
>>>>>>>> do you set something special in config when DEBUG_MEMORY is 1? It
>>>>>>>> is not by default there, so I assume you added some rules based on this
>>>>>>>> pre-processor directive.
>>>>>>>>
>>>>>>>> For memory troubleshooting, set memlog to a value lower than debug
>>>>>>>> parameter in config file and try with mem_summary=12 for a more compact
>>>>>>>> output. See more about these parameters in the wiki:
>>>>>>>>
>>>>>>>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>>>>>>>
>>>>>>>> Run kamailio for a while in normal conditions, then restart it to
>>>>>>>> get the memory usage summaries. There should be indication if there is some
>>>>>>>> leak, by seeing memory chunks allocated many times from a function used at
>>>>>>>> runtime. You can send the memory summary for a process here, we can look at
>>>>>>>> it.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> While this was running and Kamailio didn't respond to packets, it
>>>>>>>>> logged lots of lines like this:
>>>>>>>>>
>>>>>>>>
>>>>>>>> Do you have syslog to be configured in asynchronous mode? See the
>>>>>>>> notes from:
>>>>>>>>
>>>>>>>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>>>>>>>
>>>>>>>> The memdbg is less than debug value, that means printing few log
>>>>>>>> messages for each memory operation. You can make memdbg higher and rely on
>>>>>>>> memlog for memory summaries, otherwise will be lot of log messages related
>>>>>>>> to memory.
>>>>>>>>
>>>>>>>>
>>>>>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>>>>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>>>>>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>>>>>>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
>>>>>>>>> (size=128) on 1 -th hit
>>>>>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>>>>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>>>>>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>>>>>>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
>>>>>>>>> (size=128) on 1 -th hit
>>>>>>>>>
>>>>>>>> addstr() is a function used only for parsing configuration file,
>>>>>>>> as long as you can still see them, the configuration file parsing was not
>>>>>>>> finish. addstr() is not a source of leaks because it is not used at runtime.
>>>>>>>>
>>>>>>>> If you have large config file, then you can get close to the limits
>>>>>>>> of the private memory, which is set to 4MB. You can increase its value
>>>>>>>> using -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
>>>>>>>> memory).
>>>>>>>>
>>>>>>>> Over the time, the private memory can get used due to
>>>>>>>> fragmentation, you can set the mem_join parameter in config file to avoid
>>>>>>>> it (works when compiled with MEMDBG=1).
>>>>>>>>
>>>>>>>> To monitor usage of internal pkg memory, then you can use sercmd
>>>>>>>> with pkg.stats command:
>>>>>>>>
>>>>>>>>
>>>>>>>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>>>>>>>
>>>>>>>> Shared memory stats are printed by 'kamctl fifo get_statistics
>>>>>>>> shmem:'
>>>>>>>>
>>>>>>>> When you see significant increase of the memory usage, then you can
>>>>>>>> restart to get the summaries.
>>>>>>>>
>>>>>>>> You should run these commands after start, just to see the initial
>>>>>>>> usage of memory.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Daniel
>>>>>>>>
>>>>>>>> --
>>>>>>>> Daniel-Constantin Mierla - http://www.asipto.com
>>>>>>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>>>>> list
>>>>>>>> sr-users at lists.sip-router.org
>>>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> David Cunningham, Voisonics
>>>>>>> http://voisonics.com/
>>>>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>>>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>>>>>> Australia: +61 (0) 2 8063 9019
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> David Cunningham, Voisonics
>>>>>> http://voisonics.com/
>>>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>>>>> Australia: +61 (0) 2 8063 9019
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> David Cunningham, Voisonics
>>>>> http://voisonics.com/
>>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>>>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>>>>
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> David Cunningham, Voisonics
>>>> http://voisonics.com/
>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>> UK: +44 (0) 20 3298 1642
>>>> Australia: +61 (0) 2 8063 9019
>>>>
>>>
>>>
>>>
>>> --
>>> David Cunningham, Voisonics
>>> http://voisonics.com/
>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>
>>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092
>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019
>>
>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>
>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20130802/c15244a3/attachment-0001.html>
More information about the sr-users
mailing list