[SR-Users] Memory leak in 3.3.4
Daniel-Constantin Mierla
miconda at gmail.com
Wed Jul 31 13:54:26 CEST 2013
Hello,
I pushed the commit to branch 3.3 now -- forgot about it.
The code you pasted below becomes (after the two patches, thus is the
final version that should be used):
ENTER; /* everything created after here */
SAVETMPS; /* ...is a temporary variable. */
PUSHMARK(SP); /* remember the stack pointer */
m = sv_newmortal();
sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));
XPUSHs(m); /* Our reference to the stack... */
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
Cheers,
Daniel
On 7/31/13 1:48 PM, David Cunningham wrote:
> Hi Daniel,
>
> In 3.3.x the code is arranged a little differently and I'd like to
> make sure we get it right. Could you please advise?
>
>
> m = sv_newmortal();
> sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
> SvREADONLY_on(SvRV(m));
>
>
> ENTER; /* everything created after
> here */
> SAVETMPS; /* ...is a temporary
> variable. */
> PUSHMARK(SP); /* remember the stack
> pointer */
> XPUSHs(m); /* Our reference to the
> stack... */
>
> if (mystr)
> XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
> /* Our string to the stack... */
>
>
>
> On 31 July 2013 21:44, David Cunningham <dcunningham at voisonics.com
> <mailto:dcunningham at voisonics.com>> wrote:
>
> Hi Daniel,
>
> Thank you, we will try that.
>
>
>
> On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda at gmail.com
> <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
> revising the patch I noticed I was moving the initialization
> of the variable after pushing it to perl environment (from the
> perl docs, the variable should have been initialized after
> initializing the environment -- what I tried to do with
> previous patch).
>
> There is a new smaller patch to be added:
>
> http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain;h=3935fedf23f3bf2b6675182193cef6af3bbd903a
>
> Practically, the line XPUSHs(m); has to be moved after the
> line with SvREADONLY_on(SvRV(m));
>
> Cheers,
> Daniel
>
>
> On 7/31/13 8:26 AM, David Cunningham wrote:
>> Hi Daniel,
>>
>> We tried that patch, but Kamailio logged lots of errors like
>> the following. The undefined value in question is $m, which
>> should be the SIP message. Would you have any advice? Thanks.
>>
>> Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
>> [openserxs.xs:1022]: perl error: Can't call method
>> "pseudoVar" on an undefined value at Foo.pm line 247.#012
>>
>>
>> On 25 July 2013 17:11, David Cunningham
>> <dcunningham at voisonics.com
>> <mailto:dcunningham at voisonics.com>> wrote:
>>
>> Hi Daniel,
>>
>> I'll suggest that to the customer. Thank you!
>>
>>
>>
>> On 25 July 2013 15:45, Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>> can you try the attached patch? It's the same patch,
>> just for two versions, one is for 3.3.x and the other
>> for devel version
>>
>> It initializes the SIP message variable that is
>> passed to perl after creating the temporary
>> environment, so it is actually destroyed by the perl
>> embedded interpreter.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 7/25/13 1:29 AM, David Cunningham wrote:
>>> Hi Daniel,
>>>
>>> The system is running Perl 5.8.8 on Red Hat
>>> Enterprise Linux Server release 5.4. If I remember
>>> right programs running under Valgrind can have
>>> issues, so I'm not sure if the customer will want to
>>> do that. Ideally we'd do it on a test system, but
>>> I'm not sure if we have any RHEL available.
>>> I'll see what we can do. Thanks again.
>>>
>>>
>>> On 25 July 2013 04:55, Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> I would say that perl_exec() is the one with the
>>> highest chances to be the reason for the leak.
>>> Next is line would be db_mysql module, if liked
>>> with some custom mysql client library, although
>>> even in this case will be unlikely.
>>>
>>> Back to perl, the module itself does not call
>>> any malloc, so it might be the embedding Perl
>>> API that is not used properly in the module.
>>>
>>> Can you use some testbed, set children=1 and run
>>> kamailio under valgrind, then do some calls and
>>> see if it detects the source of the leak?
>>>
>>> I'm not using the perl module, I will try to
>>> check it whenever I get a chance in the next
>>> days. What version of perl do you have installed?
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 7/24/13 10:31 AM, David Cunningham wrote:
>>>> Hello,
>>>>
>>>> We don't do any kamctl commands at all. We do
>>>> have various modules loaded, as follows. The
>>>> primary functions we use Kamailio for are phone
>>>> registrations through usrloc, and routing calls
>>>> to Asterisk through logic contained in Perl via
>>>> perl_exec().
>>>> Thanks for all your advice so far!
>>>>
>>>> loadmodule "tm.so"
>>>> loadmodule "tmx.so"
>>>> loadmodule "usrloc.so"
>>>> loadmodule "auth.so"
>>>> loadmodule "auth_db.so"
>>>> loadmodule "ctl.so"
>>>> loadmodule "db_mysql.so"
>>>> loadmodule "kex.so"
>>>> loadmodule "maxfwd.so"
>>>> loadmodule "mi_fifo.so"
>>>> loadmodule "mi_rpc.so"
>>>> loadmodule "nathelper.so"
>>>> loadmodule "perl.so"
>>>> loadmodule "pv.so"
>>>> loadmodule "registrar.so"
>>>> loadmodule "rr.so"
>>>> loadmodule "sanity.so"
>>>> loadmodule "siputils.so"
>>>> loadmodule "sl.so"
>>>> loadmodule "textops.so"
>>>> loadmodule "xlog.so"
>>>>
>>>>
>>>> On 24 July 2013 16:33, Daniel-Constantin Mierla
>>>> <miconda at gmail.com <mailto:miconda at gmail.com>>
>>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>>
>>>> On 7/24/13 4:24 AM, David Cunningham wrote:
>>>>> Hello,
>>>>>
>>>>> Thank you very much for the email. In reply:
>>>>>
>>>>> 1. The system ran out of memory. Linux's
>>>>> oom-killer killed Kamailio.
>>>> then all the instructions I gave are
>>>> useless, they are for debugging kamailio's
>>>> internal memory manager, which handles pkg
>>>> and shm mallocs.
>>>>
>>>> The chances to be from kamailio itself are
>>>> very low now. Do you do lot of mi commands
>>>> (e.g., kamctl ...)? The mi api uses system
>>>> malloc, but the rest of code should use
>>>> internal memory manager which does not go
>>>> beyond the limits set with -m and -M, thus
>>>> not causing an OS memory exhaustion.
>>>>
>>>> Can you list what modules are you loading?
>>>> At some point it was a leak in libssl, in
>>>> case you use tls a lot. But could be
>>>> another external library...
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>>>
>>>>> 2. You're right, DEBUG_MEMORY is a local
>>>>> configuration setting. If defined it sets
>>>>> memdbg to -2, and memlog to -2. The debug
>>>>> setting is -1.
>>>>>
>>>>> 3. We'll try setting mem_summary=12, thanks.
>>>>>
>>>>> 4. We'll try setting asynchronous syslog,
>>>>> thanks.
>>>>>
>>>>> 5. Our configuration totals 338 lines, or
>>>>> approx 8.5kb. Is that a lot?
>>>>>
>>>>> 6. We'll try setting mem_join=1, thanks.
>>>>>
>>>>>
>>>>>
>>>>> On 23 July 2013 16:53, Daniel-Constantin
>>>>> Mierla <miconda at gmail.com
>>>>> <mailto:miconda at gmail.com>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> first, to clarify, is the system
>>>>> memory or kamailio's pkg/shm memory
>>>>> running out? If the operating system
>>>>> runs out of memory, then should be a
>>>>> leak in a library, because kamailio
>>>>> modules uses only from a pre-allocated
>>>>> chunk, not going over it.
>>>>>
>>>>>
>>>>> On 7/23/13 7:33 AM, David Cunningham
>>>>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> We're running a Kamailio 3.3.4
>>>>> system, and Kamailio is slowly
>>>>> using more and more memory. Over a
>>>>> couple of weeks it will run out of
>>>>> system memory.
>>>>>
>>>>> We tried to enable memory
>>>>> debugging doing the following, but
>>>>> it resulted in Kamailio not
>>>>> responding to any SIP packets.
>>>>> Would anyone have advice please on
>>>>> how to debug the situation?
>>>>>
>>>>> 1. In Makefile.defs set MEMDBG to
>>>>> 1 and recompile Kamailio.
>>>>> 2. In kamailio.cfg add the line:
>>>>> #!define DEBUG_MEMORY 1
>>>>>
>>>>> do you set something special in config
>>>>> when DEBUG_MEMORY is 1? It is not by
>>>>> default there, so I assume you added
>>>>> some rules based on this pre-processor
>>>>> directive.
>>>>>
>>>>> For memory troubleshooting, set memlog
>>>>> to a value lower than debug parameter
>>>>> in config file and try with
>>>>> mem_summary=12 for a more compact
>>>>> output. See more about these
>>>>> parameters in the wiki:
>>>>>
>>>>> -
>>>>> http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>>>>
>>>>> Run kamailio for a while in normal
>>>>> conditions, then restart it to get the
>>>>> memory usage summaries. There should
>>>>> be indication if there is some leak,
>>>>> by seeing memory chunks allocated many
>>>>> times from a function used at runtime.
>>>>> You can send the memory summary for a
>>>>> process here, we can look at it.
>>>>>
>>>>>
>>>>>
>>>>> While this was running and
>>>>> Kamailio didn't respond to
>>>>> packets, it logged lots of lines
>>>>> like this:
>>>>>
>>>>>
>>>>> Do you have syslog to be configured in
>>>>> asynchronous mode? See the notes from:
>>>>>
>>>>> -
>>>>> http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>>>>
>>>>> The memdbg is less than debug value,
>>>>> that means printing few log messages
>>>>> for each memory operation. You can
>>>>> make memdbg higher and rely on memlog
>>>>> for memory summaries, otherwise will
>>>>> be lot of log messages related to memory.
>>>>>
>>>>>
>>>>> Jul 22 21:32:22 hostname kamailio:
>>>>> : <core> [mem/q_malloc.c:369]:
>>>>> qm_malloc(0x4000e008, 128) called
>>>>> from <core>: cfg.lex: addstr(1438)
>>>>> Jul 22 21:32:22 hostname kamailio:
>>>>> : <core> [mem/q_malloc.c:413]:
>>>>> qm_malloc(0x4000e008, 128) returns
>>>>> address 0x40048918 frag.
>>>>> 0x40048900 (size=128) on 1 -th hit
>>>>> Jul 22 21:32:22 hostname kamailio:
>>>>> : <core> [mem/q_malloc.c:369]:
>>>>> qm_malloc(0x4000e008, 128) called
>>>>> from <core>: cfg.lex: addstr(1438)
>>>>> Jul 22 21:32:22 hostname kamailio:
>>>>> : <core> [mem/q_malloc.c:413]:
>>>>> qm_malloc(0x4000e008, 128) returns
>>>>> address 0x400489c8 frag.
>>>>> 0x400489b0 (size=128) on 1 -th hit
>>>>>
>>>>> addstr() is a function used only for
>>>>> parsing configuration file, as long as
>>>>> you can still see them, the
>>>>> configuration file parsing was not
>>>>> finish. addstr() is not a source of
>>>>> leaks because it is not used at runtime.
>>>>>
>>>>> If you have large config file, then
>>>>> you can get close to the limits of the
>>>>> private memory, which is set to 4MB.
>>>>> You can increase its value using -M
>>>>> parameter (e.g., start kamailio with
>>>>> -M 8 to set it to use 8MB of memory).
>>>>>
>>>>> Over the time, the private memory can
>>>>> get used due to fragmentation, you can
>>>>> set the mem_join parameter in config
>>>>> file to avoid it (works when compiled
>>>>> with MEMDBG=1).
>>>>>
>>>>> To monitor usage of internal pkg
>>>>> memory, then you can use sercmd with
>>>>> pkg.stats command:
>>>>>
>>>>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>>>>
>>>>> Shared memory stats are printed by
>>>>> 'kamctl fifo get_statistics shmem:'
>>>>>
>>>>> When you see significant increase of
>>>>> the memory usage, then you can restart
>>>>> to get the summaries.
>>>>>
>>>>> You should run these commands after
>>>>> start, just to see the initial usage
>>>>> of memory.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla -
>>>>> http://www.asipto.com
>>>>> http://twitter.com/#!/miconda
>>>>> <http://twitter.com/#%21/miconda> -
>>>>> http://www.linkedin.com/in/miconda
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio
>>>>> (OpenSER) - sr-users mailing list
>>>>> sr-users at lists.sip-router.org
>>>>> <mailto:sr-users at lists.sip-router.org>
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> David Cunningham, Voisonics
>>>>> http://voisonics.com/
>>>>> USA: +1 213 221 1092
>>>>> <tel:%2B1%20213%20221%201092>
>>>>> UK: +44 (0) 20 3298 1642
>>>>> <tel:%2B44%20%280%29%2020%203298%201642>
>>>>> Australia: +61 (0) 2 8063 9019
>>>>> <tel:%2B61%20%280%29%202%208063%209019>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -http://www.asipto.com
>>>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> David Cunningham, Voisonics
>>>> http://voisonics.com/
>>>> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>>> UK: +44 (0) 20 3298 1642
>>>> <tel:%2B44%20%280%29%2020%203298%201642>
>>>> Australia: +61 (0) 2 8063 9019
>>>> <tel:%2B61%20%280%29%202%208063%209019>
>>>
>>> --
>>> Daniel-Constantin Mierla -http://www.asipto.com
>>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>>
>>>
>>>
>>>
>>> --
>>> David Cunningham, Voisonics
>>> http://voisonics.com/
>>> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>> UK: +44 (0) 20 3298 1642
>>> <tel:%2B44%20%280%29%2020%203298%201642>
>>> Australia: +61 (0) 2 8063 9019
>>> <tel:%2B61%20%280%29%202%208063%209019>
>>
>> --
>> Daniel-Constantin Mierla -http://www.asipto.com
>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>
>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>> UK: +44 (0) 20 3298 1642
>> <tel:%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019
>> <tel:%2B61%20%280%29%202%208063%209019>
>>
>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>> UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019
>> <tel:%2B61%20%280%29%202%208063%209019>
>
> --
> Daniel-Constantin Mierla -http://www.asipto.com
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>
>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
> UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
> Australia: +61 (0) 2 8063 9019 <tel:%2B61%20%280%29%202%208063%209019>
>
>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20130731/717c340e/attachment-0001.html>
More information about the sr-users
mailing list