[SR-Users] Memory leak in 3.3.4

Thu Jul 25 01:29:21 CEST 2013

Hi Daniel,

The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server release
5.4. If I remember right programs running under Valgrind can have issues,
so I'm not sure if the customer will want to do that. Ideally we'd do it on
a test system, but I'm not sure if we have any RHEL available.
I'll see what we can do. Thanks again.

On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda at gmail.com> wrote:

>  Hello,
>
> I would say that perl_exec() is the one with the highest chances to be the
> reason for the leak. Next is line would be db_mysql module, if liked with
> some custom mysql client library, although even in this case will be
> unlikely.
>
> Back to perl, the module itself does not call any malloc, so it might be
> the embedding Perl API that is not used properly in the module.
>
> Can you use some testbed, set children=1 and run kamailio under valgrind,
> then do some calls and see if it detects the source of the leak?
>
> I'm not using the perl module, I will try to check it whenever I get a
> chance in the next days. What version of perl do you have installed?
>
> Cheers,
> Daniel
>
>
> On 7/24/13 10:31 AM, David Cunningham wrote:
>
> Hello,
>
> We don't do any kamctl commands at all. We do have various modules loaded,
> as follows. The primary functions we use Kamailio for are phone
> registrations through usrloc, and routing calls to Asterisk through logic
> contained in Perl via perl_exec().
> Thanks for all your advice so far!
>
> loadmodule "tm.so"
> loadmodule "tmx.so"
> loadmodule "usrloc.so"
> loadmodule "auth.so"
> loadmodule "auth_db.so"
> loadmodule "ctl.so"
> loadmodule "db_mysql.so"
> loadmodule "kex.so"
> loadmodule "maxfwd.so"
> loadmodule "mi_fifo.so"
> loadmodule "mi_rpc.so"
> loadmodule "nathelper.so"
> loadmodule "perl.so"
> loadmodule "pv.so"
> loadmodule "registrar.so"
> loadmodule "rr.so"
> loadmodule "sanity.so"
> loadmodule "siputils.so"
> loadmodule "sl.so"
> loadmodule "textops.so"
> loadmodule "xlog.so"
>
>
> On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda at gmail.com> wrote:
>
>>  Hello,
>>
>>
>> On 7/24/13 4:24 AM, David Cunningham wrote:
>>
>> Hello,
>>
>>  Thank you very much for the email. In reply:
>>
>> 1. The system ran out of memory. Linux's oom-killer killed Kamailio.
>>
>> then all the instructions I gave are useless, they are for debugging
>> kamailio's internal memory manager, which handles pkg and shm mallocs.
>>
>> The chances to be from kamailio itself are very low now. Do you do lot of
>> mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the rest
>> of code should use internal memory manager which does not go beyond the
>> limits set with -m and -M, thus not causing an OS memory exhaustion.
>>
>> Can you list what modules are you loading? At some point it was a leak in
>> libssl, in case you use tls a lot. But could be another external library...
>>
>> Cheers,
>> Daniel
>>
>>
>>
>> 2. You're right, DEBUG_MEMORY is a local configuration setting. If
>> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
>>
>> 3. We'll try setting mem_summary=12, thanks.
>>
>> 4. We'll try setting asynchronous syslog, thanks.
>>
>> 5.  Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?
>>
>> 6. We'll try setting mem_join=1, thanks.
>>
>>
>>
>> On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>
>>> Hello,
>>>
>>> first, to clarify, is the system memory or kamailio's pkg/shm memory
>>> running out? If the operating system runs out of memory, then should be a
>>> leak in a library, because kamailio modules uses only from a pre-allocated
>>> chunk, not going over it.
>>>
>>>
>>> On 7/23/13 7:33 AM, David Cunningham wrote:
>>>
>>>> Hello,
>>>>
>>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using
>>>> more and more memory. Over a couple of weeks it will run out of system
>>>> memory.
>>>>
>>>> We tried to enable memory debugging doing the following, but it
>>>> resulted in Kamailio not responding to any SIP packets. Would anyone have
>>>> advice please on how to debug the situation?
>>>>
>>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>>>> 2. In kamailio.cfg add the line:
>>>> #!define DEBUG_MEMORY 1
>>>>
>>>  do you set something special in config when DEBUG_MEMORY is 1? It is
>>> not by default there, so I assume you added some rules based on this
>>> pre-processor directive.
>>>
>>> For memory troubleshooting, set memlog to a value lower than debug
>>> parameter in config file and try with mem_summary=12 for a more compact
>>> output. See more about these parameters in the wiki:
>>>
>>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>>
>>> Run kamailio for a while in normal conditions, then restart it to get
>>> the memory usage summaries. There should be indication if there is some
>>> leak, by seeing memory chunks allocated many times from a function used at
>>> runtime. You can send the memory summary for a process here, we can look at
>>> it.
>>>
>>>
>>>
>>>> While this was running and Kamailio didn't respond to packets, it
>>>> logged lots of lines like this:
>>>>
>>>
>>>  Do you have syslog to be configured in asynchronous mode? See the notes
>>> from:
>>>
>>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>>
>>> The memdbg is less than debug value, that means printing few log
>>> messages for each memory operation. You can make memdbg higher and rely on
>>> memlog for memory summaries, otherwise will be lot of log messages related
>>> to memory.
>>>
>>>
>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
>>>> (size=128) on 1 -th hit
>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
>>>> (size=128) on 1 -th hit
>>>>
>>>  addstr() is a function used only for parsing configuration file, as
>>> long as you can still see them, the configuration file parsing was not
>>> finish. addstr() is not a source of leaks because it is not used at runtime.
>>>
>>> If you have large config file, then you can get close to the limits of
>>> the private memory, which is set to 4MB. You can increase its value using
>>> -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
>>> memory).
>>>
>>> Over the time, the private memory can get used due to fragmentation, you
>>> can set the mem_join parameter in config file to avoid it (works when
>>> compiled with MEMDBG=1).
>>>
>>> To monitor usage of internal pkg memory, then you can use sercmd with
>>> pkg.stats command:
>>>
>>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>>
>>> Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
>>>
>>> When you see significant increase of the memory usage, then you can
>>> restart to get the summaries.
>>>
>>> You should run these commands after start, just to see the initial usage
>>> of memory.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.com
>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users at lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092
>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>
>>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>
>

-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20130725/6d2477cf/attachment.html>