[SR-Users] Memory leak in 3.3.4

David Cunningham dcunningham at voisonics.com
Wed Oct 2 14:09:01 CEST 2013


Hi Daniel,

Kamailio had only been running for about 24 hours with very little traffic
on that server. Should we wait for longer and throw some calls at it?

Thanks again.



On 2 October 2013 18:39, Daniel-Constantin Mierla <miconda at gmail.com> wrote:

>  Hello,
>
> on a second check, those counter chunks for counters/statistics seems to
> be from startup, thus not part of a leak.
>
> How long have you been running kamailio? Looking at the pkg stats from
> previous email, the free part is still a lot, with only about 200k used. It
> could mean the leak was not making any effect yet.
>
> Cheers,
> Daniel
>
>
> On 10/2/13 10:19 AM, Daniel-Constantin Mierla wrote:
>
> Hello,
>
> the leak is due to keep defining counters or statistics. That should
> happen only at startup, but if you execute from perl script some function
> from a kamailio module, it may trigger this issue. Can you share what
> functions from kamailio are you execute inside the perl script?
>
> Cheers,
> Daniel
>
> On 10/2/13 9:26 AM, David Cunningham wrote:
>
>  Hi Daniel,
>
>  Thank you - I used the restart method and got a whole load of memory
> information. 10 megabytes of it! I've compressed that and attached it to
> this email.
>
>  Thanks for your help.
>
>
>
> On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>
>>  Hello,
>>
>> ahh, right, forgot that it was disabled because it was not safe to use at
>> runtime for worker processes. You have to use:
>>
>> kamcmd cfg.set_now_int core mem_dump_pkg <pid>
>>
>> then wait for a bit so that process is receiving a SIP message.
>>
>> Or, if you restart, the stats are printed in syslog for all processes.
>>
>> Another alternative is to call pkg_status() or pkg_summary() from
>> cfgutils module in config file, like:
>>
>> if(src_ip==127.0.0.1 && is_method("OPTIONS") && $rU=="pkgstats") {
>>   send_reply("200", "OK");
>>   pkg_summary();
>>   exit;
>> }
>>
>> You can use sipsak to send an options request to "sip:pkgstats at ..."
>>
>> Cheers,
>> Daniel
>>
>>
>> On 10/2/13 2:30 AM, David Cunningham wrote:
>>
>> Hi Daniel,
>>
>> When I send SIGUSR1 to a worker process nothing is logged. Any ideas
>> there? The following is the output of "sercmd pkg.stats".
>>
>> {
>>     entry: 0
>>     pid: 14570
>>     rank: 0
>>     used: 191100
>>     free: 3731504
>>     real_used: 462800
>> }
>> {
>>     entry: 1
>>     pid: 14571
>>     rank: 1
>>     used: 202656
>>     free: 3714236
>>     real_used: 480068
>> }
>> {
>>     entry: 2
>>     pid: 14572
>>     rank: 2
>>     used: 202720
>>     free: 3714220
>>     real_used: 480084
>> }
>> {
>>     entry: 3
>>     pid: 14573
>>     rank: 3
>>     used: 202664
>>     free: 3714324
>>     real_used: 479980
>> }
>> {
>>     entry: 4
>>     pid: 14574
>>     rank: 4
>>     used: 202744
>>     free: 3714100
>>     real_used: 480204
>> }
>> {
>>     entry: 5
>>     pid: 14575
>>     rank: 5
>>     used: 201908
>>     free: 3715176
>>     real_used: 479128
>> }
>> {
>>     entry: 6
>>     pid: 14576
>>     rank: 6
>>     used: 202572
>>     free: 3714368
>>     real_used: 479936
>> }
>> {
>>     entry: 7
>>     pid: 14577
>>     rank: 7
>>     used: 202740
>>     free: 3714488
>>     real_used: 479816
>> }
>> {
>>     entry: 8
>>     pid: 14578
>>     rank: 8
>>     used: 199760
>>     free: 3717084
>>     real_used: 477220
>> }
>> {
>>     entry: 9
>>     pid: 14579
>>     rank: 9
>>     used: 199440
>>     free: 3722828
>>     real_used: 471476
>> }
>> {
>>     entry: 10
>>     pid: 14580
>>     rank: 10
>>     used: 199440
>>     free: 3722828
>>     real_used: 471476
>> }
>> {
>>     entry: 11
>>     pid: 14581
>>     rank: 11
>>     used: 199440
>>     free: 3722828
>>     real_used: 471476
>> }
>> {
>>     entry: 12
>>     pid: 14582
>>     rank: 12
>>     used: 199440
>>     free: 3722828
>>     real_used: 471476
>> }
>> {
>>     entry: 13
>>     pid: 14583
>>     rank: 13
>>     used: 199440
>>     free: 3722828
>>     real_used: 471476
>> }
>> {
>>     entry: 14
>>     pid: 14584
>>     rank: 14
>>     used: 199440
>>     free: 3722828
>>     real_used: 471476
>> }
>> {
>>     entry: 15
>>     pid: 14585
>>     rank: 15
>>     used: 199440
>>     free: 3722828
>>     real_used: 471476
>> }
>> {
>>     entry: 16
>>     pid: 14586
>>     rank: 16
>>     used: 199440
>>     free: 3722828
>>     real_used: 471476
>> }
>> {
>>     entry: 17
>>     pid: 14587
>>     rank: -1
>>     used: 199400
>>     free: 3722868
>>     real_used: 471436
>> }
>> {
>>     entry: 18
>>     pid: 14588
>>     rank: -1
>>     used: 2105844
>>     free: 3722868
>>     real_used: 2337944
>> }
>> {
>>     entry: 19
>>     pid: 14589
>>     rank: -2
>>     used: 198552
>>     free: 3723236
>>     real_used: 471068
>> }
>> {
>>     entry: 20
>>     pid: 0
>>     rank: 0
>>     used: 209756
>>     free: 3712368
>>     real_used: 481936
>> }
>> {
>>     entry: 21
>>     pid: 14591
>>     rank: -4
>>     used: 253660
>>     free: 3668704
>>     real_used: 525600
>> }
>>
>>
>>
>>
>> On 1 October 2013 22:45, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>
>>>  Hello,
>>>
>>> send SIGUSR1 to one of the worker processes (e.g., UDP listener). The
>>> main process of kamailio doesn't process SIP messages, so it is not exposed
>>> to leaks.
>>>
>>> You can used 'kamcmd pkg.stats" to see statistics about usage of pkg
>>> per each process (older versions have sercmd).
>>>
>>> 'kamctl ps' gives the list of processes.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 10/1/13 2:35 PM, David Cunningham wrote:
>>>
>>>  Hi Daniel,
>>>
>>>  What I copied and pasted is all that's listed when I do a kill -SIGUSR1
>>> on the Kamailio master process. How do I get fuller logs?
>>>
>>>  Thanks for your help.
>>>
>>>
>>>
>>> On 1 October 2013 21:57, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>>
>>>> Hello,
>>>>
>>>> you gave only the first part of memory log output. The chunks listed
>>>> there are from startup, thus not part of a leak, because they are not
>>>> allocated again at runtime.
>>>>
>>>> Can you give the full logs with messages from qm_status? The leaks are
>>>> typically listed towards the end of those messages.
>>>>
>>>> Yesterday someone else reported a potential leak when using perl
>>>> module, so it might be something else there. I found one, which is related
>>>> to an error case, but perhaps you don't get that, otherwise you will notice
>>>> the error message.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 10/1/13 12:03 PM, David Cunningham wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Following up on this after a long break. It seems we are still having
>>>>> a memory leak. We've added the memory debugging options as per the
>>>>> documentation with the following output.
>>>>> I'm not sure how to read it though - can anyone help? Thank you.
>>>>>
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
>>>>> [main.c:754]: Memory status (pkg):
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>> (0x4019b008):
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>> heap size= 4194304
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>> used= 191100, used+overhead=462800, free=3731504
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>> max used (+overhead)= 480224
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>> dumping all alloc'ed. fragments:
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    0. N  address=0x401cf414 frag=0x401cf3fc size=512 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    1. N  address=0x401cf644 frag=0x401cf62c size=128 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    2. N  address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: counters.c: cnt_hash_add(331)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    3. N  address=0x401cf824 frag=0x401cf80c size=64 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: counters.c: init_counters(121)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    4. N  address=0x401cf894 frag=0x401cf87c size=8 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: route.c: init_rlist(180)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    5. N  address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    6. N  address=0x401cf93c frag=0x401cf924 size=24 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: route.c: route_add(159)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    7. N  address=0x401cf984 frag=0x401cf96c size=8 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: route.c: init_rlist(180)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    8. N  address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>    9. N  address=0x401cfa2c frag=0x401cfa14 size=24 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: route.c: route_add(159)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>   10. N  address=0x401cfa74 frag=0x401cfa5c size=8 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: db_id.c: dupl_string(50)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>   11. N  address=0x401cfaac frag=0x401cfa94 size=64 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>   12. N  address=0x401cfb1c frag=0x401cfb04 size=24 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: route.c: route_add(159)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>   13. N  address=0x401cfb64 frag=0x401cfb4c size=8 used=1
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>         alloc'd from <core>: route.c: init_rlist(180)
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>        start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>>> Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>>>   14. N  address=0x401cfb9c frag=0x401cfb84 size=64 used=1
>>>>>
>>>>>
>>>>>
>>>>   --
>>>> Daniel-Constantin Mierla - http://www.asipto.com
>>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>  Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>>>>   - more details about Kamailio trainings at http://www.asipto.com -
>>>>
>>>>
>>>
>>>
>>> --
>>> David Cunningham, Voisonics
>>> http://voisonics.com/
>>> USA: +1 213 221 1092
>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>>>   - more details about Kamailio trainings at http://www.asipto.com -
>>>
>>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>>   - more details about Kamailio trainings at http://www.asipto.com -
>>
>>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>   - more details about Kamailio trainings at http://www.asipto.com -
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>   - more details about Kamailio trainings at http://www.asipto.com -
>
>


-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131002/37de947f/attachment-0001.html>


More information about the sr-users mailing list