[SR-Users] Memory leak in 3.3.4
Daniel-Constantin Mierla
miconda at gmail.com
Wed Oct 2 10:39:34 CEST 2013
Hello,
on a second check, those counter chunks for counters/statistics seems to
be from startup, thus not part of a leak.
How long have you been running kamailio? Looking at the pkg stats from
previous email, the free part is still a lot, with only about 200k used.
It could mean the leak was not making any effect yet.
Cheers,
Daniel
On 10/2/13 10:19 AM, Daniel-Constantin Mierla wrote:
> Hello,
>
> the leak is due to keep defining counters or statistics. That should
> happen only at startup, but if you execute from perl script some
> function from a kamailio module, it may trigger this issue. Can you
> share what functions from kamailio are you execute inside the perl script?
>
> Cheers,
> Daniel
>
> On 10/2/13 9:26 AM, David Cunningham wrote:
>> Hi Daniel,
>>
>> Thank you - I used the restart method and got a whole load of memory
>> information. 10 megabytes of it! I've compressed that and attached it
>> to this email.
>>
>> Thanks for your help.
>>
>>
>>
>> On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda at gmail.com
>> <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>> ahh, right, forgot that it was disabled because it was not safe
>> to use at runtime for worker processes. You have to use:
>>
>> kamcmd cfg.set_now_int core mem_dump_pkg <pid>
>>
>> then wait for a bit so that process is receiving a SIP message.
>>
>> Or, if you restart, the stats are printed in syslog for all
>> processes.
>>
>> Another alternative is to call pkg_status() or pkg_summary() from
>> cfgutils module in config file, like:
>>
>> if(src_ip==127.0.0.1 && is_method("OPTIONS") && $rU=="pkgstats") {
>> send_reply("200", "OK");
>> pkg_summary();
>> exit;
>> }
>>
>> You can use sipsak to send an options request to "sip:pkgstats at ..."
>>
>> Cheers,
>> Daniel
>>
>>
>> On 10/2/13 2:30 AM, David Cunningham wrote:
>>> Hi Daniel,
>>>
>>> When I send SIGUSR1 to a worker process nothing is logged. Any
>>> ideas there? The following is the output of "sercmd pkg.stats".
>>>
>>> {
>>> entry: 0
>>> pid: 14570
>>> rank: 0
>>> used: 191100
>>> free: 3731504
>>> real_used: 462800
>>> }
>>> {
>>> entry: 1
>>> pid: 14571
>>> rank: 1
>>> used: 202656
>>> free: 3714236
>>> real_used: 480068
>>> }
>>> {
>>> entry: 2
>>> pid: 14572
>>> rank: 2
>>> used: 202720
>>> free: 3714220
>>> real_used: 480084
>>> }
>>> {
>>> entry: 3
>>> pid: 14573
>>> rank: 3
>>> used: 202664
>>> free: 3714324
>>> real_used: 479980
>>> }
>>> {
>>> entry: 4
>>> pid: 14574
>>> rank: 4
>>> used: 202744
>>> free: 3714100
>>> real_used: 480204
>>> }
>>> {
>>> entry: 5
>>> pid: 14575
>>> rank: 5
>>> used: 201908
>>> free: 3715176
>>> real_used: 479128
>>> }
>>> {
>>> entry: 6
>>> pid: 14576
>>> rank: 6
>>> used: 202572
>>> free: 3714368
>>> real_used: 479936
>>> }
>>> {
>>> entry: 7
>>> pid: 14577
>>> rank: 7
>>> used: 202740
>>> free: 3714488
>>> real_used: 479816
>>> }
>>> {
>>> entry: 8
>>> pid: 14578
>>> rank: 8
>>> used: 199760
>>> free: 3717084
>>> real_used: 477220
>>> }
>>> {
>>> entry: 9
>>> pid: 14579
>>> rank: 9
>>> used: 199440
>>> free: 3722828
>>> real_used: 471476
>>> }
>>> {
>>> entry: 10
>>> pid: 14580
>>> rank: 10
>>> used: 199440
>>> free: 3722828
>>> real_used: 471476
>>> }
>>> {
>>> entry: 11
>>> pid: 14581
>>> rank: 11
>>> used: 199440
>>> free: 3722828
>>> real_used: 471476
>>> }
>>> {
>>> entry: 12
>>> pid: 14582
>>> rank: 12
>>> used: 199440
>>> free: 3722828
>>> real_used: 471476
>>> }
>>> {
>>> entry: 13
>>> pid: 14583
>>> rank: 13
>>> used: 199440
>>> free: 3722828
>>> real_used: 471476
>>> }
>>> {
>>> entry: 14
>>> pid: 14584
>>> rank: 14
>>> used: 199440
>>> free: 3722828
>>> real_used: 471476
>>> }
>>> {
>>> entry: 15
>>> pid: 14585
>>> rank: 15
>>> used: 199440
>>> free: 3722828
>>> real_used: 471476
>>> }
>>> {
>>> entry: 16
>>> pid: 14586
>>> rank: 16
>>> used: 199440
>>> free: 3722828
>>> real_used: 471476
>>> }
>>> {
>>> entry: 17
>>> pid: 14587
>>> rank: -1
>>> used: 199400
>>> free: 3722868
>>> real_used: 471436
>>> }
>>> {
>>> entry: 18
>>> pid: 14588
>>> rank: -1
>>> used: 2105844
>>> free: 3722868
>>> real_used: 2337944
>>> }
>>> {
>>> entry: 19
>>> pid: 14589
>>> rank: -2
>>> used: 198552
>>> free: 3723236
>>> real_used: 471068
>>> }
>>> {
>>> entry: 20
>>> pid: 0
>>> rank: 0
>>> used: 209756
>>> free: 3712368
>>> real_used: 481936
>>> }
>>> {
>>> entry: 21
>>> pid: 14591
>>> rank: -4
>>> used: 253660
>>> free: 3668704
>>> real_used: 525600
>>> }
>>>
>>>
>>>
>>>
>>> On 1 October 2013 22:45, Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> send SIGUSR1 to one of the worker processes (e.g., UDP
>>> listener). The main process of kamailio doesn't process SIP
>>> messages, so it is not exposed to leaks.
>>>
>>> You can used 'kamcmd pkg.stats" to see statistics about
>>> usage of pkg per each process (older versions have sercmd).
>>>
>>> 'kamctl ps' gives the list of processes.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 10/1/13 2:35 PM, David Cunningham wrote:
>>>> Hi Daniel,
>>>>
>>>> What I copied and pasted is all that's listed when I do a
>>>> kill -SIGUSR1 on the Kamailio master process. How do I get
>>>> fuller logs?
>>>>
>>>> Thanks for your help.
>>>>
>>>>
>>>>
>>>> On 1 October 2013 21:57, Daniel-Constantin Mierla
>>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> you gave only the first part of memory log output. The
>>>> chunks listed there are from startup, thus not part of
>>>> a leak, because they are not allocated again at runtime.
>>>>
>>>> Can you give the full logs with messages from
>>>> qm_status? The leaks are typically listed towards the
>>>> end of those messages.
>>>>
>>>> Yesterday someone else reported a potential leak when
>>>> using perl module, so it might be something else there.
>>>> I found one, which is related to an error case, but
>>>> perhaps you don't get that, otherwise you will notice
>>>> the error message.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 10/1/13 12:03 PM, David Cunningham wrote:
>>>>
>>>> Hello,
>>>>
>>>> Following up on this after a long break. It seems
>>>> we are still having a memory leak. We've added the
>>>> memory debugging options as per the documentation
>>>> with the following output.
>>>> I'm not sure how to read it though - can anyone
>>>> help? Thank you.
>>>>
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: <core> [main.c:754]: Memory status (pkg):
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: (0x4019b008):
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: heap size= 4194304
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: used= 191100,
>>>> used+overhead=462800, free=3731504
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: max used (+overhead)= 480224
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: dumping all alloc'ed. fragments:
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 0. N address=0x401cf414
>>>> frag=0x401cf3fc size=512 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 1. N address=0x401cf644
>>>> frag=0x401cf62c size=128 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 2. N address=0x401cf6f4
>>>> frag=0x401cf6dc size=256 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> counters.c: cnt_hash_add(331)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 3. N address=0x401cf824
>>>> frag=0x401cf80c size=64 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> counters.c: init_counters(121)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 4. N address=0x401cf894
>>>> frag=0x401cf87c size=8 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> route.c: init_rlist(180)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 5. N address=0x401cf8cc
>>>> frag=0x401cf8b4 size=64 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 6. N address=0x401cf93c
>>>> frag=0x401cf924 size=24 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> route.c: route_add(159)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 7. N address=0x401cf984
>>>> frag=0x401cf96c size=8 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> route.c: init_rlist(180)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 8. N address=0x401cf9bc
>>>> frag=0x401cf9a4 size=64 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 9. N address=0x401cfa2c
>>>> frag=0x401cfa14 size=24 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> route.c: route_add(159)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 10. N address=0x401cfa74
>>>> frag=0x401cfa5c size=8 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> db_id.c: dupl_string(50)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 11. N address=0x401cfaac
>>>> frag=0x401cfa94 size=64 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 12. N address=0x401cfb1c
>>>> frag=0x401cfb04 size=24 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> route.c: route_add(159)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 13. N address=0x401cfb64
>>>> frag=0x401cfb4c size=8 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: alloc'd from <core>:
>>>> route.c: init_rlist(180)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: start check=f0f0f0f0, end
>>>> check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>>> NOTICE: qm_status: 14. N address=0x401cfb9c
>>>> frag=0x401cfb84 size=64 used=1
>>>>
>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla - http://www.asipto.com
>>>> http://twitter.com/#!/miconda
>>>> <http://twitter.com/#%21/miconda> -
>>>> http://www.linkedin.com/in/miconda
>>>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
>>>> Nov 18-20, 2013
>>>> - more details about Kamailio trainings at
>>>> http://www.asipto.com -
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> David Cunningham, Voisonics
>>>> http://voisonics.com/
>>>> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>>> UK: +44 (0) 20 3298 1642
>>>> <tel:%2B44%20%280%29%2020%203298%201642>
>>>> Australia: +61 (0) 2 8063 9019
>>>> <tel:%2B61%20%280%29%202%208063%209019>
>>>
>>> --
>>> Daniel-Constantin Mierla -http://www.asipto.com
>>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>>> - more details about Kamailio trainings athttp://www.asipto.com -
>>>
>>>
>>>
>>>
>>> --
>>> David Cunningham, Voisonics
>>> http://voisonics.com/
>>> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>> UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
>>> Australia: +61 (0) 2 8063 9019
>>> <tel:%2B61%20%280%29%202%208063%209019>
>>
>> --
>> Daniel-Constantin Mierla -http://www.asipto.com
>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>> - more details about Kamailio trainings athttp://www.asipto.com -
>>
>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092
>> UK: +44 (0) 20 3298 1642
>> Australia: +61 (0) 2 8063 9019
>
> --
> Daniel-Constantin Mierla -http://www.asipto.com
> http://twitter.com/#!/miconda -http://www.linkedin.com/in/miconda
> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
> - more details about Kamailio trainings athttp://www.asipto.com -
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131002/629ab041/attachment-0001.html>
More information about the sr-users
mailing list