[SR-Users] Memory leak in 3.3.4
David Cunningham
dcunningham at voisonics.com
Wed Oct 2 09:26:55 CEST 2013
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory
information. 10 megabytes of it! I've compressed that and attached it to
this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda at gmail.com> wrote:
> Hello,
>
> ahh, right, forgot that it was disabled because it was not safe to use at
> runtime for worker processes. You have to use:
>
> kamcmd cfg.set_now_int core mem_dump_pkg <pid>
>
> then wait for a bit so that process is receiving a SIP message.
>
> Or, if you restart, the stats are printed in syslog for all processes.
>
> Another alternative is to call pkg_status() or pkg_summary() from cfgutils
> module in config file, like:
>
> if(src_ip==127.0.0.1 && is_method("OPTIONS") && $rU=="pkgstats") {
> send_reply("200", "OK");
> pkg_summary();
> exit;
> }
>
> You can use sipsak to send an options request to "sip:pkgstats at ..."
>
> Cheers,
> Daniel
>
>
> On 10/2/13 2:30 AM, David Cunningham wrote:
>
> Hi Daniel,
>
> When I send SIGUSR1 to a worker process nothing is logged. Any ideas
> there? The following is the output of "sercmd pkg.stats".
>
> {
> entry: 0
> pid: 14570
> rank: 0
> used: 191100
> free: 3731504
> real_used: 462800
> }
> {
> entry: 1
> pid: 14571
> rank: 1
> used: 202656
> free: 3714236
> real_used: 480068
> }
> {
> entry: 2
> pid: 14572
> rank: 2
> used: 202720
> free: 3714220
> real_used: 480084
> }
> {
> entry: 3
> pid: 14573
> rank: 3
> used: 202664
> free: 3714324
> real_used: 479980
> }
> {
> entry: 4
> pid: 14574
> rank: 4
> used: 202744
> free: 3714100
> real_used: 480204
> }
> {
> entry: 5
> pid: 14575
> rank: 5
> used: 201908
> free: 3715176
> real_used: 479128
> }
> {
> entry: 6
> pid: 14576
> rank: 6
> used: 202572
> free: 3714368
> real_used: 479936
> }
> {
> entry: 7
> pid: 14577
> rank: 7
> used: 202740
> free: 3714488
> real_used: 479816
> }
> {
> entry: 8
> pid: 14578
> rank: 8
> used: 199760
> free: 3717084
> real_used: 477220
> }
> {
> entry: 9
> pid: 14579
> rank: 9
> used: 199440
> free: 3722828
> real_used: 471476
> }
> {
> entry: 10
> pid: 14580
> rank: 10
> used: 199440
> free: 3722828
> real_used: 471476
> }
> {
> entry: 11
> pid: 14581
> rank: 11
> used: 199440
> free: 3722828
> real_used: 471476
> }
> {
> entry: 12
> pid: 14582
> rank: 12
> used: 199440
> free: 3722828
> real_used: 471476
> }
> {
> entry: 13
> pid: 14583
> rank: 13
> used: 199440
> free: 3722828
> real_used: 471476
> }
> {
> entry: 14
> pid: 14584
> rank: 14
> used: 199440
> free: 3722828
> real_used: 471476
> }
> {
> entry: 15
> pid: 14585
> rank: 15
> used: 199440
> free: 3722828
> real_used: 471476
> }
> {
> entry: 16
> pid: 14586
> rank: 16
> used: 199440
> free: 3722828
> real_used: 471476
> }
> {
> entry: 17
> pid: 14587
> rank: -1
> used: 199400
> free: 3722868
> real_used: 471436
> }
> {
> entry: 18
> pid: 14588
> rank: -1
> used: 2105844
> free: 3722868
> real_used: 2337944
> }
> {
> entry: 19
> pid: 14589
> rank: -2
> used: 198552
> free: 3723236
> real_used: 471068
> }
> {
> entry: 20
> pid: 0
> rank: 0
> used: 209756
> free: 3712368
> real_used: 481936
> }
> {
> entry: 21
> pid: 14591
> rank: -4
> used: 253660
> free: 3668704
> real_used: 525600
> }
>
>
>
>
> On 1 October 2013 22:45, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>
>> Hello,
>>
>> send SIGUSR1 to one of the worker processes (e.g., UDP listener). The
>> main process of kamailio doesn't process SIP messages, so it is not exposed
>> to leaks.
>>
>> You can used 'kamcmd pkg.stats" to see statistics about usage of pkg per
>> each process (older versions have sercmd).
>>
>> 'kamctl ps' gives the list of processes.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 10/1/13 2:35 PM, David Cunningham wrote:
>>
>> Hi Daniel,
>>
>> What I copied and pasted is all that's listed when I do a kill -SIGUSR1
>> on the Kamailio master process. How do I get fuller logs?
>>
>> Thanks for your help.
>>
>>
>>
>> On 1 October 2013 21:57, Daniel-Constantin Mierla <miconda at gmail.com>wrote:
>>
>>> Hello,
>>>
>>> you gave only the first part of memory log output. The chunks listed
>>> there are from startup, thus not part of a leak, because they are not
>>> allocated again at runtime.
>>>
>>> Can you give the full logs with messages from qm_status? The leaks are
>>> typically listed towards the end of those messages.
>>>
>>> Yesterday someone else reported a potential leak when using perl module,
>>> so it might be something else there. I found one, which is related to an
>>> error case, but perhaps you don't get that, otherwise you will notice the
>>> error message.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 10/1/13 12:03 PM, David Cunningham wrote:
>>>
>>>> Hello,
>>>>
>>>> Following up on this after a long break. It seems we are still having a
>>>> memory leak. We've added the memory debugging options as per the
>>>> documentation with the following output.
>>>> I'm not sure how to read it though - can anyone help? Thank you.
>>>>
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
>>>> [main.c:754]: Memory status (pkg):
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> (0x4019b008):
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> heap size= 4194304
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> used= 191100, used+overhead=462800, free=3731504
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max
>>>> used (+overhead)= 480224
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> dumping all alloc'ed. fragments:
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 1. N address=0x401cf644 frag=0x401cf62c size=128 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: counters.c: cnt_hash_add(331)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 3. N address=0x401cf824 frag=0x401cf80c size=64 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: counters.c: init_counters(121)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 4. N address=0x401cf894 frag=0x401cf87c size=8 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: route.c: init_rlist(180)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 6. N address=0x401cf93c frag=0x401cf924 size=24 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: route.c: route_add(159)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 7. N address=0x401cf984 frag=0x401cf96c size=8 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: route.c: init_rlist(180)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: route.c: route_add(159)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: db_id.c: dupl_string(50)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: route.c: route_add(159)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> alloc'd from <core>: route.c: init_rlist(180)
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
>>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
>>>> 14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1
>>>>
>>>>
>>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.com
>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>>> - more details about Kamailio trainings at http://www.asipto.com -
>>>
>>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>> - more details about Kamailio trainings at http://www.asipto.com -
>>
>>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
> - more details about Kamailio trainings at http://www.asipto.com -
>
>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131002/05d4ea2c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kamailio-memory.txt.gz
Type: application/x-gzip
Size: 290108 bytes
Desc: not available
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131002/05d4ea2c/attachment-0001.bin>
More information about the sr-users
mailing list