[SR-Users] Memory leak in 3.3.4
Dragos Oancea
droancea at yahoo.com
Wed Oct 2 12:04:16 CEST 2013
Hi Daniel
Just want to make you aware of something on kamailio 4.0.3 .
I loaded cfgutils and used pkg_summary() . But I run into the following:
0(14112) NOTICE: <core> [mem/q_malloc.c:437]: qm_free(): qm_free(0x7fbad3882010, 0x7fbad3ab7220), called from <core>: rpc_lookup.c: destroy_rpcs(79)
0(14112) NOTICE: <core> [mem/q_malloc.c:472]: qm_free(): qm_free: freeing frag. 0x7fbad3ab71f0 alloc'ed from <core>: rpc_lookup.c: rpc_hash_add(156)
0(14112) NOTICE: <core> [mem/q_malloc.c:437]: qm_free(): qm_free(0x7fba8923e000, 0x7fba894e62b0), called from cfgutils: cfgutils.c: mod_destroy(806)
0(14112) : <core> [mem/q_malloc.c:466]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7fba894e62b0), called from cfgutils: cfgutils.c: mod_destroy(806), first free cfgutils: cfgutils.c: mod_destroy(806) - aborting
Regards,
Dragos
________________________________
From: David Cunningham <dcunningham at voisonics.com>
To: Daniel-Constantin Mierla <miconda at gmail.com>
Cc: Kamailio (SER) - Users Mailing List <sr-users at lists.sip-router.org>
Sent: Wednesday, October 2, 2013 9:26 AM
Subject: Re: [SR-Users] Memory leak in 3.3.4
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory information. 10 megabytes of it! I've compressed that and attached it to this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda at gmail.com> wrote:
Hello,
>
>ahh, right, forgot that it was disabled because it was not safe to
use at runtime for worker processes. You have to use:
>
>kamcmd cfg.set_now_int core mem_dump_pkg <pid>
>
>then wait for a bit so that process is receiving a SIP message.
>
>Or, if you restart, the stats are printed in syslog for all
processes.
>
>Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:
>
>if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
> send_reply("200", "OK");
> pkg_summary();
> exit;
>}
>
>You can use sipsak to send an options request to "sip:pkgstats at ..."
>
>Cheers,
>Daniel
>
>
>
>On 10/2/13 2:30 AM, David Cunningham wrote:
>
>Hi Daniel,
>>
>>When I send SIGUSR1 to a worker process nothing is logged. Any
ideas there? The following is the output of "sercmd pkg.stats".
>>
>>{
>> entry: 0
>> pid: 14570
>> rank: 0
>> used: 191100
>> free: 3731504
>> real_used: 462800
>>}
>>{
>> entry: 1
>> pid: 14571
>> rank: 1
>> used: 202656
>> free: 3714236
>> real_used: 480068
>>}
>>{
>> entry: 2
>> pid: 14572
>> rank: 2
>> used: 202720
>> free: 3714220
>> real_used: 480084
>>}
>>{
>> entry: 3
>> pid: 14573
>> rank: 3
>> used: 202664
>> free: 3714324
>> real_used: 479980
>>}
>>{
>> entry: 4
>> pid: 14574
>> rank: 4
>> used: 202744
>> free: 3714100
>> real_used: 480204
>>}
>>{
>> entry: 5
>> pid: 14575
>> rank: 5
>> used: 201908
>> free: 3715176
>> real_used: 479128
>>}
>>{
>> entry: 6
>> pid: 14576
>> rank: 6
>> used: 202572
>> free: 3714368
>> real_used: 479936
>>}
>>{
>> entry: 7
>> pid: 14577
>> rank: 7
>> used: 202740
>> free: 3714488
>> real_used: 479816
>>}
>>{
>> entry: 8
>> pid: 14578
>> rank: 8
>> used: 199760
>> free: 3717084
>> real_used: 477220
>>}
>>{
>> entry: 9
>> pid: 14579
>> rank: 9
>> used: 199440
>> free: 3722828
>> real_used: 471476
>>}
>>{
>> entry: 10
>> pid: 14580
>> rank: 10
>> used: 199440
>> free: 3722828
>> real_used: 471476
>>}
>>{
>> entry: 11
>> pid: 14581
>> rank: 11
>> used: 199440
>> free: 3722828
>> real_used: 471476
>>}
>>{
>> entry: 12
>> pid: 14582
>> rank: 12
>> used: 199440
>> free: 3722828
>> real_used: 471476
>>}
>>{
>> entry: 13
>> pid: 14583
>> rank: 13
>> used: 199440
>> free: 3722828
>> real_used: 471476
>>}
>>{
>> entry: 14
>> pid: 14584
>> rank: 14
>> used: 199440
>> free: 3722828
>> real_used: 471476
>>}
>>{
>> entry: 15
>> pid: 14585
>> rank: 15
>> used: 199440
>> free: 3722828
>> real_used: 471476
>>}
>>{
>> entry: 16
>> pid: 14586
>> rank: 16
>> used: 199440
>> free: 3722828
>> real_used: 471476
>>}
>>{
>> entry: 17
>> pid: 14587
>> rank: -1
>> used: 199400
>> free: 3722868
>> real_used: 471436
>>}
>>{
>> entry: 18
>> pid: 14588
>> rank: -1
>> used: 2105844
>> free: 3722868
>> real_used: 2337944
>>}
>>{
>> entry: 19
>> pid: 14589
>> rank: -2
>> used: 198552
>> free: 3723236
>> real_used: 471068
>>}
>>{
>> entry: 20
>> pid: 0
>> rank: 0
>> used: 209756
>> free: 3712368
>> real_used: 481936
>>}
>>{
>> entry: 21
>> pid: 14591
>> rank: -4
>> used: 253660
>> free: 3668704
>> real_used: 525600
>>}
>>
>>
>>
>>
>>
>>
>>On 1 October 2013 22:45, Daniel-Constantin Mierla <miconda at gmail.com> wrote:
>>
>>Hello,
>>>
>>>send SIGUSR1 to one of the worker processes (e.g., UDP
listener). The main process of kamailio doesn't process
SIP messages, so it is not exposed to leaks.
>>>
>>>You can used 'kamcmd pkg.stats" to see statistics about
usage of pkg per each process (older versions have
sercmd).
>>>
>>>'kamctl ps' gives the list of processes.
>>>
>>>Cheers,
>>>Daniel
>>>
>>>
>>>
>>>On 10/1/13 2:35 PM, David Cunningham wrote:
>>>
>>>Hi Daniel,
>>>>
>>>>
What I copied and pasted is all that's listed when I do a kill -SIGUSR1 on the Kamailio master process. How do I get fuller logs?
>>>>
>>>>
Thanks for your help.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>On 1 October 2013 21:57, Daniel-Constantin Mierla <miconda at gmail.com> wrote:
>>>>
>>>>Hello,
>>>>>
>>>>>you gave only the first part of memory log
output. The chunks listed there are from
startup, thus not part of a leak, because they
are not allocated again at runtime.
>>>>>
>>>>>Can you give the full logs with messages from
qm_status? The leaks are typically listed
towards the end of those messages.
>>>>>
>>>>>Yesterday someone else reported a potential
leak when using perl module, so it might be
something else there. I found one, which is
related to an error case, but perhaps you
don't get that, otherwise you will notice the
error message.
>>>>>
>>>>>Cheers,
>>>>>Daniel
>>>>>
>>>>>
>>>>>On 10/1/13 12:03 PM, David Cunningham
wrote:
>>>>>
>>>>>Hello,
>>>>>>
>>>>>>Following up on this after a long break.
It seems we are still having a memory
leak. We've added the memory debugging
options as per the documentation with
the following output.
>>>>>>I'm not sure how to read it though - can
anyone help? Thank you.
>>>>>>
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
<core> [main.c:754]: Memory status
(pkg):
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: (0x4019b008):
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: heap size= 4194304
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: used= 191100,
used+overhead=462800, free=3731504
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: max used (+overhead)= 480224
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: dumping all alloc'ed.
fragments:
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 0. N address=0x401cf414
frag=0x401cf3fc size=512 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 1. N address=0x401cf644
frag=0x401cf62c size=128 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 2. N address=0x401cf6f4
frag=0x401cf6dc size=256 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: counters.c:
cnt_hash_add(331)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 3. N address=0x401cf824
frag=0x401cf80c size=64 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: counters.c:
init_counters(121)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 4. N address=0x401cf894
frag=0x401cf87c size=8 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 5. N address=0x401cf8cc
frag=0x401cf8b4 size=64 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 6. N address=0x401cf93c
frag=0x401cf924 size=24 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 7. N address=0x401cf984
frag=0x401cf96c size=8 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 8. N address=0x401cf9bc
frag=0x401cf9a4 size=64 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 9. N address=0x401cfa2c
frag=0x401cfa14 size=24 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 10. N address=0x401cfa74
frag=0x401cfa5c size=8 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: db_id.c: dupl_string(50)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 11. N address=0x401cfaac
frag=0x401cfa94 size=64 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 12. N address=0x401cfb1c
frag=0x401cfb04 size=24 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 13. N address=0x401cfb64
frag=0x401cfb4c size=8 used=1
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 14. N address=0x401cfb9c
frag=0x401cfb84 size=64 used=1
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>--
>>>>>Daniel-Constantin Mierla - http://www.asipto.com
>>>>>http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>>>>> - more details about Kamailio trainings at http://www.asipto.com -
>>>>>
>>>>>
>>>>
>>>>
>>>>--
>>>>David Cunningham, Voisonics
>>>>http://voisonics.com/
>>>>USA: +1 213 221 1092
>>>>UK: +44 (0) 20 3298 1642
>>>>Australia: +61 (0) 2 8063 9019
>>>>
>>>
>>>--
Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013 - more details about Kamailio trainings at http://www.asipto.com -
>>
>>
>>--
>>David Cunningham, Voisonics
>>http://voisonics.com/
>>USA: +1 213 221 1092
>>UK: +44 (0) 20 3298 1642
>>Australia: +61 (0) 2 8063 9019
>>
>
>--
Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013 - more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users at lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131002/558edba0/attachment-0001.html>
More information about the sr-users
mailing list