[SR-Users] Memory leak in 3.3.4
Daniel-Constantin Mierla
miconda at gmail.com
Wed Oct 2 10:19:44 CEST 2013
Hello,
the leak is due to keep defining counters or statistics. That should
happen only at startup, but if you execute from perl script some
function from a kamailio module, it may trigger this issue. Can you
share what functions from kamailio are you execute inside the perl script?
Cheers,
Daniel
On 10/2/13 9:26 AM, David Cunningham wrote:
> Hi Daniel,
>
> Thank you - I used the restart method and got a whole load of memory
> information. 10 megabytes of it! I've compressed that and attached it
> to this email.
>
> Thanks for your help.
>
>
>
> On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda at gmail.com
> <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
> ahh, right, forgot that it was disabled because it was not safe to
> use at runtime for worker processes. You have to use:
>
> kamcmd cfg.set_now_int core mem_dump_pkg <pid>
>
> then wait for a bit so that process is receiving a SIP message.
>
> Or, if you restart, the stats are printed in syslog for all processes.
>
> Another alternative is to call pkg_status() or pkg_summary() from
> cfgutils module in config file, like:
>
> if(src_ip==127.0.0.1 && is_method("OPTIONS") && $rU=="pkgstats") {
> send_reply("200", "OK");
> pkg_summary();
> exit;
> }
>
> You can use sipsak to send an options request to "sip:pkgstats at ..."
>
> Cheers,
> Daniel
>
>
> On 10/2/13 2:30 AM, David Cunningham wrote:
>> Hi Daniel,
>>
>> When I send SIGUSR1 to a worker process nothing is logged. Any
>> ideas there? The following is the output of "sercmd pkg.stats".
>>
>> {
>> entry: 0
>> pid: 14570
>> rank: 0
>> used: 191100
>> free: 3731504
>> real_used: 462800
>> }
>> {
>> entry: 1
>> pid: 14571
>> rank: 1
>> used: 202656
>> free: 3714236
>> real_used: 480068
>> }
>> {
>> entry: 2
>> pid: 14572
>> rank: 2
>> used: 202720
>> free: 3714220
>> real_used: 480084
>> }
>> {
>> entry: 3
>> pid: 14573
>> rank: 3
>> used: 202664
>> free: 3714324
>> real_used: 479980
>> }
>> {
>> entry: 4
>> pid: 14574
>> rank: 4
>> used: 202744
>> free: 3714100
>> real_used: 480204
>> }
>> {
>> entry: 5
>> pid: 14575
>> rank: 5
>> used: 201908
>> free: 3715176
>> real_used: 479128
>> }
>> {
>> entry: 6
>> pid: 14576
>> rank: 6
>> used: 202572
>> free: 3714368
>> real_used: 479936
>> }
>> {
>> entry: 7
>> pid: 14577
>> rank: 7
>> used: 202740
>> free: 3714488
>> real_used: 479816
>> }
>> {
>> entry: 8
>> pid: 14578
>> rank: 8
>> used: 199760
>> free: 3717084
>> real_used: 477220
>> }
>> {
>> entry: 9
>> pid: 14579
>> rank: 9
>> used: 199440
>> free: 3722828
>> real_used: 471476
>> }
>> {
>> entry: 10
>> pid: 14580
>> rank: 10
>> used: 199440
>> free: 3722828
>> real_used: 471476
>> }
>> {
>> entry: 11
>> pid: 14581
>> rank: 11
>> used: 199440
>> free: 3722828
>> real_used: 471476
>> }
>> {
>> entry: 12
>> pid: 14582
>> rank: 12
>> used: 199440
>> free: 3722828
>> real_used: 471476
>> }
>> {
>> entry: 13
>> pid: 14583
>> rank: 13
>> used: 199440
>> free: 3722828
>> real_used: 471476
>> }
>> {
>> entry: 14
>> pid: 14584
>> rank: 14
>> used: 199440
>> free: 3722828
>> real_used: 471476
>> }
>> {
>> entry: 15
>> pid: 14585
>> rank: 15
>> used: 199440
>> free: 3722828
>> real_used: 471476
>> }
>> {
>> entry: 16
>> pid: 14586
>> rank: 16
>> used: 199440
>> free: 3722828
>> real_used: 471476
>> }
>> {
>> entry: 17
>> pid: 14587
>> rank: -1
>> used: 199400
>> free: 3722868
>> real_used: 471436
>> }
>> {
>> entry: 18
>> pid: 14588
>> rank: -1
>> used: 2105844
>> free: 3722868
>> real_used: 2337944
>> }
>> {
>> entry: 19
>> pid: 14589
>> rank: -2
>> used: 198552
>> free: 3723236
>> real_used: 471068
>> }
>> {
>> entry: 20
>> pid: 0
>> rank: 0
>> used: 209756
>> free: 3712368
>> real_used: 481936
>> }
>> {
>> entry: 21
>> pid: 14591
>> rank: -4
>> used: 253660
>> free: 3668704
>> real_used: 525600
>> }
>>
>>
>>
>>
>> On 1 October 2013 22:45, Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>> send SIGUSR1 to one of the worker processes (e.g., UDP
>> listener). The main process of kamailio doesn't process SIP
>> messages, so it is not exposed to leaks.
>>
>> You can used 'kamcmd pkg.stats" to see statistics about usage
>> of pkg per each process (older versions have sercmd).
>>
>> 'kamctl ps' gives the list of processes.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 10/1/13 2:35 PM, David Cunningham wrote:
>>> Hi Daniel,
>>>
>>> What I copied and pasted is all that's listed when I do a
>>> kill -SIGUSR1 on the Kamailio master process. How do I get
>>> fuller logs?
>>>
>>> Thanks for your help.
>>>
>>>
>>>
>>> On 1 October 2013 21:57, Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> you gave only the first part of memory log output. The
>>> chunks listed there are from startup, thus not part of a
>>> leak, because they are not allocated again at runtime.
>>>
>>> Can you give the full logs with messages from qm_status?
>>> The leaks are typically listed towards the end of those
>>> messages.
>>>
>>> Yesterday someone else reported a potential leak when
>>> using perl module, so it might be something else there.
>>> I found one, which is related to an error case, but
>>> perhaps you don't get that, otherwise you will notice
>>> the error message.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 10/1/13 12:03 PM, David Cunningham wrote:
>>>
>>> Hello,
>>>
>>> Following up on this after a long break. It seems we
>>> are still having a memory leak. We've added the
>>> memory debugging options as per the documentation
>>> with the following output.
>>> I'm not sure how to read it though - can anyone
>>> help? Thank you.
>>>
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: <core> [main.c:754]: Memory status (pkg):
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: (0x4019b008):
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: heap size= 4194304
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: used= 191100,
>>> used+overhead=462800, free=3731504
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: max used (+overhead)= 480224
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: dumping all alloc'ed. fragments:
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 0. N address=0x401cf414
>>> frag=0x401cf3fc size=512 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>:
>>> str_hash.h: str_hash_alloc(69)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 1. N address=0x401cf644
>>> frag=0x401cf62c size=128 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>:
>>> str_hash.h: str_hash_alloc(69)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 2. N address=0x401cf6f4
>>> frag=0x401cf6dc size=256 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>:
>>> counters.c: cnt_hash_add(331)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 3. N address=0x401cf824
>>> frag=0x401cf80c size=64 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>:
>>> counters.c: init_counters(121)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 4. N address=0x401cf894
>>> frag=0x401cf87c size=8 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>: route.c:
>>> init_rlist(180)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 5. N address=0x401cf8cc
>>> frag=0x401cf8b4 size=64 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>:
>>> str_hash.h: str_hash_alloc(69)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 6. N address=0x401cf93c
>>> frag=0x401cf924 size=24 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>: route.c:
>>> route_add(159)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 7. N address=0x401cf984
>>> frag=0x401cf96c size=8 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>: route.c:
>>> init_rlist(180)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 8. N address=0x401cf9bc
>>> frag=0x401cf9a4 size=64 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>:
>>> str_hash.h: str_hash_alloc(69)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 9. N address=0x401cfa2c
>>> frag=0x401cfa14 size=24 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>: route.c:
>>> route_add(159)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 10. N address=0x401cfa74
>>> frag=0x401cfa5c size=8 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>: db_id.c:
>>> dupl_string(50)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 11. N address=0x401cfaac
>>> frag=0x401cfa94 size=64 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>:
>>> str_hash.h: str_hash_alloc(69)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 12. N address=0x401cfb1c
>>> frag=0x401cfb04 size=24 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>: route.c:
>>> route_add(159)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 13. N address=0x401cfb64
>>> frag=0x401cfb4c size=8 used=1
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: alloc'd from <core>: route.c:
>>> init_rlist(180)
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: start check=f0f0f0f0, end check=
>>> c0c0c0c0, abcdefed
>>> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
>>> NOTICE: qm_status: 14. N address=0x401cfb9c
>>> frag=0x401cfb84 size=64 used=1
>>>
>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.com
>>> http://twitter.com/#!/miconda
>>> <http://twitter.com/#%21/miconda> -
>>> http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
>>> Nov 18-20, 2013
>>> - more details about Kamailio trainings at
>>> http://www.asipto.com -
>>>
>>>
>>>
>>>
>>> --
>>> David Cunningham, Voisonics
>>> http://voisonics.com/
>>> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>>> UK: +44 (0) 20 3298 1642
>>> <tel:%2B44%20%280%29%2020%203298%201642>
>>> Australia: +61 (0) 2 8063 9019
>>> <tel:%2B61%20%280%29%202%208063%209019>
>>
>> --
>> Daniel-Constantin Mierla -http://www.asipto.com
>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
>> - more details about Kamailio trainings athttp://www.asipto.com -
>>
>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
>> UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019
>> <tel:%2B61%20%280%29%202%208063%209019>
>
> --
> Daniel-Constantin Mierla -http://www.asipto.com
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
> Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
> - more details about Kamailio trainings athttp://www.asipto.com -
>
>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131002/3e98d621/attachment-0001.html>
More information about the sr-users
mailing list