Re: [SR-Users] Memory leak in 3.3.4

2 Oct 2013


      Hi Daniel
Just want to make you aware of something on kamailio 4.0.3 .
I loaded cfgutils and used pkg_summary() . But I run into the following:
0(14112) NOTICE: <core> [mem/q_malloc.c:437]: qm_free(): qm_free(0x7fbad3882010, 0x7fbad3ab7220), called from <core>: rpc_lookup.c: destroy_rpcs(79)
 0(14112) NOTICE: <core> [mem/q_malloc.c:472]: qm_free(): qm_free: freeing frag. 0x7fbad3ab71f0 alloc'ed from <core>: rpc_lookup.c: rpc_hash_add(156)
 0(14112) NOTICE: <core> [mem/q_malloc.c:437]: qm_free(): qm_free(0x7fba8923e000, 0x7fba894e62b0), called from cfgutils: cfgutils.c: mod_destroy(806)
 0(14112) : <core> [mem/q_malloc.c:466]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7fba894e62b0), called from cfgutils: cfgutils.c: mod_destroy(806), first free cfgutils: cfgutils.c: mod_destroy(806) - aborting
Regards,
Dragos
________________________________
 From: David Cunningham dcunningham@voisonics.com
To: Daniel-Constantin Mierla miconda@gmail.com 
Cc: Kamailio (SER) - Users Mailing List sr-users@lists.sip-router.org 
Sent: Wednesday, October 2, 2013 9:26 AM
Subject: Re: [SR-Users] Memory leak in 3.3.4
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory information. 10 megabytes of it! I've compressed that and attached it to this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
...
ahh, right, forgot that it was disabled because it was not safe to
use at runtime for worker processes. You have to use:
...
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
then wait for a bit so that process is receiving a SIP message.
Or, if you restart, the stats are printed in syslog for all
processes.
...
Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:
...
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
...
send_reply("200", "OK");
  pkg_summary();
  exit;
}
You can use sipsak to send an options request to "sip:pkgstats@..."
Cheers,
Daniel
On 10/2/13 2:30 AM, David Cunningham wrote:
Hi Daniel,
...
When I send SIGUSR1 to a worker process nothing is logged. Any
ideas there? The following is the output of "sercmd pkg.stats".
...
...
{
    entry: 0
    pid: 14570
    rank: 0
    used: 191100
    free: 3731504
    real_used: 462800
}
{
    entry: 1
    pid: 14571
    rank: 1
    used: 202656
    free: 3714236
    real_used: 480068
}
{
    entry: 2
    pid: 14572
    rank: 2
    used: 202720
    free: 3714220
    real_used: 480084
}
{
    entry: 3
    pid: 14573
    rank: 3
    used: 202664
    free: 3714324
    real_used: 479980
}
{
    entry: 4
    pid: 14574
    rank: 4
    used: 202744
    free: 3714100
    real_used: 480204
}
{
    entry: 5
    pid: 14575
    rank: 5
    used: 201908
    free: 3715176
    real_used: 479128
}
{
    entry: 6
    pid: 14576
    rank: 6
    used: 202572
    free: 3714368
    real_used: 479936
}
{
    entry: 7
    pid: 14577
    rank: 7
    used: 202740
    free: 3714488
    real_used: 479816
}
{
    entry: 8
    pid: 14578
    rank: 8
    used: 199760
    free: 3717084
    real_used: 477220
}
{
    entry: 9
    pid: 14579
    rank: 9
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 10
    pid: 14580
    rank: 10
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 11
    pid: 14581
    rank: 11
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 12
    pid: 14582
    rank: 12
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 13
    pid: 14583
    rank: 13
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 14
    pid: 14584
    rank: 14
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 15
    pid: 14585
    rank: 15
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 16
    pid: 14586
    rank: 16
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 17
    pid: 14587
    rank: -1
    used: 199400
    free: 3722868
    real_used: 471436
}
{
    entry: 18
    pid: 14588
    rank: -1
    used: 2105844
    free: 3722868
    real_used: 2337944
}
{
    entry: 19
    pid: 14589
    rank: -2
    used: 198552
    free: 3723236
    real_used: 471068
}
{
    entry: 20
    pid: 0
    rank: 0
    used: 209756
    free: 3712368
    real_used: 481936
}
{
    entry: 21
    pid: 14591
    rank: -4
    used: 253660
    free: 3668704
    real_used: 525600
}
On 1 October 2013 22:45, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
...
send SIGUSR1 to one of the worker processes (e.g., UDP
listener). The main process of kamailio doesn't process
              SIP messages, so it is not exposed to leaks.
...
...
...
You can used 'kamcmd pkg.stats" to see statistics about
usage of pkg  per each process (older versions have
              sercmd).
...
...
...
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
Hi Daniel,
...
What I copied and pasted is all that's listed when I do a kill -SIGUSR1 on the Kamailio master process. How do I get fuller logs?
...
...
...
...
Thanks for your help.
...
...
...
...
On 1 October 2013 21:57, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
...
you gave only the first part of memory log
output. The chunks listed there are from
                          startup, thus not part of a leak, because they
                          are not allocated again at runtime.
...
...
...
...
...
Can you give the full logs with messages from
qm_status? The leaks are typically listed
                          towards the end of those messages.
...
...
...
...
...
Yesterday someone else reported a potential
leak when using perl module, so it might be
                          something else there. I found one, which is
                          related to an error case, but perhaps you
                          don't get that, otherwise you will notice the
                          error message.
...
...
...
...
...
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham
wrote:
...
...
...
...
...
Hello,
...
Following up on this after a long break.
It seems we are still having a memory
                                leak. We've added the memory debugging
                                options as per the documentation with
                                the following output.
...
...
...
...
...
...
I'm not sure how to read it though - can
anyone help? Thank you.
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                <core> [main.c:754]: Memory status
                                (pkg):
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status: (0x4019b008):
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status: heap size= 4194304
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status: used= 191100,
                                used+overhead=462800, free=3731504
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status: max used (+overhead)= 480224
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status: dumping all alloc'ed.
                                fragments:
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      0. N  address=0x401cf414
                                frag=0x401cf3fc size=512 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: str_hash.h:
                                str_hash_alloc(69)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      1. N  address=0x401cf644
                                frag=0x401cf62c size=128 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: str_hash.h:
                                str_hash_alloc(69)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      2. N  address=0x401cf6f4
                                frag=0x401cf6dc size=256 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: counters.c:
                                cnt_hash_add(331)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      3. N  address=0x401cf824
                                frag=0x401cf80c size=64 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: counters.c:
                                init_counters(121)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      4. N  address=0x401cf894
                                frag=0x401cf87c size=8 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: route.c: init_rlist(180)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      5. N  address=0x401cf8cc
                                frag=0x401cf8b4 size=64 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: str_hash.h:
                                str_hash_alloc(69)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      6. N  address=0x401cf93c
                                frag=0x401cf924 size=24 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: route.c: route_add(159)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      7. N  address=0x401cf984
                                frag=0x401cf96c size=8 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: route.c: init_rlist(180)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      8. N  address=0x401cf9bc
                                frag=0x401cf9a4 size=64 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: str_hash.h:
                                str_hash_alloc(69)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:      9. N  address=0x401cfa2c
                                frag=0x401cfa14 size=24 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: route.c: route_add(159)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:     10. N  address=0x401cfa74
                                frag=0x401cfa5c size=8 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: db_id.c: dupl_string(50)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:     11. N  address=0x401cfaac
                                frag=0x401cfa94 size=64 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: str_hash.h:
                                str_hash_alloc(69)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:     12. N  address=0x401cfb1c
                                frag=0x401cfb04 size=24 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: route.c: route_add(159)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:     13. N  address=0x401cfb64
                                frag=0x401cfb4c size=8 used=1
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:           alloc'd from
                                <core>: route.c: init_rlist(180)
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:          start
                                check=f0f0f0f0, end check= c0c0c0c0,
                                abcdefed
...
...
...
...
...
...
Oct  1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
                                qm_status:     14. N  address=0x401cfb9c
                                frag=0x401cfb84 size=64 used=1
...
...
...
...
...
...
-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
...
...
...
...
...
- more details about Kamailio trainings at http://www.asipto.com -
-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013 - more details about Kamailio trainings at http://www.asipto.com -
...
...
-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013 - more details about Kamailio trainings at http://www.asipto.com -
-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Re: [SR-Users] Memory leak in 3.3.4