Re: [SR-Users] Memory leak in 3.3.4

2 Oct 2013

Hello,

ahh, right, forgot that it was disabled because it was not safe to use 
at runtime for worker processes. You have to use:

kamcmd cfg.set_now_int core mem_dump_pkg <pid>

then wait for a bit so that process is receiving a SIP message.

Or, if you restart, the stats are printed in syslog for all processes.

Another alternative is to call pkg_status() or pkg_summary() from 
cfgutils module in config file, like:

if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
   send_reply("200", "OK");
   pkg_summary();
   exit;
}

You can use sipsak to send an options request to "sip:pkgstats@..."

Cheers,
Daniel

On 10/2/13 2:30 AM, David Cunningham wrote:
...
  Hi Daniel,

 When I send SIGUSR1 to a worker process nothing is logged. Any ideas 
 there? The following is the output of "sercmd pkg.stats".

 {
     entry: 0
     pid: 14570
     rank: 0
     used: 191100
     free: 3731504
     real_used: 462800
 }
 {
     entry: 1
     pid: 14571
     rank: 1
     used: 202656
     free: 3714236
     real_used: 480068
 }
 {
     entry: 2
     pid: 14572
     rank: 2
     used: 202720
     free: 3714220
     real_used: 480084
 }
 {
     entry: 3
     pid: 14573
     rank: 3
     used: 202664
     free: 3714324
     real_used: 479980
 }
 {
     entry: 4
     pid: 14574
     rank: 4
     used: 202744
     free: 3714100
     real_used: 480204
 }
 {
     entry: 5
     pid: 14575
     rank: 5
     used: 201908
     free: 3715176
     real_used: 479128
 }
 {
     entry: 6
     pid: 14576
     rank: 6
     used: 202572
     free: 3714368
     real_used: 479936
 }
 {
     entry: 7
     pid: 14577
     rank: 7
     used: 202740
     free: 3714488
     real_used: 479816
 }
 {
     entry: 8
     pid: 14578
     rank: 8
     used: 199760
     free: 3717084
     real_used: 477220
 }
 {
     entry: 9
     pid: 14579
     rank: 9
     used: 199440
     free: 3722828
     real_used: 471476
 }
 {
     entry: 10
     pid: 14580
     rank: 10
     used: 199440
     free: 3722828
     real_used: 471476
 }
 {
     entry: 11
     pid: 14581
     rank: 11
     used: 199440
     free: 3722828
     real_used: 471476
 }
 {
     entry: 12
     pid: 14582
     rank: 12
     used: 199440
     free: 3722828
     real_used: 471476
 }
 {
     entry: 13
     pid: 14583
     rank: 13
     used: 199440
     free: 3722828
     real_used: 471476
 }
 {
     entry: 14
     pid: 14584
     rank: 14
     used: 199440
     free: 3722828
     real_used: 471476
 }
 {
     entry: 15
     pid: 14585
     rank: 15
     used: 199440
     free: 3722828
     real_used: 471476
 }
 {
     entry: 16
     pid: 14586
     rank: 16
     used: 199440
     free: 3722828
     real_used: 471476
 }
 {
     entry: 17
     pid: 14587
     rank: -1
     used: 199400
     free: 3722868
     real_used: 471436
 }
 {
     entry: 18
     pid: 14588
     rank: -1
     used: 2105844
     free: 3722868
     real_used: 2337944
 }
 {
     entry: 19
     pid: 14589
     rank: -2
     used: 198552
     free: 3723236
     real_used: 471068
 }
 {
     entry: 20
     pid: 0
     rank: 0
     used: 209756
     free: 3712368
     real_used: 481936
 }
 {
     entry: 21
     pid: 14591
     rank: -4
     used: 253660
     free: 3668704
     real_used: 525600
 }

 On 1 October 2013 22:45, Daniel-Constantin Mierla &lt;miconda(a)gmail.com 
 <mailto:miconda@gmail.com>> wrote:

     Hello,

     send SIGUSR1 to one of the worker processes (e.g., UDP listener).
     The main process of kamailio doesn't process SIP messages, so it
     is not exposed to leaks.

     You can used 'kamcmd pkg.stats" to see statistics about usage of
     pkg  per each process (older versions have sercmd).

     'kamctl ps' gives the list of processes.

     Cheers,
     Daniel

     On 10/1/13 2:35 PM, David Cunningham wrote:
      Hi Daniel,

     What I copied and pasted is all that's listed when I do a kill
     -SIGUSR1 on the Kamailio master process. How do I get fuller logs?

     Thanks for your help.

     On 1 October 2013 21:57, Daniel-Constantin Mierla
     &lt;miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:

         Hello,

         you gave only the first part of memory log output. The chunks
         listed there are from startup, thus not part of a leak,
         because they are not allocated again at runtime.

         Can you give the full logs with messages from qm_status? The
         leaks are typically listed towards the end of those messages.

         Yesterday someone else reported a potential leak when using
         perl module, so it might be something else there. I found
         one, which is related to an error case, but perhaps you don't
         get that, otherwise you will notice the error message.

         Cheers,
         Daniel

         On 10/1/13 12:03 PM, David Cunningham wrote:

             Hello,

             Following up on this after a long break. It seems we are
             still having a memory leak. We've added the memory
             debugging options as per the documentation with the
             following output.
             I'm not sure how to read it though - can anyone help?
             Thank you.

             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             <core> [main.c:754]: Memory status (pkg):
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status: (0x4019b008):
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status: heap size= 4194304
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status: used= 191100, used+overhead=462800, free=3731504
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status: max used (+overhead)= 480224
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status: dumping all alloc'ed. fragments:
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      0. N  address=0x401cf414 frag=0x401cf3fc
             size=512 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: str_hash.h:
             str_hash_alloc(69)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      1. N  address=0x401cf644 frag=0x401cf62c
             size=128 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: str_hash.h:
             str_hash_alloc(69)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      2. N  address=0x401cf6f4 frag=0x401cf6dc
             size=256 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: counters.c:
             cnt_hash_add(331)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      3. N  address=0x401cf824 frag=0x401cf80c
             size=64 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: counters.c:
             init_counters(121)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      4. N  address=0x401cf894 frag=0x401cf87c
             size=8 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: route.c:
             init_rlist(180)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      5. N  address=0x401cf8cc frag=0x401cf8b4
             size=64 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: str_hash.h:
             str_hash_alloc(69)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      6. N  address=0x401cf93c frag=0x401cf924
             size=24 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: route.c:
             route_add(159)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      7. N  address=0x401cf984 frag=0x401cf96c
             size=8 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: route.c:
             init_rlist(180)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      8. N  address=0x401cf9bc frag=0x401cf9a4
             size=64 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: str_hash.h:
             str_hash_alloc(69)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:      9. N  address=0x401cfa2c frag=0x401cfa14
             size=24 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: route.c:
             route_add(159)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:     10. N  address=0x401cfa74 frag=0x401cfa5c
             size=8 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: db_id.c:
             dupl_string(50)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:     11. N  address=0x401cfaac frag=0x401cfa94
             size=64 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: str_hash.h:
             str_hash_alloc(69)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:     12. N  address=0x401cfb1c frag=0x401cfb04
             size=24 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: route.c:
             route_add(159)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:     13. N  address=0x401cfb64 frag=0x401cfb4c
             size=8 used=1
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:           alloc'd from <core>: route.c:
             init_rlist(180)
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:          start check=f0f0f0f0, end check=
             c0c0c0c0, abcdefed
             Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
             qm_status:     14. N  address=0x401cfb9c frag=0x401cfb84
             size=64 used=1

         -- 
         Daniel-Constantin Mierla - http://www.asipto.com
         http://twitter.com/#!/miconda
         <http://twitter.com/#%21/miconda> -
         http://www.linkedin.com/in/miconda
         Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov
         18-20, 2013
           - more details about Kamailio trainings at
         http://www.asipto.com -

     -- 
     David Cunningham, Voisonics
     http://voisonics.com/
     USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
     UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
     Australia: +61 (0) 2 8063 9019
     <tel:%2B61%20%280%29%202%208063%209019> 
     -- 
     Daniel-Constantin Mierla -http://www.asipto.com
     http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda> 
-http://www.linkedin.com/in/miconda
     Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
        - more details about Kamailio trainings athttp://www.asipto.com  -

 -- 
 David Cunningham, Voisonics
 http://voisonics.com/
 USA: +1 213 221 1092
 UK: +44 (0) 20 3298 1642
 Australia: +61 (0) 2 8063 9019 
-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
   - more details about Kamailio trainings at http://www.asipto.com -

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Re: [SR-Users] Memory leak in 3.3.4