Hello,

On 7/24/13 4:24 AM, David Cunningham wrote:

Hello,

Thank you very much for the email. In reply:

1. The system ran out of memory. Linux's oom-killer killed Kamailio.

then all the instructions I gave are useless, they are for debugging kamailio's internal memory manager, which handles pkg and shm mallocs.

The chances to be from kamailio itself are very low now. Do you do lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the rest of code should use internal memory manager which does not go beyond the limits set with -m and -M, thus not causing an OS memory exhaustion.

Can you list what modules are you loading? At some point it was a leak in libssl, in case you use tls a lot. But could be another external library...

Cheers, Daniel

2. You're right, DEBUG_MEMORY is a local configuration setting. If

defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.

3. We'll try setting mem_summary=12, thanks.

4. We'll try setting asynchronous syslog, thanks.

5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?

6. We'll try setting mem_join=1, thanks.

On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,

first, to clarify, is the system memory or kamailio's pkg/shm
memory running out? If the operating system runs out of memory,
then should be a leak in a library, because kamailio modules uses
only from a pre-allocated chunk, not going over it.


On 7/23/13 7:33 AM, David Cunningham wrote:

    Hello,

    We're running a Kamailio 3.3.4 system, and Kamailio is slowly
    using more and more memory. Over a couple of weeks it will run
    out of system memory.

    We tried to enable memory debugging doing the following, but
    it resulted in Kamailio not responding to any SIP packets.
    Would anyone have advice please on how to debug the situation?

    1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
    2. In kamailio.cfg add the line:
    #!define DEBUG_MEMORY 1

do you set something special in config when DEBUG_MEMORY is 1? It
is not by default there, so I assume you added some rules based on
this pre-processor directive.

For memory troubleshooting, set memlog to a value lower than debug
parameter in config file and try with mem_summary=12 for a more
compact output. See more about these parameters in the wiki:

- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog

Run kamailio for a while in normal conditions, then restart it to
get the memory usage summaries. There should be indication if
there is some leak, by seeing memory chunks allocated many times
from a function used at runtime. You can send the memory summary
for a process here, we can look at it.



    While this was running and Kamailio didn't respond to packets,
    it logged lots of lines like this:


Do you have syslog to be configured in asynchronous mode? See the
notes from:

- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

The memdbg is less than debug value, that means printing few log
messages for each memory operation. You can make memdbg higher and
rely on memlog for memory summaries, otherwise will be lot of log
messages related to memory.


    Jul 22 21:32:22 hostname kamailio: : <core>
    [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called from
    <core>: cfg.lex: addstr(1438)
    Jul 22 21:32:22 hostname kamailio: : <core>
    [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
    address 0x40048918 frag. 0x40048900 (size=128) on 1 -th hit
    Jul 22 21:32:22 hostname kamailio: : <core>
    [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called from
    <core>: cfg.lex: addstr(1438)
    Jul 22 21:32:22 hostname kamailio: : <core>
    [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
    address 0x400489c8 frag. 0x400489b0 (size=128) on 1 -th hit

addstr() is a function used only for parsing configuration file,
as long as you can still see them, the configuration file parsing
was not finish. addstr() is not a source of leaks because it is
not used at runtime.

If you have large config file, then you can get close to the
limits of the private memory, which is set to 4MB. You can
increase its value using -M parameter (e.g., start kamailio with
-M 8 to set it to use 8MB of memory).

Over the time, the private memory can get used due to
fragmentation, you can set the mem_join parameter in config file
to avoid it (works when compiled with MEMDBG=1).

To monitor usage of internal pkg memory, then you can use sercmd
with pkg.stats command:

http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640

Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'

When you see significant increase of the memory usage, then you
can restart to get the summaries.

You should run these commands after start, just to see the initial
usage of memory.

Cheers,
Daniel

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
list
sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

Hello,

I would say that perl_exec() is the one with the highest chances to be the reason for the leak. Next is line would be db_mysql module, if liked with some custom mysql client library, although even in this case will be unlikely.

Back to perl, the module itself does not call any malloc, so it might be the embedding Perl API that is not used properly in the module.

Can you use some testbed, set children=1 and run kamailio under valgrind, then do some calls and see if it detects the source of the leak?

I'm not using the perl module, I will try to check it whenever I get a chance in the next days. What version of perl do you have installed?

Cheers, Daniel

On 7/24/13 10:31 AM, David Cunningham wrote:

Hello,

We don't do any kamctl commands at all. We do have various modules loaded, as follows. The primary functions we use Kamailio for are phone registrations through usrloc, and routing calls to Asterisk through logic contained in Perl via perl_exec(). Thanks for all your advice so far!

loadmodule "tm.so" loadmodule "tmx.so" loadmodule "usrloc.so" loadmodule "auth.so" loadmodule "auth_db.so" loadmodule "ctl.so" loadmodule "db_mysql.so" loadmodule "kex.so" loadmodule "maxfwd.so" loadmodule "mi_fifo.so" loadmodule "mi_rpc.so" loadmodule "nathelper.so" loadmodule "perl.so" loadmodule "pv.so" loadmodule "registrar.so" loadmodule "rr.so" loadmodule "sanity.so" loadmodule "siputils.so" loadmodule "sl.so" loadmodule "textops.so" loadmodule "xlog.so"

On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,


On 7/24/13 4:24 AM, David Cunningham wrote:

...

Hello,

Thank you very much for the email. In reply:

1. The system ran out of memory. Linux's oom-killer killed Kamailio.

then all the instructions I gave are useless, they are for
debugging kamailio's internal memory manager, which handles pkg
and shm mallocs.

The chances to be from kamailio itself are very low now. Do you do
lot of mi commands (e.g., kamctl ...)? The mi api uses system
malloc, but the rest of code should use internal memory manager
which does not go beyond the limits set with -m and -M, thus not
causing an OS memory exhaustion.

Can you list what modules are you loading? At some point it was a
leak in libssl, in case you use tls a lot. But could be another
external library...

Cheers,
Daniel

...

2. You're right, DEBUG_MEMORY is a local configuration setting.
If defined it sets memdbg to -2, and memlog to -2. The debug
setting is -1.

3. We'll try setting mem_summary=12, thanks.

4. We'll try setting asynchronous syslog, thanks.

5.  Our configuration totals 338 lines, or approx 8.5kb. Is that
a lot?

6. We'll try setting mem_join=1, thanks.



On 23 July 2013 16:53, Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    first, to clarify, is the system memory or kamailio's pkg/shm
    memory running out? If the operating system runs out of
    memory, then should be a leak in a library, because kamailio
    modules uses only from a pre-allocated chunk, not going over it.


    On 7/23/13 7:33 AM, David Cunningham wrote:

        Hello,

        We're running a Kamailio 3.3.4 system, and Kamailio is
        slowly using more and more memory. Over a couple of weeks
        it will run out of system memory.

        We tried to enable memory debugging doing the following,
        but it resulted in Kamailio not responding to any SIP
        packets. Would anyone have advice please on how to debug
        the situation?

        1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
        2. In kamailio.cfg add the line:
        #!define DEBUG_MEMORY 1

    do you set something special in config when DEBUG_MEMORY is
    1? It is not by default there, so I assume you added some
    rules based on this pre-processor directive.

    For memory troubleshooting, set memlog to a value lower than
    debug parameter in config file and try with mem_summary=12
    for a more compact output. See more about these parameters in
    the wiki:

    - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog

    Run kamailio for a while in normal conditions, then restart
    it to get the memory usage summaries. There should be
    indication if there is some leak, by seeing memory chunks
    allocated many times from a function used at runtime. You can
    send the memory summary for a process here, we can look at it.



        While this was running and Kamailio didn't respond to
        packets, it logged lots of lines like this:


    Do you have syslog to be configured in asynchronous mode? See
    the notes from:

    - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

    The memdbg is less than debug value, that means printing few
    log messages for each memory operation. You can make memdbg
    higher and rely on memlog for memory summaries, otherwise
    will be lot of log messages related to memory.


        Jul 22 21:32:22 hostname kamailio: : <core>
        [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called
        from <core>: cfg.lex: addstr(1438)
        Jul 22 21:32:22 hostname kamailio: : <core>
        [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
        address 0x40048918 frag. 0x40048900 (size=128) on 1 -th hit
        Jul 22 21:32:22 hostname kamailio: : <core>
        [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called
        from <core>: cfg.lex: addstr(1438)
        Jul 22 21:32:22 hostname kamailio: : <core>
        [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
        address 0x400489c8 frag. 0x400489b0 (size=128) on 1 -th hit

    addstr() is a function used only for parsing configuration
    file, as long as you can still see them, the configuration
    file parsing was not finish. addstr() is not a source of
    leaks because it is not used at runtime.

    If you have large config file, then you can get close to the
    limits of the private memory, which is set to 4MB. You can
    increase its value using -M parameter (e.g., start kamailio
    with -M 8 to set it to use 8MB of memory).

    Over the time, the private memory can get used due to
    fragmentation, you can set the mem_join parameter in config
    file to avoid it (works when compiled with MEMDBG=1).

    To monitor usage of internal pkg memory, then you can use
    sercmd with pkg.stats command:

    http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640

    Shared memory stats are printed by 'kamctl fifo
    get_statistics shmem:'

    When you see significant increase of the memory usage, then
    you can restart to get the summaries.

    You should run these commands after start, just to see the
    initial usage of memory.

    Cheers,
    Daniel

    -- 
    Daniel-Constantin Mierla - http://www.asipto.com
    http://twitter.com/#!/miconda
    <http://twitter.com/#%21/miconda> -
    http://www.linkedin.com/in/miconda


    _______________________________________________
    SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
    mailing list
    sr-users@lists.sip-router.org
    <mailto:sr-users@lists.sip-router.org>
    http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users




-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

Hello,

can you try the attached patch? It's the same patch, just for two versions, one is for 3.3.x and the other for devel version

It initializes the SIP message variable that is passed to perl after creating the temporary environment, so it is actually destroyed by the perl embedded interpreter.

Cheers, Daniel

On 7/25/13 1:29 AM, David Cunningham wrote:

Hi Daniel,

The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server release 5.4. If I remember right programs running under Valgrind can have issues, so I'm not sure if the customer will want to do that. Ideally we'd do it on a test system, but I'm not sure if we have any RHEL available. I'll see what we can do. Thanks again.

On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,

I would say that perl_exec() is the one with the highest chances
to be the reason for the leak. Next is line would be db_mysql
module, if liked with some custom mysql client library, although
even in this case will be unlikely.

Back to perl, the module itself does not call any malloc, so it
might be the embedding Perl API that is not used properly in the
module.

Can you use some testbed, set children=1 and run kamailio under
valgrind, then do some calls and see if it detects the source of
the leak?

I'm not using the perl module, I will try to check it whenever I
get a chance in the next days. What version of perl do you have
installed?

Cheers,
Daniel


On 7/24/13 10:31 AM, David Cunningham wrote:

...

Hello,

We don't do any kamctl commands at all. We do have various
modules loaded, as follows. The primary functions we use Kamailio
for are phone registrations through usrloc, and routing calls to
Asterisk through logic contained in Perl via perl_exec().
Thanks for all your advice so far!

loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"


On 24 July 2013 16:33, Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,


    On 7/24/13 4:24 AM, David Cunningham wrote:

...

    Hello,

    Thank you very much for the email. In reply:

    1. The system ran out of memory. Linux's oom-killer killed
    Kamailio.

    then all the instructions I gave are useless, they are for
    debugging kamailio's internal memory manager, which handles
    pkg and shm mallocs.

    The chances to be from kamailio itself are very low now. Do
    you do lot of mi commands (e.g., kamctl ...)? The mi api uses
    system malloc, but the rest of code should use internal
    memory manager which does not go beyond the limits set with
    -m and -M, thus not causing an OS memory exhaustion.

    Can you list what modules are you loading? At some point it
    was a leak in libssl, in case you use tls a lot. But could be
    another external library...

    Cheers,
    Daniel

...

    2. You're right, DEBUG_MEMORY is a local configuration
    setting. If defined it sets memdbg to -2, and memlog to -2.
    The debug setting is -1.

    3. We'll try setting mem_summary=12, thanks.

    4. We'll try setting asynchronous syslog, thanks.

    5.  Our configuration totals 338 lines, or approx 8.5kb. Is
    that a lot?

    6. We'll try setting mem_join=1, thanks.



    On 23 July 2013 16:53, Daniel-Constantin Mierla
    <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

        Hello,

        first, to clarify, is the system memory or kamailio's
        pkg/shm memory running out? If the operating system runs
        out of memory, then should be a leak in a library,
        because kamailio modules uses only from a pre-allocated
        chunk, not going over it.


        On 7/23/13 7:33 AM, David Cunningham wrote:

            Hello,

            We're running a Kamailio 3.3.4 system, and Kamailio
            is slowly using more and more memory. Over a couple
            of weeks it will run out of system memory.

            We tried to enable memory debugging doing the
            following, but it resulted in Kamailio not
            responding to any SIP packets. Would anyone have
            advice please on how to debug the situation?

            1. In Makefile.defs set MEMDBG to 1 and recompile
            Kamailio.
            2. In kamailio.cfg add the line:
            #!define DEBUG_MEMORY 1

        do you set something special in config when DEBUG_MEMORY
        is 1? It is not by default there, so I assume you added
        some rules based on this pre-processor directive.

        For memory troubleshooting, set memlog to a value lower
        than debug parameter in config file and try with
        mem_summary=12 for a more compact output. See more about
        these parameters in the wiki:

        - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog

        Run kamailio for a while in normal conditions, then
        restart it to get the memory usage summaries. There
        should be indication if there is some leak, by seeing
        memory chunks allocated many times from a function used
        at runtime. You can send the memory summary for a
        process here, we can look at it.



            While this was running and Kamailio didn't respond
            to packets, it logged lots of lines like this:


        Do you have syslog to be configured in asynchronous
        mode? See the notes from:

        - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

        The memdbg is less than debug value, that means printing
        few log messages for each memory operation. You can make
        memdbg higher and rely on memlog for memory summaries,
        otherwise will be lot of log messages related to memory.


            Jul 22 21:32:22 hostname kamailio: : <core>
            [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128)
            called from <core>: cfg.lex: addstr(1438)
            Jul 22 21:32:22 hostname kamailio: : <core>
            [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128)
            returns address 0x40048918 frag. 0x40048900
            (size=128) on 1 -th hit
            Jul 22 21:32:22 hostname kamailio: : <core>
            [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128)
            called from <core>: cfg.lex: addstr(1438)
            Jul 22 21:32:22 hostname kamailio: : <core>
            [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128)
            returns address 0x400489c8 frag. 0x400489b0
            (size=128) on 1 -th hit

        addstr() is a function used only for parsing
        configuration file, as long as you can still see them,
        the configuration file parsing was not finish. addstr()
        is not a source of leaks because it is not used at runtime.

        If you have large config file, then you can get close to
        the limits of the private memory, which is set to 4MB.
        You can increase its value using -M parameter (e.g.,
        start kamailio with -M 8 to set it to use 8MB of memory).

        Over the time, the private memory can get used due to
        fragmentation, you can set the mem_join parameter in
        config file to avoid it (works when compiled with MEMDBG=1).

        To monitor usage of internal pkg memory, then you can
        use sercmd with pkg.stats command:

        http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640

        Shared memory stats are printed by 'kamctl fifo
        get_statistics shmem:'

        When you see significant increase of the memory usage,
        then you can restart to get the summaries.

        You should run these commands after start, just to see
        the initial usage of memory.

        Cheers,
        Daniel

        -- 
        Daniel-Constantin Mierla - http://www.asipto.com
        http://twitter.com/#!/miconda
        <http://twitter.com/#%21/miconda> -
        http://www.linkedin.com/in/miconda


        _______________________________________________
        SIP Express Router (SER) and Kamailio (OpenSER) -
        sr-users mailing list
        sr-users@lists.sip-router.org
        <mailto:sr-users@lists.sip-router.org>
        http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users




    -- 
    David Cunningham, Voisonics
    http://voisonics.com/
    USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
    UK: +44 (0) 20 3298 1642
    <tel:%2B44%20%280%29%2020%203298%201642>
    Australia: +61 (0) 2 8063 9019
    <tel:%2B61%20%280%29%202%208063%209019>

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda




-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

Hi Daniel,

We tried that patch, but Kamailio logged lots of errors like the following. The undefined value in question is $m, which should be the SIP message. Would you have any advice? Thanks.

Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl [openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an undefined value at Foo.pm line 247.#012

On 25 July 2013 17:11, David Cunningham dcunningham@voisonics.com wrote:

Hi Daniel,

I'll suggest that to the customer. Thank you!

On 25 July 2013 15:45, Daniel-Constantin Mierla miconda@gmail.com wrote:

...

Hello,

can you try the attached patch? It's the same patch, just for two versions, one is for 3.3.x and the other for devel version

It initializes the SIP message variable that is passed to perl after creating the temporary environment, so it is actually destroyed by the perl embedded interpreter.

Cheers, Daniel

On 7/25/13 1:29 AM, David Cunningham wrote:

Hi Daniel,

The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server release 5.4. If I remember right programs running under Valgrind can have issues, so I'm not sure if the customer will want to do that. Ideally we'd do it on a test system, but I'm not sure if we have any RHEL available. I'll see what we can do. Thanks again.

On 25 July 2013 04:55, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

I would say that perl_exec() is the one with the highest chances to be the reason for the leak. Next is line would be db_mysql module, if liked with some custom mysql client library, although even in this case will be unlikely.

Back to perl, the module itself does not call any malloc, so it might be the embedding Perl API that is not used properly in the module.

Can you use some testbed, set children=1 and run kamailio under valgrind, then do some calls and see if it detects the source of the leak?

I'm not using the perl module, I will try to check it whenever I get a chance in the next days. What version of perl do you have installed?

Cheers, Daniel

On 7/24/13 10:31 AM, David Cunningham wrote:

Hello,

We don't do any kamctl commands at all. We do have various modules loaded, as follows. The primary functions we use Kamailio for are phone registrations through usrloc, and routing calls to Asterisk through logic contained in Perl via perl_exec(). Thanks for all your advice so far!

loadmodule "tm.so" loadmodule "tmx.so" loadmodule "usrloc.so" loadmodule "auth.so" loadmodule "auth_db.so" loadmodule "ctl.so" loadmodule "db_mysql.so" loadmodule "kex.so" loadmodule "maxfwd.so" loadmodule "mi_fifo.so" loadmodule "mi_rpc.so" loadmodule "nathelper.so" loadmodule "perl.so" loadmodule "pv.so" loadmodule "registrar.so" loadmodule "rr.so" loadmodule "sanity.so" loadmodule "siputils.so" loadmodule "sl.so" loadmodule "textops.so" loadmodule "xlog.so"

On 24 July 2013 16:33, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

On 7/24/13 4:24 AM, David Cunningham wrote:

Hello,

Thank you very much for the email. In reply:

1. The system ran out of memory. Linux's oom-killer killed Kamailio.

then all the instructions I gave are useless, they are for debugging kamailio's internal memory manager, which handles pkg and shm mallocs.

The chances to be from kamailio itself are very low now. Do you do lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the rest of code should use internal memory manager which does not go beyond the limits set with -m and -M, thus not causing an OS memory exhaustion.

Can you list what modules are you loading? At some point it was a leak in libssl, in case you use tls a lot. But could be another external library...

Cheers, Daniel

2. You're right, DEBUG_MEMORY is a local configuration setting. If

defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.

3. We'll try setting mem_summary=12, thanks.

4. We'll try setting asynchronous syslog, thanks.

5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?

6. We'll try setting mem_join=1, thanks.

On 23 July 2013 16:53, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

first, to clarify, is the system memory or kamailio's pkg/shm memory running out? If the operating system runs out of memory, then should be a leak in a library, because kamailio modules uses only from a pre-allocated chunk, not going over it.

On 7/23/13 7:33 AM, David Cunningham wrote:

...

Hello,

We're running a Kamailio 3.3.4 system, and Kamailio is slowly using more and more memory. Over a couple of weeks it will run out of system memory.

We tried to enable memory debugging doing the following, but it resulted in Kamailio not responding to any SIP packets. Would anyone have advice please on how to debug the situation?

1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:

#!define DEBUG_MEMORY 1

do you set something special in config when DEBUG_MEMORY is 1? It is not by default there, so I assume you added some rules based on this pre-processor directive.

For memory troubleshooting, set memlog to a value lower than debug parameter in config file and try with mem_summary=12 for a more compact output. See more about these parameters in the wiki:

• http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog

Run kamailio for a while in normal conditions, then restart it to get the memory usage summaries. There should be indication if there is some leak, by seeing memory chunks allocated many times from a function used at runtime. You can send the memory summary for a process here, we can look at it.

...

While this was running and Kamailio didn't respond to packets, it logged lots of lines like this:

Do you have syslog to be configured in asynchronous mode? See the notes from:

• http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

The memdbg is less than debug value, that means printing few log messages for each memory operation. You can make memdbg higher and rely on memlog for memory summaries, otherwise will be lot of log messages related to memory.

...

Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900 (size=128) on 1 -th hit Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0 (size=128) on 1 -th hit

addstr() is a function used only for parsing configuration file, as long as you can still see them, the configuration file parsing was not finish. addstr() is not a source of leaks because it is not used at runtime.

If you have large config file, then you can get close to the limits of the private memory, which is set to 4MB. You can increase its value using -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of memory).

Over the time, the private memory can get used due to fragmentation, you can set the mem_join parameter in config file to avoid it (works when compiled with MEMDBG=1).

To monitor usage of internal pkg memory, then you can use sercmd with pkg.stats command:

http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640

Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'

When you see significant increase of the memory usage, then you can restart to get the summaries.

You should run these commands after start, just to see the initial usage of memory.

Cheers, Daniel

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

Hi Daniel,

Thank you, we will try that.

On 31 July 2013 20:54, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

revising the patch I noticed I was moving the initialization of the variable after pushing it to perl environment (from the perl docs, the variable should have been initialized after initializing the environment -- what I tried to do with previous patch).

There is a new smaller patch to be added:

http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain...

Practically, the line XPUSHs(m); has to be moved after the line with SvREADONLY_on(SvRV(m));

Cheers, Daniel

On 7/31/13 8:26 AM, David Cunningham wrote:

Hi Daniel,

We tried that patch, but Kamailio logged lots of errors like the following. The undefined value in question is $m, which should be the SIP message. Would you have any advice? Thanks.

Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl [openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an undefined value at Foo.pm line 247.#012

On 25 July 2013 17:11, David Cunningham dcunningham@voisonics.com wrote:

...

Hi Daniel,

I'll suggest that to the customer. Thank you!

On 25 July 2013 15:45, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

can you try the attached patch? It's the same patch, just for two versions, one is for 3.3.x and the other for devel version

It initializes the SIP message variable that is passed to perl after creating the temporary environment, so it is actually destroyed by the perl embedded interpreter.

Cheers, Daniel

On 7/25/13 1:29 AM, David Cunningham wrote:

Hi Daniel,

The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server release 5.4. If I remember right programs running under Valgrind can have issues, so I'm not sure if the customer will want to do that. Ideally we'd do it on a test system, but I'm not sure if we have any RHEL available. I'll see what we can do. Thanks again.

On 25 July 2013 04:55, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

I would say that perl_exec() is the one with the highest chances to be the reason for the leak. Next is line would be db_mysql module, if liked with some custom mysql client library, although even in this case will be unlikely.

Back to perl, the module itself does not call any malloc, so it might be the embedding Perl API that is not used properly in the module.

Can you use some testbed, set children=1 and run kamailio under valgrind, then do some calls and see if it detects the source of the leak?

I'm not using the perl module, I will try to check it whenever I get a chance in the next days. What version of perl do you have installed?

Cheers, Daniel

On 7/24/13 10:31 AM, David Cunningham wrote:

Hello,

We don't do any kamctl commands at all. We do have various modules loaded, as follows. The primary functions we use Kamailio for are phone registrations through usrloc, and routing calls to Asterisk through logic contained in Perl via perl_exec(). Thanks for all your advice so far!

loadmodule "tm.so" loadmodule "tmx.so" loadmodule "usrloc.so" loadmodule "auth.so" loadmodule "auth_db.so" loadmodule "ctl.so" loadmodule "db_mysql.so" loadmodule "kex.so" loadmodule "maxfwd.so" loadmodule "mi_fifo.so" loadmodule "mi_rpc.so" loadmodule "nathelper.so" loadmodule "perl.so" loadmodule "pv.so" loadmodule "registrar.so" loadmodule "rr.so" loadmodule "sanity.so" loadmodule "siputils.so" loadmodule "sl.so" loadmodule "textops.so" loadmodule "xlog.so"

On 24 July 2013 16:33, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

On 7/24/13 4:24 AM, David Cunningham wrote:

Hello,

Thank you very much for the email. In reply:

1. The system ran out of memory. Linux's oom-killer killed Kamailio.

then all the instructions I gave are useless, they are for debugging kamailio's internal memory manager, which handles pkg and shm mallocs.

The chances to be from kamailio itself are very low now. Do you do lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the rest of code should use internal memory manager which does not go beyond the limits set with -m and -M, thus not causing an OS memory exhaustion.

Can you list what modules are you loading? At some point it was a leak in libssl, in case you use tls a lot. But could be another external library...

Cheers, Daniel

2. You're right, DEBUG_MEMORY is a local configuration setting. If

defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.

3. We'll try setting mem_summary=12, thanks.

4. We'll try setting asynchronous syslog, thanks.

5. Our configuration totals 338 lines, or approx 8.5kb. Is that a

lot?

6. We'll try setting mem_join=1, thanks.

On 23 July 2013 16:53, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

first, to clarify, is the system memory or kamailio's pkg/shm memory running out? If the operating system runs out of memory, then should be a leak in a library, because kamailio modules uses only from a pre-allocated chunk, not going over it.

On 7/23/13 7:33 AM, David Cunningham wrote:

> Hello, > > We're running a Kamailio 3.3.4 system, and Kamailio is slowly using > more and more memory. Over a couple of weeks it will run out of system > memory. > > We tried to enable memory debugging doing the following, but it > resulted in Kamailio not responding to any SIP packets. Would anyone have > advice please on how to debug the situation? > > 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio. > 2. In kamailio.cfg add the line: > #!define DEBUG_MEMORY 1 > do you set something special in config when DEBUG_MEMORY is 1? It is not by default there, so I assume you added some rules based on this pre-processor directive.

For memory troubleshooting, set memlog to a value lower than debug parameter in config file and try with mem_summary=12 for a more compact output. See more about these parameters in the wiki:

• http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog

Run kamailio for a while in normal conditions, then restart it to get the memory usage summaries. There should be indication if there is some leak, by seeing memory chunks allocated many times from a function used at runtime. You can send the memory summary for a process here, we can look at it.

> While this was running and Kamailio didn't respond to packets, it > logged lots of lines like this: >

Do you have syslog to be configured in asynchronous mode? See the notes from:

• http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

The memdbg is less than debug value, that means printing few log messages for each memory operation. You can make memdbg higher and rely on memlog for memory summaries, otherwise will be lot of log messages related to memory.

> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: > qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) > Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: > qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900 > (size=128) on 1 -th hit > Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: > qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) > Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: > qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0 > (size=128) on 1 -th hit > addstr() is a function used only for parsing configuration file, as long as you can still see them, the configuration file parsing was not finish. addstr() is not a source of leaks because it is not used at runtime.

If you have large config file, then you can get close to the limits of the private memory, which is set to 4MB. You can increase its value using -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of memory).

Over the time, the private memory can get used due to fragmentation, you can set the mem_join parameter in config file to avoid it (works when compiled with MEMDBG=1).

To monitor usage of internal pkg memory, then you can use sercmd with pkg.stats command:

http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640

Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'

When you see significant increase of the memory usage, then you can restart to get the summaries.

You should run these commands after start, just to see the initial usage of memory.

Cheers, Daniel

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Hello,

I pushed the commit to branch 3.3 now -- forgot about it.

The code you pasted below becomes (after the two patches, thus is the final version that should be used):

ENTER; /* everything created after here */ SAVETMPS; /* ...is a temporary variable. */ PUSHMARK(SP); /* remember the stack pointer */

m = sv_newmortal(); sv_setref_pv(m, "Kamailio::Message", (void *)_msg); SvREADONLY_on(SvRV(m));

XPUSHs(m); /* Our reference to the stack... */

if (mystr) XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));

Cheers, Daniel

On 7/31/13 1:48 PM, David Cunningham wrote:

Hi Daniel,

In 3.3.x the code is arranged a little differently and I'd like to make sure we get it right. Could you please advise?

    m = sv_newmortal();
    sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
    SvREADONLY_on(SvRV(m));


    ENTER;                          /* everything created after 

here */ SAVETMPS; /* ...is a temporary variable. */ PUSHMARK(SP); /* remember the stack pointer */ XPUSHs(m); /* Our reference to the stack... */

    if (mystr)
            XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
                                    /* Our string to the stack... */

On 31 July 2013 21:44, David Cunningham <dcunningham@voisonics.com mailto:dcunningham@voisonics.com> wrote:

Hi Daniel,

Thank you, we will try that.



On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda@gmail.com
<mailto:miconda@gmail.com>> wrote:

    Hello,

    revising the patch I noticed I was moving the initialization
    of the variable after pushing it to perl environment (from the
    perl docs, the variable should have been initialized after
    initializing the environment -- what I tried to do with
    previous patch).

    There is a new smaller patch to be added:

    http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain;h=3935fedf23f3bf2b6675182193cef6af3bbd903a

    Practically, the line XPUSHs(m); has to be moved after the
    line with SvREADONLY_on(SvRV(m));

    Cheers,
    Daniel


    On 7/31/13 8:26 AM, David Cunningham wrote:

...

    Hi Daniel,

    We tried that patch, but Kamailio logged lots of errors like
    the following. The undefined value in question is $m, which
    should be the SIP message. Would you have any advice? Thanks.

    Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
    [openserxs.xs:1022]: perl error: Can't call method
    "pseudoVar" on an undefined value at Foo.pm line 247.#012


    On 25 July 2013 17:11, David Cunningham
    <dcunningham@voisonics.com
    <mailto:dcunningham@voisonics.com>> wrote:

        Hi Daniel,

        I'll suggest that to the customer. Thank you!



        On 25 July 2013 15:45, Daniel-Constantin Mierla
        <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

            Hello,

            can you try the attached patch? It's the same patch,
            just for two versions, one is for 3.3.x and the other
            for devel version

            It initializes the SIP message variable that is
            passed to perl after creating the temporary
            environment, so it is actually destroyed by the perl
            embedded interpreter.

            Cheers,
            Daniel


            On 7/25/13 1:29 AM, David Cunningham wrote:

...

            Hi Daniel,

            The system is running Perl 5.8.8 on Red Hat
            Enterprise Linux Server release 5.4. If I remember
            right programs running under Valgrind can have
            issues, so I'm not sure if the customer will want to
            do that. Ideally we'd do it on a test system, but
            I'm not sure if we have any RHEL available.
            I'll see what we can do. Thanks again.


            On 25 July 2013 04:55, Daniel-Constantin Mierla
            <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

                Hello,

                I would say that perl_exec() is the one with the
                highest chances to be the reason for the leak.
                Next is line would be db_mysql module, if liked
                with some custom mysql client library, although
                even in this case will be unlikely.

                Back to perl, the module itself does not call
                any malloc, so it might be the embedding Perl
                API that is not used properly in the module.

                Can you use some testbed, set children=1 and run
                kamailio under valgrind, then do some calls and
                see if it detects the source of the leak?

                I'm not using the perl module, I will try to
                check it whenever I get a chance in the next
                days. What version of perl do you have installed?

                Cheers,
                Daniel


                On 7/24/13 10:31 AM, David Cunningham wrote:

...

                Hello,

                We don't do any kamctl commands at all. We do
                have various modules loaded, as follows. The
                primary functions we use Kamailio for are phone
                registrations through usrloc, and routing calls
                to Asterisk through logic contained in Perl via
                perl_exec().
                Thanks for all your advice so far!

                loadmodule "tm.so"
                loadmodule "tmx.so"
                loadmodule "usrloc.so"
                loadmodule "auth.so"
                loadmodule "auth_db.so"
                loadmodule "ctl.so"
                loadmodule "db_mysql.so"
                loadmodule "kex.so"
                loadmodule "maxfwd.so"
                loadmodule "mi_fifo.so"
                loadmodule "mi_rpc.so"
                loadmodule "nathelper.so"
                loadmodule "perl.so"
                loadmodule "pv.so"
                loadmodule "registrar.so"
                loadmodule "rr.so"
                loadmodule "sanity.so"
                loadmodule "siputils.so"
                loadmodule "sl.so"
                loadmodule "textops.so"
                loadmodule "xlog.so"


                On 24 July 2013 16:33, Daniel-Constantin Mierla
                <miconda@gmail.com <mailto:miconda@gmail.com>>
                wrote:

                    Hello,


                    On 7/24/13 4:24 AM, David Cunningham wrote:

...

                    Hello,

                    Thank you very much for the email. In reply:

                    1. The system ran out of memory. Linux's
                    oom-killer killed Kamailio.

                    then all the instructions I gave are
                    useless, they are for debugging kamailio's
                    internal memory manager, which handles pkg
                    and shm mallocs.

                    The chances to be from kamailio itself are
                    very low now. Do you do lot of mi commands
                    (e.g., kamctl ...)? The mi api uses system
                    malloc, but the rest of code should use
                    internal memory manager which does not go
                    beyond the limits set with -m and -M, thus
                    not causing an OS memory exhaustion.

                    Can you list what modules are you loading?
                    At some point it was a leak in libssl, in
                    case you use tls a lot. But could be
                    another external library...

                    Cheers,
                    Daniel

...

                    2. You're right, DEBUG_MEMORY is a local
                    configuration setting. If defined it sets
                    memdbg to -2, and memlog to -2. The debug
                    setting is -1.

                    3. We'll try setting mem_summary=12, thanks.

                    4. We'll try setting asynchronous syslog,
                    thanks.

                    5.  Our configuration totals 338 lines, or
                    approx 8.5kb. Is that a lot?

                    6. We'll try setting mem_join=1, thanks.



                    On 23 July 2013 16:53, Daniel-Constantin
                    Mierla <miconda@gmail.com
                    <mailto:miconda@gmail.com>> wrote:

                        Hello,

                        first, to clarify, is the system
                        memory or kamailio's pkg/shm memory
                        running out? If the operating system
                        runs out of memory, then should be a
                        leak in a library, because kamailio
                        modules uses only from a pre-allocated
                        chunk, not going over it.


                        On 7/23/13 7:33 AM, David Cunningham
                        wrote:

                            Hello,

                            We're running a Kamailio 3.3.4
                            system, and Kamailio is slowly
                            using more and more memory. Over a
                            couple of weeks it will run out of
                            system memory.

                            We tried to enable memory
                            debugging doing the following, but
                            it resulted in Kamailio not
                            responding to any SIP packets.
                            Would anyone have advice please on
                            how to debug the situation?

                            1. In Makefile.defs set MEMDBG to
                            1 and recompile Kamailio.
                            2. In kamailio.cfg add the line:
                            #!define DEBUG_MEMORY 1

                        do you set something special in config
                        when DEBUG_MEMORY is 1? It is not by
                        default there, so I assume you added
                        some rules based on this pre-processor
                        directive.

                        For memory troubleshooting, set memlog
                        to a value lower than debug parameter
                        in config file and try with
                        mem_summary=12 for a more compact
                        output. See more about these
                        parameters in the wiki:

                        -
                        http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog

                        Run kamailio for a while in normal
                        conditions, then restart it to get the
                        memory usage summaries. There should
                        be indication if there is some leak,
                        by seeing memory chunks allocated many
                        times from a function used at runtime.
                        You can send the memory summary for a
                        process here, we can look at it.



                            While this was running and
                            Kamailio didn't respond to
                            packets, it logged lots of lines
                            like this:


                        Do you have syslog to be configured in
                        asynchronous mode? See the notes from:

                        -
                        http://www.kamailio.org/wiki/tutorials/3.2.x/syslog

                        The memdbg is less than debug value,
                        that means printing few log messages
                        for each memory operation. You can
                        make memdbg higher and rely on memlog
                        for memory summaries, otherwise will
                        be lot of log messages related to memory.


                            Jul 22 21:32:22 hostname kamailio:
                            : <core> [mem/q_malloc.c:369]:
                            qm_malloc(0x4000e008, 128) called
                            from <core>: cfg.lex: addstr(1438)
                            Jul 22 21:32:22 hostname kamailio:
                            : <core> [mem/q_malloc.c:413]:
                            qm_malloc(0x4000e008, 128) returns
                            address 0x40048918 frag.
                            0x40048900 (size=128) on 1 -th hit
                            Jul 22 21:32:22 hostname kamailio:
                            : <core> [mem/q_malloc.c:369]:
                            qm_malloc(0x4000e008, 128) called
                            from <core>: cfg.lex: addstr(1438)
                            Jul 22 21:32:22 hostname kamailio:
                            : <core> [mem/q_malloc.c:413]:
                            qm_malloc(0x4000e008, 128) returns
                            address 0x400489c8 frag.
                            0x400489b0 (size=128) on 1 -th hit

                        addstr() is a function used only for
                        parsing configuration file, as long as
                        you can still see them, the
                        configuration file parsing was not
                        finish. addstr() is not a source of
                        leaks because it is not used at runtime.

                        If you have large config file, then
                        you can get close to the limits of the
                        private memory, which is set to 4MB.
                        You can increase its value using -M
                        parameter (e.g., start kamailio with
                        -M 8 to set it to use 8MB of memory).

                        Over the time, the private memory can
                        get used due to fragmentation, you can
                        set the mem_join parameter in config
                        file to avoid it (works when compiled
                        with MEMDBG=1).

                        To monitor usage of internal pkg
                        memory, then you can use sercmd with
                        pkg.stats command:

                        http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640

                        Shared memory stats are printed by
                        'kamctl fifo get_statistics shmem:'

                        When you see significant increase of
                        the memory usage, then you can restart
                        to get the summaries.

                        You should run these commands after
                        start, just to see the initial usage
                        of memory.

                        Cheers,
                        Daniel

                        -- 
                        Daniel-Constantin Mierla -
                        http://www.asipto.com
                        http://twitter.com/#!/miconda
                        <http://twitter.com/#%21/miconda> -
                        http://www.linkedin.com/in/miconda


                        _______________________________________________
                        SIP Express Router (SER) and Kamailio
                        (OpenSER) - sr-users mailing list
                        sr-users@lists.sip-router.org
                        <mailto:sr-users@lists.sip-router.org>
                        http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users




                    -- 
                    David Cunningham, Voisonics
                    http://voisonics.com/
                    USA: +1 213 221 1092
                    <tel:%2B1%20213%20221%201092>
                    UK: +44 (0) 20 3298 1642
                    <tel:%2B44%20%280%29%2020%203298%201642>
                    Australia: +61 (0) 2 8063 9019
                    <tel:%2B61%20%280%29%202%208063%209019>

                    -- 
                    Daniel-Constantin Mierla -http://www.asipto.com
                    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda




                -- 
                David Cunningham, Voisonics
                http://voisonics.com/
                USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
                UK: +44 (0) 20 3298 1642
                <tel:%2B44%20%280%29%2020%203298%201642>
                Australia: +61 (0) 2 8063 9019
                <tel:%2B61%20%280%29%202%208063%209019>

                -- 
                Daniel-Constantin Mierla -http://www.asipto.com
                http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda




            -- 
            David Cunningham, Voisonics
            http://voisonics.com/
            USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
            UK: +44 (0) 20 3298 1642
            <tel:%2B44%20%280%29%2020%203298%201642>
            Australia: +61 (0) 2 8063 9019
            <tel:%2B61%20%280%29%202%208063%209019>

            -- 
            Daniel-Constantin Mierla -http://www.asipto.com
            http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda




        -- 
        David Cunningham, Voisonics
        http://voisonics.com/
        USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
        UK: +44 (0) 20 3298 1642
        <tel:%2B44%20%280%29%2020%203298%201642>
        Australia: +61 (0) 2 8063 9019
        <tel:%2B61%20%280%29%202%208063%209019>




    -- 
    David Cunningham, Voisonics
    http://voisonics.com/
    USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
    UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
    Australia: +61 (0) 2 8063 9019
    <tel:%2B61%20%280%29%202%208063%209019>

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda




-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <tel:%2B61%20%280%29%202%208063%209019>

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

Hello,

Following up on this after a long break. It seems we are still having a memory leak. We've added the memory debugging options as per the documentation with the following output. I'm not sure how to read it though - can anyone help? Thank you.

Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core> [main.c:754]: Memory status (pkg): Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: (0x4019b008): Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: heap size= 4194304 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: used= 191100, used+overhead=462800, free=3731504 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max used (+overhead)= 480224 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: dumping all alloc'ed. fragments: Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 1. N address=0x401cf644 frag=0x401cf62c size=128 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: counters.c: cnt_hash_add(331) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 3. N address=0x401cf824 frag=0x401cf80c size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: counters.c: init_counters(121) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 4. N address=0x401cf894 frag=0x401cf87c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 6. N address=0x401cf93c frag=0x401cf924 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 7. N address=0x401cf984 frag=0x401cf96c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: db_id.c: dupl_string(50) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1

On 2 August 2013 11:34, David Cunningham dcunningham@voisonics.com wrote:

Hi Daniel,

Thank you, we will try that.

On 31 July 2013 21:54, Daniel-Constantin Mierla miconda@gmail.com wrote:

...

Hello,

I pushed the commit to branch 3.3 now -- forgot about it.

The code you pasted below becomes (after the two patches, thus is the final version that should be used):

ENTER;                /* everything created after here */
SAVETMPS;            /* ...is a temporary variable.   */
PUSHMARK(SP);        /* remember the stack pointer    */

m = sv_newmortal();
sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));

XPUSHs(m);            /* Our reference to the stack... */

if (mystr)
    XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));

Cheers, Daniel

On 7/31/13 1:48 PM, David Cunningham wrote:

Hi Daniel,

In 3.3.x the code is arranged a little differently and I'd like to make sure we get it right. Could you please advise?

    m = sv_newmortal();
    sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
    SvREADONLY_on(SvRV(m));


    ENTER;                          /* everything created after here

*/ SAVETMPS; /* ...is a temporary variable. */ PUSHMARK(SP); /* remember the stack pointer */ XPUSHs(m); /* Our reference to the stack... */

    if (mystr)
            XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
                                    /* Our string to the stack... */

On 31 July 2013 21:44, David Cunningham dcunningham@voisonics.comwrote:

...

Hi Daniel,

Thank you, we will try that.

On 31 July 2013 20:54, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

revising the patch I noticed I was moving the initialization of the variable after pushing it to perl environment (from the perl docs, the variable should have been initialized after initializing the environment -- what I tried to do with previous patch).

There is a new smaller patch to be added:

http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plain...

Practically, the line XPUSHs(m); has to be moved after the line with SvREADONLY_on(SvRV(m));

Cheers, Daniel

On 7/31/13 8:26 AM, David Cunningham wrote:

Hi Daniel,

We tried that patch, but Kamailio logged lots of errors like the following. The undefined value in question is $m, which should be the SIP message. Would you have any advice? Thanks.

Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl [openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an undefined value at Foo.pm line 247.#012

On 25 July 2013 17:11, David Cunningham dcunningham@voisonics.comwrote:

...

Hi Daniel,

I'll suggest that to the customer. Thank you!

On 25 July 2013 15:45, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

can you try the attached patch? It's the same patch, just for two versions, one is for 3.3.x and the other for devel version

It initializes the SIP message variable that is passed to perl after creating the temporary environment, so it is actually destroyed by the perl embedded interpreter.

Cheers, Daniel

On 7/25/13 1:29 AM, David Cunningham wrote:

Hi Daniel,

The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server release 5.4. If I remember right programs running under Valgrind can have issues, so I'm not sure if the customer will want to do that. Ideally we'd do it on a test system, but I'm not sure if we have any RHEL available. I'll see what we can do. Thanks again.

On 25 July 2013 04:55, Daniel-Constantin Mierla miconda@gmail.comwrote:

> Hello, > > I would say that perl_exec() is the one with the highest chances to > be the reason for the leak. Next is line would be db_mysql module, if liked > with some custom mysql client library, although even in this case will be > unlikely. > > Back to perl, the module itself does not call any malloc, so it > might be the embedding Perl API that is not used properly in the module. > > Can you use some testbed, set children=1 and run kamailio under > valgrind, then do some calls and see if it detects the source of the leak? > > I'm not using the perl module, I will try to check it whenever I get > a chance in the next days. What version of perl do you have installed? > > Cheers, > Daniel > > > On 7/24/13 10:31 AM, David Cunningham wrote: > > Hello, > > We don't do any kamctl commands at all. We do have various modules > loaded, as follows. The primary functions we use Kamailio for are phone > registrations through usrloc, and routing calls to Asterisk through logic > contained in Perl via perl_exec(). > Thanks for all your advice so far! > > loadmodule "tm.so" > loadmodule "tmx.so" > loadmodule "usrloc.so" > loadmodule "auth.so" > loadmodule "auth_db.so" > loadmodule "ctl.so" > loadmodule "db_mysql.so" > loadmodule "kex.so" > loadmodule "maxfwd.so" > loadmodule "mi_fifo.so" > loadmodule "mi_rpc.so" > loadmodule "nathelper.so" > loadmodule "perl.so" > loadmodule "pv.so" > loadmodule "registrar.so" > loadmodule "rr.so" > loadmodule "sanity.so" > loadmodule "siputils.so" > loadmodule "sl.so" > loadmodule "textops.so" > loadmodule "xlog.so" > > > On 24 July 2013 16:33, Daniel-Constantin Mierla miconda@gmail.comwrote: > >> Hello, >> >> >> On 7/24/13 4:24 AM, David Cunningham wrote: >> >> Hello, >> >> Thank you very much for the email. In reply: >> >> 1. The system ran out of memory. Linux's oom-killer killed Kamailio. >> >> then all the instructions I gave are useless, they are for >> debugging kamailio's internal memory manager, which handles pkg and shm >> mallocs. >> >> The chances to be from kamailio itself are very low now. Do you do >> lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but >> the rest of code should use internal memory manager which does not go >> beyond the limits set with -m and -M, thus not causing an OS memory >> exhaustion. >> >> Can you list what modules are you loading? At some point it was a >> leak in libssl, in case you use tls a lot. But could be another external >> library... >> >> Cheers, >> Daniel >> >> >> >> 2. You're right, DEBUG_MEMORY is a local configuration setting. If >> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1. >> >> 3. We'll try setting mem_summary=12, thanks. >> >> 4. We'll try setting asynchronous syslog, thanks. >> >> 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a >> lot? >> >> 6. We'll try setting mem_join=1, thanks. >> >> >> >> On 23 July 2013 16:53, Daniel-Constantin Mierla miconda@gmail.comwrote: >> >>> Hello, >>> >>> first, to clarify, is the system memory or kamailio's pkg/shm >>> memory running out? If the operating system runs out of memory, then should >>> be a leak in a library, because kamailio modules uses only from a >>> pre-allocated chunk, not going over it. >>> >>> >>> On 7/23/13 7:33 AM, David Cunningham wrote: >>> >>>> Hello, >>>> >>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly >>>> using more and more memory. Over a couple of weeks it will run out of >>>> system memory. >>>> >>>> We tried to enable memory debugging doing the following, but it >>>> resulted in Kamailio not responding to any SIP packets. Would anyone have >>>> advice please on how to debug the situation? >>>> >>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio. >>>> 2. In kamailio.cfg add the line: >>>> #!define DEBUG_MEMORY 1 >>>> >>> do you set something special in config when DEBUG_MEMORY is 1? It >>> is not by default there, so I assume you added some rules based on this >>> pre-processor directive. >>> >>> For memory troubleshooting, set memlog to a value lower than debug >>> parameter in config file and try with mem_summary=12 for a more compact >>> output. See more about these parameters in the wiki: >>> >>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog >>> >>> Run kamailio for a while in normal conditions, then restart it to >>> get the memory usage summaries. There should be indication if there is some >>> leak, by seeing memory chunks allocated many times from a function used at >>> runtime. You can send the memory summary for a process here, we can look at >>> it. >>> >>> >>> >>>> While this was running and Kamailio didn't respond to packets, it >>>> logged lots of lines like this: >>>> >>> >>> Do you have syslog to be configured in asynchronous mode? See the >>> notes from: >>> >>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog >>> >>> The memdbg is less than debug value, that means printing few log >>> messages for each memory operation. You can make memdbg higher and rely on >>> memlog for memory summaries, otherwise will be lot of log messages related >>> to memory. >>> >>> >>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: >>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) >>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: >>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900 >>>> (size=128) on 1 -th hit >>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: >>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) >>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: >>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0 >>>> (size=128) on 1 -th hit >>>> >>> addstr() is a function used only for parsing configuration file, >>> as long as you can still see them, the configuration file parsing was not >>> finish. addstr() is not a source of leaks because it is not used at runtime. >>> >>> If you have large config file, then you can get close to the >>> limits of the private memory, which is set to 4MB. You can increase its >>> value using -M parameter (e.g., start kamailio with -M 8 to set it to use >>> 8MB of memory). >>> >>> Over the time, the private memory can get used due to >>> fragmentation, you can set the mem_join parameter in config file to avoid >>> it (works when compiled with MEMDBG=1). >>> >>> To monitor usage of internal pkg memory, then you can use sercmd >>> with pkg.stats command: >>> >>> >>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640 >>> >>> Shared memory stats are printed by 'kamctl fifo get_statistics >>> shmem:' >>> >>> When you see significant increase of the memory usage, then you >>> can restart to get the summaries. >>> >>> You should run these commands after start, just to see the initial >>> usage of memory. >>> >>> Cheers, >>> Daniel >>> >>> -- >>> Daniel-Constantin Mierla - http://www.asipto.com >>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>> >>> >>> _______________________________________________ >>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing >>> list >>> sr-users@lists.sip-router.org >>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>> >> >> >> >> -- >> David Cunningham, Voisonics >> http://voisonics.com/ >> USA: +1 213 221 1092 <%2B1%20213%20221%201092> >> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> >> Australia: +61 (0) 2 8063 9019 >> >> >> -- >> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> >> > > > -- > David Cunningham, Voisonics > http://voisonics.com/ > USA: +1 213 221 1092 <%2B1%20213%20221%201092> > UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> > Australia: +61 (0) 2 8063 9019 > > > -- > Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > >

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

Hi Daniel,

What I copied and pasted is all that's listed when I do a kill -SIGUSR1 on the Kamailio master process. How do I get fuller logs?

Thanks for your help.

On 1 October 2013 21:57, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

you gave only the first part of memory log output. The chunks listed there are from startup, thus not part of a leak, because they are not allocated again at runtime.

Can you give the full logs with messages from qm_status? The leaks are typically listed towards the end of those messages.

Yesterday someone else reported a potential leak when using perl module, so it might be something else there. I found one, which is related to an error case, but perhaps you don't get that, otherwise you will notice the error message.

Cheers, Daniel

On 10/1/13 12:03 PM, David Cunningham wrote:

...

Hello,

Following up on this after a long break. It seems we are still having a memory leak. We've added the memory debugging options as per the documentation with the following output. I'm not sure how to read it though - can anyone help? Thank you.

Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core> [main.c:754]: Memory status (pkg): Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: (0x4019b008): Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: heap size= 4194304 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: used= 191100, used+overhead=462800, free=3731504 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max used (+overhead)= 480224 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: dumping all alloc'ed. fragments: Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:

1. N address=0x401cf644 frag=0x401cf62c size=128 used=1

Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: counters.c: cnt_hash_add(331) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 3. N address=0x401cf824 frag=0x401cf80c size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: counters.c: init_counters(121) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 4. N address=0x401cf894 frag=0x401cf87c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 6. N address=0x401cf93c frag=0x401cf924 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 7. N address=0x401cf984 frag=0x401cf96c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: db_id.c: dupl_string(50) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/**micondahttp://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

Hi Daniel,

When I send SIGUSR1 to a worker process nothing is logged. Any ideas there? The following is the output of "sercmd pkg.stats".

{ entry: 0 pid: 14570 rank: 0 used: 191100 free: 3731504 real_used: 462800 } { entry: 1 pid: 14571 rank: 1 used: 202656 free: 3714236 real_used: 480068 } { entry: 2 pid: 14572 rank: 2 used: 202720 free: 3714220 real_used: 480084 } { entry: 3 pid: 14573 rank: 3 used: 202664 free: 3714324 real_used: 479980 } { entry: 4 pid: 14574 rank: 4 used: 202744 free: 3714100 real_used: 480204 } { entry: 5 pid: 14575 rank: 5 used: 201908 free: 3715176 real_used: 479128 } { entry: 6 pid: 14576 rank: 6 used: 202572 free: 3714368 real_used: 479936 } { entry: 7 pid: 14577 rank: 7 used: 202740 free: 3714488 real_used: 479816 } { entry: 8 pid: 14578 rank: 8 used: 199760 free: 3717084 real_used: 477220 } { entry: 9 pid: 14579 rank: 9 used: 199440 free: 3722828 real_used: 471476 } { entry: 10 pid: 14580 rank: 10 used: 199440 free: 3722828 real_used: 471476 } { entry: 11 pid: 14581 rank: 11 used: 199440 free: 3722828 real_used: 471476 } { entry: 12 pid: 14582 rank: 12 used: 199440 free: 3722828 real_used: 471476 } { entry: 13 pid: 14583 rank: 13 used: 199440 free: 3722828 real_used: 471476 } { entry: 14 pid: 14584 rank: 14 used: 199440 free: 3722828 real_used: 471476 } { entry: 15 pid: 14585 rank: 15 used: 199440 free: 3722828 real_used: 471476 } { entry: 16 pid: 14586 rank: 16 used: 199440 free: 3722828 real_used: 471476 } { entry: 17 pid: 14587 rank: -1 used: 199400 free: 3722868 real_used: 471436 } { entry: 18 pid: 14588 rank: -1 used: 2105844 free: 3722868 real_used: 2337944 } { entry: 19 pid: 14589 rank: -2 used: 198552 free: 3723236 real_used: 471068 } { entry: 20 pid: 0 rank: 0 used: 209756 free: 3712368 real_used: 481936 } { entry: 21 pid: 14591 rank: -4 used: 253660 free: 3668704 real_used: 525600 }

On 1 October 2013 22:45, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

send SIGUSR1 to one of the worker processes (e.g., UDP listener). The main process of kamailio doesn't process SIP messages, so it is not exposed to leaks.

You can used 'kamcmd pkg.stats" to see statistics about usage of pkg per each process (older versions have sercmd).

'kamctl ps' gives the list of processes.

Cheers, Daniel

On 10/1/13 2:35 PM, David Cunningham wrote:

Hi Daniel,

What I copied and pasted is all that's listed when I do a kill -SIGUSR1 on the Kamailio master process. How do I get fuller logs?

Thanks for your help.

On 1 October 2013 21:57, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

you gave only the first part of memory log output. The chunks listed there are from startup, thus not part of a leak, because they are not allocated again at runtime.

Can you give the full logs with messages from qm_status? The leaks are typically listed towards the end of those messages.

Yesterday someone else reported a potential leak when using perl module, so it might be something else there. I found one, which is related to an error case, but perhaps you don't get that, otherwise you will notice the error message.

Cheers, Daniel

On 10/1/13 12:03 PM, David Cunningham wrote:

...

Hello,

Following up on this after a long break. It seems we are still having a memory leak. We've added the memory debugging options as per the documentation with the following output. I'm not sure how to read it though - can anyone help? Thank you.

Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core> [main.c:754]: Memory status (pkg): Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: (0x4019b008): Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: heap size= 4194304 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: used= 191100, used+overhead=462800, free=3731504 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max used (+overhead)= 480224 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: dumping all alloc'ed. fragments: Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:

1. N address=0x401cf644 frag=0x401cf62c size=128 used=1

Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: counters.c: cnt_hash_add(331) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 3. N address=0x401cf824 frag=0x401cf80c size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: counters.c: init_counters(121) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 4. N address=0x401cf894 frag=0x401cf87c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 6. N address=0x401cf93c frag=0x401cf924 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 7. N address=0x401cf984 frag=0x401cf96c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: db_id.c: dupl_string(50) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

Hi Daniel,

Thank you - I used the restart method and got a whole load of memory information. 10 megabytes of it! I've compressed that and attached it to this email.

Thanks for your help.

On 2 October 2013 17:17, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

ahh, right, forgot that it was disabled because it was not safe to use at runtime for worker processes. You have to use:

kamcmd cfg.set_now_int core mem_dump_pkg <pid>

then wait for a bit so that process is receiving a SIP message.

Or, if you restart, the stats are printed in syslog for all processes.

Another alternative is to call pkg_status() or pkg_summary() from cfgutils module in config file, like:

if(src_ip==127.0.0.1 && is_method("OPTIONS") && $rU=="pkgstats") { send_reply("200", "OK"); pkg_summary(); exit; }

You can use sipsak to send an options request to "sip:pkgstats@..."

Cheers, Daniel

On 10/2/13 2:30 AM, David Cunningham wrote:

Hi Daniel,

When I send SIGUSR1 to a worker process nothing is logged. Any ideas there? The following is the output of "sercmd pkg.stats".

{ entry: 0 pid: 14570 rank: 0 used: 191100 free: 3731504 real_used: 462800 } { entry: 1 pid: 14571 rank: 1 used: 202656 free: 3714236 real_used: 480068 } { entry: 2 pid: 14572 rank: 2 used: 202720 free: 3714220 real_used: 480084 } { entry: 3 pid: 14573 rank: 3 used: 202664 free: 3714324 real_used: 479980 } { entry: 4 pid: 14574 rank: 4 used: 202744 free: 3714100 real_used: 480204 } { entry: 5 pid: 14575 rank: 5 used: 201908 free: 3715176 real_used: 479128 } { entry: 6 pid: 14576 rank: 6 used: 202572 free: 3714368 real_used: 479936 } { entry: 7 pid: 14577 rank: 7 used: 202740 free: 3714488 real_used: 479816 } { entry: 8 pid: 14578 rank: 8 used: 199760 free: 3717084 real_used: 477220 } { entry: 9 pid: 14579 rank: 9 used: 199440 free: 3722828 real_used: 471476 } { entry: 10 pid: 14580 rank: 10 used: 199440 free: 3722828 real_used: 471476 } { entry: 11 pid: 14581 rank: 11 used: 199440 free: 3722828 real_used: 471476 } { entry: 12 pid: 14582 rank: 12 used: 199440 free: 3722828 real_used: 471476 } { entry: 13 pid: 14583 rank: 13 used: 199440 free: 3722828 real_used: 471476 } { entry: 14 pid: 14584 rank: 14 used: 199440 free: 3722828 real_used: 471476 } { entry: 15 pid: 14585 rank: 15 used: 199440 free: 3722828 real_used: 471476 } { entry: 16 pid: 14586 rank: 16 used: 199440 free: 3722828 real_used: 471476 } { entry: 17 pid: 14587 rank: -1 used: 199400 free: 3722868 real_used: 471436 } { entry: 18 pid: 14588 rank: -1 used: 2105844 free: 3722868 real_used: 2337944 } { entry: 19 pid: 14589 rank: -2 used: 198552 free: 3723236 real_used: 471068 } { entry: 20 pid: 0 rank: 0 used: 209756 free: 3712368 real_used: 481936 } { entry: 21 pid: 14591 rank: -4 used: 253660 free: 3668704 real_used: 525600 }

On 1 October 2013 22:45, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

send SIGUSR1 to one of the worker processes (e.g., UDP listener). The main process of kamailio doesn't process SIP messages, so it is not exposed to leaks.

You can used 'kamcmd pkg.stats" to see statistics about usage of pkg per each process (older versions have sercmd).

'kamctl ps' gives the list of processes.

Cheers, Daniel

On 10/1/13 2:35 PM, David Cunningham wrote:

Hi Daniel,

What I copied and pasted is all that's listed when I do a kill -SIGUSR1 on the Kamailio master process. How do I get fuller logs?

Thanks for your help.

On 1 October 2013 21:57, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

you gave only the first part of memory log output. The chunks listed there are from startup, thus not part of a leak, because they are not allocated again at runtime.

Can you give the full logs with messages from qm_status? The leaks are typically listed towards the end of those messages.

Yesterday someone else reported a potential leak when using perl module, so it might be something else there. I found one, which is related to an error case, but perhaps you don't get that, otherwise you will notice the error message.

Cheers, Daniel

On 10/1/13 12:03 PM, David Cunningham wrote:

...

Hello,

Following up on this after a long break. It seems we are still having a memory leak. We've added the memory debugging options as per the documentation with the following output. I'm not sure how to read it though - can anyone help? Thank you.

Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core> [main.c:754]: Memory status (pkg): Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: (0x4019b008): Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: heap size= 4194304 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: used= 191100, used+overhead=462800, free=3731504 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max used (+overhead)= 480224 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: dumping all alloc'ed. fragments: Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:

1. N address=0x401cf644 frag=0x401cf62c size=128 used=1

Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: counters.c: cnt_hash_add(331) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 3. N address=0x401cf824 frag=0x401cf80c size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: counters.c: init_counters(121) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 4. N address=0x401cf894 frag=0x401cf87c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 6. N address=0x401cf93c frag=0x401cf924 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 7. N address=0x401cf984 frag=0x401cf96c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: db_id.c: dupl_string(50) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: route_add(159) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1 Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: alloc'd from <core>: route.c: init_rlist(180) Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

Hello,

on a second check, those counter chunks for counters/statistics seems to be from startup, thus not part of a leak.

How long have you been running kamailio? Looking at the pkg stats from previous email, the free part is still a lot, with only about 200k used. It could mean the leak was not making any effect yet.

Cheers, Daniel

On 10/2/13 10:19 AM, Daniel-Constantin Mierla wrote:

Hello,

the leak is due to keep defining counters or statistics. That should happen only at startup, but if you execute from perl script some function from a kamailio module, it may trigger this issue. Can you share what functions from kamailio are you execute inside the perl script?

Cheers, Daniel

On 10/2/13 9:26 AM, David Cunningham wrote:

...

Hi Daniel,

Thank you - I used the restart method and got a whole load of memory information. 10 megabytes of it! I've compressed that and attached it to this email.

Thanks for your help.

On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,

ahh, right, forgot that it was disabled because it was not safe
to use at runtime for worker processes. You have to use:

kamcmd cfg.set_now_int core mem_dump_pkg <pid>

then wait for a bit so that process is receiving a SIP message.

Or, if you restart, the stats are printed in syslog for all
processes.

Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:

if(src_ip==127.0.0.1 && is_method("OPTIONS") && $rU=="pkgstats") {
  send_reply("200", "OK");
  pkg_summary();
  exit;
}

You can use sipsak to send an options request to "sip:pkgstats@..."

Cheers,
Daniel


On 10/2/13 2:30 AM, David Cunningham wrote:

...

Hi Daniel,

When I send SIGUSR1 to a worker process nothing is logged. Any
ideas there? The following is the output of "sercmd pkg.stats".

{
    entry: 0
    pid: 14570
    rank: 0
    used: 191100
    free: 3731504
    real_used: 462800
}
{
    entry: 1
    pid: 14571
    rank: 1
    used: 202656
    free: 3714236
    real_used: 480068
}
{
    entry: 2
    pid: 14572
    rank: 2
    used: 202720
    free: 3714220
    real_used: 480084
}
{
    entry: 3
    pid: 14573
    rank: 3
    used: 202664
    free: 3714324
    real_used: 479980
}
{
    entry: 4
    pid: 14574
    rank: 4
    used: 202744
    free: 3714100
    real_used: 480204
}
{
    entry: 5
    pid: 14575
    rank: 5
    used: 201908
    free: 3715176
    real_used: 479128
}
{
    entry: 6
    pid: 14576
    rank: 6
    used: 202572
    free: 3714368
    real_used: 479936
}
{
    entry: 7
    pid: 14577
    rank: 7
    used: 202740
    free: 3714488
    real_used: 479816
}
{
    entry: 8
    pid: 14578
    rank: 8
    used: 199760
    free: 3717084
    real_used: 477220
}
{
    entry: 9
    pid: 14579
    rank: 9
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 10
    pid: 14580
    rank: 10
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 11
    pid: 14581
    rank: 11
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 12
    pid: 14582
    rank: 12
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 13
    pid: 14583
    rank: 13
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 14
    pid: 14584
    rank: 14
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 15
    pid: 14585
    rank: 15
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 16
    pid: 14586
    rank: 16
    used: 199440
    free: 3722828
    real_used: 471476
}
{
    entry: 17
    pid: 14587
    rank: -1
    used: 199400
    free: 3722868
    real_used: 471436
}
{
    entry: 18
    pid: 14588
    rank: -1
    used: 2105844
    free: 3722868
    real_used: 2337944
}
{
    entry: 19
    pid: 14589
    rank: -2
    used: 198552
    free: 3723236
    real_used: 471068
}
{
    entry: 20
    pid: 0
    rank: 0
    used: 209756
    free: 3712368
    real_used: 481936
}
{
    entry: 21
    pid: 14591
    rank: -4
    used: 253660
    free: 3668704
    real_used: 525600
}




On 1 October 2013 22:45, Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    send SIGUSR1 to one of the worker processes (e.g., UDP
    listener). The main process of kamailio doesn't process SIP
    messages, so it is not exposed to leaks.

    You can used 'kamcmd pkg.stats" to see statistics about
    usage of pkg  per each process (older versions have sercmd).

    'kamctl ps' gives the list of processes.

    Cheers,
    Daniel


    On 10/1/13 2:35 PM, David Cunningham wrote:

...

    Hi Daniel,

    What I copied and pasted is all that's listed when I do a
    kill -SIGUSR1 on the Kamailio master process. How do I get
    fuller logs?

    Thanks for your help.



    On 1 October 2013 21:57, Daniel-Constantin Mierla
    <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

        Hello,

        you gave only the first part of memory log output. The
        chunks listed there are from startup, thus not part of
        a leak, because they are not allocated again at runtime.

        Can you give the full logs with messages from
        qm_status? The leaks are typically listed towards the
        end of those messages.

        Yesterday someone else reported a potential leak when
        using perl module, so it might be something else there.
        I found one, which is related to an error case, but
        perhaps you don't get that, otherwise you will notice
        the error message.

        Cheers,
        Daniel


        On 10/1/13 12:03 PM, David Cunningham wrote:

            Hello,

            Following up on this after a long break. It seems
            we are still having a memory leak. We've added the
            memory debugging options as per the documentation
            with the following output.
            I'm not sure how to read it though - can anyone
            help? Thank you.

            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: <core> [main.c:754]: Memory status (pkg):
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: (0x4019b008):
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: heap size= 4194304
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: used= 191100,
            used+overhead=462800, free=3731504
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: max used (+overhead)= 480224
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: dumping all alloc'ed. fragments:
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  0. N  address=0x401cf414
            frag=0x401cf3fc size=512 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            str_hash.h: str_hash_alloc(69)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  1. N  address=0x401cf644
            frag=0x401cf62c size=128 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            str_hash.h: str_hash_alloc(69)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  2. N  address=0x401cf6f4
            frag=0x401cf6dc size=256 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            counters.c: cnt_hash_add(331)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  3. N  address=0x401cf824
            frag=0x401cf80c size=64 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            counters.c: init_counters(121)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  4. N  address=0x401cf894
            frag=0x401cf87c size=8 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            route.c: init_rlist(180)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  5. N  address=0x401cf8cc
            frag=0x401cf8b4 size=64 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            str_hash.h: str_hash_alloc(69)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  6. N  address=0x401cf93c
            frag=0x401cf924 size=24 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            route.c: route_add(159)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  7. N  address=0x401cf984
            frag=0x401cf96c size=8 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            route.c: init_rlist(180)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  8. N  address=0x401cf9bc
            frag=0x401cf9a4 size=64 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            str_hash.h: str_hash_alloc(69)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:  9. N  address=0x401cfa2c
            frag=0x401cfa14 size=24 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            route.c: route_add(159)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: 10. N  address=0x401cfa74
            frag=0x401cfa5c size=8 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            db_id.c: dupl_string(50)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: 11. N  address=0x401cfaac
            frag=0x401cfa94 size=64 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            str_hash.h: str_hash_alloc(69)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: 12. N  address=0x401cfb1c
            frag=0x401cfb04 size=24 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            route.c: route_add(159)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: 13. N  address=0x401cfb64
            frag=0x401cfb4c size=8 used=1
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:     alloc'd from <core>:
            route.c: init_rlist(180)
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status:    start check=f0f0f0f0, end
            check= c0c0c0c0, abcdefed
            Oct  1 05:58:48 sip0-test /sbin/kamailio[14570]:
            NOTICE: qm_status: 14. N  address=0x401cfb9c
            frag=0x401cfb84 size=64 used=1



        -- 
        Daniel-Constantin Mierla - http://www.asipto.com
        http://twitter.com/#!/miconda
        <http://twitter.com/#%21/miconda> -
        http://www.linkedin.com/in/miconda
        Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
        Nov 18-20, 2013
          - more details about Kamailio trainings at
        http://www.asipto.com -




    -- 
    David Cunningham, Voisonics
    http://voisonics.com/
    USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
    UK: +44 (0) 20 3298 1642
    <tel:%2B44%20%280%29%2020%203298%201642>
    Australia: +61 (0) 2 8063 9019
    <tel:%2B61%20%280%29%202%208063%209019>

    -- 
    Daniel-Constantin Mierla -http://www.asipto.com
    http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
    Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
       - more details about Kamailio trainings athttp://www.asipto.com  -




-- 
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>

-- 
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
   - more details about Kamailio trainings athttp://www.asipto.com  -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla -http://www.asipto.com http://twitter.com/#!/miconda -http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings athttp://www.asipto.com -

Hello,

yes, you should run it for a while, with traffic, otherwise the leak cannot be discovered.

You can watch the processes with rank 1,2,3 and see when the free memory is decreasing. Then get the pkg status logs. Get also the list of running processes with 'kamctl ps' so I know at what pid to look for the leaks.

Cheers, Daniel

On 10/2/13 2:09 PM, David Cunningham wrote:

Hi Daniel,

Kamailio had only been running for about 24 hours with very little traffic on that server. Should we wait for longer and throw some calls at it?

Thanks again.

Hi Daniel,

Here are the 'kamctl ps' and pkg status logs. During the 24 hours it ran for lots of REGISTER packets came through and memory use definitely increased. Thank you.

Process:: ID=0 PID=6822 Type=attendant Process:: ID=1 PID=6824 Type=udp receiver child=0 sock=67.58.182.98:5060 Process:: ID=2 PID=6825 Type=udp receiver child=1 sock=67.58.182.98:5060 Process:: ID=3 PID=6826 Type=udp receiver child=2 sock=67.58.182.98:5060 Process:: ID=4 PID=6827 Type=udp receiver child=3 sock=67.58.182.98:5060 Process:: ID=5 PID=6828 Type=udp receiver child=4 sock=67.58.182.98:5060 Process:: ID=6 PID=6829 Type=udp receiver child=5 sock=67.58.182.98:5060 Process:: ID=7 PID=6830 Type=udp receiver child=6 sock=67.58.182.98:5060 Process:: ID=8 PID=6831 Type=udp receiver child=7 sock=67.58.182.98:5060 Process:: ID=9 PID=6832 Type=udp receiver child=0 sock=67.58.182.98:5061 Process:: ID=10 PID=6833 Type=udp receiver child=1 sock=67.58.182.98:5061 Process:: ID=11 PID=6834 Type=udp receiver child=2 sock=67.58.182.98:5061 Process:: ID=12 PID=6835 Type=udp receiver child=3 sock=67.58.182.98:5061 Process:: ID=13 PID=6836 Type=udp receiver child=4 sock=67.58.182.98:5061 Process:: ID=14 PID=6837 Type=udp receiver child=5 sock=67.58.182.98:5061 Process:: ID=15 PID=6838 Type=udp receiver child=6 sock=67.58.182.98:5061 Process:: ID=16 PID=6839 Type=udp receiver child=7 sock=67.58.182.98:5061 Process:: ID=17 PID=6840 Type=udp receiver child=0 sock=127.0.0.1:5060 Process:: ID=18 PID=6841 Type=udp receiver child=1 sock=127.0.0.1:5060 Process:: ID=19 PID=6842 Type=udp receiver child=2 sock=127.0.0.1:5060 Process:: ID=20 PID=6843 Type=udp receiver child=3 sock=127.0.0.1:5060 Process:: ID=21 PID=6844 Type=udp receiver child=4 sock=127.0.0.1:5060 Process:: ID=22 PID=6845 Type=udp receiver child=5 sock=127.0.0.1:5060 Process:: ID=23 PID=6846 Type=udp receiver child=6 sock=127.0.0.1:5060 Process:: ID=24 PID=6847 Type=udp receiver child=7 sock=127.0.0.1:5060 Process:: ID=25 PID=6848 Type=slow timer Process:: ID=26 PID=6849 Type=timer Process:: ID=27 PID=6850 Type=ctl handler Process:: ID=28 PID=6851 Type=MI FIFO Process:: ID=29 PID=6852 Type=tcp main process

Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: summarizing all alloc'ed. fragments: Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 8192 bytes from mi_fifo: mi_writer.c: mi_writer_init(57) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 964 bytes from db_mysql: km_my_con.c: db_mysql_new_connection(62) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 512 bytes from <core>: counters.c: cnt_hash_add(382) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from auth: auth_mod.c: generate_random_secret(221) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 76 bytes from auth: auth_mod.c: generate_random_secret(220) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 60 bytes from <core>: script_cb.c: add_callback(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: select.c: register_select_table(461) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c: cfg_new_group(86) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 228 bytes from <core>: socket_info.c: fix_sock_str(436) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from <core>: udp_server.c: udp_rcv_loop(434) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from db_mysql: km_my_con.c: db_mysql_new_connection(53) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 84 bytes from <core>: name_alias.h: add_alias(101) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 112 bytes from <core>: name_alias.h: add_alias(99) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 44 bytes from <core>: socket_info.c: fix_hostname(1377) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 9 size= 392 bytes from <core>: counters.c: grp_hash_add(233) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 24 bytes from <core>: socket_info.c: fix_socket_list(1581) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 900 bytes from <core>: receive.c: receive_msg(120) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 336 bytes from <core>: parser/msg_parser.c: get_hdr_field(171) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 96 bytes from <core>: parser/parse_via.c: parse_via(2429) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 116 bytes from <core>: parser/msg_parser.c: get_hdr_field(131) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 12 bytes from rtpproxy: rtpproxy.c: child_init(907) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: parser/parse_to.c: parse_to_param(287) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: parser/msg_parser.c: get_hdr_field(150) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 328 bytes from <core>: parser/msg_parser.c: parse_headers(341) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 240 bytes from sanity: sanity.c: check_parse_uris(767) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: route.c: route_new_list(234) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: sr_module.c: fix_param(1142) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 56 bytes from <core>: switch.c: mk_switch_cond_table(54) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 4 size= 248 bytes from <core>: sr_module.c: fix_param(1203) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 72 bytes from <core>: mod_fix.c: fixup_regexp_null(219) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 15 size= 228 bytes from textops: textops.c: fixup_method(1705) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 4 size= 140 bytes from textops: textops.c: hname_fixup(1634) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 8 size= 448 bytes from <core>: pvapi.c: pv_parse_format(954) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 44 bytes from <core>: rvalue.c: mk_rval_expr_v(2545) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 9000 bytes from <core>: rvalue.c: mk_rval_expr2(2669) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: route_struct.c: mk_elem(127) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 18 size= 16200 bytes from <core>: rvalue.c: mk_rval_expr1(2611) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 58 size= 52200 bytes from <core>: rvalue.c: mk_rval_expr_v(2534) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 180 size= 22032 bytes from <core>: route_struct.c: mk_action(155) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from <core>: re.c: subst_str(533) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from xlog: xlog.c: xdbg_fixup_helper(346) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 86 size= 1752 bytes from <core>: rvalue.c: rval_get_str(1260) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 46 size= 1592 bytes from <core>: sr_module.c: fix_param(1108) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: sr_module.c: init_modules(971) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 7 size= 140 bytes from <core>: pvapi.c: tr_table_add(1660) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 21 size= 9828 bytes from <core>: sr_module.c: sr_cmd_exports_convert(234) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 23 size= 1656 bytes from <core>: sr_module.c: register_module(278) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 23 size= 856 bytes from <core>: sr_module.c: load_module(485) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 312 bytes from <core>: modparam.c: set_mod_param_regex(124) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: cfg.lex: sr_push_yy_state(1692) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: re.c: subst_parser(315) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 80 bytes from <core>: re.c: subst_parser(306) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: re.c: subst_parser(289) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 293 size= 38296 bytes from <core>: cfg.lex: addstr(1439) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 14 size= 276 bytes from <core>: cfg.lex: pp_define(1799) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 20 bytes from <core>: cfg.lex: sr_push_yy_state(1682) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 44 bytes from <core>: socket_info.c: new_sock_info(251) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 528 bytes from <core>: socket_info.c: new_sock_info(246) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 256 bytes from <core>: rpc_lookup.c: rpc_hash_add(156) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 63 size= 7048 bytes from <core>: rpc_lookup.c: rpc_hash_add(110) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 256 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 10240 bytes from <core>: pvapi.c: pv_init_buffer(1831) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 40 bytes from <core>: pvapi.c: pv_init_buffer(1821) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 12 bytes from <core>: nonsip_hooks.c: init_nonsip_hooks(57) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 248 bytes from <core>: db_id.c: dupl_string(50) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 240 bytes from <core>: route.c: route_add(159) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 6 size= 384 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 36 bytes from <core>: route.c: init_rlist(180) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: counters.c: init_counters(121) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 73 size= 9044 bytes from <core>: counters.c: cnt_hash_add(331) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 640 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: -----------------------------

On 3 October 2013 11:14, David Cunningham dcunningham@voisonics.com wrote:

Thanks Daniel. We will provide feedback when we have let it run for a while.

On 2 October 2013 22:51, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

yes, you should run it for a while, with traffic, otherwise the leak cannot be discovered.

You can watch the processes with rank 1,2,3 and see when the free memory is decreasing. Then get the pkg status logs. Get also the list of running processes with 'kamctl ps' so I know at what pid to look for the leaks.

Cheers, Daniel

On 10/2/13 2:09 PM, David Cunningham wrote:

...

Hi Daniel,

Kamailio had only been running for about 24 hours with very little traffic on that server. Should we wait for longer and throw some calls at it?

Thanks again.

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/**micondahttp://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

Hi Daniel,

How do I get that information please? The information above includes what I got from the pkgstats module. Is it by restarting Kamailio?

On 4 October 2013 16:42, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

none of the chunks there are indication of a leak. Can you give the output of pkg.stats for pid 6824? Which of the processes has the highest value of used pkg memory?

Cheers, Daniel

On 10/4/13 4:25 AM, David Cunningham wrote:

Hi Daniel,

Here are the 'kamctl ps' and pkg status logs. During the 24 hours it ran for lots of REGISTER packets came through and memory use definitely increased. Thank you.

Process:: ID=0 PID=6822 Type=attendant Process:: ID=1 PID=6824 Type=udp receiver child=0 sock=67.58.182.98:5060 Process:: ID=2 PID=6825 Type=udp receiver child=1 sock=67.58.182.98:5060 Process:: ID=3 PID=6826 Type=udp receiver child=2 sock=67.58.182.98:5060 Process:: ID=4 PID=6827 Type=udp receiver child=3 sock=67.58.182.98:5060 Process:: ID=5 PID=6828 Type=udp receiver child=4 sock=67.58.182.98:5060 Process:: ID=6 PID=6829 Type=udp receiver child=5 sock=67.58.182.98:5060 Process:: ID=7 PID=6830 Type=udp receiver child=6 sock=67.58.182.98:5060 Process:: ID=8 PID=6831 Type=udp receiver child=7 sock=67.58.182.98:5060 Process:: ID=9 PID=6832 Type=udp receiver child=0 sock=67.58.182.98:5061 Process:: ID=10 PID=6833 Type=udp receiver child=1 sock=67.58.182.98:5061 Process:: ID=11 PID=6834 Type=udp receiver child=2 sock=67.58.182.98:5061 Process:: ID=12 PID=6835 Type=udp receiver child=3 sock=67.58.182.98:5061 Process:: ID=13 PID=6836 Type=udp receiver child=4 sock=67.58.182.98:5061 Process:: ID=14 PID=6837 Type=udp receiver child=5 sock=67.58.182.98:5061 Process:: ID=15 PID=6838 Type=udp receiver child=6 sock=67.58.182.98:5061 Process:: ID=16 PID=6839 Type=udp receiver child=7 sock=67.58.182.98:5061 Process:: ID=17 PID=6840 Type=udp receiver child=0 sock=127.0.0.1:5060 Process:: ID=18 PID=6841 Type=udp receiver child=1 sock=127.0.0.1:5060 Process:: ID=19 PID=6842 Type=udp receiver child=2 sock=127.0.0.1:5060 Process:: ID=20 PID=6843 Type=udp receiver child=3 sock=127.0.0.1:5060 Process:: ID=21 PID=6844 Type=udp receiver child=4 sock=127.0.0.1:5060 Process:: ID=22 PID=6845 Type=udp receiver child=5 sock=127.0.0.1:5060 Process:: ID=23 PID=6846 Type=udp receiver child=6 sock=127.0.0.1:5060 Process:: ID=24 PID=6847 Type=udp receiver child=7 sock=127.0.0.1:5060 Process:: ID=25 PID=6848 Type=slow timer Process:: ID=26 PID=6849 Type=timer Process:: ID=27 PID=6850 Type=ctl handler Process:: ID=28 PID=6851 Type=MI FIFO Process:: ID=29 PID=6852 Type=tcp main process

Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: summarizing all alloc'ed. fragments: Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 8192 bytes from mi_fifo: mi_writer.c: mi_writer_init(57) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 964 bytes from db_mysql: km_my_con.c: db_mysql_new_connection(62) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 512 bytes from <core>: counters.c: cnt_hash_add(382) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from auth: auth_mod.c: generate_random_secret(221) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 76 bytes from auth: auth_mod.c: generate_random_secret(220) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 60 bytes from <core>: script_cb.c: add_callback(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: select.c: register_select_table(461) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c: cfg_new_group(86) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 228 bytes from <core>: socket_info.c: fix_sock_str(436) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from <core>: udp_server.c: udp_rcv_loop(434) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from db_mysql: km_my_con.c: db_mysql_new_connection(53) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 84 bytes from <core>: name_alias.h: add_alias(101) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 112 bytes from <core>: name_alias.h: add_alias(99) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 44 bytes from <core>: socket_info.c: fix_hostname(1377) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 9 size= 392 bytes from <core>: counters.c: grp_hash_add(233) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 24 bytes from <core>: socket_info.c: fix_socket_list(1581) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 900 bytes from <core>: receive.c: receive_msg(120) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 336 bytes from <core>: parser/msg_parser.c: get_hdr_field(171) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 96 bytes from <core>: parser/parse_via.c: parse_via(2429) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 116 bytes from <core>: parser/msg_parser.c: get_hdr_field(131) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 12 bytes from rtpproxy: rtpproxy.c: child_init(907) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: parser/parse_to.c: parse_to_param(287) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: parser/msg_parser.c: get_hdr_field(150) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 328 bytes from <core>: parser/msg_parser.c: parse_headers(341) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 240 bytes from sanity: sanity.c: check_parse_uris(767) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: route.c: route_new_list(234) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: sr_module.c: fix_param(1142) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 56 bytes from <core>: switch.c: mk_switch_cond_table(54) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 4 size= 248 bytes from <core>: sr_module.c: fix_param(1203) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 72 bytes from <core>: mod_fix.c: fixup_regexp_null(219) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 15 size= 228 bytes from textops: textops.c: fixup_method(1705) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 4 size= 140 bytes from textops: textops.c: hname_fixup(1634) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 8 size= 448 bytes from <core>: pvapi.c: pv_parse_format(954) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 44 bytes from <core>: rvalue.c: mk_rval_expr_v(2545) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 9000 bytes from <core>: rvalue.c: mk_rval_expr2(2669) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: route_struct.c: mk_elem(127) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 18 size= 16200 bytes from <core>: rvalue.c: mk_rval_expr1(2611) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 58 size= 52200 bytes from <core>: rvalue.c: mk_rval_expr_v(2534) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 180 size= 22032 bytes from <core>: route_struct.c: mk_action(155) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from <core>: re.c: subst_str(533) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from xlog: xlog.c: xdbg_fixup_helper(346) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 86 size= 1752 bytes from <core>: rvalue.c: rval_get_str(1260) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 46 size= 1592 bytes from <core>: sr_module.c: fix_param(1108) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: sr_module.c: init_modules(971) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 7 size= 140 bytes from <core>: pvapi.c: tr_table_add(1660) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 21 size= 9828 bytes from <core>: sr_module.c: sr_cmd_exports_convert(234) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 23 size= 1656 bytes from <core>: sr_module.c: register_module(278) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 23 size= 856 bytes from <core>: sr_module.c: load_module(485) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 312 bytes from <core>: modparam.c: set_mod_param_regex(124) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: cfg.lex: sr_push_yy_state(1692) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: re.c: subst_parser(315) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 80 bytes from <core>: re.c: subst_parser(306) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: re.c: subst_parser(289) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 293 size= 38296 bytes from <core>: cfg.lex: addstr(1439) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 14 size= 276 bytes from <core>: cfg.lex: pp_define(1799) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 20 bytes from <core>: cfg.lex: sr_push_yy_state(1682) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 44 bytes from <core>: socket_info.c: new_sock_info(251) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 528 bytes from <core>: socket_info.c: new_sock_info(246) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 256 bytes from <core>: rpc_lookup.c: rpc_hash_add(156) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 63 size= 7048 bytes from <core>: rpc_lookup.c: rpc_hash_add(110) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 256 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 10240 bytes from <core>: pvapi.c: pv_init_buffer(1831) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 40 bytes from <core>: pvapi.c: pv_init_buffer(1821) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 12 bytes from <core>: nonsip_hooks.c: init_nonsip_hooks(57) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 248 bytes from <core>: db_id.c: dupl_string(50) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 240 bytes from <core>: route.c: route_add(159) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 6 size= 384 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 36 bytes from <core>: route.c: init_rlist(180) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: counters.c: init_counters(121) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 73 size= 9044 bytes from <core>: counters.c: cnt_hash_add(331) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 640 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:

---

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

Hi Daniel,

I was looking to give you the information you requested in your previous reply for PID 6824. Hopefully this is it:

# sercmd pkg.stats pid 6824 { entry: 1 pid: 6824 rank: 1 used: 209836 free: 3704080 real_used: 490224 }

On 10 October 2013 21:41, Daniel-Constantin Mierla miconda@gmail.comwrote:

Hello,

getting pkg stats for a process:

sercmd pkg.stats pid 1234

replace 1234 with the real PID value. There are options to use rank or index, see:

• http://kamailio.org/docs/modules/stable/modules/kex.html#idp213872

Is it what you were looking for or did I misunderstood?

Cheers, Daniel

On 10/10/13 6:48 AM, David Cunningham wrote:

Hi Daniel,

How do I get that information please? The information above includes what I got from the pkgstats module. Is it by restarting Kamailio?

On 4 October 2013 16:42, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

none of the chunks there are indication of a leak. Can you give the output of pkg.stats for pid 6824? Which of the processes has the highest value of used pkg memory?

Cheers, Daniel

On 10/4/13 4:25 AM, David Cunningham wrote:

Hi Daniel,

Here are the 'kamctl ps' and pkg status logs. During the 24 hours it ran for lots of REGISTER packets came through and memory use definitely increased. Thank you.

Process:: ID=0 PID=6822 Type=attendant Process:: ID=1 PID=6824 Type=udp receiver child=0 sock=67.58.182.98:5060 Process:: ID=2 PID=6825 Type=udp receiver child=1 sock=67.58.182.98:5060 Process:: ID=3 PID=6826 Type=udp receiver child=2 sock=67.58.182.98:5060 Process:: ID=4 PID=6827 Type=udp receiver child=3 sock=67.58.182.98:5060 Process:: ID=5 PID=6828 Type=udp receiver child=4 sock=67.58.182.98:5060 Process:: ID=6 PID=6829 Type=udp receiver child=5 sock=67.58.182.98:5060 Process:: ID=7 PID=6830 Type=udp receiver child=6 sock=67.58.182.98:5060 Process:: ID=8 PID=6831 Type=udp receiver child=7 sock=67.58.182.98:5060 Process:: ID=9 PID=6832 Type=udp receiver child=0 sock=67.58.182.98:5061 Process:: ID=10 PID=6833 Type=udp receiver child=1 sock= 67.58.182.98:5061 Process:: ID=11 PID=6834 Type=udp receiver child=2 sock= 67.58.182.98:5061 Process:: ID=12 PID=6835 Type=udp receiver child=3 sock= 67.58.182.98:5061 Process:: ID=13 PID=6836 Type=udp receiver child=4 sock= 67.58.182.98:5061 Process:: ID=14 PID=6837 Type=udp receiver child=5 sock= 67.58.182.98:5061 Process:: ID=15 PID=6838 Type=udp receiver child=6 sock= 67.58.182.98:5061 Process:: ID=16 PID=6839 Type=udp receiver child=7 sock= 67.58.182.98:5061 Process:: ID=17 PID=6840 Type=udp receiver child=0 sock=127.0.0.1:5060 Process:: ID=18 PID=6841 Type=udp receiver child=1 sock=127.0.0.1:5060 Process:: ID=19 PID=6842 Type=udp receiver child=2 sock=127.0.0.1:5060 Process:: ID=20 PID=6843 Type=udp receiver child=3 sock=127.0.0.1:5060 Process:: ID=21 PID=6844 Type=udp receiver child=4 sock=127.0.0.1:5060 Process:: ID=22 PID=6845 Type=udp receiver child=5 sock=127.0.0.1:5060 Process:: ID=23 PID=6846 Type=udp receiver child=6 sock=127.0.0.1:5060 Process:: ID=24 PID=6847 Type=udp receiver child=7 sock=127.0.0.1:5060 Process:: ID=25 PID=6848 Type=slow timer Process:: ID=26 PID=6849 Type=timer Process:: ID=27 PID=6850 Type=ctl handler Process:: ID=28 PID=6851 Type=MI FIFO Process:: ID=29 PID=6852 Type=tcp main process

Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: summarizing all alloc'ed. fragments: Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 8192 bytes from mi_fifo: mi_writer.c: mi_writer_init(57) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 964 bytes from db_mysql: km_my_con.c: db_mysql_new_connection(62) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 512 bytes from <core>: counters.c: cnt_hash_add(382) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from auth: auth_mod.c: generate_random_secret(221) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 76 bytes from auth: auth_mod.c: generate_random_secret(220) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 60 bytes from <core>: script_cb.c: add_callback(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: select.c: register_select_table(461) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c: cfg_new_group(86) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 228 bytes from <core>: socket_info.c: fix_sock_str(436) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from <core>: udp_server.c: udp_rcv_loop(434) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from db_mysql: km_my_con.c: db_mysql_new_connection(53) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 84 bytes from <core>: name_alias.h: add_alias(101) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 112 bytes from <core>: name_alias.h: add_alias(99) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 44 bytes from <core>: socket_info.c: fix_hostname(1377) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 9 size= 392 bytes from <core>: counters.c: grp_hash_add(233) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 24 bytes from <core>: socket_info.c: fix_socket_list(1581) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 900 bytes from <core>: receive.c: receive_msg(120) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 336 bytes from <core>: parser/msg_parser.c: get_hdr_field(171) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 96 bytes from <core>: parser/parse_via.c: parse_via(2429) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 116 bytes from <core>: parser/msg_parser.c: get_hdr_field(131) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 12 bytes from rtpproxy: rtpproxy.c: child_init(907) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: parser/parse_to.c: parse_to_param(287) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: parser/msg_parser.c: get_hdr_field(150) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 328 bytes from <core>: parser/msg_parser.c: parse_headers(341) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 240 bytes from sanity: sanity.c: check_parse_uris(767) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: route.c: route_new_list(234) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: sr_module.c: fix_param(1142) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 56 bytes from <core>: switch.c: mk_switch_cond_table(54) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 4 size= 248 bytes from <core>: sr_module.c: fix_param(1203) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 72 bytes from <core>: mod_fix.c: fixup_regexp_null(219) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 15 size= 228 bytes from textops: textops.c: fixup_method(1705) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 4 size= 140 bytes from textops: textops.c: hname_fixup(1634) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 8 size= 448 bytes from <core>: pvapi.c: pv_parse_format(954) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 44 bytes from <core>: rvalue.c: mk_rval_expr_v(2545) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 9000 bytes from <core>: rvalue.c: mk_rval_expr2(2669) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: route_struct.c: mk_elem(127) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 18 size= 16200 bytes from <core>: rvalue.c: mk_rval_expr1(2611) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 58 size= 52200 bytes from <core>: rvalue.c: mk_rval_expr_v(2534) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 180 size= 22032 bytes from <core>: route_struct.c: mk_action(155) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from <core>: re.c: subst_str(533) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from xlog: xlog.c: xdbg_fixup_helper(346) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 86 size= 1752 bytes from <core>: rvalue.c: rval_get_str(1260) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 46 size= 1592 bytes from <core>: sr_module.c: fix_param(1108) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: sr_module.c: init_modules(971) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 7 size= 140 bytes from <core>: pvapi.c: tr_table_add(1660) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 21 size= 9828 bytes from <core>: sr_module.c: sr_cmd_exports_convert(234) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 23 size= 1656 bytes from <core>: sr_module.c: register_module(278) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 23 size= 856 bytes from <core>: sr_module.c: load_module(485) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 312 bytes from <core>: modparam.c: set_mod_param_regex(124) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: cfg.lex: sr_push_yy_state(1692) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: re.c: subst_parser(315) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 80 bytes from <core>: re.c: subst_parser(306) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: re.c: subst_parser(289) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 293 size= 38296 bytes from <core>: cfg.lex: addstr(1439) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 14 size= 276 bytes from <core>: cfg.lex: pp_define(1799) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 20 bytes from <core>: cfg.lex: sr_push_yy_state(1682) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 44 bytes from <core>: socket_info.c: new_sock_info(251) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 528 bytes from <core>: socket_info.c: new_sock_info(246) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 256 bytes from <core>: rpc_lookup.c: rpc_hash_add(156) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 63 size= 7048 bytes from <core>: rpc_lookup.c: rpc_hash_add(110) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 256 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 10240 bytes from <core>: pvapi.c: pv_init_buffer(1831) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 40 bytes from <core>: pvapi.c: pv_init_buffer(1821) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 12 bytes from <core>: nonsip_hooks.c: init_nonsip_hooks(57) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 248 bytes from <core>: db_id.c: dupl_string(50) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 240 bytes from <core>: route.c: route_add(159) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 6 size= 384 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 36 bytes from <core>: route.c: init_rlist(180) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: counters.c: init_counters(121) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 73 size= 9044 bytes from <core>: counters.c: cnt_hash_add(331) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 640 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:

---

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

Hi Daniel,

Thanks for the reply. Perhaps what we're seeing is normal, and the memory use is meant to increase as time progresses. Would you expect to see an ongoing memory use increase, or when should it stop increasing?

On 11 October 2013 00:49, Daniel-Constantin Mierla miconda@gmail.comwrote:

Hello,

On 10/10/13 1:14 PM, David Cunningham wrote:

Hi Daniel,

I was looking to give you the information you requested in your previous reply for PID 6824. Hopefully this is it:

# sercmd pkg.stats pid 6824 { entry: 1 pid: 6824 rank: 1 used: 209836 free: 3704080 real_used: 490224 }

used memory is like 200k out of 4MB, so it is not big and the log you sent with memory chunks didn't reveal a potential leaking. Maybe it is a corner case not hit so far.

You need to monitor for a while and when you see that used memory value has increased with few more 100k (or getting at 1MB) and doesn't go back low, then dump again the log with memory chunks and sent it over.

Cheers, Daniel

On 10 October 2013 21:41, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

getting pkg stats for a process:

sercmd pkg.stats pid 1234

replace 1234 with the real PID value. There are options to use rank or index, see:

• http://kamailio.org/docs/modules/stable/modules/kex.html#idp213872

Is it what you were looking for or did I misunderstood?

Cheers, Daniel

On 10/10/13 6:48 AM, David Cunningham wrote:

Hi Daniel,

How do I get that information please? The information above includes what I got from the pkgstats module. Is it by restarting Kamailio?

On 4 October 2013 16:42, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hello,

none of the chunks there are indication of a leak. Can you give the output of pkg.stats for pid 6824? Which of the processes has the highest value of used pkg memory?

Cheers, Daniel

On 10/4/13 4:25 AM, David Cunningham wrote:

Hi Daniel,

Here are the 'kamctl ps' and pkg status logs. During the 24 hours it ran for lots of REGISTER packets came through and memory use definitely increased. Thank you.

Process:: ID=0 PID=6822 Type=attendant Process:: ID=1 PID=6824 Type=udp receiver child=0 sock= 67.58.182.98:5060 Process:: ID=2 PID=6825 Type=udp receiver child=1 sock= 67.58.182.98:5060 Process:: ID=3 PID=6826 Type=udp receiver child=2 sock= 67.58.182.98:5060 Process:: ID=4 PID=6827 Type=udp receiver child=3 sock= 67.58.182.98:5060 Process:: ID=5 PID=6828 Type=udp receiver child=4 sock= 67.58.182.98:5060 Process:: ID=6 PID=6829 Type=udp receiver child=5 sock= 67.58.182.98:5060 Process:: ID=7 PID=6830 Type=udp receiver child=6 sock= 67.58.182.98:5060 Process:: ID=8 PID=6831 Type=udp receiver child=7 sock= 67.58.182.98:5060 Process:: ID=9 PID=6832 Type=udp receiver child=0 sock= 67.58.182.98:5061 Process:: ID=10 PID=6833 Type=udp receiver child=1 sock= 67.58.182.98:5061 Process:: ID=11 PID=6834 Type=udp receiver child=2 sock= 67.58.182.98:5061 Process:: ID=12 PID=6835 Type=udp receiver child=3 sock= 67.58.182.98:5061 Process:: ID=13 PID=6836 Type=udp receiver child=4 sock= 67.58.182.98:5061 Process:: ID=14 PID=6837 Type=udp receiver child=5 sock= 67.58.182.98:5061 Process:: ID=15 PID=6838 Type=udp receiver child=6 sock= 67.58.182.98:5061 Process:: ID=16 PID=6839 Type=udp receiver child=7 sock= 67.58.182.98:5061 Process:: ID=17 PID=6840 Type=udp receiver child=0 sock=127.0.0.1:5060 Process:: ID=18 PID=6841 Type=udp receiver child=1 sock=127.0.0.1:5060 Process:: ID=19 PID=6842 Type=udp receiver child=2 sock=127.0.0.1:5060 Process:: ID=20 PID=6843 Type=udp receiver child=3 sock=127.0.0.1:5060 Process:: ID=21 PID=6844 Type=udp receiver child=4 sock=127.0.0.1:5060 Process:: ID=22 PID=6845 Type=udp receiver child=5 sock=127.0.0.1:5060 Process:: ID=23 PID=6846 Type=udp receiver child=6 sock=127.0.0.1:5060 Process:: ID=24 PID=6847 Type=udp receiver child=7 sock=127.0.0.1:5060 Process:: ID=25 PID=6848 Type=slow timer Process:: ID=26 PID=6849 Type=timer Process:: ID=27 PID=6850 Type=ctl handler Process:: ID=28 PID=6851 Type=MI FIFO Process:: ID=29 PID=6852 Type=tcp main process

Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: summarizing all alloc'ed. fragments: Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 8192 bytes from mi_fifo: mi_writer.c: mi_writer_init(57) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 964 bytes from db_mysql: km_my_con.c: db_mysql_new_connection(62) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 512 bytes from <core>: counters.c: cnt_hash_add(382) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from auth: auth_mod.c: generate_random_secret(221) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 76 bytes from auth: auth_mod.c: generate_random_secret(220) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 60 bytes from <core>: script_cb.c: add_callback(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 8 bytes from <core>: select.c: register_select_table(461) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c: cfg_new_group(86) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 228 bytes from <core>: socket_info.c: fix_sock_str(436) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from <core>: udp_server.c: udp_rcv_loop(434) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from db_mysql: km_my_con.c: db_mysql_new_connection(53) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 84 bytes from <core>: name_alias.h: add_alias(101) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 112 bytes from <core>: name_alias.h: add_alias(99) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 44 bytes from <core>: socket_info.c: fix_hostname(1377) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 9 size= 392 bytes from <core>: counters.c: grp_hash_add(233) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 24 bytes from <core>: socket_info.c: fix_socket_list(1581) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 900 bytes from <core>: receive.c: receive_msg(120) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 336 bytes from <core>: parser/msg_parser.c: get_hdr_field(171) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 96 bytes from <core>: parser/parse_via.c: parse_via(2429) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 116 bytes from <core>: parser/msg_parser.c: get_hdr_field(131) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 12 bytes from rtpproxy: rtpproxy.c: child_init(907) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: parser/parse_to.c: parse_to_param(287) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: parser/msg_parser.c: get_hdr_field(150) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 328 bytes from <core>: parser/msg_parser.c: parse_headers(341) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 240 bytes from sanity: sanity.c: check_parse_uris(767) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 16 bytes from <core>: route.c: route_new_list(234) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: sr_module.c: fix_param(1142) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 56 bytes from <core>: switch.c: mk_switch_cond_table(54) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 4 size= 248 bytes from <core>: sr_module.c: fix_param(1203) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 72 bytes from <core>: mod_fix.c: fixup_regexp_null(219) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 15 size= 228 bytes from textops: textops.c: fixup_method(1705) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 4 size= 140 bytes from textops: textops.c: hname_fixup(1634) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 8 size= 448 bytes from <core>: pvapi.c: pv_parse_format(954) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 44 bytes from <core>: rvalue.c: mk_rval_expr_v(2545) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 9000 bytes from <core>: rvalue.c: mk_rval_expr2(2669) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 64 bytes from <core>: route_struct.c: mk_elem(127) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 18 size= 16200 bytes from <core>: rvalue.c: mk_rval_expr1(2611) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 58 size= 52200 bytes from <core>: rvalue.c: mk_rval_expr_v(2534) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 180 size= 22032 bytes from <core>: route_struct.c: mk_action(155) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 28 bytes from <core>: re.c: subst_str(533) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from xlog: xlog.c: xdbg_fixup_helper(346) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 86 size= 1752 bytes from <core>: rvalue.c: rval_get_str(1260) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 46 size= 1592 bytes from <core>: sr_module.c: fix_param(1108) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: sr_module.c: init_modules(971) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 7 size= 140 bytes from <core>: pvapi.c: tr_table_add(1660) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 21 size= 9828 bytes from <core>: sr_module.c: sr_cmd_exports_convert(234) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 23 size= 1656 bytes from <core>: sr_module.c: register_module(278) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 23 size= 856 bytes from <core>: sr_module.c: load_module(485) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 312 bytes from <core>: modparam.c: set_mod_param_regex(124) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: cfg.lex: sr_push_yy_state(1692) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 24 bytes from <core>: re.c: subst_parser(315) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 80 bytes from <core>: re.c: subst_parser(306) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 32 bytes from <core>: re.c: subst_parser(289) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 293 size= 38296 bytes from <core>: cfg.lex: addstr(1439) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 14 size= 276 bytes from <core>: cfg.lex: pp_define(1799) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 20 bytes from <core>: cfg.lex: sr_push_yy_state(1682) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 44 bytes from <core>: socket_info.c: new_sock_info(251) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 528 bytes from <core>: socket_info.c: new_sock_info(246) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 256 bytes from <core>: rpc_lookup.c: rpc_hash_add(156) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 63 size= 7048 bytes from <core>: rpc_lookup.c: rpc_hash_add(110) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 256 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 10240 bytes from <core>: pvapi.c: pv_init_buffer(1831) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 40 bytes from <core>: pvapi.c: pv_init_buffer(1821) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 12 bytes from <core>: nonsip_hooks.c: init_nonsip_hooks(57) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 248 bytes from <core>: db_id.c: dupl_string(50) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 10 size= 240 bytes from <core>: route.c: route_add(159) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 6 size= 384 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 5 size= 36 bytes from <core>: route.c: init_rlist(180) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 1 size= 64 bytes from <core>: counters.c: init_counters(121) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 73 size= 9044 bytes from <core>: counters.c: cnt_hash_add(331) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count= 2 size= 640 bytes from <core>: str_hash.h: str_hash_alloc(69) Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:

---

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

Hi Daniel,

Thanks for the reply again. Looking at the email history, I'm not sure how we decided it was definitely a pkg memory problem. What we see is the output of "ps aux" as follows:

root@sip0-test:~# ps aux | egrep -i "kamailio|mem" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6794 0.0 0.0 22480 1868 ? Ss Oct02 0:12 /opt/testuser/current/sbin/testuser_safe_kamailio testuser 6822 0.0 0.2 556528 4580 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6824 0.3 8.7 825552 180244 ? S Oct02 56:40 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6825 0.3 8.7 825536 180776 ? S Oct02 56:20 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6826 0.3 8.7 825912 180296 ? S Oct02 55:54 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6827 0.3 8.7 825744 180580 ? S Oct02 56:19 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6828 0.3 8.7 825536 180092 ? S Oct02 56:25 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6829 0.3 8.7 825536 180632 ? S Oct02 56:21 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6830 0.3 8.7 825472 180968 ? S Oct02 56:37 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6831 0.3 8.7 825276 180272 ? S Oct02 56:41 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6832 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6833 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6834 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6835 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6836 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6837 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6838 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6839 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6840 0.0 0.0 556528 1776 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6841 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6842 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6843 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6844 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6845 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6846 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6847 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6848 0.0 0.0 556528 1676 ? S Oct02 0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6849 0.0 0.1 556528 3568 ? S Oct02 5:28 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6850 0.0 0.0 556612 1600 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6851 0.0 0.0 556532 1188 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6852 0.0 0.0 556528 1360 ? S Oct02 0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid

You'll see for example that process 6824 is using 8.7% of memory, which is much more than it was using 5 days ago. Yet if I run the same sercmd again I get (exactly!) the same numbers:

root@sip0-test:~# sercmd pkg.stats pid 6824 { entry: 1 pid: 6824 rank: 1 used: 209836 free: 3704080 real_used: 490224 }

Any ideas?

On 12 October 2013 00:23, Daniel-Constantin Mierla miconda@gmail.comwrote:

Hi David,

On 10/10/13 11:36 PM, David Cunningham wrote:

...

Hi Daniel,

Thanks for the reply. Perhaps what we're seeing is normal, and the memory use is meant to increase as time progresses. Would you expect to see an ongoing memory use increase, or when should it stop increasing?

private memory (pkg) should stay rather constant. It increases when

there is a sip message processed, but once is sent out, it should come back around the average.

There are couple of functions that can fill the private memory and keep it up, such as doing an sql_query() that returns a big data and the result is not freed (sql_result_free()). It is not actually a leak as the next sql_query() will free previous result, but in case you have such query for some corner case that is not executed frequently, then the memory can stay filled in. Another example will be storing very large value in a $var(...) (e.g., $var(x) ).

This is private memory, per process, which is meant for temporary operations. Shared memory (shm) can increase over the time, being the place where the dynamic data required at runtime is stored (e.g., location records, hash tables, transactions) - so as you get more traffic or more phones using kamailio, more shm is used. But your problem was reported for pkg.

Anyhow, keep an eye on the pkg.stats and if you see constant increase which is substantial, then get a mem log dump.

Cheers, Daniel

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/**micondahttp://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

Hi Daniel,

Configuration sent to you privately. Thanks.

On 16 October 2013 20:28, Daniel-Constantin Mierla miconda@gmail.comwrote:

Hello,

indeed, it looks like a system memory leak, for what so ever reason the discussion was misled to pkg.

Can you send me the list of modules you load (loadmodule lines) as well as the functions from your perl script that are related to kamailio internal functions or variables? You can send directly to me, without mailing list if there is something you want to protect from public eyes.

Cheers, Daniel

On 10/15/13 5:12 AM, David Cunningham wrote:

Hi Daniel,

Thanks for the reply again. Looking at the email history, I'm not sure how we decided it was definitely a pkg memory problem. What we see is the output of "ps aux" as follows:

root@sip0-test:~# ps aux | egrep -i "kamailio|mem" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6794 0.0 0.0 22480 1868 ? Ss Oct02 0:12 /opt/testuser/current/sbin/testuser_safe_kamailio testuser 6822 0.0 0.2 556528 4580 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6824 0.3 8.7 825552 180244 ? S Oct02 56:40 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6825 0.3 8.7 825536 180776 ? S Oct02 56:20 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6826 0.3 8.7 825912 180296 ? S Oct02 55:54 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6827 0.3 8.7 825744 180580 ? S Oct02 56:19 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6828 0.3 8.7 825536 180092 ? S Oct02 56:25 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6829 0.3 8.7 825536 180632 ? S Oct02 56:21 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6830 0.3 8.7 825472 180968 ? S Oct02 56:37 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6831 0.3 8.7 825276 180272 ? S Oct02 56:41 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6832 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6833 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6834 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6835 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6836 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6837 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6838 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6839 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6840 0.0 0.0 556528 1776 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6841 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6842 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6843 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6844 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6845 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6846 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6847 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6848 0.0 0.0 556528 1676 ? S Oct02 0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6849 0.0 0.1 556528 3568 ? S Oct02 5:28 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6850 0.0 0.0 556612 1600 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6851 0.0 0.0 556532 1188 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6852 0.0 0.0 556528 1360 ? S Oct02 0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid

You'll see for example that process 6824 is using 8.7% of memory, which is much more than it was using 5 days ago. Yet if I run the same sercmd again I get (exactly!) the same numbers:

root@sip0-test:~# sercmd pkg.stats pid 6824 { entry: 1 pid: 6824 rank: 1 used: 209836 free: 3704080 real_used: 490224 }

Any ideas?

On 12 October 2013 00:23, Daniel-Constantin Mierla miconda@gmail.comwrote:

...

Hi David,

On 10/10/13 11:36 PM, David Cunningham wrote:

...

Hi Daniel,

Thanks for the reply. Perhaps what we're seeing is normal, and the memory use is meant to increase as time progresses. Would you expect to see an ongoing memory use increase, or when should it stop increasing?

private memory (pkg) should stay rather constant. It increases when

there is a sip message processed, but once is sent out, it should come back around the average.

There are couple of functions that can fill the private memory and keep it up, such as doing an sql_query() that returns a big data and the result is not freed (sql_result_free()). It is not actually a leak as the next sql_query() will free previous result, but in case you have such query for some corner case that is not executed frequently, then the memory can stay filled in. Another example will be storing very large value in a $var(...) (e.g., $var(x) ).

This is private memory, per process, which is meant for temporary operations. Shared memory (shm) can increase over the time, being the place where the dynamic data required at runtime is stored (e.g., location records, hash tables, transactions) - so as you get more traffic or more phones using kamailio, more shm is used. But your problem was reported for pkg.

Anyhow, keep an eye on the pkg.stats and if you see constant increase which is substantial, then get a mem log dump.

Cheers, Daniel

-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -

-- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019

-- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013

• more details about Kamailio trainings at http://www.asipto.com -