Re: [SR-Users] OOM Errors

I am not sure if anyone is using purple module anymore, because, iirc, it doesn't compile with latest version of libpurple given there were changes in the lib api and nobody updated the module. Also, I understood that yahoo discontinued old protocol, not sure what are other protocols that work, so maybe only xmpp/jabber (as a client) is still working.

I guess we should ask on mailing lists, wait a bit and then move the module to obsolete folder if nobody is using it and none wants to refresh it for the last version of libpurple.

Cheers, Daniel

On 19/09/16 22:17, Phil Lavin wrote:

Thanks, Daniel. That seems to be the logical explanation. I’ll apply the patch and see how it goes.

Looking back, that code was based on a bit of code from modules/purple/mapping.c which also does not free the result, in a few places, in the case of 0 rows in the result. We don’t use “purple” but it’s probably worth fixing that one up too.

Cheers

*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Sent:* 19 September 2016 20:04 *To:* Phil Lavin phil.lavin@cloudcall.com; Kamailio (SER) - Users Mailing List sr-users@lists.sip-router.org *Subject:* Re: [SR-Users] OOM Errors

Hello,

yes, the db related allocated chunks look to be quite high in number. That can be possible if there are db results with lots of records, but in this case there are more related to column structure than row structure.

Tracing back in the history of commits, I noticed a place where db result is not freed if no row is returned by query, in the presence module, patch done by you. It was not backported to 4.3, iirc, but if you did it in your branch, try with the latest patch pushed few minutes ago in master and 4.4.

Cheers, Daniel

On 19/09/16 20:20, Phil Lavin wrote:

Nevermind – I found this in the docs.



kamcmd cfg.set_now_int core mem_dump_pkg 17645

kamcmd cfg.set_now_int core mem_dump_pkg 1



There were 27526 allocations. I have taken and parsed the output,
summing the memory usage, grouping it by file/function. Parsed
output is at
https://gist.github.com/phil-lavin/d2607a4eed36065e01ed42c43a0d27ad



Do any of those stick out to you as being far higher than they
should be? My feeling is that it’s related to the db_* stuff.





Cheers



*From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org] *On
Behalf Of *Phil Lavin
*Sent:* 19 September 2016 18:15
*To:* miconda@gmail.com <mailto:miconda@gmail.com>; Kamailio (SER)
- Users Mailing List <sr-users@lists.sip-router.org>
<mailto:sr-users@lists.sip-router.org>
*Subject:* Re: [SR-Users] OOM Errors



4.3





*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Sent:* 19 September 2016 17:47
*To:* Phil Lavin <phil.lavin@cloudcall.com
<mailto:phil.lavin@cloudcall.com>>; Kamailio (SER) - Users Mailing
List <sr-users@lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>>
*Subject:* Re: [SR-Users] OOM Errors



The rpc command is in 4.4, what version are you using?

Cheers,
Daniel



On 19/09/16 18:11, Phil Lavin wrote:

    Having adding some graphing, I see this memory issue on
    another production Kamailio server. A few of the processes are
    using significantly more pkg memory than others. What I assume
    is happening is that memory is leaking and will eventually
    start hitting OOM errors. See
    https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9
    for pkg memory stats output.



    When was corex.pkg_summary added? I’m getting not found error:



    error: 500 - command corex.pkg_summary not found





    *From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org]
    *On Behalf Of *Phil Lavin
    *Sent:* 19 September 2016 14:36
    *To:* miconda@gmail.com <mailto:miconda@gmail.com>; Kamailio
    (SER) - Users Mailing List <sr-users@lists.sip-router.org>
    <mailto:sr-users@lists.sip-router.org>
    *Subject:* Re: [SR-Users] OOM Errors



    Thanks, Daniel.



    Kamailio has been restarted now so there’s not much of any use
    in the pkg memory output.



    I’m going to add some monitoring/graphing on this and see if
    there’s an upward trend in usage.





    *From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org]
    *On Behalf Of *Daniel-Constantin Mierla
    *Sent:* 19 September 2016 13:41
    *To:* Kamailio (SER) - Users Mailing List
    <sr-users@lists.sip-router.org
    <mailto:sr-users@lists.sip-router.org>>
    *Subject:* Re: [SR-Users] OOM Errors



    Hello,

    it seems that the pkg is out of free space, not shm. You can
    check it with:

    kamcmd pkg.stats

    and look for the same pid that printed the log messages.

    To troubleshoot, it is good to print the pkg summary, see:

      -
    https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary

    Then get the log messages from syslog printed by the command
    and send them over here.

    Cheers,
    Daniel



    On 19/09/16 14:02, Phil Lavin wrote:

        Hi all,



        Kamailio stopped performing some functions earlier today,
        citing OOM errors. An excerpt from the log is below.



        SHM usage, at the time, looked fine. Also see below for
        that output.



        A restart of Kamailio resolved the issue. Is this SHM
        related or OS memory manager related?



        root@ua-proxy-01:~# kamctl stats shmem

        shmem:fragments = 910

        shmem:free_size = 104131224

        shmem:max_used_size =  51590800

        shmem:real_used_size = 30086504

        shmem:total_size = 134217728

        shmem:used_size =  19558800



        Sep 19 11:35:56 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:161]: insert_new_lump_before(): out of
        memory

        Sep 19 11:40:10 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [msg_translator.c:2390]:
        build_res_buf_from_sip_req(): out of memory; needs 423

        Sep 19 11:40:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [msg_translator.c:2390]:
        build_res_buf_from_sip_req(): out of memory; needs 453

        Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:136]: insert_new_lump_after(): out of
        memory

        Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [ut.c:163]: as_asciiz(): Out of memory

        Sep 19 11:42:57 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [ut.c:163]: as_asciiz(): Out of memory

        Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:361]: anchor_lump(): out of memory

        Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:361]: anchor_lump(): out of memory

        Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:306]: del_lump(): out of memory

        Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:306]: del_lump(): out of memory

        Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [msg_translator.c:326]: received_builder(): out of
        memory

        Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [msg_translator.c:326]: received_builder(): out of
        memory

        Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [msg_translator.c:326]: received_builder(): out of
        memory

        Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:136]: insert_new_lump_after(): out of
        memory

        Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [ut.c:163]: as_asciiz(): Out of memory

        Sep 19 11:44:18 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:161]: insert_new_lump_before(): out of
        memory

        Sep 19 11:45:07 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [ut.c:163]: as_asciiz(): Out of memory

        Sep 19 11:45:30 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        acc [acc_extra.c:267]: extra2strar(): extra2strar: out of
        memory.

        Sep 19 11:45:43 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:186]: insert_subst_lump_after(): out
        of memory

        Sep 19 11:45:52 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
        <core> [data_lump.c:161]: insert_new_lump_before(): out of
        memory






        _______________________________________________

        SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list

        sr-users@lists.sip-router.org
        <mailto:sr-users@lists.sip-router.org>

        http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users



    -- 

    Daniel-Constantin Mierla

    http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

    Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com



-- 

Daniel-Constantin Mierla

http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com

-- Daniel-Constantin Mierla http://twitter.com/#!/miconda http://twitter.com/#%21/miconda - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com