[SR-Users] OOM Errors

Daniel-Constantin Mierla miconda at gmail.com
Mon Sep 19 18:46:40 CEST 2016 

The rpc command is in 4.4, what version are you using?

Cheers,
Daniel


On 19/09/16 18:11, Phil Lavin wrote:
>
> Having adding some graphing, I see this memory issue on another
> production Kamailio server. A few of the processes are using
> significantly more pkg memory than others. What I assume is happening
> is that memory is leaking and will eventually start hitting OOM
> errors. See
> https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9
> for pkg memory stats output.
>
>  
>
> When was corex.pkg_summary added? I’m getting not found error:
>
>  
>
> error: 500 - command corex.pkg_summary not found
>
>  
>
>  
>
> *From:*sr-users [mailto:sr-users-bounces at lists.sip-router.org] *On
> Behalf Of *Phil Lavin
> *Sent:* 19 September 2016 14:36
> *To:* miconda at gmail.com; Kamailio (SER) - Users Mailing List
> <sr-users at lists.sip-router.org>
> *Subject:* Re: [SR-Users] OOM Errors
>
>  
>
> Thanks, Daniel.
>
>  
>
> Kamailio has been restarted now so there’s not much of any use in the
> pkg memory output.
>
>  
>
> I’m going to add some monitoring/graphing on this and see if there’s
> an upward trend in usage.
>
>  
>
>  
>
> *From:*sr-users [mailto:sr-users-bounces at lists.sip-router.org] *On
> Behalf Of *Daniel-Constantin Mierla
> *Sent:* 19 September 2016 13:41
> *To:* Kamailio (SER) - Users Mailing List
> <sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>>
> *Subject:* Re: [SR-Users] OOM Errors
>
>  
>
> Hello,
>
> it seems that the pkg is out of free space, not shm. You can check it
> with:
>
> kamcmd pkg.stats
>
> and look for the same pid that printed the log messages.
>
> To troubleshoot, it is good to print the pkg summary, see:
>
>   -
> https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary
>
> Then get the log messages from syslog printed by the command and send
> them over here.
>
> Cheers,
> Daniel
>
>  
>
> On 19/09/16 14:02, Phil Lavin wrote:
>
>     Hi all,
>
>      
>
>     Kamailio stopped performing some functions earlier today, citing
>     OOM errors. An excerpt from the log is below.
>
>      
>
>     SHM usage, at the time, looked fine. Also see below for that output.
>
>      
>
>     A restart of Kamailio resolved the issue. Is this SHM related or
>     OS memory manager related?
>
>      
>
>     root at ua-proxy-01:~# kamctl stats shmem
>
>     shmem:fragments = 910
>
>     shmem:free_size = 104131224
>
>     shmem:max_used_size =  51590800
>
>     shmem:real_used_size = 30086504
>
>     shmem:total_size = 134217728
>
>     shmem:used_size =  19558800
>
>      
>
>     Sep 19 11:35:56 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:161]: insert_new_lump_before(): out of memory
>
>     Sep 19 11:40:10 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [msg_translator.c:2390]: build_res_buf_from_sip_req(): out of
>     memory; needs 423
>
>     Sep 19 11:40:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [msg_translator.c:2390]: build_res_buf_from_sip_req(): out of
>     memory; needs 453
>
>     Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:136]: insert_new_lump_after(): out of memory
>
>     Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [ut.c:163]: as_asciiz(): Out of memory
>
>     Sep 19 11:42:57 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [ut.c:163]: as_asciiz(): Out of memory
>
>     Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:361]: anchor_lump(): out of memory
>
>     Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:361]: anchor_lump(): out of memory
>
>     Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:306]: del_lump(): out of memory
>
>     Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:306]: del_lump(): out of memory
>
>     Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [msg_translator.c:326]: received_builder(): out of memory
>
>     Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [msg_translator.c:326]: received_builder(): out of memory
>
>     Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [msg_translator.c:326]: received_builder(): out of memory
>
>     Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:136]: insert_new_lump_after(): out of memory
>
>     Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [ut.c:163]: as_asciiz(): Out of memory
>
>     Sep 19 11:44:18 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:161]: insert_new_lump_before(): out of memory
>
>     Sep 19 11:45:07 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [ut.c:163]: as_asciiz(): Out of memory
>
>     Sep 19 11:45:30 ua-proxy-01 /sbin/kamailio[20218]: ERROR: acc
>     [acc_extra.c:267]: extra2strar(): extra2strar: out of memory.
>
>     Sep 19 11:45:43 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:186]: insert_subst_lump_after(): out of memory
>
>     Sep 19 11:45:52 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
>     [data_lump.c:161]: insert_new_lump_before(): out of memory
>
>      
>
>
>
>     _______________________________________________
>
>     SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>
>     sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>
>
>     http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>  
>
> -- 
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20160919/5235a123/attachment.html>

More information about the sr-users mailing list