[SR-Users] OOM Errors
Daniel-Constantin Mierla
miconda at gmail.com
Tue Sep 20 09:49:35 CEST 2016
I am not sure if anyone is using purple module anymore, because, iirc,
it doesn't compile with latest version of libpurple given there were
changes in the lib api and nobody updated the module. Also, I understood
that yahoo discontinued old protocol, not sure what are other protocols
that work, so maybe only xmpp/jabber (as a client) is still working.
I guess we should ask on mailing lists, wait a bit and then move the
module to obsolete folder if nobody is using it and none wants to
refresh it for the last version of libpurple.
Cheers,
Daniel
On 19/09/16 22:17, Phil Lavin wrote:
>
> Thanks, Daniel. That seems to be the logical explanation. I’ll apply
> the patch and see how it goes.
>
>
>
> Looking back, that code was based on a bit of code from
> modules/purple/mapping.c which also does not free the result, in a few
> places, in the case of 0 rows in the result. We don’t use “purple” but
> it’s probably worth fixing that one up too.
>
>
>
>
>
> Cheers
>
>
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* 19 September 2016 20:04
> *To:* Phil Lavin <phil.lavin at cloudcall.com>; Kamailio (SER) - Users
> Mailing List <sr-users at lists.sip-router.org>
> *Subject:* Re: [SR-Users] OOM Errors
>
>
>
> Hello,
>
> yes, the db related allocated chunks look to be quite high in number.
> That can be possible if there are db results with lots of records, but
> in this case there are more related to column structure than row
> structure.
>
> Tracing back in the history of commits, I noticed a place where db
> result is not freed if no row is returned by query, in the presence
> module, patch done by you. It was not backported to 4.3, iirc, but if
> you did it in your branch, try with the latest patch pushed few
> minutes ago in master and 4.4.
>
> Cheers,
> Daniel
>
>
>
> On 19/09/16 20:20, Phil Lavin wrote:
>
> Nevermind – I found this in the docs.
>
>
>
> kamcmd cfg.set_now_int core mem_dump_pkg 17645
>
> kamcmd cfg.set_now_int core mem_dump_pkg 1
>
>
>
> There were 27526 allocations. I have taken and parsed the output,
> summing the memory usage, grouping it by file/function. Parsed
> output is at
> https://gist.github.com/phil-lavin/d2607a4eed36065e01ed42c43a0d27ad
>
>
>
> Do any of those stick out to you as being far higher than they
> should be? My feeling is that it’s related to the db_* stuff.
>
>
>
>
>
> Cheers
>
>
>
> *From:*sr-users [mailto:sr-users-bounces at lists.sip-router.org] *On
> Behalf Of *Phil Lavin
> *Sent:* 19 September 2016 18:15
> *To:* miconda at gmail.com <mailto:miconda at gmail.com>; Kamailio (SER)
> - Users Mailing List <sr-users at lists.sip-router.org>
> <mailto:sr-users at lists.sip-router.org>
> *Subject:* Re: [SR-Users] OOM Errors
>
>
>
> 4.3
>
>
>
>
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* 19 September 2016 17:47
> *To:* Phil Lavin <phil.lavin at cloudcall.com
> <mailto:phil.lavin at cloudcall.com>>; Kamailio (SER) - Users Mailing
> List <sr-users at lists.sip-router.org
> <mailto:sr-users at lists.sip-router.org>>
> *Subject:* Re: [SR-Users] OOM Errors
>
>
>
> The rpc command is in 4.4, what version are you using?
>
> Cheers,
> Daniel
>
>
>
> On 19/09/16 18:11, Phil Lavin wrote:
>
> Having adding some graphing, I see this memory issue on
> another production Kamailio server. A few of the processes are
> using significantly more pkg memory than others. What I assume
> is happening is that memory is leaking and will eventually
> start hitting OOM errors. See
> https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9
> for pkg memory stats output.
>
>
>
> When was corex.pkg_summary added? I’m getting not found error:
>
>
>
> error: 500 - command corex.pkg_summary not found
>
>
>
>
>
> *From:*sr-users [mailto:sr-users-bounces at lists.sip-router.org]
> *On Behalf Of *Phil Lavin
> *Sent:* 19 September 2016 14:36
> *To:* miconda at gmail.com <mailto:miconda at gmail.com>; Kamailio
> (SER) - Users Mailing List <sr-users at lists.sip-router.org>
> <mailto:sr-users at lists.sip-router.org>
> *Subject:* Re: [SR-Users] OOM Errors
>
>
>
> Thanks, Daniel.
>
>
>
> Kamailio has been restarted now so there’s not much of any use
> in the pkg memory output.
>
>
>
> I’m going to add some monitoring/graphing on this and see if
> there’s an upward trend in usage.
>
>
>
>
>
> *From:*sr-users [mailto:sr-users-bounces at lists.sip-router.org]
> *On Behalf Of *Daniel-Constantin Mierla
> *Sent:* 19 September 2016 13:41
> *To:* Kamailio (SER) - Users Mailing List
> <sr-users at lists.sip-router.org
> <mailto:sr-users at lists.sip-router.org>>
> *Subject:* Re: [SR-Users] OOM Errors
>
>
>
> Hello,
>
> it seems that the pkg is out of free space, not shm. You can
> check it with:
>
> kamcmd pkg.stats
>
> and look for the same pid that printed the log messages.
>
> To troubleshoot, it is good to print the pkg summary, see:
>
> -
> https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary
>
> Then get the log messages from syslog printed by the command
> and send them over here.
>
> Cheers,
> Daniel
>
>
>
> On 19/09/16 14:02, Phil Lavin wrote:
>
> Hi all,
>
>
>
> Kamailio stopped performing some functions earlier today,
> citing OOM errors. An excerpt from the log is below.
>
>
>
> SHM usage, at the time, looked fine. Also see below for
> that output.
>
>
>
> A restart of Kamailio resolved the issue. Is this SHM
> related or OS memory manager related?
>
>
>
> root at ua-proxy-01:~# kamctl stats shmem
>
> shmem:fragments = 910
>
> shmem:free_size = 104131224
>
> shmem:max_used_size = 51590800
>
> shmem:real_used_size = 30086504
>
> shmem:total_size = 134217728
>
> shmem:used_size = 19558800
>
>
>
> Sep 19 11:35:56 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:161]: insert_new_lump_before(): out of
> memory
>
> Sep 19 11:40:10 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [msg_translator.c:2390]:
> build_res_buf_from_sip_req(): out of memory; needs 423
>
> Sep 19 11:40:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [msg_translator.c:2390]:
> build_res_buf_from_sip_req(): out of memory; needs 453
>
> Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:136]: insert_new_lump_after(): out of
> memory
>
> Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [ut.c:163]: as_asciiz(): Out of memory
>
> Sep 19 11:42:57 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [ut.c:163]: as_asciiz(): Out of memory
>
> Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:361]: anchor_lump(): out of memory
>
> Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:361]: anchor_lump(): out of memory
>
> Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:306]: del_lump(): out of memory
>
> Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:306]: del_lump(): out of memory
>
> Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [msg_translator.c:326]: received_builder(): out of
> memory
>
> Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [msg_translator.c:326]: received_builder(): out of
> memory
>
> Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [msg_translator.c:326]: received_builder(): out of
> memory
>
> Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:136]: insert_new_lump_after(): out of
> memory
>
> Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [ut.c:163]: as_asciiz(): Out of memory
>
> Sep 19 11:44:18 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:161]: insert_new_lump_before(): out of
> memory
>
> Sep 19 11:45:07 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [ut.c:163]: as_asciiz(): Out of memory
>
> Sep 19 11:45:30 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> acc [acc_extra.c:267]: extra2strar(): extra2strar: out of
> memory.
>
> Sep 19 11:45:43 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:186]: insert_subst_lump_after(): out
> of memory
>
> Sep 19 11:45:52 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
> <core> [data_lump.c:161]: insert_new_lump_before(): out of
> memory
>
>
>
>
>
>
> _______________________________________________
>
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>
> sr-users at lists.sip-router.org
> <mailto:sr-users at lists.sip-router.org>
>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
> --
>
> Daniel-Constantin Mierla
>
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
>
> --
>
> Daniel-Constantin Mierla
>
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20160920/0604b099/attachment.html>
More information about the sr-users
mailing list