[SR-Users] Out of memory in UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB, swap:131408kB
Daniel-Constantin Mierla
miconda at gmail.com
Thu Oct 6 17:05:46 CEST 2016
Hello,
are you using pike.top rpc command? I noticed in the code that it uses
system malloc, but I haven't investigated further yet, first to see if
this would be a possibility ...
Cheers,
Daniel
On 06/10/16 16:33, Jurijs Ivolga wrote:
> Hi Daniel,
>
> We do not do any external operations.
>
> We are using janson 2.7 everywhere. I will try to update to latest
> janson version tomorrow.
> All json operation is pretty much same, we are using only jansson_get.
>
> In attachment you can see memory consumption. On the right 2 servers
> which are faced internet on the left which don't face internet. As you
> can see memory consumption is pretty dramatic.
>
> Thank you for your help!
>
> With kind regards,
>
> Jurijs
>
> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
> are you doing different external operations than on the other
> instances, like mi/rpc commands.
>
> From the list of the modules you exposed, I think jansson has the
> higher probability to work with system memory. Are you doing
> different json operations in config that in the other instances of
> kamailio? Are you using same version of libjansson everywhere?
>
> Cheers,
> Daniel
>
> On 06/10/16 13:46, Jurijs Ivolga wrote:
>> Hi Daniel,
>>
>> This modules what we are using:
>>
>> loadmodule "mi_fifo.so"
>> loadmodule "kex.so"
>> loadmodule "corex.so"
>> loadmodule "tm.so"
>> loadmodule "tmx.so"
>> loadmodule "sl.so"
>> loadmodule "rr.so"
>> loadmodule "pv.so"
>> loadmodule "maxfwd.so"
>> loadmodule "textops.so"
>> loadmodule "siputils.so"
>> loadmodule "xlog.so"
>> loadmodule "sanity.so"
>> loadmodule "ctl.so"
>> loadmodule "cfg_rpc.so"
>> loadmodule "mi_rpc.so"
>> loadmodule "dispatcher.so"
>> loadmodule "utils.so"
>> loadmodule "path.so"
>> loadmodule "ipops.so"
>> loadmodule "jansson.so"
>> loadmodule "auth.so"
>> loadmodule "nathelper.so"
>> loadmodule "tls.so"
>> loadmodule "htable.so"
>> loadmodule "pike.so"
>>
>> We have several other Kamailio instances but they are not faced
>> to internet and they do not have such memory issue. That other
>> Kamailio instances have same modules, except modules listed
>> below. So if you think that issue is inside external library,
>> probably we need to check first modules from list below.
>>
>> loadmodule "ipops.so"
>> loadmodule "auth.so"
>> loadmodule "nathelper.so"
>> loadmodule "pike.so"
>>
>> But maybe this other Kamailio instances do not have this memory
>> issue, just because they did not face to internet and did not
>> have same load as instances with memory issue.
>>
>> kamailio -v
>> version: kamailio 4.4.3 (x86_64/linux) 5a2195
>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
>> SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
>> DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
>> 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> id: 5a2195
>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
>>
>>
>> With kind regards,
>>
>>
>> Jurijs
>>
>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>> it looks like a leak from the system memory, not from
>> kamailio's pkg or shm memory. This can be due to an improper
>> use of an external library (e.g., libxml2) by a kamailio
>> module or because of a problem in the library.
>>
>> Can you list the modules used in your config (the loadmodule
>> lines)? I will try to guess from the list which one relying
>> on external libs with higher risk of leak issues.
>>
>> Also, provide the version of kamailio you are using (kamailio
>> -v).
>>
>> Cheers,
>> Daniel
>>
>>
>> On 04/10/16 15:42, Jurijs Ivolga wrote:
>>> Hi,
>>>
>>> Our Kamailio server is crashing once per week, with
>>> following error:
>>>
>>> Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of
>>> memory in UB 210: OOM killed process 12261 (kamailio) score
>>> 0 vm:1614768kB, rss:280200kB, swap:131408kB
>>>
>>> Core dump was never created, probably it is because of my
>>> environment, but I will try to get it.
>>>
>>> Server constantly eats memory, maybe some kind of memory leak?
>>>
>>> Any help is highly appreciated!
>>>
>>> Jurijs
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users at lists.sip-router.org
>>> <mailto:sr-users at lists.sip-router.org>
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>>
>> --
>> Daniel-Constantin Mierla
>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>> <http://www.linkedin.com/in/miconda>
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>
>> _______________________________________________ SIP Express
>> Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> <mailto:sr-users at lists.sip-router.org>
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
> <http://www.linkedin.com/in/miconda>
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20161006/6dfa1452/attachment.html>
More information about the sr-users
mailing list