[SR-Users] Out of memory in UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB, swap:131408kB
Jurijs Ivolga
jurijs.ivolga at gmail.com
Fri Oct 7 15:45:21 CEST 2016
Hi Daniel,
I tried to compile with memory manager in debugging mode like this:
MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson json
snmpstats tls utils uuid'
make all
/etc/init.d/kamailioa stop
make install
/etc/init.d/kamailioa start
But still I'm getting nothing on:
kamailio -I | grep DBG_QM_MALLOC
Any hints?
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:39 AM, Daniel-Constantin Mierla <miconda at gmail.com
> wrote:
> Hello,
>
> There are some results when searching for memory leak on openssl 1.0.1,
> but might not be valid for this case.
>
> Would you be able to run kamailio through valgrind? It may slow down a bit
> the processing, but may be the fastest way to catch the system memory leak.
> Maybe you have an instance will less traffic and you can run through
> valgrind for a while, you don't have to wait until you get the OOM.
>
> Cheers,
> Daniel
>
> On 07/10/16 09:27, Jurijs Ivolga wrote:
>
> Hi Daniel,
>
> openssl.x86_64
> 1.0.1e-48.el6_8.1
> CentOS release 6.8
>
> With kind regards,
>
> Jurijs
>
> On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>> Hello,
>>
>> the tls module in kamailio is using shm memory, but can be something
>> internal for libssl. What operating system do you use and what is the
>> libssl version?
>>
>> Cheers,
>> Daniel
>>
>> On 06/10/16 17:43, Jurijs Ivolga wrote:
>>
>> Hi Daniel,
>>
>> I do not use puke.top rpc command. Maybe this issue related to TLS?
>> Servers what has this problem are utilizing TLS heavily, servers which do
>> not has this problem use UDP.
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> are you using pike.top rpc command? I noticed in the code that it uses
>>> system malloc, but I haven't investigated further yet, first to see if this
>>> would be a possibility ...
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 06/10/16 16:33, Jurijs Ivolga wrote:
>>>
>>> Hi Daniel,
>>>
>>> We do not do any external operations.
>>>
>>> We are using janson 2.7 everywhere. I will try to update to latest
>>> janson version tomorrow.
>>> All json operation is pretty much same, we are using only jansson_get.
>>>
>>> In attachment you can see memory consumption. On the right 2 servers
>>> which are faced internet on the left which don't face internet. As you can
>>> see memory consumption is pretty dramatic.
>>>
>>> Thank you for your help!
>>>
>>> With kind regards,
>>>
>>> Jurijs
>>>
>>> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla <
>>> miconda at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> are you doing different external operations than on the other
>>>> instances, like mi/rpc commands.
>>>>
>>>> From the list of the modules you exposed, I think jansson has the
>>>> higher probability to work with system memory. Are you doing different json
>>>> operations in config that in the other instances of kamailio? Are you using
>>>> same version of libjansson everywhere?
>>>>
>>>> Cheers,
>>>> Daniel
>>>> On 06/10/16 13:46, Jurijs Ivolga wrote:
>>>>
>>>> Hi Daniel,
>>>>
>>>> This modules what we are using:
>>>>
>>>> loadmodule "mi_fifo.so"
>>>> loadmodule "kex.so"
>>>> loadmodule "corex.so"
>>>> loadmodule "tm.so"
>>>> loadmodule "tmx.so"
>>>> loadmodule "sl.so"
>>>> loadmodule "rr.so"
>>>> loadmodule "pv.so"
>>>> loadmodule "maxfwd.so"
>>>> loadmodule "textops.so"
>>>> loadmodule "siputils.so"
>>>> loadmodule "xlog.so"
>>>> loadmodule "sanity.so"
>>>> loadmodule "ctl.so"
>>>> loadmodule "cfg_rpc.so"
>>>> loadmodule "mi_rpc.so"
>>>> loadmodule "dispatcher.so"
>>>> loadmodule "utils.so"
>>>> loadmodule "path.so"
>>>> loadmodule "ipops.so"
>>>> loadmodule "jansson.so"
>>>> loadmodule "auth.so"
>>>> loadmodule "nathelper.so"
>>>> loadmodule "tls.so"
>>>> loadmodule "htable.so"
>>>> loadmodule "pike.so"
>>>>
>>>> We have several other Kamailio instances but they are not faced to
>>>> internet and they do not have such memory issue. That other Kamailio
>>>> instances have same modules, except modules listed below. So if you think
>>>> that issue is inside external library, probably we need to check first
>>>> modules from list below.
>>>>
>>>> loadmodule "ipops.so"
>>>> loadmodule "auth.so"
>>>> loadmodule "nathelper.so"
>>>> loadmodule "pike.so"
>>>>
>>>> But maybe this other Kamailio instances do not have this memory issue,
>>>> just because they did not face to internet and did not have same load as
>>>> instances with memory issue.
>>>>
>>>> kamailio -v
>>>> version: kamailio 4.4.3 (x86_64/linux) 5a2195
>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
>>>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
>>>> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
>>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
>>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>> id: 5a2195
>>>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
>>>>
>>>>
>>>> With kind regards,
>>>>
>>>>
>>>> Jurijs
>>>>
>>>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla <
>>>> miconda at gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> it looks like a leak from the system memory, not from kamailio's pkg
>>>>> or shm memory. This can be due to an improper use of an external library
>>>>> (e.g., libxml2) by a kamailio module or because of a problem in the library.
>>>>>
>>>>> Can you list the modules used in your config (the loadmodule lines)? I
>>>>> will try to guess from the list which one relying on external libs with
>>>>> higher risk of leak issues.
>>>>>
>>>>> Also, provide the version of kamailio you are using (kamailio -v).
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>> On 04/10/16 15:42, Jurijs Ivolga wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Our Kamailio server is crashing once per week, with following error:
>>>>>
>>>>> Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB
>>>>> 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
>>>>> rss:280200kB, swap:131408kB
>>>>>
>>>>> Core dump was never created, probably it is because of my environment,
>>>>> but I will try to get it.
>>>>>
>>>>> Server constantly eats memory, maybe some kind of memory leak?
>>>>>
>>>>> Any help is highly appreciated!
>>>>>
>>>>> Jurijs
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>>>
>>>>> _______________________________________________ SIP Express Router
>>>>> (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>> sr-users at lists.sip-router.org http://lists.sip-router.org/cg
>>>>> i-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>>
>>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>
>>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>
>> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20161007/d232c841/attachment.html>
More information about the sr-users
mailing list