[SR-Users] Out of memory in UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB, swap:131408kB
Jurijs Ivolga
jurijs.ivolga at gmail.com
Mon Oct 10 12:06:36 CEST 2016
Hi Daniel,
I left valgrind running for little while, not sure if this will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla <miconda at gmail.com>
wrote:
> Hello,
>
> that's the way it was done for older versions of kamailio.
>
> In master and 4.4 the memory debugging is turned on and it is reflected by
> the presence of DBG_SR_MEMORY in the output of 'kamailio -v'.
>
> Anyhow, what you reported is not a leak inside kamailio memory manager,
> but a leak of using system memory, so it is not affected by DBG_SR_MEMORY
> and cannot be troubleshooted using the mechanisms for pkg and shm managers.
>
> Cheers,
> Daniel
>
> On 07/10/16 15:45, Jurijs Ivolga wrote:
>
> Hi Daniel,
>
> I tried to compile with memory manager in debugging mode like this:
>
> MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson json
> snmpstats tls utils uuid'
> make all
> /etc/init.d/kamailioa stop
> make install
> /etc/init.d/kamailioa start
>
> But still I'm getting nothing on:
>
> kamailio -I | grep DBG_QM_MALLOC
>
> Any hints?
>
> With kind regards,
>
> Jurijs
>
> On Fri, Oct 7, 2016 at 10:39 AM, Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>> Hello,
>>
>> There are some results when searching for memory leak on openssl 1.0.1,
>> but might not be valid for this case.
>>
>> Would you be able to run kamailio through valgrind? It may slow down a
>> bit the processing, but may be the fastest way to catch the system memory
>> leak. Maybe you have an instance will less traffic and you can run through
>> valgrind for a while, you don't have to wait until you get the OOM.
>>
>> Cheers,
>> Daniel
>>
>> On 07/10/16 09:27, Jurijs Ivolga wrote:
>>
>> Hi Daniel,
>>
>> openssl.x86_64
>> 1.0.1e-48.el6_8.1
>> CentOS release 6.8
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> the tls module in kamailio is using shm memory, but can be something
>>> internal for libssl. What operating system do you use and what is the
>>> libssl version?
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 06/10/16 17:43, Jurijs Ivolga wrote:
>>>
>>> Hi Daniel,
>>>
>>> I do not use puke.top rpc command. Maybe this issue related to TLS?
>>> Servers what has this problem are utilizing TLS heavily, servers which do
>>> not has this problem use UDP.
>>>
>>> With kind regards,
>>>
>>> Jurijs
>>>
>>> On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla <
>>> miconda at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> are you using pike.top rpc command? I noticed in the code that it uses
>>>> system malloc, but I haven't investigated further yet, first to see if this
>>>> would be a possibility ...
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 06/10/16 16:33, Jurijs Ivolga wrote:
>>>>
>>>> Hi Daniel,
>>>>
>>>> We do not do any external operations.
>>>>
>>>> We are using janson 2.7 everywhere. I will try to update to latest
>>>> janson version tomorrow.
>>>> All json operation is pretty much same, we are using only jansson_get.
>>>>
>>>> In attachment you can see memory consumption. On the right 2 servers
>>>> which are faced internet on the left which don't face internet. As you can
>>>> see memory consumption is pretty dramatic.
>>>>
>>>> Thank you for your help!
>>>>
>>>> With kind regards,
>>>>
>>>> Jurijs
>>>>
>>>> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla <
>>>> miconda at gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> are you doing different external operations than on the other
>>>>> instances, like mi/rpc commands.
>>>>>
>>>>> From the list of the modules you exposed, I think jansson has the
>>>>> higher probability to work with system memory. Are you doing different json
>>>>> operations in config that in the other instances of kamailio? Are you using
>>>>> same version of libjansson everywhere?
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>> On 06/10/16 13:46, Jurijs Ivolga wrote:
>>>>>
>>>>> Hi Daniel,
>>>>>
>>>>> This modules what we are using:
>>>>>
>>>>> loadmodule "mi_fifo.so"
>>>>> loadmodule "kex.so"
>>>>> loadmodule "corex.so"
>>>>> loadmodule "tm.so"
>>>>> loadmodule "tmx.so"
>>>>> loadmodule "sl.so"
>>>>> loadmodule "rr.so"
>>>>> loadmodule "pv.so"
>>>>> loadmodule "maxfwd.so"
>>>>> loadmodule "textops.so"
>>>>> loadmodule "siputils.so"
>>>>> loadmodule "xlog.so"
>>>>> loadmodule "sanity.so"
>>>>> loadmodule "ctl.so"
>>>>> loadmodule "cfg_rpc.so"
>>>>> loadmodule "mi_rpc.so"
>>>>> loadmodule "dispatcher.so"
>>>>> loadmodule "utils.so"
>>>>> loadmodule "path.so"
>>>>> loadmodule "ipops.so"
>>>>> loadmodule "jansson.so"
>>>>> loadmodule "auth.so"
>>>>> loadmodule "nathelper.so"
>>>>> loadmodule "tls.so"
>>>>> loadmodule "htable.so"
>>>>> loadmodule "pike.so"
>>>>>
>>>>> We have several other Kamailio instances but they are not faced to
>>>>> internet and they do not have such memory issue. That other Kamailio
>>>>> instances have same modules, except modules listed below. So if you think
>>>>> that issue is inside external library, probably we need to check first
>>>>> modules from list below.
>>>>>
>>>>> loadmodule "ipops.so"
>>>>> loadmodule "auth.so"
>>>>> loadmodule "nathelper.so"
>>>>> loadmodule "pike.so"
>>>>>
>>>>> But maybe this other Kamailio instances do not have this memory issue,
>>>>> just because they did not face to internet and did not have same load as
>>>>> instances with memory issue.
>>>>>
>>>>> kamailio -v
>>>>> version: kamailio 4.4.3 (x86_64/linux) 5a2195
>>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
>>>>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
>>>>> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
>>>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
>>>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>>> id: 5a2195
>>>>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
>>>>>
>>>>>
>>>>> With kind regards,
>>>>>
>>>>>
>>>>> Jurijs
>>>>>
>>>>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla <
>>>>> miconda at gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> it looks like a leak from the system memory, not from kamailio's pkg
>>>>>> or shm memory. This can be due to an improper use of an external library
>>>>>> (e.g., libxml2) by a kamailio module or because of a problem in the library.
>>>>>>
>>>>>> Can you list the modules used in your config (the loadmodule lines)?
>>>>>> I will try to guess from the list which one relying on external libs with
>>>>>> higher risk of leak issues.
>>>>>>
>>>>>> Also, provide the version of kamailio you are using (kamailio -v).
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>> On 04/10/16 15:42, Jurijs Ivolga wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Our Kamailio server is crashing once per week, with following error:
>>>>>>
>>>>>> Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in
>>>>>> UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
>>>>>> rss:280200kB, swap:131408kB
>>>>>>
>>>>>> Core dump was never created, probably it is because of my
>>>>>> environment, but I will try to get it.
>>>>>>
>>>>>> Server constantly eats memory, maybe some kind of memory leak?
>>>>>>
>>>>>> Any help is highly appreciated!
>>>>>>
>>>>>> Jurijs
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>>>>
>>>>>> _______________________________________________ SIP Express Router
>>>>>> (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>>> sr-users at lists.sip-router.org http://lists.sip-router.org/cg
>>>>>> i-bin/mailman/listinfo/sr-users
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>>>
>>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>>
>>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>
>>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>
>> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20161010/8d7a7afd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: memory.zip
Type: application/zip
Size: 636363 bytes
Desc: not available
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20161010/8d7a7afd/attachment-0001.zip>
More information about the sr-users
mailing list