[SR-Users] Out of memory in UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB, swap:131408kB
Daniel-Constantin Mierla
miconda at gmail.com
Fri Oct 7 18:15:15 CEST 2016
Hello,
that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and it is reflected
by the presence of DBG_SR_MEMORY in the output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside kamailio memory manager,
but a leak of using system memory, so it is not affected by
DBG_SR_MEMORY and cannot be troubleshooted using the mechanisms for pkg
and shm managers.
Cheers,
Daniel
On 07/10/16 15:45, Jurijs Ivolga wrote:
> Hi Daniel,
>
> I tried to compile with memory manager in debugging mode like this:
>
> MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson json
> snmpstats tls utils uuid'
> make all
> /etc/init.d/kamailioa stop
> make install
> /etc/init.d/kamailioa start
>
> But still I'm getting nothing on:
>
> kamailio -I | grep DBG_QM_MALLOC
>
> Any hints?
>
> With kind regards,
>
> Jurijs
>
> On Fri, Oct 7, 2016 at 10:39 AM, Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
> There are some results when searching for memory leak on openssl
> 1.0.1, but might not be valid for this case.
>
> Would you be able to run kamailio through valgrind? It may slow
> down a bit the processing, but may be the fastest way to catch the
> system memory leak. Maybe you have an instance will less traffic
> and you can run through valgrind for a while, you don't have to
> wait until you get the OOM.
>
> Cheers,
> Daniel
>
>
> On 07/10/16 09:27, Jurijs Ivolga wrote:
>> Hi Daniel,
>>
>> openssl.x86_64
>> 1.0.1e-48.el6_8.1
>> CentOS release 6.8
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>> the tls module in kamailio is using shm memory, but can be
>> something internal for libssl. What operating system do you
>> use and what is the libssl version?
>>
>> Cheers,
>> Daniel
>>
>>
>> On 06/10/16 17:43, Jurijs Ivolga wrote:
>>> Hi Daniel,
>>>
>>> I do not use puke.top rpc command. Maybe this issue related
>>> to TLS? Servers what has this problem are utilizing TLS
>>> heavily, servers which do not has this problem use UDP.
>>>
>>> With kind regards,
>>>
>>> Jurijs
>>>
>>> On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> are you using pike.top rpc command? I noticed in the
>>> code that it uses system malloc, but I haven't
>>> investigated further yet, first to see if this would be
>>> a possibility ...
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 06/10/16 16:33, Jurijs Ivolga wrote:
>>>> Hi Daniel,
>>>>
>>>> We do not do any external operations.
>>>>
>>>> We are using janson 2.7 everywhere. I will try to
>>>> update to latest janson version tomorrow.
>>>> All json operation is pretty much same, we are using
>>>> only jansson_get.
>>>>
>>>> In attachment you can see memory consumption. On the
>>>> right 2 servers which are faced internet on the left
>>>> which don't face internet. As you can see memory
>>>> consumption is pretty dramatic.
>>>>
>>>> Thank you for your help!
>>>>
>>>> With kind regards,
>>>>
>>>> Jurijs
>>>>
>>>> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin
>>>> Mierla <miconda at gmail.com <mailto:miconda at gmail.com>>
>>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> are you doing different external operations than on
>>>> the other instances, like mi/rpc commands.
>>>>
>>>> From the list of the modules you exposed, I think
>>>> jansson has the higher probability to work with
>>>> system memory. Are you doing different json
>>>> operations in config that in the other instances of
>>>> kamailio? Are you using same version of libjansson
>>>> everywhere?
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 06/10/16 13:46, Jurijs Ivolga wrote:
>>>>> Hi Daniel,
>>>>>
>>>>> This modules what we are using:
>>>>>
>>>>> loadmodule "mi_fifo.so"
>>>>> loadmodule "kex.so"
>>>>> loadmodule "corex.so"
>>>>> loadmodule "tm.so"
>>>>> loadmodule "tmx.so"
>>>>> loadmodule "sl.so"
>>>>> loadmodule "rr.so"
>>>>> loadmodule "pv.so"
>>>>> loadmodule "maxfwd.so"
>>>>> loadmodule "textops.so"
>>>>> loadmodule "siputils.so"
>>>>> loadmodule "xlog.so"
>>>>> loadmodule "sanity.so"
>>>>> loadmodule "ctl.so"
>>>>> loadmodule "cfg_rpc.so"
>>>>> loadmodule "mi_rpc.so"
>>>>> loadmodule "dispatcher.so"
>>>>> loadmodule "utils.so"
>>>>> loadmodule "path.so"
>>>>> loadmodule "ipops.so"
>>>>> loadmodule "jansson.so"
>>>>> loadmodule "auth.so"
>>>>> loadmodule "nathelper.so"
>>>>> loadmodule "tls.so"
>>>>> loadmodule "htable.so"
>>>>> loadmodule "pike.so"
>>>>>
>>>>> We have several other Kamailio instances but they
>>>>> are not faced to internet and they do not have
>>>>> such memory issue. That other Kamailio instances
>>>>> have same modules, except modules listed below.
>>>>> So if you think that issue is inside external
>>>>> library, probably we need to check first modules
>>>>> from list below.
>>>>>
>>>>> loadmodule "ipops.so"
>>>>> loadmodule "auth.so"
>>>>> loadmodule "nathelper.so"
>>>>> loadmodule "pike.so"
>>>>>
>>>>> But maybe this other Kamailio instances do not
>>>>> have this memory issue, just because they did not
>>>>> face to internet and did not have same load as
>>>>> instances with memory issue.
>>>>>
>>>>> kamailio -v
>>>>> version: kamailio 4.4.3 (x86_64/linux) 5a2195
>>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP,
>>>>> TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
>>>>> USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
>>>>> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
>>>>> DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
>>>>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
>>>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE
>>>>> 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE
>>>>> 65535, DEFAULT PKG_SIZE 8MB
>>>>> poll method support: poll, epoll_lt, epoll_et,
>>>>> sigio_rt, select.
>>>>> id: 5a2195
>>>>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
>>>>>
>>>>>
>>>>> With kind regards,
>>>>>
>>>>>
>>>>> Jurijs
>>>>>
>>>>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin
>>>>> Mierla <miconda at gmail.com
>>>>> <mailto:miconda at gmail.com>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> it looks like a leak from the system memory,
>>>>> not from kamailio's pkg or shm memory. This
>>>>> can be due to an improper use of an external
>>>>> library (e.g., libxml2) by a kamailio module
>>>>> or because of a problem in the library.
>>>>>
>>>>> Can you list the modules used in your config
>>>>> (the loadmodule lines)? I will try to guess
>>>>> from the list which one relying on external
>>>>> libs with higher risk of leak issues.
>>>>>
>>>>> Also, provide the version of kamailio you are
>>>>> using (kamailio -v).
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>>> On 04/10/16 15:42, Jurijs Ivolga wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Our Kamailio server is crashing once per
>>>>>> week, with following error:
>>>>>>
>>>>>> Oct 1 06:25:06 kamailio kernel:
>>>>>> [26982632.803789] Out of memory in UB 210:
>>>>>> OOM killed process 12261 (kamailio) score 0
>>>>>> vm:1614768kB, rss:280200kB, swap:131408kB
>>>>>>
>>>>>> Core dump was never created, probably it is
>>>>>> because of my environment, but I will try to
>>>>>> get it.
>>>>>>
>>>>>> Server constantly eats memory, maybe some
>>>>>> kind of memory leak?
>>>>>>
>>>>>> Any help is highly appreciated!
>>>>>>
>>>>>> Jurijs
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>>> sr-users at lists.sip-router.org
>>>>>> <mailto:sr-users at lists.sip-router.org>
>>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla
>>>>> http://twitter.com/#!/miconda
>>>>> <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>>>>> <http://www.linkedin.com/in/miconda>
>>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio
>>>>> (OpenSER) - sr-users mailing list
>>>>> sr-users at lists.sip-router.org
>>>>> <mailto:sr-users at lists.sip-router.org>
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>>>>>
>>>>>
>>>> --
>>>> Daniel-Constantin Mierla
>>>> http://twitter.com/#!/miconda
>>>> <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>>>> <http://www.linkedin.com/in/miconda>
>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>>
>>> --
>>> Daniel-Constantin Mierla
>>> http://twitter.com/#!/miconda
>>> <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>>> <http://www.linkedin.com/in/miconda>
>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>
>> --
>> Daniel-Constantin Mierla
>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>> <http://www.linkedin.com/in/miconda>
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
> <http://www.linkedin.com/in/miconda>
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20161007/12b88bb5/attachment.html>
More information about the sr-users
mailing list