Re: [SR-Users] Out of memory in UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB, swap:131408kB

Hi Daniel, I tried to compile with memory manager in debugging mode like this: MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson json snmpstats tls utils uuid' make all /etc/init.d/kamailioa stop make install /etc/init.d/kamailioa start But still I'm getting nothing on: kamailio -I | grep DBG_QM_MALLOC Any hints? With kind regards, Jurijs On Fri, Oct 7, 2016 at 10:39 AM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com > Hello, > > There are some results when searching for memory leak on openssl 1.0.1, > but might not be valid for this case. > > Would you be able to run kamailio through valgrind? It may slow down a bit > the processing, but may be the fastest way to catch the system memory leak. > Maybe you have an instance will less traffic and you can run through > valgrind for a while, you don't have to wait until you get the OOM. > > Cheers, > Daniel > > On 07/10/16 09:27, Jurijs Ivolga wrote: > > Hi Daniel, > > openssl.x86_64 > 1.0.1e-48.el6_8.1 > CentOS release 6.8 > > With kind regards, > > Jurijs > > On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla < > miconda(a)gmail.com> >> Hello, >> >> the tls module in kamailio is using shm memory, but can be something >> internal for libssl. What operating system do you use and what is the >> libssl version? >> >> Cheers, >> Daniel >> >> On 06/10/16 17:43, Jurijs Ivolga wrote: >> >> Hi Daniel, >> >> I do not use puke.top rpc command. Maybe this issue related to TLS? >> Servers what has this problem are utilizing TLS heavily, servers which do >> not has this problem use UDP. >> >> With kind regards, >> >> Jurijs >> >> On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla < >> miconda(a)gmail.com>> >>> Hello, >>> >>> are you using pike.top rpc command? I noticed in the code that it uses >>> system malloc, but I haven't investigated further yet, first to see if this >>> would be a possibility ... >>> >>> Cheers, >>> Daniel >>> >>> On 06/10/16 16:33, Jurijs Ivolga wrote: >>> >>> Hi Daniel, >>> >>> We do not do any external operations. >>> >>> We are using janson 2.7 everywhere. I will try to update to latest >>> janson version tomorrow. >>> All json operation is pretty much same, we are using only jansson_get. >>> >>> In attachment you can see memory consumption. On the right 2 servers >>> which are faced internet on the left which don't face internet. As you can >>> see memory consumption is pretty dramatic. >>> >>> Thank you for your help! >>> >>> With kind regards, >>> >>> Jurijs >>> >>> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla < >>> miconda(a)gmail.com>>> >>>> Hello, >>>> >>>> are you doing different external operations than on the other >>>> instances, like mi/rpc commands. >>>> >>>> From the list of the modules you exposed, I think jansson has the >>>> higher probability to work with system memory. Are you doing different json >>>> operations in config that in the other instances of kamailio? Are you using >>>> same version of libjansson everywhere? >>>> >>>> Cheers, >>>> Daniel >>>> On 06/10/16 13:46, Jurijs Ivolga wrote: >>>> >>>> Hi Daniel, >>>> >>>> This modules what we are using: >>>> >>>> loadmodule "mi_fifo.so" >>>> loadmodule "kex.so" >>>> loadmodule "corex.so" >>>> loadmodule "tm.so" >>>> loadmodule "tmx.so" >>>> loadmodule "sl.so" >>>> loadmodule "rr.so" >>>> loadmodule "pv.so" >>>> loadmodule "maxfwd.so" >>>> loadmodule "textops.so" >>>> loadmodule "siputils.so" >>>> loadmodule "xlog.so" >>>> loadmodule "sanity.so" >>>> loadmodule "ctl.so" >>>> loadmodule "cfg_rpc.so" >>>> loadmodule "mi_rpc.so" >>>> loadmodule "dispatcher.so" >>>> loadmodule "utils.so" >>>> loadmodule "path.so" >>>> loadmodule "ipops.so" >>>> loadmodule "jansson.so" >>>> loadmodule "auth.so" >>>> loadmodule "nathelper.so" >>>> loadmodule "tls.so" >>>> loadmodule "htable.so" >>>> loadmodule "pike.so" >>>> >>>> We have several other Kamailio instances but they are not faced to >>>> internet and they do not have such memory issue. That other Kamailio >>>> instances have same modules, except modules listed below. So if you think >>>> that issue is inside external library, probably we need to check first >>>> modules from list below. >>>> >>>> loadmodule "ipops.so" >>>> loadmodule "auth.so" >>>> loadmodule "nathelper.so" >>>> loadmodule "pike.so" >>>> >>>> But maybe this other Kamailio instances do not have this memory issue, >>>> just because they did not face to internet and did not have same load as >>>> instances with memory issue. >>>> >>>> kamailio -v >>>> version: kamailio 4.4.3 (x86_64/linux) 5a2195 >>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, >>>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, >>>> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, >>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, >>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES >>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, >>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB >>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>>> id: 5a2195 >>>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7 >>>> >>>> >>>> With kind regards, >>>> >>>> >>>> Jurijs >>>> >>>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla < >>>> miconda(a)gmail.com>>>> >>>>> Hello, >>>>> >>>>> it looks like a leak from the system memory, not from kamailio's pkg >>>>> or shm memory. This can be due to an improper use of an external library >>>>> (e.g., libxml2) by a kamailio module or because of a problem in the library. >>>>> >>>>> Can you list the modules used in your config (the loadmodule lines)? I >>>>> will try to guess from the list which one relying on external libs with >>>>> higher risk of leak issues. >>>>> >>>>> Also, provide the version of kamailio you are using (kamailio -v). >>>>> >>>>> Cheers, >>>>> Daniel >>>>> >>>>> On 04/10/16 15:42, Jurijs Ivolga wrote: >>>>> >>>>> Hi, >>>>> >>>>> Our Kamailio server is crashing once per week, with following error: >>>>> >>>>> Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB >>>>> 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, >>>>> rss:280200kB, swap:131408kB >>>>> >>>>> Core dump was never created, probably it is because of my environment, >>>>> but I will try to get it. >>>>> >>>>> Server constantly eats memory, maybe some kind of memory leak? >>>>> >>>>> Any help is highly appreciated! >>>>> >>>>> Jurijs >>>>> >>>>> >>>>> _______________________________________________ >>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>>> >>>>> -- >>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com >>>>> >>>>> _______________________________________________ SIP Express Router >>>>> (SER) and Kamailio (OpenSER) - sr-users mailing list >>>>> sr-users(a)lists.sip-router.org http://lists.sip-router.org/cg >>>>> i-bin/mailman/listinfo/sr-users >>>> >>>> -- >>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com >>>> >>>> -- >>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com >>> >>> -- >> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com >> >> -- > Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com > >