Re: [SR-Users] Out of memory in UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB, swap:131408kB

Hi Daniel, openssl.x86_64 1.0.1e-48.el6_8.1 CentOS release 6.8 With kind regards, Jurijs On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com > Hello, > > the tls module in kamailio is using shm memory, but can be something > internal for libssl. What operating system do you use and what is the > libssl version? > > Cheers, > Daniel > > On 06/10/16 17:43, Jurijs Ivolga wrote: > > Hi Daniel, > > I do not use puke.top rpc command. Maybe this issue related to TLS? > Servers what has this problem are utilizing TLS heavily, servers which do > not has this problem use UDP. > > With kind regards, > > Jurijs > > On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla < > miconda(a)gmail.com> >> Hello, >> >> are you using pike.top rpc command? I noticed in the code that it uses >> system malloc, but I haven't investigated further yet, first to see if this >> would be a possibility ... >> >> Cheers, >> Daniel >> >> On 06/10/16 16:33, Jurijs Ivolga wrote: >> >> Hi Daniel, >> >> We do not do any external operations. >> >> We are using janson 2.7 everywhere. I will try to update to latest janson >> version tomorrow. >> All json operation is pretty much same, we are using only jansson_get. >> >> In attachment you can see memory consumption. On the right 2 servers >> which are faced internet on the left which don't face internet. As you can >> see memory consumption is pretty dramatic. >> >> Thank you for your help! >> >> With kind regards, >> >> Jurijs >> >> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla < >> miconda(a)gmail.com>> >>> Hello, >>> >>> are you doing different external operations than on the other instances, >>> like mi/rpc commands. >>> >>> From the list of the modules you exposed, I think jansson has the higher >>> probability to work with system memory. Are you doing different json >>> operations in config that in the other instances of kamailio? Are you using >>> same version of libjansson everywhere? >>> >>> Cheers, >>> Daniel >>> On 06/10/16 13:46, Jurijs Ivolga wrote: >>> >>> Hi Daniel, >>> >>> This modules what we are using: >>> >>> loadmodule "mi_fifo.so" >>> loadmodule "kex.so" >>> loadmodule "corex.so" >>> loadmodule "tm.so" >>> loadmodule "tmx.so" >>> loadmodule "sl.so" >>> loadmodule "rr.so" >>> loadmodule "pv.so" >>> loadmodule "maxfwd.so" >>> loadmodule "textops.so" >>> loadmodule "siputils.so" >>> loadmodule "xlog.so" >>> loadmodule "sanity.so" >>> loadmodule "ctl.so" >>> loadmodule "cfg_rpc.so" >>> loadmodule "mi_rpc.so" >>> loadmodule "dispatcher.so" >>> loadmodule "utils.so" >>> loadmodule "path.so" >>> loadmodule "ipops.so" >>> loadmodule "jansson.so" >>> loadmodule "auth.so" >>> loadmodule "nathelper.so" >>> loadmodule "tls.so" >>> loadmodule "htable.so" >>> loadmodule "pike.so" >>> >>> We have several other Kamailio instances but they are not faced to >>> internet and they do not have such memory issue. That other Kamailio >>> instances have same modules, except modules listed below. So if you think >>> that issue is inside external library, probably we need to check first >>> modules from list below. >>> >>> loadmodule "ipops.so" >>> loadmodule "auth.so" >>> loadmodule "nathelper.so" >>> loadmodule "pike.so" >>> >>> But maybe this other Kamailio instances do not have this memory issue, >>> just because they did not face to internet and did not have same load as >>> instances with memory issue. >>> >>> kamailio -v >>> version: kamailio 4.4.3 (x86_64/linux) 5a2195 >>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, >>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, >>> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, >>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, >>> USE_DST_BLACKLIST, HAVE_RESOLV_RES >>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, >>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB >>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>> id: 5a2195 >>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7 >>> >>> >>> With kind regards, >>> >>> >>> Jurijs >>> >>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla < >>> miconda(a)gmail.com>>> >>>> Hello, >>>> >>>> it looks like a leak from the system memory, not from kamailio's pkg or >>>> shm memory. This can be due to an improper use of an external library >>>> (e.g., libxml2) by a kamailio module or because of a problem in the library. >>>> >>>> Can you list the modules used in your config (the loadmodule lines)? I >>>> will try to guess from the list which one relying on external libs with >>>> higher risk of leak issues. >>>> >>>> Also, provide the version of kamailio you are using (kamailio -v). >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 04/10/16 15:42, Jurijs Ivolga wrote: >>>> >>>> Hi, >>>> >>>> Our Kamailio server is crashing once per week, with following error: >>>> >>>> Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB >>>> 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, >>>> rss:280200kB, swap:131408kB >>>> >>>> Core dump was never created, probably it is because of my environment, >>>> but I will try to get it. >>>> >>>> Server constantly eats memory, maybe some kind of memory leak? >>>> >>>> Any help is highly appreciated! >>>> >>>> Jurijs >>>> >>>> >>>> _______________________________________________ >>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> -- >>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com >>>> >>>> _______________________________________________ SIP Express Router >>>> (SER) and Kamailio (OpenSER) - sr-users mailing list >>>> sr-users(a)lists.sip-router.org http://lists.sip-router.org/cg >>>> i-bin/mailman/listinfo/sr-users >>> >>> -- >>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com >>> >>> -- >> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com >> >> -- > Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com > >