7 Oct
2016
7 Oct
'16
9:39 a.m.
Hello,
There are some results when searching for memory leak on openssl 1.0.1,
but might not be valid for this case.
Would you be able to run kamailio through valgrind? It may slow down a
bit the processing, but may be the fastest way to catch the system
memory leak. Maybe you have an instance will less traffic and you can
run through valgrind for a while, you don't have to wait until you get
the OOM.
Cheers,
Daniel
On 07/10/16 09:27, Jurijs Ivolga wrote:
Hi Daniel,
openssl.x86_64
1.0.1e-48.el6_8.1
CentOS release 6.8
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
the tls module in kamailio is using shm memory, but can be
something internal for libssl. What operating system do you use
and what is the libssl version?
Cheers,
Daniel
On 06/10/16 17:43, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
I do not use puke.top rpc command. Maybe this issue related to
TLS? Servers what has this problem are utilizing TLS heavily,
servers which do not has this problem use UDP.
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
are you using pike.top rpc command? I noticed in the code
that it uses system malloc, but I haven't investigated
further yet, first to see if this would be a possibility ...
Cheers,
Daniel
On 06/10/16 16:33, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
We do not do any external operations.
We are using janson 2.7 everywhere. I will try to update to
latest janson version tomorrow.
All json operation is pretty much same, we are using only
jansson_get.
In attachment you can see memory consumption. On the right 2
servers which are faced internet on the left which don't
face internet. As you can see memory consumption is pretty
dramatic.
Thank you for your help!
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
are you doing different external operations than on the
other instances, like mi/rpc commands.
From the list of the modules you exposed, I think
jansson has the higher probability to work with system
memory. Are you doing different json operations in
config that in the other instances of kamailio? Are you
using same version of libjansson everywhere?
Cheers,
Daniel
On 06/10/16 13:46, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
This modules what we are using:
loadmodule "mi_fifo.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dispatcher.so"
loadmodule "utils.so"
loadmodule "path.so"
loadmodule "ipops.so"
loadmodule "jansson.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
We have several other Kamailio instances but they are
not faced to internet and they do not have such memory
issue. That other Kamailio instances have same
modules, except modules listed below. So if you think
that issue is inside external library, probably we need
to check first modules from list below.
loadmodule "ipops.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "pike.so"
But maybe this other Kamailio instances do not have
this memory issue, just because they did not face to
internet and did not have same load as instances with
memory issue.
kamailio -v
version: kamailio 4.4.3 (x86_64/linux) 5a2195
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP,
TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et,
sigio_rt, select.
id: 5a2195
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin
Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>>
wrote:
Hello,
it looks like a leak from the system memory, not
from kamailio's pkg or shm memory. This can be due
to an improper use of an external library (e.g.,
libxml2) by a kamailio module or because of a
problem in the library.
Can you list the modules used in your config (the
loadmodule lines)? I will try to guess from the
list which one relying on external libs with higher
risk of leak issues.
Also, provide the version of kamailio you are using
(kamailio -v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
> Hi,
>
> Our Kamailio server is crashing once per week,
> with following error:
>
> Oct 1 06:25:06 kamailio kernel: [26982632.803789]
> Out of memory in UB 210: OOM killed process 12261
> (kamailio) score 0 vm:1614768kB, rss:280200kB,
> swap:131408kB
>
> Core dump was never created, probably it is
> because of my environment, but I will try to get it.
>
> Server constantly eats memory, maybe some kind of
> memory leak?
>
> Any help is highly appreciated!
>
> Jurijs
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
list
> sr-users(a)lists.sip-router.org
> <mailto:sr-users@lists.sip-router.org>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
_______________________________________________ SIP
Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com