Out of memory in UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB, swap:131408kB
4 Oct
2016
4 Oct
'16
3:42 p.m.
Hi,
Our Kamailio server is crashing once per week, with following error:
Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB 210:
OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB,
swap:131408kB
Core dump was never created, probably it is because of my environment, but
I will try to get it.
Server constantly eats memory, maybe some kind of memory leak?
Any help is highly appreciated!
Jurijs
6 Oct
6 Oct
11:52 a.m.
Hello,
it looks like a leak from the system memory, not from kamailio's pkg or
shm memory. This can be due to an improper use of an external library
(e.g., libxml2) by a kamailio module or because of a problem in the library.
Can you list the modules used in your config (the loadmodule lines)? I
will try to guess from the list which one relying on external libs with
higher risk of leak issues.
Also, provide the version of kamailio you are using (kamailio -v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
Hi,
Our Kamailio server is crashing once per week, with following error:
Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB
210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
rss:280200kB, swap:131408kB
Core dump was never created, probably it is because of my environment,
but I will try to get it.
Server constantly eats memory, maybe some kind of memory leak?
Any help is highly appreciated!
Jurijs
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
1:46 p.m.
Hi Daniel,
This modules what we are using:
loadmodule "mi_fifo.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dispatcher.so"
loadmodule "utils.so"
loadmodule "path.so"
loadmodule "ipops.so"
loadmodule "jansson.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
We have several other Kamailio instances but they are not faced to internet
and they do not have such memory issue. That other Kamailio instances have
same modules, except modules listed below. So if you think that issue is
inside external library, probably we need to check first modules from list
below.
loadmodule "ipops.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "pike.so"
But maybe this other Kamailio instances do not have this memory issue, just
because they did not face to internet and did not have same load as
instances with memory issue.
kamailio -v
version: kamailio 4.4.3 (x86_64/linux) 5a2195
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 5a2195
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
Hello,
it looks like a leak from the system memory, not from kamailio's pkg or
shm memory. This can be due to an improper use of an external library
(e.g., libxml2) by a kamailio module or because of a problem in the library.
Can you list the modules used in your config (the loadmodule lines)? I
will try to guess from the list which one relying on external libs with
higher risk of leak issues.
Also, provide the version of kamailio you are using (kamailio -v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
Hi,
Our Kamailio server is crashing once per week, with following error:
Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB
210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
rss:280200kB, swap:131408kB
Core dump was never created, probably it is because of my environment, but
I will try to get it.
Server constantly eats memory, maybe some kind of memory leak?
Any help is highly appreciated!
Jurijs
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
4:17 p.m.
Hello,
are you doing different external operations than on the other instances,
like mi/rpc commands.
From the list of the modules you exposed, I think
jansson has the higher
probability to work with system memory. Are you doing
different json
operations in config that in the other instances of kamailio? Are you
using same version of libjansson everywhere?
Cheers,
Daniel
On 06/10/16 13:46, Jurijs Ivolga wrote:
Hi Daniel,
This modules what we are using:
loadmodule "mi_fifo.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dispatcher.so"
loadmodule "utils.so"
loadmodule "path.so"
loadmodule "ipops.so"
loadmodule "jansson.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
We have several other Kamailio instances but they are not faced to
internet and they do not have such memory issue. That other Kamailio
instances have same modules, except modules listed below. So if you
think that issue is inside external library, probably we need to check
first modules from list below.
loadmodule "ipops.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "pike.so"
But maybe this other Kamailio instances do not have this memory issue,
just because they did not face to internet and did not have same load
as instances with memory issue.
kamailio -v
version: kamailio 4.4.3 (x86_64/linux) 5a2195
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 5a2195
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
it looks like a leak from the system memory, not from kamailio's
pkg or shm memory. This can be due to an improper use of an
external library (e.g., libxml2) by a kamailio module or because
of a problem in the library.
Can you list the modules used in your config (the loadmodule
lines)? I will try to guess from the list which one relying on
external libs with higher risk of leak issues.
Also, provide the version of kamailio you are using (kamailio -v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi,
Our Kamailio server is crashing once per week, with following error:
Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory
in UB 210: OOM killed process 12261 (kamailio) score 0
vm:1614768kB, rss:280200kB, swap:131408kB
Core dump was never created, probably it is because of my
environment, but I will try to get it.
Server constantly eats memory, maybe some kind of memory leak?
Any help is highly appreciated!
Jurijs
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
_______________________________________________ SIP Express Router
(SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
4:33 p.m.
Hi Daniel,
We do not do any external operations.
We are using janson 2.7 everywhere. I will try to update to latest janson
version tomorrow.
All json operation is pretty much same, we are using only jansson_get.
In attachment you can see memory consumption. On the right 2 servers which
are faced internet on the left which don't face internet. As you can see
memory consumption is pretty dramatic.
Thank you for your help!
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
are you doing different external operations than on the other instances,
like mi/rpc commands.
From the list of the modules you exposed, I think jansson has the higher
probability to work with system memory. Are you doing different json
operations in config that in the other instances of kamailio? Are you using
same version of libjansson everywhere?
Cheers,
Daniel
On 06/10/16 13:46, Jurijs Ivolga wrote:
Hi Daniel,
This modules what we are using:
loadmodule "mi_fifo.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dispatcher.so"
loadmodule "utils.so"
loadmodule "path.so"
loadmodule "ipops.so"
loadmodule "jansson.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
We have several other Kamailio instances but they are not faced to
internet and they do not have such memory issue. That other Kamailio
instances have same modules, except modules listed below. So if you think
that issue is inside external library, probably we need to check first
modules from list below.
loadmodule "ipops.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "pike.so"
But maybe this other Kamailio instances do not have this memory issue,
just because they did not face to internet and did not have same load as
instances with memory issue.
kamailio -v
version: kamailio 4.4.3 (x86_64/linux) 5a2195
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 5a2195
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
it looks like a leak from the system memory, not from kamailio's pkg or
shm memory. This can be due to an improper use of an external library
(e.g., libxml2) by a kamailio module or because of a problem in the library.
Can you list the modules used in your config (the loadmodule lines)? I
will try to guess from the list which one relying on external libs with
higher risk of leak issues.
Also, provide the version of kamailio you are using (kamailio -v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
Hi,
Our Kamailio server is crashing once per week, with following error:
Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB
210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
rss:280200kB, swap:131408kB
Core dump was never created, probably it is because of my environment,
but I will try to get it.
Server constantly eats memory, maybe some kind of memory leak?
Any help is highly appreciated!
Jurijs
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
_______________________________________________ SIP Express Router (SER)
and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org http://lists.sip-router.org/cg
i-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
5:05 p.m.
Hello,
are you using pike.top rpc command? I noticed in the code that it uses
system malloc, but I haven't investigated further yet, first to see if
this would be a possibility ...
Cheers,
Daniel
On 06/10/16 16:33, Jurijs Ivolga wrote:
Hi Daniel,
We do not do any external operations.
We are using janson 2.7 everywhere. I will try to update to latest
janson version tomorrow.
All json operation is pretty much same, we are using only jansson_get.
In attachment you can see memory consumption. On the right 2 servers
which are faced internet on the left which don't face internet. As you
can see memory consumption is pretty dramatic.
Thank you for your help!
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
are you doing different external operations than on the other
instances, like mi/rpc commands.
From the list of the modules you exposed, I think jansson has the
higher probability to work with system memory. Are you doing
different json operations in config that in the other instances of
kamailio? Are you using same version of libjansson everywhere?
Cheers,
Daniel
On 06/10/16 13:46, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
This modules what we are using:
loadmodule "mi_fifo.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dispatcher.so"
loadmodule "utils.so"
loadmodule "path.so"
loadmodule "ipops.so"
loadmodule "jansson.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
We have several other Kamailio instances but they are not faced
to internet and they do not have such memory issue. That other
Kamailio instances have same modules, except modules listed
below. So if you think that issue is inside external library,
probably we need to check first modules from list below.
loadmodule "ipops.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "pike.so"
But maybe this other Kamailio instances do not have this memory
issue, just because they did not face to internet and did not
have same load as instances with memory issue.
kamailio -v
version: kamailio 4.4.3 (x86_64/linux) 5a2195
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN
16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 5a2195
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
it looks like a leak from the system memory, not from
kamailio's pkg or shm memory. This can be due to an improper
use of an external library (e.g., libxml2) by a kamailio
module or because of a problem in the library.
Can you list the modules used in your config (the loadmodule
lines)? I will try to guess from the list which one relying
on external libs with higher risk of leak issues.
Also, provide the version of kamailio you are using (kamailio
-v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi,
Our Kamailio server is crashing once per week, with
following error:
Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of
memory in UB 210: OOM killed process 12261 (kamailio) score
0 vm:1614768kB, rss:280200kB, swap:131408kB
Core dump was never created, probably it is because of my
environment, but I will try to get it.
Server constantly eats memory, maybe some kind of memory leak?
Any help is highly appreciated!
Jurijs
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
_______________________________________________ SIP Express
Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
5:43 p.m.
Hi Daniel,
I do not use puke.top rpc command. Maybe this issue related to TLS? Servers
what has this problem are utilizing TLS heavily, servers which do not has
this problem use UDP.
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
are you using pike.top rpc command? I noticed in the code that it uses
system malloc, but I haven't investigated further yet, first to see if this
would be a possibility ...
Cheers,
Daniel
On 06/10/16 16:33, Jurijs Ivolga wrote:
Hi Daniel,
We do not do any external operations.
We are using janson 2.7 everywhere. I will try to update to latest janson
version tomorrow.
All json operation is pretty much same, we are using only jansson_get.
In attachment you can see memory consumption. On the right 2 servers which
are faced internet on the left which don't face internet. As you can see
memory consumption is pretty dramatic.
Thank you for your help!
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
are you doing different external operations than on the other instances,
like mi/rpc commands.
From the list of the modules you exposed, I think jansson has the higher
probability to work with system memory. Are you doing different json
operations in config that in the other instances of kamailio? Are you using
same version of libjansson everywhere?
Cheers,
Daniel
On 06/10/16 13:46, Jurijs Ivolga wrote:
Hi Daniel,
This modules what we are using:
loadmodule "mi_fifo.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dispatcher.so"
loadmodule "utils.so"
loadmodule "path.so"
loadmodule "ipops.so"
loadmodule "jansson.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
We have several other Kamailio instances but they are not faced to
internet and they do not have such memory issue. That other Kamailio
instances have same modules, except modules listed below. So if you think
that issue is inside external library, probably we need to check first
modules from list below.
loadmodule "ipops.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "pike.so"
But maybe this other Kamailio instances do not have this memory issue,
just because they did not face to internet and did not have same load as
instances with memory issue.
kamailio -v
version: kamailio 4.4.3 (x86_64/linux) 5a2195
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 5a2195
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hello,
it looks like a leak from the system memory, not from kamailio's pkg or
shm memory. This can be due to an improper use of an external library
(e.g., libxml2) by a kamailio module or because of a problem in the library.
Can you list the modules used in your config (the loadmodule lines)? I
will try to guess from the list which one relying on external libs with
higher risk of leak issues.
Also, provide the version of kamailio you are using (kamailio -v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
Hi,
Our Kamailio server is crashing once per week, with following error:
Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB
210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
rss:280200kB, swap:131408kB
Core dump was never created, probably it is because of my environment,
but I will try to get it.
Server constantly eats memory, maybe some kind of memory leak?
Any help is highly appreciated!
Jurijs
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
_______________________________________________ SIP Express Router
(SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org http://lists.sip-router.org/cg
i-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
--
7 Oct
7 Oct
9:20 a.m.
Hello,
the tls module in kamailio is using shm memory, but can be something
internal for libssl. What operating system do you use and what is the
libssl version?
Cheers,
Daniel
On 06/10/16 17:43, Jurijs Ivolga wrote:
Hi Daniel,
I do not use puke.top rpc command. Maybe this issue related to TLS?
Servers what has this problem are utilizing TLS heavily, servers which
do not has this problem use UDP.
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
are you using pike.top rpc command? I noticed in the code that it
uses system malloc, but I haven't investigated further yet, first
to see if this would be a possibility ...
Cheers,
Daniel
On 06/10/16 16:33, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
We do not do any external operations.
We are using janson 2.7 everywhere. I will try to update to
latest janson version tomorrow.
All json operation is pretty much same, we are using only
jansson_get.
In attachment you can see memory consumption. On the right 2
servers which are faced internet on the left which don't face
internet. As you can see memory consumption is pretty dramatic.
Thank you for your help!
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
are you doing different external operations than on the other
instances, like mi/rpc commands.
From the list of the modules you exposed, I think jansson has
the higher probability to work with system memory. Are you
doing different json operations in config that in the other
instances of kamailio? Are you using same version of
libjansson everywhere?
Cheers,
Daniel
On 06/10/16 13:46, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
This modules what we are using:
loadmodule "mi_fifo.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dispatcher.so"
loadmodule "utils.so"
loadmodule "path.so"
loadmodule "ipops.so"
loadmodule "jansson.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
We have several other Kamailio instances but they are not
faced to internet and they do not have such memory issue.
That other Kamailio instances have same modules, except
modules listed below. So if you think that issue is inside
external library, probably we need to check first modules
from list below.
loadmodule "ipops.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "pike.so"
But maybe this other Kamailio instances do not have this
memory issue, just because they did not face to internet and
did not have same load as instances with memory issue.
kamailio -v
version: kamailio 4.4.3 (x86_64/linux) 5a2195
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC,
TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT
PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 5a2195
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
it looks like a leak from the system memory, not from
kamailio's pkg or shm memory. This can be due to an
improper use of an external library (e.g., libxml2) by a
kamailio module or because of a problem in the library.
Can you list the modules used in your config (the
loadmodule lines)? I will try to guess from the list
which one relying on external libs with higher risk of
leak issues.
Also, provide the version of kamailio you are using
(kamailio -v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi,
Our Kamailio server is crashing once per week, with
following error:
Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out
of memory in UB 210: OOM killed process 12261
(kamailio) score 0 vm:1614768kB, rss:280200kB,
swap:131408kB
Core dump was never created, probably it is because of
my environment, but I will try to get it.
Server constantly eats memory, maybe some kind of
memory leak?
Any help is highly appreciated!
Jurijs
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
_______________________________________________ SIP
Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
9:27 a.m.
Hi Daniel,
openssl.x86_64
1.0.1e-48.el6_8.1
CentOS release 6.8
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
> Hello,
>
> the tls module in kamailio is using shm memory, but can be something
> internal for libssl. What operating system do you use and what is the
> libssl version?
>
> Cheers,
> Daniel
>
> On 06/10/16 17:43, Jurijs Ivolga wrote:
>
> Hi Daniel,
>
> I do not use puke.top rpc command. Maybe this issue related to TLS?
> Servers what has this problem are utilizing TLS heavily, servers which do
> not has this problem use UDP.
>
> With kind regards,
>
> Jurijs
>
> On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla <
> miconda(a)gmail.com wrote:
>
>> Hello,
>>
>> are you using pike.top rpc command? I noticed in the code that it uses
>> system malloc, but I haven't investigated further yet, first to see if this
>> would be a possibility ...
>>
>> Cheers,
>> Daniel
>>
>> On 06/10/16 16:33, Jurijs Ivolga wrote:
>>
>> Hi Daniel,
>>
>> We do not do any external operations.
>>
>> We are using janson 2.7 everywhere. I will try to update to latest janson
>> version tomorrow.
>> All json operation is pretty much same, we are using only jansson_get.
>>
>> In attachment you can see memory consumption. On the right 2 servers
>> which are faced internet on the left which don't face internet. As you can
>> see memory consumption is pretty dramatic.
>>
>> Thank you for your help!
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla <
>> miconda(a)gmail.com wrote:
>>
>>> Hello,
>>>
>>> are you doing different external operations than on the other instances,
>>> like mi/rpc commands.
>>>
>>> From the list of the modules you exposed, I think jansson has the higher
>>> probability to work with system memory. Are you doing different json
>>> operations in config that in the other instances of kamailio? Are you using
>>> same version of libjansson everywhere?
>>>
>>> Cheers,
>>> Daniel
>>> On 06/10/16 13:46, Jurijs Ivolga wrote:
>>>
>>> Hi Daniel,
>>>
>>> This modules what we are using:
>>>
>>> loadmodule "mi_fifo.so"
>>> loadmodule "kex.so"
>>> loadmodule "corex.so"
>>> loadmodule "tm.so"
>>> loadmodule "tmx.so"
>>> loadmodule "sl.so"
>>> loadmodule "rr.so"
>>> loadmodule "pv.so"
>>> loadmodule "maxfwd.so"
>>> loadmodule "textops.so"
>>> loadmodule "siputils.so"
>>> loadmodule "xlog.so"
>>> loadmodule "sanity.so"
>>> loadmodule "ctl.so"
>>> loadmodule "cfg_rpc.so"
>>> loadmodule "mi_rpc.so"
>>> loadmodule "dispatcher.so"
>>> loadmodule "utils.so"
>>> loadmodule "path.so"
>>> loadmodule "ipops.so"
>>> loadmodule "jansson.so"
>>> loadmodule "auth.so"
>>> loadmodule "nathelper.so"
>>> loadmodule "tls.so"
>>> loadmodule "htable.so"
>>> loadmodule "pike.so"
>>>
>>> We have several other Kamailio instances but they are not faced to
>>> internet and they do not have such memory issue. That other Kamailio
>>> instances have same modules, except modules listed below. So if you think
>>> that issue is inside external library, probably we need to check first
>>> modules from list below.
>>>
>>> loadmodule "ipops.so"
>>> loadmodule "auth.so"
>>> loadmodule "nathelper.so"
>>> loadmodule "pike.so"
>>>
>>> But maybe this other Kamailio instances do not have this memory issue,
>>> just because they did not face to internet and did not have same load as
>>> instances with memory issue.
>>>
>>> kamailio -v
>>> version: kamailio 4.4.3 (x86_64/linux) 5a2195
>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>>> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>> id: 5a2195
>>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
>>>
>>>
>>> With kind regards,
>>>
>>>
>>> Jurijs
>>>
>>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla <
>>> miconda(a)gmail.com wrote:
>>>
>>>> Hello,
>>>>
>>>> it looks like a leak from the system memory, not from kamailio's pkg
or
>>>> shm memory. This can be due to an improper use of an external library
>>>> (e.g., libxml2) by a kamailio module or because of a problem in the
library.
>>>>
>>>> Can you list the modules used in your config (the loadmodule lines)? I
>>>> will try to guess from the list which one relying on external libs with
>>>> higher risk of leak issues.
>>>>
>>>> Also, provide the version of kamailio you are using (kamailio -v).
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 04/10/16 15:42, Jurijs Ivolga wrote:
>>>>
>>>> Hi,
>>>>
>>>> Our Kamailio server is crashing once per week, with following error:
>>>>
>>>> Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in UB
>>>> 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
>>>> rss:280200kB, swap:131408kB
>>>>
>>>> Core dump was never created, probably it is because of my environment,
>>>> but I will try to get it.
>>>>
>>>> Server constantly eats memory, maybe some kind of memory leak?
>>>>
>>>> Any help is highly appreciated!
>>>>
>>>> Jurijs
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
>>>>
>>>> _______________________________________________ SIP Express Router
>>>> (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>> sr-users(a)lists.sip-router.org http://lists.sip-router.org/cg
>>>> i-bin/mailman/listinfo/sr-users
>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>
>>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>
>> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
9:39 a.m.
Hello,
There are some results when searching for memory leak on openssl 1.0.1,
but might not be valid for this case.
Would you be able to run kamailio through valgrind? It may slow down a
bit the processing, but may be the fastest way to catch the system
memory leak. Maybe you have an instance will less traffic and you can
run through valgrind for a while, you don't have to wait until you get
the OOM.
Cheers,
Daniel
On 07/10/16 09:27, Jurijs Ivolga wrote:
Hi Daniel,
openssl.x86_64
1.0.1e-48.el6_8.1
CentOS release 6.8
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
the tls module in kamailio is using shm memory, but can be
something internal for libssl. What operating system do you use
and what is the libssl version?
Cheers,
Daniel
On 06/10/16 17:43, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
I do not use puke.top rpc command. Maybe this issue related to
TLS? Servers what has this problem are utilizing TLS heavily,
servers which do not has this problem use UDP.
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
are you using pike.top rpc command? I noticed in the code
that it uses system malloc, but I haven't investigated
further yet, first to see if this would be a possibility ...
Cheers,
Daniel
On 06/10/16 16:33, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
We do not do any external operations.
We are using janson 2.7 everywhere. I will try to update to
latest janson version tomorrow.
All json operation is pretty much same, we are using only
jansson_get.
In attachment you can see memory consumption. On the right 2
servers which are faced internet on the left which don't
face internet. As you can see memory consumption is pretty
dramatic.
Thank you for your help!
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
are you doing different external operations than on the
other instances, like mi/rpc commands.
From the list of the modules you exposed, I think
jansson has the higher probability to work with system
memory. Are you doing different json operations in
config that in the other instances of kamailio? Are you
using same version of libjansson everywhere?
Cheers,
Daniel
On 06/10/16 13:46, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
This modules what we are using:
loadmodule "mi_fifo.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dispatcher.so"
loadmodule "utils.so"
loadmodule "path.so"
loadmodule "ipops.so"
loadmodule "jansson.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
We have several other Kamailio instances but they are
not faced to internet and they do not have such memory
issue. That other Kamailio instances have same
modules, except modules listed below. So if you think
that issue is inside external library, probably we need
to check first modules from list below.
loadmodule "ipops.so"
loadmodule "auth.so"
loadmodule "nathelper.so"
loadmodule "pike.so"
But maybe this other Kamailio instances do not have
this memory issue, just because they did not face to
internet and did not have same load as instances with
memory issue.
kamailio -v
version: kamailio 4.4.3 (x86_64/linux) 5a2195
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP,
TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et,
sigio_rt, select.
id: 5a2195
compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin
Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>>
wrote:
Hello,
it looks like a leak from the system memory, not
from kamailio's pkg or shm memory. This can be due
to an improper use of an external library (e.g.,
libxml2) by a kamailio module or because of a
problem in the library.
Can you list the modules used in your config (the
loadmodule lines)? I will try to guess from the
list which one relying on external libs with higher
risk of leak issues.
Also, provide the version of kamailio you are using
(kamailio -v).
Cheers,
Daniel
On 04/10/16 15:42, Jurijs Ivolga wrote:
> Hi,
>
> Our Kamailio server is crashing once per week,
> with following error:
>
> Oct 1 06:25:06 kamailio kernel: [26982632.803789]
> Out of memory in UB 210: OOM killed process 12261
> (kamailio) score 0 vm:1614768kB, rss:280200kB,
> swap:131408kB
>
> Core dump was never created, probably it is
> because of my environment, but I will try to get it.
>
> Server constantly eats memory, maybe some kind of
> memory leak?
>
> Any help is highly appreciated!
>
> Jurijs
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
list
> sr-users(a)lists.sip-router.org
> <mailto:sr-users@lists.sip-router.org>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
_______________________________________________ SIP
Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
3:45 p.m.
Hi Daniel,
I tried to compile with memory manager in debugging mode like this:
MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson json
snmpstats tls utils uuid'
make all
/etc/init.d/kamailioa stop
make install
/etc/init.d/kamailioa start
But still I'm getting nothing on:
kamailio -I | grep DBG_QM_MALLOC
Any hints?
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:39 AM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
> Hello,
>
> There are some results when searching for memory leak on openssl 1.0.1,
> but might not be valid for this case.
>
> Would you be able to run kamailio through valgrind? It may slow down a bit
> the processing, but may be the fastest way to catch the system memory leak.
> Maybe you have an instance will less traffic and you can run through
> valgrind for a while, you don't have to wait until you get the OOM.
>
> Cheers,
> Daniel
>
> On 07/10/16 09:27, Jurijs Ivolga wrote:
>
> Hi Daniel,
>
> openssl.x86_64
> 1.0.1e-48.el6_8.1
> CentOS release 6.8
>
> With kind regards,
>
> Jurijs
>
> On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla <
> miconda(a)gmail.com wrote:
>
>> Hello,
>>
>> the tls module in kamailio is using shm memory, but can be something
>> internal for libssl. What operating system do you use and what is the
>> libssl version?
>>
>> Cheers,
>> Daniel
>>
>> On 06/10/16 17:43, Jurijs Ivolga wrote:
>>
>> Hi Daniel,
>>
>> I do not use puke.top rpc command. Maybe this issue related to TLS?
>> Servers what has this problem are utilizing TLS heavily, servers which do
>> not has this problem use UDP.
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla <
>> miconda(a)gmail.com wrote:
>>
>>> Hello,
>>>
>>> are you using pike.top rpc command? I noticed in the code that it uses
>>> system malloc, but I haven't investigated further yet, first to see if
this
>>> would be a possibility ...
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 06/10/16 16:33, Jurijs Ivolga wrote:
>>>
>>> Hi Daniel,
>>>
>>> We do not do any external operations.
>>>
>>> We are using janson 2.7 everywhere. I will try to update to latest
>>> janson version tomorrow.
>>> All json operation is pretty much same, we are using only jansson_get.
>>>
>>> In attachment you can see memory consumption. On the right 2 servers
>>> which are faced internet on the left which don't face internet. As you
can
>>> see memory consumption is pretty dramatic.
>>>
>>> Thank you for your help!
>>>
>>> With kind regards,
>>>
>>> Jurijs
>>>
>>> On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla <
>>> miconda(a)gmail.com wrote:
>>>
>>>> Hello,
>>>>
>>>> are you doing different external operations than on the other
>>>> instances, like mi/rpc commands.
>>>>
>>>> From the list of the modules you exposed, I think jansson has the
>>>> higher probability to work with system memory. Are you doing different
json
>>>> operations in config that in the other instances of kamailio? Are you
using
>>>> same version of libjansson everywhere?
>>>>
>>>> Cheers,
>>>> Daniel
>>>> On 06/10/16 13:46, Jurijs Ivolga wrote:
>>>>
>>>> Hi Daniel,
>>>>
>>>> This modules what we are using:
>>>>
>>>> loadmodule "mi_fifo.so"
>>>> loadmodule "kex.so"
>>>> loadmodule "corex.so"
>>>> loadmodule "tm.so"
>>>> loadmodule "tmx.so"
>>>> loadmodule "sl.so"
>>>> loadmodule "rr.so"
>>>> loadmodule "pv.so"
>>>> loadmodule "maxfwd.so"
>>>> loadmodule "textops.so"
>>>> loadmodule "siputils.so"
>>>> loadmodule "xlog.so"
>>>> loadmodule "sanity.so"
>>>> loadmodule "ctl.so"
>>>> loadmodule "cfg_rpc.so"
>>>> loadmodule "mi_rpc.so"
>>>> loadmodule "dispatcher.so"
>>>> loadmodule "utils.so"
>>>> loadmodule "path.so"
>>>> loadmodule "ipops.so"
>>>> loadmodule "jansson.so"
>>>> loadmodule "auth.so"
>>>> loadmodule "nathelper.so"
>>>> loadmodule "tls.so"
>>>> loadmodule "htable.so"
>>>> loadmodule "pike.so"
>>>>
>>>> We have several other Kamailio instances but they are not faced to
>>>> internet and they do not have such memory issue. That other Kamailio
>>>> instances have same modules, except modules listed below. So if you
think
>>>> that issue is inside external library, probably we need to check first
>>>> modules from list below.
>>>>
>>>> loadmodule "ipops.so"
>>>> loadmodule "auth.so"
>>>> loadmodule "nathelper.so"
>>>> loadmodule "pike.so"
>>>>
>>>> But maybe this other Kamailio instances do not have this memory issue,
>>>> just because they did not face to internet and did not have same load as
>>>> instances with memory issue.
>>>>
>>>> kamailio -v
>>>> version: kamailio 4.4.3 (x86_64/linux) 5a2195
>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
>>>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
>>>> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
>>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
>>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>> id: 5a2195
>>>> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
>>>>
>>>>
>>>> With kind regards,
>>>>
>>>>
>>>> Jurijs
>>>>
>>>> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla <
>>>> miconda(a)gmail.com wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> it looks like a leak from the system memory, not from kamailio's
pkg
>>>>> or shm memory. This can be due to an improper use of an external
library
>>>>> (e.g., libxml2) by a kamailio module or because of a problem in the
library.
>>>>>
>>>>> Can you list the modules used in your config (the loadmodule lines)?
I
>>>>> will try to guess from the list which one relying on external libs
with
>>>>> higher risk of leak issues.
>>>>>
>>>>> Also, provide the version of kamailio you are using (kamailio -v).
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>> On 04/10/16 15:42, Jurijs Ivolga wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> Our Kamailio server is crashing once per week, with following error:
>>>>>
>>>>> Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in
UB
>>>>> 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
>>>>> rss:280200kB, swap:131408kB
>>>>>
>>>>> Core dump was never created, probably it is because of my
environment,
>>>>> but I will try to get it.
>>>>>
>>>>> Server constantly eats memory, maybe some kind of memory leak?
>>>>>
>>>>> Any help is highly appreciated!
>>>>>
>>>>> Jurijs
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
>>>>>
>>>>> _______________________________________________ SIP Express Router
>>>>> (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>> sr-users(a)lists.sip-router.org http://lists.sip-router.org/cg
>>>>> i-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
>>>>
>>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>
>>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>
>> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
6:15 p.m.
Hello,
that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and it is reflected
by the presence of DBG_SR_MEMORY in the output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside kamailio memory manager,
but a leak of using system memory, so it is not affected by
DBG_SR_MEMORY and cannot be troubleshooted using the mechanisms for pkg
and shm managers.
Cheers,
Daniel
On 07/10/16 15:45, Jurijs Ivolga wrote:
Hi Daniel,
I tried to compile with memory manager in debugging mode like this:
MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson json
snmpstats tls utils uuid'
make all
/etc/init.d/kamailioa stop
make install
/etc/init.d/kamailioa start
But still I'm getting nothing on:
kamailio -I | grep DBG_QM_MALLOC
Any hints?
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:39 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
There are some results when searching for memory leak on openssl
1.0.1, but might not be valid for this case.
Would you be able to run kamailio through valgrind? It may slow
down a bit the processing, but may be the fastest way to catch the
system memory leak. Maybe you have an instance will less traffic
and you can run through valgrind for a while, you don't have to
wait until you get the OOM.
Cheers,
Daniel
On 07/10/16 09:27, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
openssl.x86_64
1.0.1e-48.el6_8.1
CentOS release 6.8
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
the tls module in kamailio is using shm memory, but can be
something internal for libssl. What operating system do you
use and what is the libssl version?
Cheers,
Daniel
On 06/10/16 17:43, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
I do not use puke.top rpc command. Maybe this issue related
to TLS? Servers what has this problem are utilizing TLS
heavily, servers which do not has this problem use UDP.
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
are you using pike.top rpc command? I noticed in the
code that it uses system malloc, but I haven't
investigated further yet, first to see if this would be
a possibility ...
Cheers,
Daniel
On 06/10/16 16:33, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
We do not do any external operations.
We are using janson 2.7 everywhere. I will try to
update to latest janson version tomorrow.
All json operation is pretty much same, we are using
only jansson_get.
In attachment you can see memory consumption. On the
right 2 servers which are faced internet on the left
which don't face internet. As you can see memory
consumption is pretty dramatic.
Thank you for your help!
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin
Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>>
wrote:
Hello,
are you doing different external operations than on
the other instances, like mi/rpc commands.
From the list of the modules you exposed, I think
jansson has the higher probability to work with
system memory. Are you doing different json
operations in config that in the other instances of
kamailio? Are you using same version of libjansson
everywhere?
Cheers,
Daniel
On 06/10/16 13:46, Jurijs Ivolga wrote:
> Hi Daniel,
>
> This modules what we are using:
>
> loadmodule "mi_fifo.so"
> loadmodule "kex.so"
> loadmodule "corex.so"
> loadmodule "tm.so"
> loadmodule "tmx.so"
> loadmodule "sl.so"
> loadmodule "rr.so"
> loadmodule "pv.so"
> loadmodule "maxfwd.so"
> loadmodule "textops.so"
> loadmodule "siputils.so"
> loadmodule "xlog.so"
> loadmodule "sanity.so"
> loadmodule "ctl.so"
> loadmodule "cfg_rpc.so"
> loadmodule "mi_rpc.so"
> loadmodule "dispatcher.so"
> loadmodule "utils.so"
> loadmodule "path.so"
> loadmodule "ipops.so"
> loadmodule "jansson.so"
> loadmodule "auth.so"
> loadmodule "nathelper.so"
> loadmodule "tls.so"
> loadmodule "htable.so"
> loadmodule "pike.so"
>
> We have several other Kamailio instances but they
> are not faced to internet and they do not have
> such memory issue. That other Kamailio instances
> have same modules, except modules listed below.
> So if you think that issue is inside external
> library, probably we need to check first modules
> from list below.
>
> loadmodule "ipops.so"
> loadmodule "auth.so"
> loadmodule "nathelper.so"
> loadmodule "pike.so"
>
> But maybe this other Kamailio instances do not
> have this memory issue, just because they did not
> face to internet and did not have same load as
> instances with memory issue.
>
> kamailio -v
> version: kamailio 4.4.3 (x86_64/linux) 5a2195
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP,
> TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
> USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
> DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
> USE_DST_BLACKLIST, HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE
> 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE
> 65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et,
> sigio_rt, select.
> id: 5a2195
> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
>
>
> With kind regards,
>
>
> Jurijs
>
> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin
> Mierla <miconda(a)gmail.com
> <mailto:miconda@gmail.com>> wrote:
>
> Hello,
>
> it looks like a leak from the system memory,
> not from kamailio's pkg or shm memory. This
> can be due to an improper use of an external
> library (e.g., libxml2) by a kamailio module
> or because of a problem in the library.
>
> Can you list the modules used in your config
> (the loadmodule lines)? I will try to guess
> from the list which one relying on external
> libs with higher risk of leak issues.
>
> Also, provide the version of kamailio you are
> using (kamailio -v).
>
> Cheers,
> Daniel
>
>
> On 04/10/16 15:42, Jurijs Ivolga wrote:
>> Hi,
>>
>> Our Kamailio server is crashing once per
>> week, with following error:
>>
>> Oct 1 06:25:06 kamailio kernel:
>> [26982632.803789] Out of memory in UB 210:
>> OOM killed process 12261 (kamailio) score 0
>> vm:1614768kB, rss:280200kB, swap:131408kB
>>
>> Core dump was never created, probably it is
>> because of my environment, but I will try to
>> get it.
>>
>> Server constantly eats memory, maybe some
>> kind of memory leak?
>>
>> Any help is highly appreciated!
>>
>> Jurijs
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list
>> sr-users(a)lists.sip-router.org
>> <mailto:sr-users@lists.sip-router.org>
>>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda
> <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
> <http://www.linkedin.com/in/miconda>
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio
> (OpenSER) - sr-users mailing list
> sr-users(a)lists.sip-router.org
> <mailto:sr-users@lists.sip-router.org>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
>
>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
10 Oct
10 Oct
12:06 p.m.
Hi Daniel,
I left valgrind running for little while, not sure if this will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla <miconda(a)gmail.com>
wrote:
Hello,
that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and it is reflected by
the presence of DBG_SR_MEMORY in the output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside kamailio memory manager,
but a leak of using system memory, so it is not affected by DBG_SR_MEMORY
and cannot be troubleshooted using the mechanisms for pkg and shm managers.
Cheers,
Daniel
On 07/10/16 15:45, Jurijs Ivolga wrote:
Hi Daniel,
I tried to compile with memory manager in debugging mode like this:
MEMDBG=1 make cfg 'include_modules= textopsx db_mysql jansson json
snmpstats tls utils uuid'
make all
/etc/init.d/kamailioa stop
make install
/etc/init.d/kamailioa start
But still I'm getting nothing on:
kamailio -I | grep DBG_QM_MALLOC
Any hints?
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:39 AM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
There are some results when searching for memory leak on openssl 1.0.1,
but might not be valid for this case.
Would you be able to run kamailio through valgrind? It may slow down a
bit the processing, but may be the fastest way to catch the system memory
leak. Maybe you have an instance will less traffic and you can run through
valgrind for a while, you don't have to wait until you get the OOM.
Cheers,
Daniel
On 07/10/16 09:27, Jurijs Ivolga wrote:
Hi Daniel,
openssl.x86_64
1.0.1e-48.el6_8.1
CentOS release 6.8
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 10:20 AM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hello,
the tls module in kamailio is using shm memory, but can be something
internal for libssl. What operating system do you use and what is the
libssl version?
Cheers,
Daniel
On 06/10/16 17:43, Jurijs Ivolga wrote:
Hi Daniel,
I do not use puke.top rpc command. Maybe this issue related to TLS?
Servers what has this problem are utilizing TLS heavily, servers which do
not has this problem use UDP.
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 6:05 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
-- Hello,
are you using pike.top rpc command? I noticed in the code that it uses
system malloc, but I haven't investigated further yet, first to see if this
would be a possibility ...
Cheers,
Daniel
On 06/10/16 16:33, Jurijs Ivolga wrote:
Hi Daniel,
We do not do any external operations.
We are using janson 2.7 everywhere. I will try to update to latest
janson version tomorrow.
All json operation is pretty much same, we are using only jansson_get.
In attachment you can see memory consumption. On the right 2 servers
which are faced internet on the left which don't face internet. As you can
see memory consumption is pretty dramatic.
Thank you for your help!
With kind regards,
Jurijs
On Thu, Oct 6, 2016 at 5:17 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
> Hello,
>
> are you doing different external operations than on the other
> instances, like mi/rpc commands.
>
> From the list of the modules you exposed, I think jansson has the
> higher probability to work with system memory. Are you doing different json
> operations in config that in the other instances of kamailio? Are you using
> same version of libjansson everywhere?
>
> Cheers,
> Daniel
> On 06/10/16 13:46, Jurijs Ivolga wrote:
>
> Hi Daniel,
>
> This modules what we are using:
>
> loadmodule "mi_fifo.so"
> loadmodule "kex.so"
> loadmodule "corex.so"
> loadmodule "tm.so"
> loadmodule "tmx.so"
> loadmodule "sl.so"
> loadmodule "rr.so"
> loadmodule "pv.so"
> loadmodule "maxfwd.so"
> loadmodule "textops.so"
> loadmodule "siputils.so"
> loadmodule "xlog.so"
> loadmodule "sanity.so"
> loadmodule "ctl.so"
> loadmodule "cfg_rpc.so"
> loadmodule "mi_rpc.so"
> loadmodule "dispatcher.so"
> loadmodule "utils.so"
> loadmodule "path.so"
> loadmodule "ipops.so"
> loadmodule "jansson.so"
> loadmodule "auth.so"
> loadmodule "nathelper.so"
> loadmodule "tls.so"
> loadmodule "htable.so"
> loadmodule "pike.so"
>
> We have several other Kamailio instances but they are not faced to
> internet and they do not have such memory issue. That other Kamailio
> instances have same modules, except modules listed below. So if you think
> that issue is inside external library, probably we need to check first
> modules from list below.
>
> loadmodule "ipops.so"
> loadmodule "auth.so"
> loadmodule "nathelper.so"
> loadmodule "pike.so"
>
> But maybe this other Kamailio instances do not have this memory issue,
> just because they did not face to internet and did not have same load as
> instances with memory issue.
>
> kamailio -v
> version: kamailio 4.4.3 (x86_64/linux) 5a2195
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
> PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
> USE_DST_BLACKLIST, HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: 5a2195
> compiled on 08:30:51 Sep 15 2016 with gcc 4.4.7
>
>
> With kind regards,
>
>
> Jurijs
>
> On Thu, Oct 6, 2016 at 12:52 PM, Daniel-Constantin Mierla <
> miconda(a)gmail.com> wrote:
>
>> Hello,
>>
>> it looks like a leak from the system memory, not from kamailio's pkg
>> or shm memory. This can be due to an improper use of an external library
>> (e.g., libxml2) by a kamailio module or because of a problem in the library.
>>
>> Can you list the modules used in your config (the loadmodule lines)?
>> I will try to guess from the list which one relying on external libs with
>> higher risk of leak issues.
>>
>> Also, provide the version of kamailio you are using (kamailio -v).
>>
>> Cheers,
>> Daniel
>>
>> On 04/10/16 15:42, Jurijs Ivolga wrote:
>>
>> Hi,
>>
>> Our Kamailio server is crashing once per week, with following error:
>>
>> Oct 1 06:25:06 kamailio kernel: [26982632.803789] Out of memory in
>> UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB,
>> rss:280200kB, swap:131408kB
>>
>> Core dump was never created, probably it is because of my
>> environment, but I will try to get it.
>>
>> Server constantly eats memory, maybe some kind of memory leak?
>>
>> Any help is highly appreciated!
>>
>> Jurijs
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>
>> _______________________________________________ SIP Express Router
>> (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users(a)lists.sip-router.org http://lists.sip-router.org/cg
>> i-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
> --
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
--
11 Oct
11 Oct
9:55 a.m.
Hello,
from the logs, it seems to be related to curl library, I see many
reports like:
==16459== 189,318 bytes in 167 blocks are possibly lost in loss record
681 of 683
==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
==16459== by 0xB71AE62: PK11_CreateGenericObject (in
/usr/lib64/libnss3.so)
==16459== by 0xA0674DF: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA067666: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA069141: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA0601C4: Curl_ssl_connect (in /usr/lib64/libcurl.so.4.1.1)
That's like almost 200KB lost in this report.
From the list of the modules, I see you have utils and
I guess you use
http query function from there, is it?
Cheers,
Daniel
On 10/10/16 12:06, Jurijs Ivolga wrote:
Hi Daniel,
I left valgrind running for little while, not sure if this will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and it is
reflected by the presence of DBG_SR_MEMORY in the output of
'kamailio -v'.
Anyhow, what you reported is not a leak inside kamailio memory
manager, but a leak of using system memory, so it is not affected
by DBG_SR_MEMORY and cannot be troubleshooted using the mechanisms
for pkg and shm managers.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
1:43 p.m.
Hi Daniel,
You are correct we are using heavily http_query.
I found following bug report:
https://bugs.centos.org/view.php?id=9391
I will try to update to libcurl 7.44 and check if this help.
Thank you a lot Daniel!
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
from the logs, it seems to be related to curl library, I see many reports
like:
==16459== 189,318 bytes in 167 blocks are possibly lost in loss record 681
of 683
==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
==16459== by 0xB71AE62: PK11_CreateGenericObject (in
/usr/lib64/libnss3.so)
==16459== by 0xA0674DF: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA067666: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA069141: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA0601C4: Curl_ssl_connect (in
/usr/lib64/libcurl.so.4.1.1)
That's like almost 200KB lost in this report.
From the list of the modules, I see you have utils and I guess you use
http query function from there, is it?
Cheers,
Daniel
On 10/10/16 12:06, Jurijs Ivolga wrote:
Hi Daniel,
I left valgrind running for little while, not sure if this will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and it is reflected
by the presence of DBG_SR_MEMORY in the output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside kamailio memory manager,
but a leak of using system memory, so it is not affected by DBG_SR_MEMORY
and cannot be troubleshooted using the mechanisms for pkg and shm managers.
Cheers,
Daniel
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
12 Oct
12 Oct
3:04 p.m.
Hi Daniel,
Thank you a lot, it looks that issue is solved, after updating libcurl.
I was using following manual for updating libcurl, in case if somebody will
have same issue.
https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-cen…
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga <jurijs.ivolga(a)gmail.com>
wrote:
Hi Daniel,
You are correct we are using heavily http_query.
I found following bug report:
https://bugs.centos.org/view.php?id=9391
I will try to update to libcurl 7.44 and check if this help.
Thank you a lot Daniel!
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
from the logs, it seems to be related to curl library, I see many reports
like:
==16459== 189,318 bytes in 167 blocks are possibly lost in loss record
681 of 683
==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
==16459== by 0xB71AE62: PK11_CreateGenericObject (in
/usr/lib64/libnss3.so)
==16459== by 0xA0674DF: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA067666: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA069141: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA0601C4: Curl_ssl_connect (in
/usr/lib64/libcurl.so.4.1.1)
That's like almost 200KB lost in this report.
From the list of the modules, I see you have utils and I guess you use
http query function from there, is it?
Cheers,
Daniel
On 10/10/16 12:06, Jurijs Ivolga wrote:
Hi Daniel,
I left valgrind running for little while, not sure if this will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and it is reflected
by the presence of DBG_SR_MEMORY in the output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside kamailio memory manager,
but a leak of using system memory, so it is not affected by DBG_SR_MEMORY
and cannot be troubleshooted using the mechanisms for pkg and shm managers.
Cheers,
Daniel
--
Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
3:30 p.m.
Hello,
ok, so that was...
Maybe it would be good to add a note to the docs of the module about
this issue so people become aware of it. I guess the other http_*
modules are affected. Pull requests or other suggestions are welcome, of
course!
Cheers,
Daniel
On 12/10/16 15:04, Jurijs Ivolga wrote:
Hi Daniel,
Thank you a lot, it looks that issue is solved, after updating libcurl.
I was using following manual for updating libcurl, in case if somebody
will have same issue.
https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-cen…
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga
<jurijs.ivolga(a)gmail.com <mailto:jurijs.ivolga@gmail.com>> wrote:
Hi Daniel,
You are correct we are using heavily http_query.
I found following bug report:
https://bugs.centos.org/view.php?id=9391
<https://bugs.centos.org/view.php?id=9391>
I will try to update to libcurl 7.44 and check if this help.
Thank you a lot Daniel!
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
from the logs, it seems to be related to curl library, I see
many reports like:
==16459== 189,318 bytes in 167 blocks are possibly lost in
loss record 681 of 683
==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
==16459== by 0xB71AE62: PK11_CreateGenericObject (in
/usr/lib64/libnss3.so)
==16459== by 0xA0674DF: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA067666: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA069141: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA0601C4: Curl_ssl_connect (in
/usr/lib64/libcurl.so.4.1.1)
That's like almost 200KB lost in this report.
From the list of the modules, I see you have utils and I guess
you use http query function from there, is it?
Cheers,
Daniel
On 10/10/16 12:06, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
I left valgrind running for little while, not sure if this
will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and
it is reflected by the presence of DBG_SR_MEMORY in the
output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside kamailio
memory manager, but a leak of using system memory, so it
is not affected by DBG_SR_MEMORY and cannot be
troubleshooted using the mechanisms for pkg and shm managers.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
3:35 p.m.
Hi Daniel,
I will test couple more days and later I will try to add some comments to
utils, http_client & HTTP_ASYNC_CLIENT(not sure if there any other modules
which use curl, but I will check) modules when I will be 200% sure that
issue is gone.
With kind regards,
Jurijs
On Wed, Oct 12, 2016 at 4:30 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
> Hello,
>
> ok, so that was...
>
> Maybe it would be good to add a note to the docs of the module about this
> issue so people become aware of it. I guess the other http_* modules are
> affected. Pull requests or other suggestions are welcome, of course!
>
> Cheers,
> Daniel
>
> On 12/10/16 15:04, Jurijs Ivolga wrote:
>
> Hi Daniel,
>
> Thank you a lot, it looks that issue is solved, after updating libcurl.
>
> I was using following manual for updating libcurl, in case if somebody
> will have same issue.
>
> https://www.digitalocean.com/community/questions/how-to-
> upgrade-curl-in-centos6
>
> With kind regards,
>
> Jurijs
>
> On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga <jurijs.ivolga(a)gmail.com>
wrote:
>
>> Hi Daniel,
>>
>> You are correct we are using heavily http_query.
>>
>> I found following bug report:
>>
>> https://bugs.centos.org/view.php?id=9391
>>
>> I will try to update to libcurl 7.44 and check if this help.
>>
>> Thank you a lot Daniel!
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla <
>> miconda(a)gmail.com wrote:
>>
>>> Hello,
>>>
>>> from the logs, it seems to be related to curl library, I see many
>>> reports like:
>>>
>>> ==16459== 189,318 bytes in 167 blocks are possibly lost in loss record
>>> 681 of 683
>>> ==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
>>> ==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
>>> ==16459== by 0xB71AE62: PK11_CreateGenericObject (in
>>> /usr/lib64/libnss3.so)
>>> ==16459== by 0xA0674DF: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>> ==16459== by 0xA067666: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>> ==16459== by 0xA069141: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>> ==16459== by 0xA0601C4: Curl_ssl_connect (in
>>> /usr/lib64/libcurl.so.4.1.1)
>>>
>>> That's like almost 200KB lost in this report.
>>>
>>> From the list of the modules, I see you have utils and I guess you use
>>> http query function from there, is it?
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 10/10/16 12:06, Jurijs Ivolga wrote:
>>>
>>> Hi Daniel,
>>>
>>> I left valgrind running for little while, not sure if this will be
>>> enough.
>>>
>>> Please find attached log file.
>>>
>>> Thank you a lot for your help!
>>>
>>> With kind regards,
>>>
>>> Jurijs
>>>
>>> On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla <
>>> miconda(a)gmail.com wrote:
>>>
>>>> Hello,
>>>>
>>>> that's the way it was done for older versions of kamailio.
>>>>
>>>> In master and 4.4 the memory debugging is turned on and it is reflected
>>>> by the presence of DBG_SR_MEMORY in the output of 'kamailio -v'.
>>>>
>>>> Anyhow, what you reported is not a leak inside kamailio memory manager,
>>>> but a leak of using system memory, so it is not affected by
DBG_SR_MEMORY
>>>> and cannot be troubleshooted using the mechanisms for pkg and shm
managers.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>
>>> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
3:47 p.m.
Hello,
thanks -- coming in my mind now, xcap_client should use also the curl
library.
Cheers,
Daniel
On 12/10/16 15:35, Jurijs Ivolga wrote:
Hi Daniel,
I will test couple more days and later I will try to add some comments
to utils, http_client & HTTP_ASYNC_CLIENT(not sure if there any other
modules which use curl, but I will check) modules when I will be 200%
sure that issue is gone.
With kind regards,
Jurijs
On Wed, Oct 12, 2016 at 4:30 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
ok, so that was...
Maybe it would be good to add a note to the docs of the module
about this issue so people become aware of it. I guess the other
http_* modules are affected. Pull requests or other suggestions
are welcome, of course!
Cheers,
Daniel
On 12/10/16 15:04, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
Thank you a lot, it looks that issue is solved, after updating
libcurl.
I was using following manual for updating libcurl, in case if
somebody will have same issue.
https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-cen…
<https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-centos6>
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga
<jurijs.ivolga(a)gmail.com <mailto:jurijs.ivolga@gmail.com>> wrote:
Hi Daniel,
You are correct we are using heavily http_query.
I found following bug report:
https://bugs.centos.org/view.php?id=9391
<https://bugs.centos.org/view.php?id=9391>
I will try to update to libcurl 7.44 and check if this help.
Thank you a lot Daniel!
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
from the logs, it seems to be related to curl library, I
see many reports like:
==16459== 189,318 bytes in 167 blocks are possibly lost
in loss record 681 of 683
==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
==16459== by 0xB71AE62: PK11_CreateGenericObject (in
/usr/lib64/libnss3.so)
==16459== by 0xA0674DF: ??? (in
/usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA067666: ??? (in
/usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA069141: ??? (in
/usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA0601C4: Curl_ssl_connect (in
/usr/lib64/libcurl.so.4.1.1)
That's like almost 200KB lost in this report.
From the list of the modules, I see you have utils and I
guess you use http query function from there, is it?
Cheers,
Daniel
On 10/10/16 12:06, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
I left valgrind running for little while, not sure if
this will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
that's the way it was done for older versions of
kamailio.
In master and 4.4 the memory debugging is turned on
and it is reflected by the presence of DBG_SR_MEMORY
in the output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside
kamailio memory manager, but a leak of using system
memory, so it is not affected by DBG_SR_MEMORY and
cannot be troubleshooted using the mechanisms for
pkg and shm managers.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
21 Oct
21 Oct
11:22 a.m.
Hi Daniel,
After 8 days there is still small memory leak, 200MB of Ram wasn't freed
during this 8 days, please check diagram below.
1) Is this something usual?
2) Memory leak seems very small, maybe I should never worry about this?
What you think?
[image: Inline image 2]
Thank
you!
With kind regards,
Jurijs
On Wed, Oct 12, 2016 at 4:47 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
> Hello,
>
> thanks -- coming in my mind now, xcap_client should use also the curl
> library.
> Cheers,
> Daniel
>
>
> On 12/10/16 15:35, Jurijs Ivolga wrote:
>
> Hi Daniel,
>
> I will test couple more days and later I will try to add some comments to
> utils, http_client & HTTP_ASYNC_CLIENT(not sure if there any other modules
> which use curl, but I will check) modules when I will be 200% sure that
> issue is gone.
>
> With kind regards,
>
> Jurijs
>
> On Wed, Oct 12, 2016 at 4:30 PM, Daniel-Constantin Mierla <
> miconda(a)gmail.com wrote:
>
>> Hello,
>>
>> ok, so that was...
>>
>> Maybe it would be good to add a note to the docs of the module about this
>> issue so people become aware of it. I guess the other http_* modules are
>> affected. Pull requests or other suggestions are welcome, of course!
>>
>> Cheers,
>> Daniel
>>
>> On 12/10/16 15:04, Jurijs Ivolga wrote:
>>
>> Hi Daniel,
>>
>> Thank you a lot, it looks that issue is solved, after updating libcurl.
>>
>> I was using following manual for updating libcurl, in case if somebody
>> will have same issue.
>>
>> https://www.digitalocean.com/community/questions/how-to-upgr
>> ade-curl-in-centos6
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga <jurijs.ivolga(a)gmail.com>
> wrote:
>>
>>> Hi Daniel,
>>>
>>> You are correct we are using heavily http_query.
>>>
>>> I found following bug report:
>>>
>>> https://bugs.centos.org/view.php?id=9391
>>>
>>> I will try to update to libcurl 7.44 and check if this help.
>>>
>>> Thank you a lot Daniel!
>>>
>>> With kind regards,
>>>
>>> Jurijs
>>>
>>> On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla <
>>> miconda(a)gmail.com wrote:
>>>
>>>> Hello,
>>>>
>>>> from the logs, it seems to be related to curl library, I see many
>>>> reports like:
>>>>
>>>> ==16459== 189,318 bytes in 167 blocks are possibly lost in loss record
>>>> 681 of 683
>>>> ==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
>>>> ==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
>>>> ==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
>>>> ==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
>>>> ==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
>>>> ==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
>>>> ==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
>>>> ==16459== by 0xB71AE62: PK11_CreateGenericObject (in
>>>> /usr/lib64/libnss3.so)
>>>> ==16459== by 0xA0674DF: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>>> ==16459== by 0xA067666: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>>> ==16459== by 0xA069141: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>>> ==16459== by 0xA0601C4: Curl_ssl_connect (in
>>>> /usr/lib64/libcurl.so.4.1.1)
>>>>
>>>> That's like almost 200KB lost in this report.
>>>>
>>>> From the list of the modules, I see you have utils and I guess you use
>>>> http query function from there, is it?
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 10/10/16 12:06, Jurijs Ivolga wrote:
>>>>
>>>> Hi Daniel,
>>>>
>>>> I left valgrind running for little while, not sure if this will be
>>>> enough.
>>>>
>>>> Please find attached log file.
>>>>
>>>> Thank you a lot for your help!
>>>>
>>>> With kind regards,
>>>>
>>>> Jurijs
>>>>
>>>> On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla <
>>>> miconda(a)gmail.com wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> that's the way it was done for older versions of kamailio.
>>>>>
>>>>> In master and 4.4 the memory debugging is turned on and it is
>>>>> reflected by the presence of DBG_SR_MEMORY in the output of
'kamailio -v'.
>>>>>
>>>>> Anyhow, what you reported is not a leak inside kamailio memory
>>>>> manager, but a leak of using system memory, so it is not affected by
>>>>> DBG_SR_MEMORY and cannot be troubleshooted using the mechanisms for
pkg and
>>>>> shm managers.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
>>>>
>>>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>
>> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
11:36 a.m.
Hello,
On 21/10/16 11:22, Jurijs Ivolga wrote:
Hi Daniel,
After 8 days there is still small memory leak, 200MB of Ram wasn't
freed during this 8 days, please check diagram below.
1) Is this something usual?
2) Memory leak seems very small, maybe I should never worry about
this? What you think?
200MB in 8 days it's still relevant and a leak. Maybe you can run again
with valgrind...
Unless it is unclaimed memory by kernel:
-
https://www.kamailio.org/wiki/tutorials/troubleshooting/memory#os_memory_re…
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
26 Oct
26 Oct
1:12 p.m.
Hi Daniel,
I ran again with valgrind and still there is some issues.
It will be really nice if you can take a look. :)
With kind regards,
Jurijs
On Fri, Oct 21, 2016 at 12:36 PM, Daniel-Constantin Mierla <
miconda(a)gmail.com> wrote:
Hello,
On 21/10/16 11:22, Jurijs Ivolga wrote:
Hi Daniel,
After 8 days there is still small memory leak, 200MB of Ram wasn't
freed during this 8 days, please check diagram below.
1) Is this something usual?
2) Memory leak seems very small, maybe I should never worry about
this? What you think?
200MB in 8 days it's still relevant and a leak. Maybe you can run again
with valgrind...
Unless it is unclaimed memory by kernel:
-
https://www.kamailio.org/wiki/tutorials/troubleshooting/
memory#os_memory_reports
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com
7 Nov
7 Nov
10:42 a.m.
Hi Daniel,
I found modules what are impacted by this leak:
http_client
utils
xcap
http_async_client
auth_identity
xcap_client
I would like to update documentations, but this is first time when I'm
updating documentation.
I believe I need to update xml file for module. Should I then regenerate
readme file as stated here:
https://www.kamailio.org/wiki/devel/how-to/module-readme
Or not?
Then I believe to create pull request with my changes. Correct?
Maybe you have manual for this?
With kind regards,
Jurijs
On Wed, Oct 12, 2016 at 4:30 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
> Hello,
>
> ok, so that was...
>
> Maybe it would be good to add a note to the docs of the module about this
> issue so people become aware of it. I guess the other http_* modules are
> affected. Pull requests or other suggestions are welcome, of course!
>
> Cheers,
> Daniel
>
> On 12/10/16 15:04, Jurijs Ivolga wrote:
>
> Hi Daniel,
>
> Thank you a lot, it looks that issue is solved, after updating libcurl.
>
> I was using following manual for updating libcurl, in case if somebody
> will have same issue.
>
> https://www.digitalocean.com/community/questions/how-to-
> upgrade-curl-in-centos6
>
> With kind regards,
>
> Jurijs
>
> On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga <jurijs.ivolga(a)gmail.com>
wrote:
>
>> Hi Daniel,
>>
>> You are correct we are using heavily http_query.
>>
>> I found following bug report:
>>
>> https://bugs.centos.org/view.php?id=9391
>>
>> I will try to update to libcurl 7.44 and check if this help.
>>
>> Thank you a lot Daniel!
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla <
>> miconda(a)gmail.com wrote:
>>
>>> Hello,
>>>
>>> from the logs, it seems to be related to curl library, I see many
>>> reports like:
>>>
>>> ==16459== 189,318 bytes in 167 blocks are possibly lost in loss record
>>> 681 of 683
>>> ==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
>>> ==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
>>> ==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
>>> ==16459== by 0xB71AE62: PK11_CreateGenericObject (in
>>> /usr/lib64/libnss3.so)
>>> ==16459== by 0xA0674DF: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>> ==16459== by 0xA067666: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>> ==16459== by 0xA069141: ??? (in /usr/lib64/libcurl.so.4.1.1)
>>> ==16459== by 0xA0601C4: Curl_ssl_connect (in
>>> /usr/lib64/libcurl.so.4.1.1)
>>>
>>> That's like almost 200KB lost in this report.
>>>
>>> From the list of the modules, I see you have utils and I guess you use
>>> http query function from there, is it?
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 10/10/16 12:06, Jurijs Ivolga wrote:
>>>
>>> Hi Daniel,
>>>
>>> I left valgrind running for little while, not sure if this will be
>>> enough.
>>>
>>> Please find attached log file.
>>>
>>> Thank you a lot for your help!
>>>
>>> With kind regards,
>>>
>>> Jurijs
>>>
>>> On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla <
>>> miconda(a)gmail.com wrote:
>>>
>>>> Hello,
>>>>
>>>> that's the way it was done for older versions of kamailio.
>>>>
>>>> In master and 4.4 the memory debugging is turned on and it is reflected
>>>> by the presence of DBG_SR_MEMORY in the output of 'kamailio -v'.
>>>>
>>>> Anyhow, what you reported is not a leak inside kamailio memory manager,
>>>> but a leak of using system memory, so it is not affected by
DBG_SR_MEMORY
>>>> and cannot be troubleshooted using the mechanisms for pkg and shm
managers.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>>>
>>> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
>
>
10:44 a.m.
New subject: Out of memory in UB 210: OOM killed process 12261 (kamailio) score 0 vm:1614768kB, rss:280200kB, swap:131408kB
I think we need to update the Make system instead of the docs to avoid this version.
Or implement some code like we have for OpenSSL that outputs warnings at runtime.
/O
On 07 Nov 2016, at 10:42, Jurijs Ivolga
<jurijs.ivolga(a)gmail.com> wrote:
Hi Daniel,
I found modules what are impacted by this leak:
http_client
utils
xcap
http_async_client
auth_identity
xcap_client
I would like to update documentations, but this is first time when I'm updating
documentation.
I believe I need to update xml file for module. Should I then regenerate readme file as
stated here:
https://www.kamailio.org/wiki/devel/how-to/module-readme
<https://www.kamailio.org/wiki/devel/how-to/module-readme>
Or not?
Then I believe to create pull request with my changes. Correct?
Maybe you have manual for this?
With kind regards,
Jurijs
On Wed, Oct 12, 2016 at 4:30 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
ok, so that was...
Maybe it would be good to add a note to the docs of the module about this issue so people
become aware of it. I guess the other http_* modules are affected. Pull requests or other
suggestions are welcome, of course!
Cheers,
Daniel
On 12/10/16 15:04, Jurijs Ivolga wrote:
Hi Daniel,
Thank you a lot, it looks that issue is solved, after updating libcurl.
I was using following manual for updating libcurl, in case if somebody will have same
issue.
https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-cen…
<https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-centos6>
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga <jurijs.ivolga(a)gmail.com
<mailto:jurijs.ivolga@gmail.com>> wrote:
Hi Daniel,
You are correct we are using heavily http_query.
I found following bug report:
https://bugs.centos.org/view.php?id=9391
<https://bugs.centos.org/view.php?id=9391>
I will try to update to libcurl 7.44 and check if this help.
Thank you a lot Daniel!
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
from the logs, it seems to be related to curl library, I see many reports like:
==16459== 189,318 bytes in 167 blocks are possibly lost in loss record 681 of 683
==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
==16459== by 0xB71AE62: PK11_CreateGenericObject (in /usr/lib64/libnss3.so)
==16459== by 0xA0674DF: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA067666: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA069141: ??? (in /usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA0601C4: Curl_ssl_connect (in /usr/lib64/libcurl.so.4.1.1)
That's like almost 200KB lost in this report.
From the list of the modules, I see you have utils and I guess you use http query
function from there, is it?
Cheers,
Daniel
On 10/10/16 12:06, Jurijs Ivolga wrote:
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#!/miconda> -
http://www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
<http://www.asipto.com/>
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users Hi Daniel,
I left valgrind running for little while, not sure if this will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
that's the way it was done for older versions of kamailio.
In master and 4.4 the memory debugging is turned on and it is reflected by the presence
of DBG_SR_MEMORY in the output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside kamailio memory manager, but a leak of
using system memory, so it is not affected by DBG_SR_MEMORY and cannot be troubleshooted
using the mechanisms for pkg and shm managers.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
<http://www.asipto.com/>--
8 Nov
8 Nov
9:40 a.m.
I think this issue was exposed only on centos/redhad packages, which is
harder to detect as we set the OS type to linux. If someone comes up
with a solution, I am ok, otherwise I think it's fine to add some
remarks in the docs.
Cheers,
Daniel
On 07/11/16 10:44, Olle E. Johansson wrote:
I think we need to update the Make system instead of
the docs to avoid
this version.
Or implement some code like we have for OpenSSL that outputs warnings
at runtime.
/O
> On 07 Nov 2016, at 10:42, Jurijs Ivolga <jurijs.ivolga(a)gmail.com
> <mailto:jurijs.ivolga@gmail.com>> wrote:
>
> Hi Daniel,
>
> I found modules what are impacted by this leak:
>
> http_client
> utils
> xcap
> http_async_client
> auth_identity
> xcap_client
>
> I would like to update documentations, but this is first time when
> I'm updating documentation.
>
> I believe I need to update xml file for module. Should I then
> regenerate readme file as stated here:
>
> https://www.kamailio.org/wiki/devel/how-to/module-readme
>
> Or not?
>
> Then I believe to create pull request with my changes. Correct?
>
> Maybe you have manual for this?
>
> With kind regards,
>
> Jurijs
>
> On Wed, Oct 12, 2016 at 4:30 PM, Daniel-Constantin Mierla
> <miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
>
> Hello,
>
> ok, so that was...
>
> Maybe it would be good to add a note to the docs of the module
> about this issue so people become aware of it. I guess the other
> http_* modules are affected. Pull requests or other suggestions
> are welcome, of course!
>
> Cheers,
> Daniel
>
>
> On 12/10/16 15:04, Jurijs Ivolga wrote:
>> Hi Daniel,
>>
>> Thank you a lot, it looks that issue is solved, after updating
>> libcurl.
>>
>> I was using following manual for updating libcurl, in case if
>> somebody will have same issue.
>>
>>
https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-cen…
>>
<https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-centos6>
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga
>> <jurijs.ivolga(a)gmail.com <mailto:jurijs.ivolga@gmail.com>>
wrote:
>>
>> Hi Daniel,
>>
>> You are correct we are using heavily http_query.
>>
>> I found following bug report:
>>
>> https://bugs.centos.org/view.php?id=9391
>> <https://bugs.centos.org/view.php?id=9391>
>>
>> I will try to update to libcurl 7.44 and check if this help.
>>
>> Thank you a lot Daniel!
>>
>> With kind regards,
>>
>> Jurijs
>>
>> On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla
>> <miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
>>
>> Hello,
>>
>> from the logs, it seems to be related to curl library, I
>> see many reports like:
>>
>> ==16459== 189,318 bytes in 167 blocks are possibly lost
>> in loss record 681 of 683
>> ==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
>> ==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
>> ==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
>> ==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
>> ==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
>> ==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
>> ==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
>> ==16459== by 0xB71AE62: PK11_CreateGenericObject (in
>> /usr/lib64/libnss3.so)
>> ==16459== by 0xA0674DF: ??? (in
>> /usr/lib64/libcurl.so.4.1.1)
>> ==16459== by 0xA067666: ??? (in
>> /usr/lib64/libcurl.so.4.1.1)
>> ==16459== by 0xA069141: ??? (in
>> /usr/lib64/libcurl.so.4.1.1)
>> ==16459== by 0xA0601C4: Curl_ssl_connect (in
>> /usr/lib64/libcurl.so.4.1.1)
>>
>> That's like almost 200KB lost in this report.
>>
>> From the list of the modules, I see you have utils and I
>> guess you use http query function from there, is it?
>>
>> Cheers,
>> Daniel
>>
>>
>> On 10/10/16 12:06, Jurijs Ivolga wrote:
>>> Hi Daniel,
>>>
>>> I left valgrind running for little while, not sure if
>>> this will be enough.
>>>
>>> Please find attached log file.
>>>
>>> Thank you a lot for your help!
>>>
>>> With kind regards,
>>>
>>> Jurijs
>>>
>>> On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin
>>> Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>>
>>> wrote:
>>>
>>> Hello,
>>>
>>> that's the way it was done for older versions of
>>> kamailio.
>>>
>>> In master and 4.4 the memory debugging is turned on
>>> and it is reflected by the presence of
>>> DBG_SR_MEMORY in the output of 'kamailio -v'.
>>>
>>> Anyhow, what you reported is not a leak inside
>>> kamailio memory manager, but a leak of using system
>>> memory, so it is not affected by DBG_SR_MEMORY and
>>> cannot be troubleshooted using the mechanisms for
>>> pkg and shm managers.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierla
>> http://twitter.com/#!/miconda
>> <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
>> <http://www.linkedin.com/in/miconda>
>> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 -
http://www.asipto.com <http://www.asipto.com/>
>>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
> <http://www.linkedin.com/in/miconda>
> Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
<http://www.asipto.com/>
>
> _______________________________________________ SIP Express Router
> (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Nov 28-30, 2016 - http://www.asipto.com
9:38 a.m.
Hello,
you don't have to generate the readme files anymore, they are generated
automatically on the server after committing the xml files.
For creating the pull requests, just search the web/github about how to
do it for github -- there are some good manuals in place hosted by
github.com, but I don't have the links at hand right now.
Cheers,
Daniel
On 07/11/16 10:42, Jurijs Ivolga wrote:
Hi Daniel,
I found modules what are impacted by this leak:
http_client
utils
xcap
http_async_client
auth_identity
xcap_client
I would like to update documentations, but this is first time when I'm
updating documentation.
I believe I need to update xml file for module. Should I then
regenerate readme file as stated here:
https://www.kamailio.org/wiki/devel/how-to/module-readme
Or not?
Then I believe to create pull request with my changes. Correct?
Maybe you have manual for this?
With kind regards,
Jurijs
On Wed, Oct 12, 2016 at 4:30 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
ok, so that was...
Maybe it would be good to add a note to the docs of the module
about this issue so people become aware of it. I guess the other
http_* modules are affected. Pull requests or other suggestions
are welcome, of course!
Cheers,
Daniel
On 12/10/16 15:04, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Nov 28-30, 2016 - http://www.asipto.com
Hi Daniel,
Thank you a lot, it looks that issue is solved, after updating
libcurl.
I was using following manual for updating libcurl, in case if
somebody will have same issue.
https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-cen…
<https://www.digitalocean.com/community/questions/how-to-upgrade-curl-in-centos6>
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 2:43 PM, Jurijs Ivolga
<jurijs.ivolga(a)gmail.com <mailto:jurijs.ivolga@gmail.com>> wrote:
Hi Daniel,
You are correct we are using heavily http_query.
I found following bug report:
https://bugs.centos.org/view.php?id=9391
<https://bugs.centos.org/view.php?id=9391>
I will try to update to libcurl 7.44 and check if this help.
Thank you a lot Daniel!
With kind regards,
Jurijs
On Tue, Oct 11, 2016 at 10:55 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
from the logs, it seems to be related to curl library, I
see many reports like:
==16459== 189,318 bytes in 167 blocks are possibly lost
in loss record 681 of 683
==16459== at 0x4C26FEF: calloc (vg_replace_malloc.c:711)
==16459== by 0x104BB699: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AA537: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104AB81E: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B0B88: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0x104B77E1: ??? (in /usr/lib64/libnsspem.so)
==16459== by 0xB71ABC9: ??? (in /usr/lib64/libnss3.so)
==16459== by 0xB71AE62: PK11_CreateGenericObject (in
/usr/lib64/libnss3.so)
==16459== by 0xA0674DF: ??? (in
/usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA067666: ??? (in
/usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA069141: ??? (in
/usr/lib64/libcurl.so.4.1.1)
==16459== by 0xA0601C4: Curl_ssl_connect (in
/usr/lib64/libcurl.so.4.1.1)
That's like almost 200KB lost in this report.
From the list of the modules, I see you have utils and I
guess you use http query function from there, is it?
Cheers,
Daniel
On 10/10/16 12:06, Jurijs Ivolga wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
Hi Daniel,
I left valgrind running for little while, not sure if
this will be enough.
Please find attached log file.
Thank you a lot for your help!
With kind regards,
Jurijs
On Fri, Oct 7, 2016 at 7:15 PM, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
that's the way it was done for older versions of
kamailio.
In master and 4.4 the memory debugging is turned on
and it is reflected by the presence of DBG_SR_MEMORY
in the output of 'kamailio -v'.
Anyhow, what you reported is not a leak inside
kamailio memory manager, but a leak of using system
memory, so it is not affected by DBG_SR_MEMORY and
cannot be troubleshooted using the mechanisms for
pkg and shm managers.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
2752
days inactive
2787
days old
25 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Jurijs Ivolga -
Olle E. Johansson