[SR-Users] Kamailio 4.0.4 slow memory leak

Tue Aug 26 22:42:56 CEST 2014

I have attached is the output of "kamcmd cfg.set_now_int core mem_dump_shm 1" to this email.  I had to restart Kamailio to load ctl and ctl_rpc, so let me know if I need to do another one after some time passes and calls are processed.

Also, here is the output of "kamctl mi get_statistics shmem:"

shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768

Thanks!
-Tim

From: Daniel-Constantin Mierla [mailto:miconda at gmail.com]
Sent: Tuesday, August 26, 2014 10:46 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

It is not needed to see the memory operations logs. Just to dump the chunks in memory.

You would need ctl and ctl_rpc modules (iirc -- they should be in the default config file), then run:

kamcmd cfg.set_now_int core mem_dump_shm 1

Extract the logs of the dump from the shared memory and send them over.

Send also the output for:

kamctl mi get_statistics shmem:

Cheers,
Daniel
On 26/08/14 17:34, Heenan, Timothy Steven wrote:
Thank you.

I have recompiled with MEMDBG, however I am not observing any additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.

Here are my log settings from Kamailio.cfg

debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
...
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")

From: Daniel-Constantin Mierla [mailto:miconda at gmail.com]
Sent: Tuesday, August 26, 2014 3:54 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

I quickly looked over the modules that use shared memory and I couldn't spot a commit related to any leak.

Have you had the chance to compile with  MEMDBG?

Cheers,
Daniel
On 19/08/14 19:14, Heenan, Timothy Steven wrote:
Do you think any of these modules possibly causing this issue?

-Tim

From: sr-users-bounces at lists.sip-router.org<mailto:sr-users-bounces at lists.sip-router.org> [mailto:sr-users-bounces at lists.sip-router.org] On Behalf Of Heenan, Timothy Steven
Sent: Tuesday, August 12, 2014 2:37 PM
To: 'miconda at gmail.com<mailto:miconda at gmail.com>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hi Daniel,

Here's a list of the modules I'm running:

mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace

We're setting the memory allocation to 256 ( -m 256 -M 64 ).

I'll see about recompiling with MEMDBG=1.

Regards,
- Tim

From: sr-users-bounces at lists.sip-router.org<mailto:sr-users-bounces at lists.sip-router.org> [mailto:sr-users-bounces at lists.sip-router.org] On Behalf Of Daniel-Constantin Mierla
Sent: Tuesday, August 12, 2014 4:17 AM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

can you give the list of loaded modules? How much memory did you allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
On 12/08/14 01:40, Heenan, Timothy Steven wrote:
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.

Thank you in advance,
Tim Heenan
Engineer I - VoIPWholesale | Windstream
timothy.heenan at windstream.com<mailto:email at windstream.com> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116

________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.

_______________________________________________

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list

sr-users at lists.sip-router.org<mailto:sr-users at lists.sip-router.org>

http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140826/9ae943b7/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: kamailio_mem.txt
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140826/9ae943b7/attachment-0001.txt>