After restarting Kamailio on September 3rd, the problem has again gotten to the point of no call processing. Here are the memory statistics; additionally, I can see the virtual memory has ballooned up to 400mb again. shmem:fragments = 2378 shmem:free_size = 209840 shmem:max_used_size = 268393216 shmem:real_used_size = 268225616 shmem:total_size = 268435456 shmem:used_size = 220534792 -Tim *From:*sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] *On Behalf Of *Heenan, Timothy Steven *Sent:* Wednesday, September 03, 2014 3:28 PM *To:* &#39;miconda(a)gmail.com&#39;com'; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak I have about 460 destination sets. In each set I could have from 1 address up to 6. When I took that initial memory log, Kamailio had been restarted shortly beforehand. I just took another one after the system had decayed so Kamailio was responding to Invites with “SIP/2.0 500 No error (2/SL).” (exhibiting the issue after a few days) and the log is about 400Mb! I’ll attach a small snippet. Seems to be going on and on about the dispatcher module. -Tim *From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Sent:* Thursday, August 28, 2014 5:03 AM *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak Looking at the logs, there are 1336 URIs, which seems ok given your numbers, because the module keeps also the previous set of records (needed because at reload time there can be kamailio worker processes using the records -- perhaps we can improve a bit here, I will review the current reload code a bit, since it was a contribution from another developer quite long time ago). Some more questions: - how many destination sets (groups) do you have? Can you estimate the minimum and maximum addresses in a set? - when did you take the memory log? In other words, for how long was kamailio running? Did you wait enough to notice the steady increase of memory usage? Cheers, Daniel On 28/08/14 01:56, Heenan, Timothy Steven wrote: Thank you for the help. For dispatcher, I’m using a database that contains about 700 records. A reload is performed via cronjob every 5 minutes. The command being used is: kamctl dispatcher reload Thanks, -Tim *From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Sent:* Wednesday, August 27, 2014 5:22 PM *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak The logs suggest some leaks in dispatcher module -- it allocates shared memory only for caching routing records. Few more questions I need to get the answers in order to troubleshoot further. How many records do you have in dispatcher table? Is it in database or the text file? How ofter do you reload? What is the command you use for reload? Cheers, Daniel On 26/08/14 22:42, Heenan, Timothy Steven wrote: I have attached is the output of “kamcmd cfg.set_now_int core mem_dump_shm 1” to this email. I had to restart Kamailio to load ctl and ctl_rpc, so let me know if I need to do another one after some time passes and calls are processed. Also, here is the output of “kamctl mi get_statistics shmem:” shmem:fragments = 398 shmem:free_size = 259170720 shmem:max_used_size = 9265184 shmem:real_used_size = 9264736 shmem:total_size = 268435456 shmem:used_size = 8559768 Thanks! -Tim *From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Sent:* Tuesday, August 26, 2014 10:46 AM *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak It is not needed to see the memory operations logs. Just to dump the chunks in memory. You would need ctl and ctl_rpc modules (iirc -- they should be in the default config file), then run: kamcmd cfg.set_now_int core mem_dump_shm 1 Extract the logs of the dump from the shared memory and send them over. Send also the output for: kamctl mi get_statistics shmem: Cheers, Daniel On 26/08/14 17:34, Heenan, Timothy Steven wrote: Thank you. I have recompiled with MEMDBG, however I am not observing any additional output in my log files. I think I have something set incorrectly in kamailio.cfg. Here are my log settings from Kamailio.cfg debug=3 log_stderror=no log_facility=LOG_LOCAL0 memdbg=2 memlog=2 … modparam("debugger", "cfgtrace", 1) modparam("debugger", "log_level", 2) modparam("debugger", "cfgpkgcheck", 1) modparam("debugger", "mod_level", "core=2") modparam("debugger", "mod_level", "tm=2") modparam("debugger", "mod_level", "dispatcher=2") modparam("debugger", "mod_level", "siptrace=2") *From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Sent:* Tuesday, August 26, 2014 3:54 AM *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak Hello, I quickly looked over the modules that use shared memory and I couldn't spot a commit related to any leak. Have you had the chance to compile with MEMDBG? Cheers, Daniel On 19/08/14 19:14, Heenan, Timothy Steven wrote: Do you think any of these modules possibly causing this issue? -Tim *From:*sr-users-bounces@lists.sip-router.org <mailto:sr-users-bounces@lists.sip-router.org> [mailto:sr-users-bounces@lists.sip-router.org] *On Behalf Of *Heenan, Timothy Steven *Sent:* Tuesday, August 12, 2014 2:37 PM *To:* &#39;miconda(a)gmail.com <mailto:miconda@gmail.com>'; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak Hi Daniel, Here’s a list of the modules I’m running: mi_fifo db_mysql sl kex tm tmx rr xlog maxfwd usrloc registrar textops pv acc permissions siputils auth lcr dispatcher sanity debugger siptrace We’re setting the memory allocation to 256 ( -m 256 –M 64 ). I’ll see about recompiling with MEMDBG=1. Regards, - Tim *From:*sr-users-bounces@lists.sip-router.org <mailto:sr-users-bounces@lists.sip-router.org> [mailto:sr-users-bounces@lists.sip-router.org] *On Behalf Of *Daniel-Constantin Mierla *Sent:* Tuesday, August 12, 2014 4:17 AM *To:* Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak Hello, can you give the list of loaded modules? How much memory did you allocate (-m parameter value)? It will narrow down searches to see if there was anything similar fixed since 4.0.4. To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed. Cheers, Daniel On 12/08/14 01:40, Heenan, Timothy Steven wrote: Greetings, I’m running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I’m observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs: ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory. What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls. -- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Next Kamailio Advanced Trainings 2014 -http://www.asipto.com Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA ------------------------------------------------------------------------ This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments. -- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Next Kamailio Advanced Trainings 2014 -http://www.asipto.com Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

I’ve tested this patch pretty thoroughly. Afterwards, I’ve pushed it to our production environment and have not seen the problem recur. *This has fixed the problem! * Thanks for the patch Daniel, I would definitely support its inclusion in a backport for Kamailio 4.0.X . Thanks! -Tim *From:*sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] *On Behalf Of *Heenan, Timothy Steven *Sent:* Monday, September 15, 2014 10:50 AM *To:* &#39;miconda(a)gmail.com&#39;com'; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak Thank you! I will test this on my 4.0.4 setup and report back how it works. Thank you, *Tim Heenan* *Engineer I - VoIP**Wholesale | Windstream* timothy.heenan(a)windstream.com <mailto:email@windstream.com>| windstreambusiness.com <http://www.windstreambusiness.com/> o: 847.348.1338 | f: 847.963.0116 *From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Sent:* Thursday, September 11, 2014 3:45 PM *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak I pushed a patch to master branch, hopefully fixing the leak: - http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=7fb8c88… Before backporting, I need to have it tested and be sure is not having side effects. For 4.0.x, if you installed from sources, either try to cherry pick the commit via git or edit the file: modules/dispatcher/dispatch.c and replace existing function destroy_list(...) with: void destroy_list(int list_id) { ds_set_t *sp = NULL; ds_set_t *sp1 = NULL; ds_dest_t *dest = NULL; sp = ds_lists[list_id]; while(sp) { sp1 = sp->next; for(dest = sp->dlist; dest!= NULL; dest=dest->next) { if(dest->uri.s!=NULL) { shm_free(dest->uri.s); dest->uri.s = NULL; } } if (sp->dlist != NULL) shm_free(sp->dlist); shm_free(sp); sp = sp1; } ds_lists[list_id] = NULL; } Let me know the results and if all ok, I will backport. Cheers, Daniel On 11/09/14 18:03, Heenan, Timothy Steven wrote: Thanks for your help Daniel! -Tim *From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Sent:* Thursday, September 11, 2014 9:59 AM *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak Short note to say that last log you sent had more leads, and apparently there is a leak at least in 4.0. I need a bit more time to look at it and check if still on master and 4.1 (hopefully event later today) - I will come back with an update about it soon. Daniel On 08/09/14 18:11, Daniel-Constantin Mierla wrote: Were you taking the memory logs after the errors, before restarting? I didn't get the chance to look over the previous email, I will do it soon. Daniel On 08/09/14 17:48, Heenan, Timothy Steven wrote: After restarting Kamailio on September 3rd, the problem has again gotten to the point of no call processing. Here are the memory statistics; additionally, I can see the virtual memory has ballooned up to 400mb again. shmem:fragments = 2378 shmem:free_size = 209840 shmem:max_used_size = 268393216 shmem:real_used_size = 268225616 shmem:total_size = 268435456 shmem:used_size = 220534792 -Tim *From:*sr-users-bounces@lists.sip-router.org <mailto:sr-users-bounces@lists.sip-router.org> [mailto:sr-users-bounces@lists.sip-router.org] *On Behalf Of *Heenan, Timothy Steven *Sent:* Wednesday, September 03, 2014 3:28 PM *To:* &#39;miconda(a)gmail.com <mailto:miconda@gmail.com>'; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak I have about 460 destination sets. In each set I could have from 1 address up to 6. When I took that initial memory log, Kamailio had been restarted shortly beforehand. I just took another one after the system had decayed so Kamailio was responding to Invites with “SIP/2.0 500 No error (2/SL).” (exhibiting the issue after a few days) and the log is about 400Mb! I’ll attach a small snippet. Seems to be going on and on about the dispatcher module. -Tim *From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com] *Sent:* Thursday, August 28, 2014 5:03 AM *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak Looking at the logs, there are 1336 URIs, which seems ok given your numbers, because the module keeps also the previous set of records (needed because at reload time there can be kamailio worker processes using the records -- perhaps we can improve a bit here, I will review the current reload code a bit, since it was a contribution from another developer quite long time ago). Some more questions: - how many destination sets (groups) do you have? Can you estimate the minimum and maximum addresses in a set? - when did you take the memory log? In other words, for how long was kamailio running? Did you wait enough to notice the steady increase of memory usage? Cheers, Daniel On 28/08/14 01:56, Heenan, Timothy Steven wrote: Thank you for the help. For dispatcher, I’m using a database that contains about 700 records. A reload is performed via cronjob every 5 minutes. The command being used is: kamctl dispatcher reload Thanks, -Tim -- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Next Kamailio Advanced Trainings 2014 -http://www.asipto.com Sep 22-25, Berlin, Germany ------------------------------------------------------------------------ This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments. -- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda Next Kamailio Advanced Trainings 2014 -http://www.asipto.com Sep 22-25, Berlin, Germany