[SR-Users] Kamailio 4.0.4 slow memory leak

Daniel-Constantin Mierla miconda at gmail.com
Mon Sep 8 18:11:20 CEST 2014


Were you taking the memory logs after the errors, before restarting?

I didn't get the chance to look over the previous email, I will do it soon.

Daniel

On 08/09/14 17:48, Heenan, Timothy Steven wrote:
>
> After restarting Kamailio on September 3rd, the problem has again 
> gotten to the point of no call processing.
>
> Here are the memory statistics; additionally, I can see the virtual 
> memory has ballooned up to 400mb again.
>
> shmem:fragments = 2378
>
> shmem:free_size = 209840
>
> shmem:max_used_size = 268393216
>
> shmem:real_used_size = 268225616
>
> shmem:total_size = 268435456
>
> shmem:used_size = 220534792
>
> -Tim
>
> *From:*sr-users-bounces at lists.sip-router.org 
> [mailto:sr-users-bounces at lists.sip-router.org] *On Behalf Of *Heenan, 
> Timothy Steven
> *Sent:* Wednesday, September 03, 2014 3:28 PM
> *To:* 'miconda at gmail.com'; Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> I have about 460 destination sets. In each set I could have from 1 
> address up to 6.
>
> When I took that initial memory log, Kamailio had been restarted 
> shortly beforehand. I just took another one after the system had 
> decayed so Kamailio was responding to Invites with “SIP/2.0 500 No 
> error (2/SL).” (exhibiting the issue after a few days) and the log is 
> about 400Mb! I’ll attach a small snippet.
>
> Seems to be going on and on about the dispatcher module.
>
> -Tim
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* Thursday, August 28, 2014 5:03 AM
> *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> Looking at the logs, there are 1336 URIs, which seems ok given your 
> numbers, because the module keeps also the previous set of records 
> (needed because at reload time there can be kamailio worker processes 
> using the records -- perhaps we can improve a bit here, I will review 
> the current reload code a bit, since it was a contribution from 
> another developer quite long time ago).
>
> Some more questions:
> - how many destination sets (groups) do you have? Can you estimate the 
> minimum and maximum addresses in a set?
> - when did you take the memory log? In other words, for how long was 
> kamailio running? Did you wait enough to notice the steady increase of 
> memory usage?
>
> Cheers,
> Daniel
>
> On 28/08/14 01:56, Heenan, Timothy Steven wrote:
>
>     Thank you for the help.
>
>     For dispatcher, I’m using a database that contains about 700 records.
>
>     A reload is performed via cronjob every 5 minutes. The command
>     being used is:
>
>     kamctl dispatcher reload
>
>     Thanks,
>
>     -Tim
>
>     *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
>     *Sent:* Wednesday, August 27, 2014 5:22 PM
>     *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
>     *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
>     The logs suggest some leaks in dispatcher module -- it allocates
>     shared memory only for caching routing records. Few more questions
>     I need to get the answers in order to troubleshoot further.
>
>     How many records do you have in dispatcher table? Is it in
>     database or the text file?
>
>     How ofter do you reload? What is the command you use for reload?
>
>     Cheers,
>     Daniel
>
>     On 26/08/14 22:42, Heenan, Timothy Steven wrote:
>
>         I have attached is the output of “kamcmd cfg.set_now_int core
>         mem_dump_shm 1” to this email.  I had to restart Kamailio to
>         load ctl and ctl_rpc, so let me know if I need to do another
>         one after some time passes and calls are processed.
>
>         Also, here is the output of “kamctl mi get_statistics shmem:”
>
>         shmem:fragments = 398
>
>         shmem:free_size = 259170720
>
>         shmem:max_used_size = 9265184
>
>         shmem:real_used_size = 9264736
>
>         shmem:total_size = 268435456
>
>         shmem:used_size = 8559768
>
>         Thanks!
>
>         -Tim
>
>         *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
>         *Sent:* Tuesday, August 26, 2014 10:46 AM
>         *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
>         *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
>         It is not needed to see the memory operations logs. Just to
>         dump the chunks in memory.
>
>         You would need ctl and ctl_rpc modules (iirc -- they should be
>         in the default config file), then run:
>
>         kamcmd cfg.set_now_int core mem_dump_shm 1
>
>         Extract the logs of the dump from the shared memory and send
>         them over.
>
>         Send also the output for:
>
>         kamctl mi get_statistics shmem:
>
>         Cheers,
>         Daniel
>
>         On 26/08/14 17:34, Heenan, Timothy Steven wrote:
>
>             Thank you.
>
>             I have recompiled with MEMDBG, however I am not observing
>             any additional output in my log files.
>
>             I think I have something set incorrectly in kamailio.cfg.
>
>             Here are my log settings from Kamailio.cfg
>
>             debug=3
>
>             log_stderror=no
>
>             log_facility=LOG_LOCAL0
>
>             memdbg=2
>
>             memlog=2
>
>>
>             modparam("debugger", "cfgtrace", 1)
>
>             modparam("debugger", "log_level", 2)
>
>             modparam("debugger", "cfgpkgcheck", 1)
>
>             modparam("debugger", "mod_level", "core=2")
>
>             modparam("debugger", "mod_level", "tm=2")
>
>             modparam("debugger", "mod_level", "dispatcher=2")
>
>             modparam("debugger", "mod_level", "siptrace=2")
>
>             *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
>             *Sent:* Tuesday, August 26, 2014 3:54 AM
>             *To:* Heenan, Timothy Steven; Kamailio (SER) - Users
>             Mailing List
>             *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
>             Hello,
>
>             I quickly looked over the modules that use shared memory
>             and I couldn't spot a commit related to any leak.
>
>             Have you had the chance to compile with  MEMDBG?
>
>             Cheers,
>             Daniel
>
>             On 19/08/14 19:14, Heenan, Timothy Steven wrote:
>
>                 Do you think any of these modules possibly causing
>                 this issue?
>
>                 -Tim
>
>                 *From:*sr-users-bounces at lists.sip-router.org
>                 <mailto:sr-users-bounces at lists.sip-router.org>
>                 [mailto:sr-users-bounces at lists.sip-router.org] *On
>                 Behalf Of *Heenan, Timothy Steven
>                 *Sent:* Tuesday, August 12, 2014 2:37 PM
>                 *To:* 'miconda at gmail.com <mailto:miconda at gmail.com>';
>                 Kamailio (SER) - Users Mailing List
>                 *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
>                 Hi Daniel,
>
>                 Here’s a list of the modules I’m running:
>
>                 mi_fifo
>
>                 db_mysql
>
>                 sl
>
>                 kex
>
>                 tm
>
>                 tmx
>
>                 rr
>
>                 xlog
>
>                 maxfwd
>
>                 usrloc
>
>                 registrar
>
>                 textops
>
>                 pv
>
>                 acc
>
>                 permissions
>
>                 siputils
>
>                 auth
>
>                 lcr
>
>                 dispatcher
>
>                 sanity
>
>                 debugger
>
>                 siptrace
>
>                 We’re setting the memory allocation to 256 ( -m 256 –M
>                 64 ).
>
>                 I’ll see about recompiling with MEMDBG=1.
>
>                 Regards,
>
>                 - Tim
>
>                 *From:*sr-users-bounces at lists.sip-router.org
>                 <mailto:sr-users-bounces at lists.sip-router.org>
>                 [mailto:sr-users-bounces at lists.sip-router.org] *On
>                 Behalf Of *Daniel-Constantin Mierla
>                 *Sent:* Tuesday, August 12, 2014 4:17 AM
>                 *To:* Kamailio (SER) - Users Mailing List
>                 *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
>                 Hello,
>
>                 can you give the list of loaded modules? How much
>                 memory did you allocate (-m parameter value)?
>
>                 It will narrow down searches to see if there was
>                 anything similar fixed since 4.0.4.
>
>                 To troubleshoot easier, would be good to recompile
>                 with MEMDBG=1, then the details of chunks in memory
>                 can be dumped and analysed.
>
>                 Cheers,
>                 Daniel
>
>                 On 12/08/14 01:40, Heenan, Timothy Steven wrote:
>
>                     Greetings,
>
>                     I’m running into a slow memory leak on my kamailio
>                     4.0.4 SIP proxies. I’m observing a steady increase
>                     in the memory consumption until there is no more
>                     left. Kamailio then starts repeating this in the logs:
>
>                     ERROR: dispatcher [dispatch.c:279]:
>                     add_dest2list(): no more memory.
>
>                     What would be the best way to debug this kind of a
>                     memory leak? The proxy does not handle any
>                     registrations but does route a fair amount of calls.
>
>
>
>     -- 
>
>     Daniel-Constantin Mierla
>
>     http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
>
>     Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
>
>     Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
>
>     ------------------------------------------------------------------------
>
>     This email message and any attachments are for the sole use of the
>     intended recipient(s). Any unauthorized review, use, disclosure or
>     distribution is prohibited. If you are not the intended recipient,
>     please contact the sender by reply email and destroy all copies of
>     the original message and any attachments.
>
> -- 
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda  <http://twitter.com/#%21/miconda>  -http://www.linkedin.com/in/miconda
> Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
> Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140908/ed056707/attachment.html>


More information about the sr-users mailing list