[SR-Users] Kamailio 4.0.4 slow memory leak
Daniel-Constantin Mierla
miconda at gmail.com
Mon Sep 8 18:11:20 CEST 2014
Were you taking the memory logs after the errors, before restarting?
I didn't get the chance to look over the previous email, I will do it soon.
Daniel
On 08/09/14 17:48, Heenan, Timothy Steven wrote:
>
> After restarting Kamailio on September 3rd, the problem has again
> gotten to the point of no call processing.
>
> Here are the memory statistics; additionally, I can see the virtual
> memory has ballooned up to 400mb again.
>
> shmem:fragments = 2378
>
> shmem:free_size = 209840
>
> shmem:max_used_size = 268393216
>
> shmem:real_used_size = 268225616
>
> shmem:total_size = 268435456
>
> shmem:used_size = 220534792
>
> -Tim
>
> *From:*sr-users-bounces at lists.sip-router.org
> [mailto:sr-users-bounces at lists.sip-router.org] *On Behalf Of *Heenan,
> Timothy Steven
> *Sent:* Wednesday, September 03, 2014 3:28 PM
> *To:* 'miconda at gmail.com'; Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> I have about 460 destination sets. In each set I could have from 1
> address up to 6.
>
> When I took that initial memory log, Kamailio had been restarted
> shortly beforehand. I just took another one after the system had
> decayed so Kamailio was responding to Invites with “SIP/2.0 500 No
> error (2/SL).” (exhibiting the issue after a few days) and the log is
> about 400Mb! I’ll attach a small snippet.
>
> Seems to be going on and on about the dispatcher module.
>
> -Tim
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* Thursday, August 28, 2014 5:03 AM
> *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> Looking at the logs, there are 1336 URIs, which seems ok given your
> numbers, because the module keeps also the previous set of records
> (needed because at reload time there can be kamailio worker processes
> using the records -- perhaps we can improve a bit here, I will review
> the current reload code a bit, since it was a contribution from
> another developer quite long time ago).
>
> Some more questions:
> - how many destination sets (groups) do you have? Can you estimate the
> minimum and maximum addresses in a set?
> - when did you take the memory log? In other words, for how long was
> kamailio running? Did you wait enough to notice the steady increase of
> memory usage?
>
> Cheers,
> Daniel
>
> On 28/08/14 01:56, Heenan, Timothy Steven wrote:
>
> Thank you for the help.
>
> For dispatcher, I’m using a database that contains about 700 records.
>
> A reload is performed via cronjob every 5 minutes. The command
> being used is:
>
> kamctl dispatcher reload
>
> Thanks,
>
> -Tim
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* Wednesday, August 27, 2014 5:22 PM
> *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> The logs suggest some leaks in dispatcher module -- it allocates
> shared memory only for caching routing records. Few more questions
> I need to get the answers in order to troubleshoot further.
>
> How many records do you have in dispatcher table? Is it in
> database or the text file?
>
> How ofter do you reload? What is the command you use for reload?
>
> Cheers,
> Daniel
>
> On 26/08/14 22:42, Heenan, Timothy Steven wrote:
>
> I have attached is the output of “kamcmd cfg.set_now_int core
> mem_dump_shm 1” to this email. I had to restart Kamailio to
> load ctl and ctl_rpc, so let me know if I need to do another
> one after some time passes and calls are processed.
>
> Also, here is the output of “kamctl mi get_statistics shmem:”
>
> shmem:fragments = 398
>
> shmem:free_size = 259170720
>
> shmem:max_used_size = 9265184
>
> shmem:real_used_size = 9264736
>
> shmem:total_size = 268435456
>
> shmem:used_size = 8559768
>
> Thanks!
>
> -Tim
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* Tuesday, August 26, 2014 10:46 AM
> *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> It is not needed to see the memory operations logs. Just to
> dump the chunks in memory.
>
> You would need ctl and ctl_rpc modules (iirc -- they should be
> in the default config file), then run:
>
> kamcmd cfg.set_now_int core mem_dump_shm 1
>
> Extract the logs of the dump from the shared memory and send
> them over.
>
> Send also the output for:
>
> kamctl mi get_statistics shmem:
>
> Cheers,
> Daniel
>
> On 26/08/14 17:34, Heenan, Timothy Steven wrote:
>
> Thank you.
>
> I have recompiled with MEMDBG, however I am not observing
> any additional output in my log files.
>
> I think I have something set incorrectly in kamailio.cfg.
>
> Here are my log settings from Kamailio.cfg
>
> debug=3
>
> log_stderror=no
>
> log_facility=LOG_LOCAL0
>
> memdbg=2
>
> memlog=2
>
> …
>
> modparam("debugger", "cfgtrace", 1)
>
> modparam("debugger", "log_level", 2)
>
> modparam("debugger", "cfgpkgcheck", 1)
>
> modparam("debugger", "mod_level", "core=2")
>
> modparam("debugger", "mod_level", "tm=2")
>
> modparam("debugger", "mod_level", "dispatcher=2")
>
> modparam("debugger", "mod_level", "siptrace=2")
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* Tuesday, August 26, 2014 3:54 AM
> *To:* Heenan, Timothy Steven; Kamailio (SER) - Users
> Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> Hello,
>
> I quickly looked over the modules that use shared memory
> and I couldn't spot a commit related to any leak.
>
> Have you had the chance to compile with MEMDBG?
>
> Cheers,
> Daniel
>
> On 19/08/14 19:14, Heenan, Timothy Steven wrote:
>
> Do you think any of these modules possibly causing
> this issue?
>
> -Tim
>
> *From:*sr-users-bounces at lists.sip-router.org
> <mailto:sr-users-bounces at lists.sip-router.org>
> [mailto:sr-users-bounces at lists.sip-router.org] *On
> Behalf Of *Heenan, Timothy Steven
> *Sent:* Tuesday, August 12, 2014 2:37 PM
> *To:* 'miconda at gmail.com <mailto:miconda at gmail.com>';
> Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> Hi Daniel,
>
> Here’s a list of the modules I’m running:
>
> mi_fifo
>
> db_mysql
>
> sl
>
> kex
>
> tm
>
> tmx
>
> rr
>
> xlog
>
> maxfwd
>
> usrloc
>
> registrar
>
> textops
>
> pv
>
> acc
>
> permissions
>
> siputils
>
> auth
>
> lcr
>
> dispatcher
>
> sanity
>
> debugger
>
> siptrace
>
> We’re setting the memory allocation to 256 ( -m 256 –M
> 64 ).
>
> I’ll see about recompiling with MEMDBG=1.
>
> Regards,
>
> - Tim
>
> *From:*sr-users-bounces at lists.sip-router.org
> <mailto:sr-users-bounces at lists.sip-router.org>
> [mailto:sr-users-bounces at lists.sip-router.org] *On
> Behalf Of *Daniel-Constantin Mierla
> *Sent:* Tuesday, August 12, 2014 4:17 AM
> *To:* Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> Hello,
>
> can you give the list of loaded modules? How much
> memory did you allocate (-m parameter value)?
>
> It will narrow down searches to see if there was
> anything similar fixed since 4.0.4.
>
> To troubleshoot easier, would be good to recompile
> with MEMDBG=1, then the details of chunks in memory
> can be dumped and analysed.
>
> Cheers,
> Daniel
>
> On 12/08/14 01:40, Heenan, Timothy Steven wrote:
>
> Greetings,
>
> I’m running into a slow memory leak on my kamailio
> 4.0.4 SIP proxies. I’m observing a steady increase
> in the memory consumption until there is no more
> left. Kamailio then starts repeating this in the logs:
>
> ERROR: dispatcher [dispatch.c:279]:
> add_dest2list(): no more memory.
>
> What would be the best way to debug this kind of a
> memory leak? The proxy does not handle any
> registrations but does route a fair amount of calls.
>
>
>
> --
>
> Daniel-Constantin Mierla
>
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>
> Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
>
> Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
>
> ------------------------------------------------------------------------
>
> This email message and any attachments are for the sole use of the
> intended recipient(s). Any unauthorized review, use, disclosure or
> distribution is prohibited. If you are not the intended recipient,
> please contact the sender by reply email and destroy all copies of
> the original message and any attachments.
>
> --
> Daniel-Constantin Mierla
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
> Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
> Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140908/ed056707/attachment.html>
More information about the sr-users
mailing list