[SR-Users] Kamailio 4.0.4 slow memory leak
Daniel-Constantin Mierla
miconda at gmail.com
Thu Aug 28 00:21:48 CEST 2014
The logs suggest some leaks in dispatcher module -- it allocates shared
memory only for caching routing records. Few more questions I need to
get the answers in order to troubleshoot further.
How many records do you have in dispatcher table? Is it in database or
the text file?
How ofter do you reload? What is the command you use for reload?
Cheers,
Daniel
On 26/08/14 22:42, Heenan, Timothy Steven wrote:
>
> I have attached is the output of “kamcmd cfg.set_now_int core
> mem_dump_shm 1” to this email. I had to restart Kamailio to load ctl
> and ctl_rpc, so let me know if I need to do another one after some
> time passes and calls are processed.
>
> Also, here is the output of “kamctl mi get_statistics shmem:”
>
> shmem:fragments = 398
>
> shmem:free_size = 259170720
>
> shmem:max_used_size = 9265184
>
> shmem:real_used_size = 9264736
>
> shmem:total_size = 268435456
>
> shmem:used_size = 8559768
>
> Thanks!
>
> -Tim
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* Tuesday, August 26, 2014 10:46 AM
> *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> It is not needed to see the memory operations logs. Just to dump the
> chunks in memory.
>
> You would need ctl and ctl_rpc modules (iirc -- they should be in the
> default config file), then run:
>
> kamcmd cfg.set_now_int core mem_dump_shm 1
>
> Extract the logs of the dump from the shared memory and send them over.
>
> Send also the output for:
>
> kamctl mi get_statistics shmem:
>
> Cheers,
> Daniel
>
> On 26/08/14 17:34, Heenan, Timothy Steven wrote:
>
> Thank you.
>
> I have recompiled with MEMDBG, however I am not observing any
> additional output in my log files.
>
> I think I have something set incorrectly in kamailio.cfg.
>
> Here are my log settings from Kamailio.cfg
>
> debug=3
>
> log_stderror=no
>
> log_facility=LOG_LOCAL0
>
> memdbg=2
>
> memlog=2
>
> …
>
> modparam("debugger", "cfgtrace", 1)
>
> modparam("debugger", "log_level", 2)
>
> modparam("debugger", "cfgpkgcheck", 1)
>
> modparam("debugger", "mod_level", "core=2")
>
> modparam("debugger", "mod_level", "tm=2")
>
> modparam("debugger", "mod_level", "dispatcher=2")
>
> modparam("debugger", "mod_level", "siptrace=2")
>
> *From:*Daniel-Constantin Mierla [mailto:miconda at gmail.com]
> *Sent:* Tuesday, August 26, 2014 3:54 AM
> *To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> Hello,
>
> I quickly looked over the modules that use shared memory and I
> couldn't spot a commit related to any leak.
>
> Have you had the chance to compile with MEMDBG?
>
> Cheers,
> Daniel
>
> On 19/08/14 19:14, Heenan, Timothy Steven wrote:
>
> Do you think any of these modules possibly causing this issue?
>
> -Tim
>
> *From:*sr-users-bounces at lists.sip-router.org
> <mailto:sr-users-bounces at lists.sip-router.org>
> [mailto:sr-users-bounces at lists.sip-router.org] *On Behalf Of
> *Heenan, Timothy Steven
> *Sent:* Tuesday, August 12, 2014 2:37 PM
> *To:* 'miconda at gmail.com <mailto:miconda at gmail.com>'; Kamailio
> (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> Hi Daniel,
>
> Here’s a list of the modules I’m running:
>
> mi_fifo
>
> db_mysql
>
> sl
>
> kex
>
> tm
>
> tmx
>
> rr
>
> xlog
>
> maxfwd
>
> usrloc
>
> registrar
>
> textops
>
> pv
>
> acc
>
> permissions
>
> siputils
>
> auth
>
> lcr
>
> dispatcher
>
> sanity
>
> debugger
>
> siptrace
>
> We’re setting the memory allocation to 256 ( -m 256 –M 64 ).
>
> I’ll see about recompiling with MEMDBG=1.
>
> Regards,
>
> - Tim
>
> *From:*sr-users-bounces at lists.sip-router.org
> <mailto:sr-users-bounces at lists.sip-router.org>
> [mailto:sr-users-bounces at lists.sip-router.org] *On Behalf Of
> *Daniel-Constantin Mierla
> *Sent:* Tuesday, August 12, 2014 4:17 AM
> *To:* Kamailio (SER) - Users Mailing List
> *Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
>
> Hello,
>
> can you give the list of loaded modules? How much memory did
> you allocate (-m parameter value)?
>
> It will narrow down searches to see if there was anything
> similar fixed since 4.0.4.
>
> To troubleshoot easier, would be good to recompile with
> MEMDBG=1, then the details of chunks in memory can be dumped
> and analysed.
>
> Cheers,
> Daniel
>
> On 12/08/14 01:40, Heenan, Timothy Steven wrote:
>
> Greetings,
>
> I’m running into a slow memory leak on my kamailio 4.0.4
> SIP proxies. I’m observing a steady increase in the memory
> consumption until there is no more left. Kamailio then
> starts repeating this in the logs:
>
> ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no
> more memory.
>
> What would be the best way to debug this kind of a memory
> leak? The proxy does not handle any registrations but does
> route a fair amount of calls.
>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140828/5d5112dc/attachment.html>
More information about the sr-users
mailing list