[SR-Users] Kamailio 4.0.4 slow memory leak

Mon Sep 15 17:49:55 CEST 2014

Thank you!

I will test this on my 4.0.4 setup and report back how it works.

Thank you,

Tim Heenan
Engineer I - VoIP Wholesale | Windstream
timothy.heenan at windstream.com<mailto:email at windstream.com> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116

From: Daniel-Constantin Mierla [mailto:miconda at gmail.com]
Sent: Thursday, September 11, 2014 3:45 PM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

I pushed a patch to master branch, hopefully fixing the leak:

- http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=7fb8c88c1d4aeb50d1e637697132ab0994dcdb28

Before backporting, I need to have it tested and be sure is not having side effects.

For 4.0.x, if you installed from sources, either try to cherry pick the commit via git or edit the file:

modules/dispatcher/dispatch.c

and replace existing function destroy_list(...) with:

void destroy_list(int list_id)
{
    ds_set_t  *sp = NULL;
    ds_set_t  *sp1 = NULL;
    ds_dest_t *dest = NULL;

    sp = ds_lists[list_id];

    while(sp)
    {
        sp1 = sp->next;
        for(dest = sp->dlist; dest!= NULL; dest=dest->next)
        {
            if(dest->uri.s!=NULL)
            {
                shm_free(dest->uri.s);
                dest->uri.s = NULL;
            }
        }
        if (sp->dlist != NULL)
            shm_free(sp->dlist);
        shm_free(sp);
        sp = sp1;
    }

    ds_lists[list_id]  = NULL;
}

Let me know the results and if all ok, I will backport.

Cheers,
Daniel
On 11/09/14 18:03, Heenan, Timothy Steven wrote:
Thanks for your help Daniel!

-Tim

From: Daniel-Constantin Mierla [mailto:miconda at gmail.com]
Sent: Thursday, September 11, 2014 9:59 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Short note to say that last log you sent had more leads, and apparently there is a leak at least in 4.0.

I need a bit more time to look at it and check if still on master and 4.1 (hopefully event later today) - I will come back with an update about it soon.

Daniel
On 08/09/14 18:11, Daniel-Constantin Mierla wrote:
Were you taking the memory logs after the errors, before restarting?

I didn't get the chance to look over the previous email, I will do it soon.

Daniel
On 08/09/14 17:48, Heenan, Timothy Steven wrote:
After restarting Kamailio on September 3rd, the problem has again gotten to the point of no call processing.

Here are the memory statistics; additionally, I can see the virtual memory has ballooned up to 400mb again.

shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792

-Tim

From: sr-users-bounces at lists.sip-router.org<mailto:sr-users-bounces at lists.sip-router.org> [mailto:sr-users-bounces at lists.sip-router.org] On Behalf Of Heenan, Timothy Steven
Sent: Wednesday, September 03, 2014 3:28 PM
To: 'miconda at gmail.com<mailto:miconda at gmail.com>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

I have about 460 destination sets. In each set I could have from 1 address up to 6.

When I took that initial memory log, Kamailio had been restarted shortly beforehand. I just took another one after the system had decayed so Kamailio was responding to Invites with "SIP/2.0 500 No error (2/SL)." (exhibiting the issue after a few days) and the log is about 400Mb! I'll attach a small snippet.
Seems to be going on and on about the dispatcher module.

-Tim

From: Daniel-Constantin Mierla [mailto:miconda at gmail.com]
Sent: Thursday, August 28, 2014 5:03 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Looking at the logs, there are 1336 URIs, which seems ok given your numbers, because the module keeps also the previous set of records (needed because at reload time there can be kamailio worker processes using the records -- perhaps we can improve a bit here, I will review the current reload code a bit, since it was a contribution from another developer quite long time ago).

Some more questions:
- how many destination sets (groups) do you have? Can you estimate the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was kamailio running? Did you wait enough to notice the steady increase of memory usage?

Cheers,
Daniel
On 28/08/14 01:56, Heenan, Timothy Steven wrote:
Thank you for the help.

For dispatcher, I'm using a database that contains about 700 records.

A reload is performed via cronjob every 5 minutes. The command being used is:

kamctl dispatcher reload

Thanks,
-Tim

--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany

________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.

--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140915/be11f251/attachment.html>