[SR-Users] reload to memory is much slower and has problms after upgrade to 3.3.2 and using mem_join=1

Uri Shacked ushacked at gmail.com
Thu Nov 22 08:51:13 CET 2012


Hi,

>> Btw, is the reload time constant now? Even if you run couple of times?

yes, the reload time is constant - 56 sec. i tested for 100 reloads in an
hour and it was OK.

>> What are the values for 'kamctl mi get_statistics shmem:'?

i configured kamailio to start with 4Gb and after reload the shmem
(real_used) take around 30% of it.
but, after 20 reloads it grows in 1%. so, after the 100 reloads the
real_used take around 34%-35% of shmem.
i made the choise to compile again with f_malloc and not use mem_join. the
reloads are faster, it uses less real_size (12% and not 30%) and the
increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i
will keep track on it and update.

thanks a lot.

Btw, what do you think? would you use f_malloc with no mem_join or q_malloc
with join?

BR,
Uri


On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla <miconda at gmail.com
> wrote:

>  Hello,
>
>
> On 11/21/12 1:33 PM, Uri Shacked wrote:
>
>  Hi,
>
> I recompiled with MEMDBG=1 and installed.
> here are the results for reloading 5 million rows with MTREE:
>
> mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around
> 1.2Gb.
> mem_join=0 --> takes 10 seconds and the real_used_size of shmem is around
> 2Gb
>  does it seems normal?
> 56 seconds is a lot of time......
>
> the join is done for the free operation, meaning that most of the time is
> spent when freeing the old tree from memory. The new values will be used
> after loading the database records, then the old tree is destroyed (this
> involves the join operation). Also, the sip routing is not affected,
> loading the new records and destroying old memory tree is done in the
> MI/RPC process.
>
> In other words, while the MI/RPC process takes care of loading new data
> and destroying the old one, the SIP routing is not affected at all.
>
> Even when the reload command is executed, the old tree is used until all
> the records are loaded in a new tree. At that moment, the pointer to the
> active tree is changed from the old tree to the new tree (a very simple
> sequence of assignments, very fast). Routing will use the new tree and the
> Mi/RPC process starts destroying the old tree.
>
>
>
> by the way, when the f_malloc was used, the size of the real_used shmem
> was twice smaller.
>
>
> The overhead when storing small values is significant for q_malloc, each
> fragment keeping references (pointers) to file name and line where it was
> allocated and freed. In addition it keeps information to get to previews
> and next fragment, resulting in faster join.
>
> It is some space to improve, in order to make less overhead (like a
> compile time option not to keep info about file and line of malloc/free). I
> will think about what can be done for the next release.
>
> Btw, is the reload time constant now? Even if you run couple of times?
>
> What are the values for 'kamctl mi get_statistics shmem:'?
>
> Cheers,
> Daniel
>
>
> Thanks.
>  On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>>  Hello,
>>
>>
>> On 11/20/12 7:34 PM, Uri Shacked wrote:
>>
>>  Hi,
>>
>> can you be a litle more specific of the steps of the install and where do
>> i make the changes?
>>
>>
>>  in the source tree, edit the file Makefile.defs and set:
>>
>> MEMDBG=1
>>
>> then run:
>>
>> make all
>> make install
>>
>>
>>  some words of what is the diff between f_malloc and q_malloc will be
>> great :-).
>>
>>
>>  q_malloc is more debugging purposes, keeping more information for each
>> chunk, therefore the overhead is a bit higher than with f_malloc, but
>> because keeps more details, it is faster to find the fragments that can be
>> joined.
>>
>> Cheers,
>> Daniel
>>
>>
>>
>> thanks,
>> Uri
>>
>>  On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>>  Hello,
>>>
>>> ok, I will look over it. At this moment the f_malloc (which is enabled
>>> for 3.3) has a pretty inefficient mem join implementation, can you try with
>>> q_malloc? Edit Makefile.defs and set:
>>>
>>> MEMDBG=1
>>>
>>> Then compile and install.
>>>
>>> The join operation should be faster, let's see if you get blocking
>>> issues with this one.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 11/20/12 2:57 PM, Uri Shacked wrote:
>>>
>>>  Daniel hi,
>>>
>>> I attached 2 txt files.
>>> One with mem_join=1, the other with mem_join=0, and the info you asked
>>> for.
>>> Let me know if it is OK.
>>>
>>> Thanks,
>>> Uri
>>>
>>>  On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla <
>>> miconda at gmail.com> wrote:
>>>
>>>>  Hello,
>>>>
>>>> if you set memjoin to 0, do you see any difference?
>>>>
>>>> Can you try again (with memjoin 1 as well as 0) and send the output of:
>>>>
>>>> kamctl mi get_statistics shmem:
>>>>
>>>> before executing the reload commands?
>>>>
>>>> When it gets to 100%, can you see which process is using the cpu and
>>>> attach to it with:
>>>>
>>>> gdb /path/to/kamailio PID
>>>>
>>>> then do:
>>>>
>>>> bt full
>>>>
>>>> and send output here?
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 11/18/12 4:09 PM, Uri Shacked wrote:
>>>>
>>>>   After some testing I notice the following:
>>>> First reload of 5 million records after kamailio started took about 9
>>>> sec.
>>>> Second reload (4 minutes after the first one) took 60 sec.
>>>> The third one (again about 4 minutes after the secind) got kamailio to
>>>> use 100% cpu and after 13 minutes! i killed it.....
>>>>
>>>> I can understand that the memory manger works harder, still, any ideas
>>>> on how to use mem_join and keep on reloading data.
>>>> (in real life our data loads 5 million records once a day when almost
>>>> no traffic. still after a few days it stops...)
>>>>
>>>> Thanks,
>>>> Uri
>>>>
>>>>
>>>>
>>>> On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked <ushacked at gmail.com>wrote:
>>>>
>>>>>   Hi,
>>>>>
>>>>> I am using MTREE and DIALPLAN modules to load lots of info to
>>>>> kamailio. (6 million rows).
>>>>>
>>>>> When kamailio was running with 3.2.1 (no mem_join=1 option), the used
>>>>> size was increasing but the process of loading the data was fast eanough.
>>>>>
>>>>> I upgraded to 3.3.2 and set mem_join=1. Now the loading process take
>>>>> about 10 time longer and sometimes stops kamailio from responding to
>>>>> traffic.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Uri
>>>>>
>>>>
>>>>
>>>>
>>>>  _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>
>>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20121122/393586fb/attachment-0001.htm>


More information about the sr-users mailing list