[SR-Users] reload to memory is much slower and has problms after upgrade to 3.3.2 and using mem_join=1

Uri Shacked ushacked at gmail.com
Sun Mar 3 18:36:27 CET 2013 

Hi,

I have this memory issue again.

The status now is that I compiled kamailio as followed here and use
mem_join=1.

I reload my data from DB every minute! I know i reload a lot, will take
care of that (will update you all on April :-)).

When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600
reloads per day) the shmem is about 24%.

The traffic is not the issue.

Still, can defrag of memory improve? Ideas for defrag?

Thanks,

Uri

On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked <ushacked at gmail.com> wrote:

>
> OK
> >>ok - it's 56sec or 36sec?
> It was 56 sec.
>
> Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the
> shmem real used is 25% of 4Gb.
> When mem_join=, the reload take 30 sec and the shmem real used is 17%.
>
>
> Looks good with mem_join=1.
>
> Thanks,
> Uri
>
>
> On Mon, Nov 26, 2012 at 5:43 PM, Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>>  Hello,
>>
>>
>> On 11/22/12 8:51 AM, Uri Shacked wrote:
>>
>>
>> Hi,
>>
>> >> Btw, is the reload time constant now? Even if you run couple of times?
>>
>> yes, the reload time is constant - 56 sec. i tested for 100 reloads in an
>> hour and it was OK.
>>
>>
>> ok - it's 56sec or 36sec?
>>
>>
>>
>> >> What are the values for 'kamctl mi get_statistics shmem:'?
>>
>> i configured kamailio to start with 4Gb and after reload the shmem
>> (real_used) take around 30% of it.
>> but, after 20 reloads it grows in 1%. so, after the 100 reloads the
>> real_used take around 34%-35% of shmem.
>> i made the choise to compile again with f_malloc and not use mem_join.
>> the reloads are faster, it uses less real_size (12% and not 30%) and the
>> increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i
>> will keep track on it and update.
>>
>> thanks a lot.
>>
>> Btw, what do you think? would you use f_malloc with no mem_join or
>> q_malloc with join?
>>
>>
>> Can you try with master branch? I switched to q_malloc with no debug
>> info. That means the overhead should be lower and the joining faster.
>>
>> You can do it also with 3.3, you have to edit Makefile.defs:
>> - be sure MEMDBG=0
>> - replace the next block:
>>
>> ifeq ($(MEMDBG), 1)
>>     C_DEFS+= -DDBG_QM_MALLOC
>>     C_DEFS+= -DMEM_JOIN_FREE
>> else
>>     C_DEFS+= -DF_MALLOC
>>     C_DEFS+= -DMEM_JOIN_FREE
>> endif
>>
>> with:
>>
>> ifeq ($(MEMDBG), 1)
>>     C_DEFS+= -DDBG_QM_MALLOC
>>     C_DEFS+= -DMEM_JOIN_FREE
>> else
>>     C_DEFS+= -DMEM_JOIN_FREE
>> endif
>>
>> Practically is removal of line     C_DEFS+= -DF_MALLOC
>>
>> Then recompile and reinstall..
>>
>> Cheers,
>> Daniel
>>
>>
>> BR,
>> Uri
>>
>>
>>  On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>>  Hello,
>>>
>>>
>>> On 11/21/12 1:33 PM, Uri Shacked wrote:
>>>
>>>  Hi,
>>>
>>>  I recompiled with MEMDBG=1 and installed.
>>> here are the results for reloading 5 million rows with MTREE:
>>>
>>> mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around
>>> 1.2Gb.
>>> mem_join=0 --> takes 10 seconds and the real_used_size of shmem is
>>> around 2Gb
>>>  does it seems normal?
>>> 56 seconds is a lot of time......
>>>
>>> the join is done for the free operation, meaning that most of the time
>>> is spent when freeing the old tree from memory. The new values will be used
>>> after loading the database records, then the old tree is destroyed (this
>>> involves the join operation). Also, the sip routing is not affected,
>>> loading the new records and destroying old memory tree is done in the
>>> MI/RPC process.
>>>
>>> In other words, while the MI/RPC process takes care of loading new data
>>> and destroying the old one, the SIP routing is not affected at all.
>>>
>>> Even when the reload command is executed, the old tree is used until all
>>> the records are loaded in a new tree. At that moment, the pointer to the
>>> active tree is changed from the old tree to the new tree (a very simple
>>> sequence of assignments, very fast). Routing will use the new tree and the
>>> Mi/RPC process starts destroying the old tree.
>>>
>>>
>>>
>>> by the way, when the f_malloc was used, the size of the real_used shmem
>>> was twice smaller.
>>>
>>>
>>>  The overhead when storing small values is significant for q_malloc,
>>> each fragment keeping references (pointers) to file name and line where it
>>> was allocated and freed. In addition it keeps information to get to
>>> previews and next fragment, resulting in faster join.
>>>
>>> It is some space to improve, in order to make less overhead (like a
>>> compile time option not to keep info about file and line of malloc/free). I
>>> will think about what can be done for the next release.
>>>
>>> Btw, is the reload time constant now? Even if you run couple of times?
>>>
>>> What are the values for 'kamctl mi get_statistics shmem:'?
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> Thanks.
>>>  On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla <
>>> miconda at gmail.com> wrote:
>>>
>>>>  Hello,
>>>>
>>>>
>>>> On 11/20/12 7:34 PM, Uri Shacked wrote:
>>>>
>>>>  Hi,
>>>>
>>>> can you be a litle more specific of the steps of the install and where
>>>> do i make the changes?
>>>>
>>>>
>>>>  in the source tree, edit the file Makefile.defs and set:
>>>>
>>>> MEMDBG=1
>>>>
>>>> then run:
>>>>
>>>> make all
>>>> make install
>>>>
>>>>
>>>>  some words of what is the diff between f_malloc and q_malloc will be
>>>> great :-).
>>>>
>>>>
>>>>  q_malloc is more debugging purposes, keeping more information for each
>>>> chunk, therefore the overhead is a bit higher than with f_malloc, but
>>>> because keeps more details, it is faster to find the fragments that can be
>>>> joined.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>>
>>>> thanks,
>>>> Uri
>>>>
>>>>  On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla <
>>>> miconda at gmail.com> wrote:
>>>>
>>>>>  Hello,
>>>>>
>>>>> ok, I will look over it. At this moment the f_malloc (which is enabled
>>>>> for 3.3) has a pretty inefficient mem join implementation, can you try with
>>>>> q_malloc? Edit Makefile.defs and set:
>>>>>
>>>>> MEMDBG=1
>>>>>
>>>>> Then compile and install.
>>>>>
>>>>> The join operation should be faster, let's see if you get blocking
>>>>> issues with this one.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>>> On 11/20/12 2:57 PM, Uri Shacked wrote:
>>>>>
>>>>>  Daniel hi,
>>>>>
>>>>> I attached 2 txt files.
>>>>> One with mem_join=1, the other with mem_join=0, and the info you asked
>>>>> for.
>>>>> Let me know if it is OK.
>>>>>
>>>>> Thanks,
>>>>> Uri
>>>>>
>>>>>  On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla <
>>>>> miconda at gmail.com> wrote:
>>>>>
>>>>>>  Hello,
>>>>>>
>>>>>> if you set memjoin to 0, do you see any difference?
>>>>>>
>>>>>> Can you try again (with memjoin 1 as well as 0) and send the output
>>>>>> of:
>>>>>>
>>>>>> kamctl mi get_statistics shmem:
>>>>>>
>>>>>> before executing the reload commands?
>>>>>>
>>>>>> When it gets to 100%, can you see which process is using the cpu and
>>>>>> attach to it with:
>>>>>>
>>>>>> gdb /path/to/kamailio PID
>>>>>>
>>>>>> then do:
>>>>>>
>>>>>> bt full
>>>>>>
>>>>>> and send output here?
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 11/18/12 4:09 PM, Uri Shacked wrote:
>>>>>>
>>>>>>   After some testing I notice the following:
>>>>>> First reload of 5 million records after kamailio started took about 9
>>>>>> sec.
>>>>>> Second reload (4 minutes after the first one) took 60 sec.
>>>>>> The third one (again about 4 minutes after the secind) got kamailio
>>>>>> to use 100% cpu and after 13 minutes! i killed it.....
>>>>>>
>>>>>> I can understand that the memory manger works harder, still, any
>>>>>> ideas on how to use mem_join and keep on reloading data.
>>>>>> (in real life our data loads 5 million records once a day when almost
>>>>>> no traffic. still after a few days it stops...)
>>>>>>
>>>>>> Thanks,
>>>>>> Uri
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked <ushacked at gmail.com>wrote:
>>>>>>
>>>>>>>   Hi,
>>>>>>>
>>>>>>> I am using MTREE and DIALPLAN modules to load lots of info to
>>>>>>> kamailio. (6 million rows).
>>>>>>>
>>>>>>> When kamailio was running with 3.2.1 (no mem_join=1 option), the
>>>>>>> used size was increasing but the process of loading the data was fast
>>>>>>> eanough.
>>>>>>>
>>>>>>> I upgraded to 3.3.2 and set mem_join=1. Now the loading process take
>>>>>>> about 10 time longer and sometimes stops kamailio from responding to
>>>>>>> traffic.
>>>>>>>
>>>>>>> Any ideas?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Uri
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>  _______________________________________________
>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>
>>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20130303/f7724dc2/attachment-0001.htm>

More information about the sr-users mailing list