[SR-Users] reload to memory is much slower and has problms after upgrade to 3.3.2 and using mem_join=1

Uri Shacked ushacked at gmail.com
Sun Nov 24 10:01:11 CET 2013


Hi,

It has been a while.

Still, I have some problems with reloading data into shared memory.

I can obviously notice that there is always an increase of shmem after each
reload.



When my system starts, it has 4G of memory reserved and approximately 0.8G
is populated at start.



After 30 days, when reloading 15-25 times per day only part of the 0.8G,
the shmem size is approximately 3.5G.



As I mentioned before, the traffic does not seems to be the issue. How do
I, or can I, check the size of memory occupied by traffic only?



Any ideas how to handle the reloads?



I am using kamailio 3.3.2 with modules like ACC, DIALOG, CARRIERROUTE and
more….





Thanks,

BR,

Uri


On Sun, Mar 3, 2013 at 7:36 PM, Uri Shacked <ushacked at gmail.com> wrote:

>
> Hi,
>
> I have this memory issue again.
>
> The status now is that I compiled kamailio as followed here and use
> mem_join=1.
>
> I reload my data from DB every minute! I know i reload a lot, will take
> care of that (will update you all on April :-)).
>
> When kamailio starts the shmem is about 17% of 4Gb and after 7 days (600
> reloads per day) the shmem is about 24%.
>
> The traffic is not the issue.
>
> Still, can defrag of memory improve? Ideas for defrag?
>
> Thanks,
>
> Uri
>
> On Tue, Nov 27, 2012 at 12:37 PM, Uri Shacked <ushacked at gmail.com> wrote:
>
>>
>> OK
>> >>ok - it's 56sec or 36sec?
>> It was 56 sec.
>>
>> Now, after recompile, when cfg mem_join=0 the reload takes 8 sec and the
>> shmem real used is 25% of 4Gb.
>> When mem_join=, the reload take 30 sec and the shmem real used is 17%.
>>
>>
>> Looks good with mem_join=1.
>>
>> Thanks,
>> Uri
>>
>>
>>  On Mon, Nov 26, 2012 at 5:43 PM, Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>>  Hello,
>>>
>>>
>>> On 11/22/12 8:51 AM, Uri Shacked wrote:
>>>
>>>
>>> Hi,
>>>
>>> >> Btw, is the reload time constant now? Even if you run couple of times?
>>>
>>> yes, the reload time is constant - 56 sec. i tested for 100 reloads in
>>> an hour and it was OK.
>>>
>>>
>>> ok - it's 56sec or 36sec?
>>>
>>>
>>>
>>> >> What are the values for 'kamctl mi get_statistics shmem:'?
>>>
>>> i configured kamailio to start with 4Gb and after reload the shmem
>>> (real_used) take around 30% of it.
>>> but, after 20 reloads it grows in 1%. so, after the 100 reloads the
>>> real_used take around 34%-35% of shmem.
>>> i made the choise to compile again with f_malloc and not use mem_join.
>>> the reloads are faster, it uses less real_size (12% and not 30%) and the
>>> increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i
>>> will keep track on it and update.
>>>
>>> thanks a lot.
>>>
>>> Btw, what do you think? would you use f_malloc with no mem_join or
>>> q_malloc with join?
>>>
>>>
>>> Can you try with master branch? I switched to q_malloc with no debug
>>> info. That means the overhead should be lower and the joining faster.
>>>
>>> You can do it also with 3.3, you have to edit Makefile.defs:
>>> - be sure MEMDBG=0
>>> - replace the next block:
>>>
>>> ifeq ($(MEMDBG), 1)
>>>     C_DEFS+= -DDBG_QM_MALLOC
>>>     C_DEFS+= -DMEM_JOIN_FREE
>>> else
>>>     C_DEFS+= -DF_MALLOC
>>>     C_DEFS+= -DMEM_JOIN_FREE
>>> endif
>>>
>>> with:
>>>
>>> ifeq ($(MEMDBG), 1)
>>>     C_DEFS+= -DDBG_QM_MALLOC
>>>     C_DEFS+= -DMEM_JOIN_FREE
>>> else
>>>     C_DEFS+= -DMEM_JOIN_FREE
>>> endif
>>>
>>> Practically is removal of line     C_DEFS+= -DF_MALLOC
>>>
>>> Then recompile and reinstall..
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> BR,
>>> Uri
>>>
>>>
>>>  On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla <
>>> miconda at gmail.com> wrote:
>>>
>>>>  Hello,
>>>>
>>>>
>>>> On 11/21/12 1:33 PM, Uri Shacked wrote:
>>>>
>>>>  Hi,
>>>>
>>>>  I recompiled with MEMDBG=1 and installed.
>>>> here are the results for reloading 5 million rows with MTREE:
>>>>
>>>> mem_join=1 -->takes 56 seconds and the real_used_size of shmem is
>>>> around 1.2Gb.
>>>> mem_join=0 --> takes 10 seconds and the real_used_size of shmem is
>>>> around 2Gb
>>>>  does it seems normal?
>>>> 56 seconds is a lot of time......
>>>>
>>>> the join is done for the free operation, meaning that most of the time
>>>> is spent when freeing the old tree from memory. The new values will be used
>>>> after loading the database records, then the old tree is destroyed (this
>>>> involves the join operation). Also, the sip routing is not affected,
>>>> loading the new records and destroying old memory tree is done in the
>>>> MI/RPC process.
>>>>
>>>> In other words, while the MI/RPC process takes care of loading new data
>>>> and destroying the old one, the SIP routing is not affected at all.
>>>>
>>>> Even when the reload command is executed, the old tree is used until
>>>> all the records are loaded in a new tree. At that moment, the pointer to
>>>> the active tree is changed from the old tree to the new tree (a very simple
>>>> sequence of assignments, very fast). Routing will use the new tree and the
>>>> Mi/RPC process starts destroying the old tree.
>>>>
>>>>
>>>>
>>>> by the way, when the f_malloc was used, the size of the real_used shmem
>>>> was twice smaller.
>>>>
>>>>
>>>>  The overhead when storing small values is significant for q_malloc,
>>>> each fragment keeping references (pointers) to file name and line where it
>>>> was allocated and freed. In addition it keeps information to get to
>>>> previews and next fragment, resulting in faster join.
>>>>
>>>> It is some space to improve, in order to make less overhead (like a
>>>> compile time option not to keep info about file and line of malloc/free). I
>>>> will think about what can be done for the next release.
>>>>
>>>> Btw, is the reload time constant now? Even if you run couple of times?
>>>>
>>>> What are the values for 'kamctl mi get_statistics shmem:'?
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> Thanks.
>>>>  On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla <
>>>> miconda at gmail.com> wrote:
>>>>
>>>>>  Hello,
>>>>>
>>>>>
>>>>> On 11/20/12 7:34 PM, Uri Shacked wrote:
>>>>>
>>>>>  Hi,
>>>>>
>>>>> can you be a litle more specific of the steps of the install and where
>>>>> do i make the changes?
>>>>>
>>>>>
>>>>>  in the source tree, edit the file Makefile.defs and set:
>>>>>
>>>>> MEMDBG=1
>>>>>
>>>>> then run:
>>>>>
>>>>> make all
>>>>> make install
>>>>>
>>>>>
>>>>>  some words of what is the diff between f_malloc and q_malloc will be
>>>>> great :-).
>>>>>
>>>>>
>>>>>  q_malloc is more debugging purposes, keeping more information for
>>>>> each chunk, therefore the overhead is a bit higher than with f_malloc, but
>>>>> because keeps more details, it is faster to find the fragments that can be
>>>>> joined.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>>>
>>>>> thanks,
>>>>> Uri
>>>>>
>>>>>  On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla <
>>>>> miconda at gmail.com> wrote:
>>>>>
>>>>>>  Hello,
>>>>>>
>>>>>> ok, I will look over it. At this moment the f_malloc (which is
>>>>>> enabled for 3.3) has a pretty inefficient mem join implementation, can you
>>>>>> try with q_malloc? Edit Makefile.defs and set:
>>>>>>
>>>>>> MEMDBG=1
>>>>>>
>>>>>> Then compile and install.
>>>>>>
>>>>>> The join operation should be faster, let's see if you get blocking
>>>>>> issues with this one.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>>
>>>>>> On 11/20/12 2:57 PM, Uri Shacked wrote:
>>>>>>
>>>>>>  Daniel hi,
>>>>>>
>>>>>> I attached 2 txt files.
>>>>>> One with mem_join=1, the other with mem_join=0, and the info you
>>>>>> asked for.
>>>>>> Let me know if it is OK.
>>>>>>
>>>>>> Thanks,
>>>>>> Uri
>>>>>>
>>>>>>  On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla <
>>>>>> miconda at gmail.com> wrote:
>>>>>>
>>>>>>>  Hello,
>>>>>>>
>>>>>>> if you set memjoin to 0, do you see any difference?
>>>>>>>
>>>>>>> Can you try again (with memjoin 1 as well as 0) and send the output
>>>>>>> of:
>>>>>>>
>>>>>>> kamctl mi get_statistics shmem:
>>>>>>>
>>>>>>> before executing the reload commands?
>>>>>>>
>>>>>>> When it gets to 100%, can you see which process is using the cpu and
>>>>>>> attach to it with:
>>>>>>>
>>>>>>> gdb /path/to/kamailio PID
>>>>>>>
>>>>>>> then do:
>>>>>>>
>>>>>>> bt full
>>>>>>>
>>>>>>> and send output here?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>
>>>>>>> On 11/18/12 4:09 PM, Uri Shacked wrote:
>>>>>>>
>>>>>>>   After some testing I notice the following:
>>>>>>> First reload of 5 million records after kamailio started took about
>>>>>>> 9 sec.
>>>>>>> Second reload (4 minutes after the first one) took 60 sec.
>>>>>>> The third one (again about 4 minutes after the secind) got kamailio
>>>>>>> to use 100% cpu and after 13 minutes! i killed it.....
>>>>>>>
>>>>>>> I can understand that the memory manger works harder, still, any
>>>>>>> ideas on how to use mem_join and keep on reloading data.
>>>>>>> (in real life our data loads 5 million records once a day when
>>>>>>> almost no traffic. still after a few days it stops...)
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Uri
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked <ushacked at gmail.com>wrote:
>>>>>>>
>>>>>>>>   Hi,
>>>>>>>>
>>>>>>>> I am using MTREE and DIALPLAN modules to load lots of info to
>>>>>>>> kamailio. (6 million rows).
>>>>>>>>
>>>>>>>> When kamailio was running with 3.2.1 (no mem_join=1 option), the
>>>>>>>> used size was increasing but the process of loading the data was fast
>>>>>>>> eanough.
>>>>>>>>
>>>>>>>> I upgraded to 3.3.2 and set mem_join=1. Now the loading process
>>>>>>>> take about 10 time longer and sometimes stops kamailio from responding to
>>>>>>>> traffic.
>>>>>>>>
>>>>>>>> Any ideas?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Uri
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>  _______________________________________________
>>>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>
>>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20131124/ebd366a1/attachment-0001.html>


More information about the sr-users mailing list