[SR-Users] reload to memory is much slower and has problms after upgrade to 3.3.2 and using mem_join=1
Daniel-Constantin Mierla
miconda at gmail.com
Mon Nov 26 16:43:08 CET 2012
Hello,
On 11/22/12 8:51 AM, Uri Shacked wrote:
>
> Hi,
> >> Btw, is the reload time constant now? Even if you run couple of times?
> yes, the reload time is constant - 56 sec. i tested for 100 reloads in
> an hour and it was OK.
ok - it's 56sec or 36sec?
> >> What are the values for 'kamctl mi get_statistics shmem:'?
> i configured kamailio to start with 4Gb and after reload the shmem
> (real_used) take around 30% of it.
> but, after 20 reloads it grows in 1%. so, after the 100 reloads the
> real_used take around 34%-35% of shmem.
> i made the choise to compile again with f_malloc and not use mem_join.
> the reloads are faster, it uses less real_size (12% and not 30%) and
> the increasment of it is around 1% for 5 reloads (i do 5 reloads a
> week). i will keep track on it and update.
> thanks a lot.
> Btw, what do you think? would you use f_malloc with no mem_join or
> q_malloc with join?
Can you try with master branch? I switched to q_malloc with no debug
info. That means the overhead should be lower and the joining faster.
You can do it also with 3.3, you have to edit Makefile.defs:
- be sure MEMDBG=0
- replace the next block:
ifeq ($(MEMDBG), 1)
C_DEFS+= -DDBG_QM_MALLOC
C_DEFS+= -DMEM_JOIN_FREE
else
C_DEFS+= -DF_MALLOC
C_DEFS+= -DMEM_JOIN_FREE
endif
with:
ifeq ($(MEMDBG), 1)
C_DEFS+= -DDBG_QM_MALLOC
C_DEFS+= -DMEM_JOIN_FREE
else
C_DEFS+= -DMEM_JOIN_FREE
endif
Practically is removal of line C_DEFS+= -DF_MALLOC
Then recompile and reinstall..
Cheers,
Daniel
> BR,
> Uri
>
> On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
>
> On 11/21/12 1:33 PM, Uri Shacked wrote:
>> Hi,
>> I recompiled with MEMDBG=1 and installed.
>> here are the results for reloading 5 million rows with MTREE:
>> mem_join=1 -->takes 56 seconds and the real_used_size of shmem is
>> around 1.2Gb.
>> mem_join=0 --> takes 10 seconds and the real_used_size of shmem
>> is around 2Gb
>> does it seems normal?
>> 56 seconds is a lot of time......
> the join is done for the free operation, meaning that most of the
> time is spent when freeing the old tree from memory. The new
> values will be used after loading the database records, then the
> old tree is destroyed (this involves the join operation). Also,
> the sip routing is not affected, loading the new records and
> destroying old memory tree is done in the MI/RPC process.
>
> In other words, while the MI/RPC process takes care of loading new
> data and destroying the old one, the SIP routing is not affected
> at all.
>
> Even when the reload command is executed, the old tree is used
> until all the records are loaded in a new tree. At that moment,
> the pointer to the active tree is changed from the old tree to the
> new tree (a very simple sequence of assignments, very fast).
> Routing will use the new tree and the Mi/RPC process starts
> destroying the old tree.
>
>
>> by the way, when the f_malloc was used, the size of the real_used
>> shmem was twice smaller.
>
> The overhead when storing small values is significant for
> q_malloc, each fragment keeping references (pointers) to file name
> and line where it was allocated and freed. In addition it keeps
> information to get to previews and next fragment, resulting in
> faster join.
>
> It is some space to improve, in order to make less overhead (like
> a compile time option not to keep info about file and line of
> malloc/free). I will think about what can be done for the next
> release.
>
> Btw, is the reload time constant now? Even if you run couple of times?
>
> What are the values for 'kamctl mi get_statistics shmem:'?
>
> Cheers,
> Daniel
>
>> Thanks.
>> On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>>
>> On 11/20/12 7:34 PM, Uri Shacked wrote:
>>> Hi,
>>> can you be a litle more specific of the steps of the install
>>> and where do i make the changes?
>>
>> in the source tree, edit the file Makefile.defs and set:
>>
>> MEMDBG=1
>>
>> then run:
>>
>> make all
>> make install
>>
>>
>>> some words of what is the diff between f_malloc and q_malloc
>>> will be great :-).
>>
>> q_malloc is more debugging purposes, keeping more information
>> for each chunk, therefore the overhead is a bit higher than
>> with f_malloc, but because keeps more details, it is faster
>> to find the fragments that can be joined.
>>
>> Cheers,
>> Daniel
>>
>>
>>> thanks,
>>> Uri
>>>
>>> On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> ok, I will look over it. At this moment the f_malloc
>>> (which is enabled for 3.3) has a pretty inefficient mem
>>> join implementation, can you try with q_malloc? Edit
>>> Makefile.defs and set:
>>>
>>> MEMDBG=1
>>>
>>> Then compile and install.
>>>
>>> The join operation should be faster, let's see if you
>>> get blocking issues with this one.
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 11/20/12 2:57 PM, Uri Shacked wrote:
>>>> Daniel hi,
>>>> I attached 2 txt files.
>>>> One with mem_join=1, the other with mem_join=0, and the
>>>> info you asked for.
>>>> Let me know if it is OK.
>>>> Thanks,
>>>> Uri
>>>>
>>>> On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin
>>>> Mierla <miconda at gmail.com <mailto:miconda at gmail.com>>
>>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> if you set memjoin to 0, do you see any difference?
>>>>
>>>> Can you try again (with memjoin 1 as well as 0) and
>>>> send the output of:
>>>>
>>>> kamctl mi get_statistics shmem:
>>>>
>>>> before executing the reload commands?
>>>>
>>>> When it gets to 100%, can you see which process is
>>>> using the cpu and attach to it with:
>>>>
>>>> gdb /path/to/kamailio PID
>>>>
>>>> then do:
>>>>
>>>> bt full
>>>>
>>>> and send output here?
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 11/18/12 4:09 PM, Uri Shacked wrote:
>>>>> After some testing I notice the following:
>>>>> First reload of 5 million records after kamailio
>>>>> started took about 9 sec.
>>>>> Second reload (4 minutes after the first one) took
>>>>> 60 sec.
>>>>> The third one (again about 4 minutes after the
>>>>> secind) got kamailio to use 100% cpu and after 13
>>>>> minutes! i killed it.....
>>>>> I can understand that the memory manger works
>>>>> harder, still, any ideas on how to use mem_join
>>>>> and keep on reloading data.
>>>>> (in real life our data loads 5 million
>>>>> records once a day when almost no traffic. still
>>>>> after a few days it stops...)
>>>>> Thanks,
>>>>> Uri
>>>>>
>>>>>
>>>>> On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked
>>>>> <ushacked at gmail.com <mailto:ushacked at gmail.com>>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I am using MTREE and DIALPLAN modules to load
>>>>> lots of info to kamailio. (6 million rows).
>>>>>
>>>>> When kamailio was running with 3.2.1 (no
>>>>> mem_join=1 option), the used size was
>>>>> increasing but the process of loading the data
>>>>> was fast eanough.
>>>>>
>>>>> I upgraded to 3.3.2 and set mem_join=1. Now
>>>>> the loading process take about 10 time longer
>>>>> and sometimes stops kamailio from responding
>>>>> to traffic.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Uri
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>> sr-users at lists.sip-router.org <mailto:sr-users at lists.sip-router.org>
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -http://www.asipto.com
>>>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierla -http://www.asipto.com
>>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierla -http://www.asipto.com
>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>>
>>
>
> --
> Daniel-Constantin Mierla -http://www.asipto.com
> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
>
>
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20121126/559b56c0/attachment-0001.htm>
More information about the sr-users
mailing list