[SR-Users] Shared memory fragmentation

[Daniel-Constantin Mierla]

Hello,

On 6/11/12 10:03 AM, Morten Isaksen wrote:
> Hi Daniel and Marius,
>
> Thanks for your feedback.
>
> I hope I have solved the issue. Due to a bug in the perl script that
> generated the route table some destinations got added instead of
> updated to the table. The result of this bug was that about 50
> destinations each was 1082 times in the route table.

ok, good that you found it!

Cheers,
Daniel
>
> /Morten
>
> On Fri, Jun 8, 2012 at 4:51 PM, Marius Zbihlei <marius.zbihlei at 1and1.ro> wrote:
>> On 06/08/2012 03:09 PM, Morten Isaksen wrote:
>>> Hi,
>> Hello,
>>
>>
>>> We have 2 Kamailio 3.0.3 servers that has been running with
>>> carrierroute for about 2 years without any problems. They have 128 MB
>>> shared memory and
>> The shared memory is too little. Keep in mind that this is shared by all
>> children, so it will not increase with the creating of more children. Also
>> keep in mind that the Linux memory alocator (like for mmap) is lazy, so
>> allocating 1 GB by the application will not use 1 GB of RAM, but instead
>> pages will be delivered to the application as it needs it. To force
>> allocation there is the shm_force_alloc core parameter to really force
>> allocation of memory by the system bye writing 0's to the memory reagion.
>> Also there is the mlock_pages parameter that will force the allocated memory
>> pages to not be swappable.
>>
>>> modparam("carrierroute", "config_source", "db")
>>> modparam("carrierroute", "db_url", "<DBURL>")
>>> modparam("carrierroute", "fetch_rows", 500)
>>
>> Fetch_rows is affects the private memory used when loading data from a data
>> source that has the fetch result capability. This decreases the private
>> memory requirements for reloading the table.
>>
>>> The carrierroute table is about 91K lines and have been growing slowly.
>>
>>  From experience this takes around 60-70 MB of shared memory to hold it.
>>   This depends a lot based on system (32 vs 64 for example), structure etc.
>>
>>
>>> Suddenly we get this ERROR: carrierroute [cr_data.c:585]: could not
>>> allocate shared memory from available pool after a few "kamctl cr
>>> reload".
>>>
>>> I increased the shared memory to 256 MB but with the same result. I
>>> have now increased it to 512 MB and it seems to work better now.
>>>
>>> I have noticed this. After a restart the shmem counters is like this:
>>>
>>> shmem:total_size = 536870912
>>> shmem:used_size = 28486752
>>> shmem:real_used_size = 40147128
>>> shmem:max_used_size = 41135424
>>> shmem:free_size = 496723784
>>> shmem:fragments = 555
>>>
>>> And after the first "kamctl cr reload" it is like this:
>>>
>>> shmem:total_size = 536870912
>>> shmem:used_size = 28619016
>>> shmem:real_used_size = 51842768
>>> shmem:max_used_size = 76993616
>>> shmem:free_size = 485028144
>>> shmem:fragments = 722063
>>>
>> I would worry if the used_size or real_used_size would have changed a lot.
>> The difference from used_size to real_used_size is due to fragmentation but
>> this will be manageable if the number of fragments is kept constant. As
>> Daniel suggested, for carrierroute reload we need double the amount of
>> memory as it holds the old and the new tables until it swaps the pointers
>> (so that the applications doesn't block when reloading the table).
>>
>>
>>> Notice the increase in fragments. Sequentials "kamctl cr reload" does
>>> not change the fragments allot.
>>>
>>> Any ideas?
>>
>> I would recommend using kamailio with at least 1 GB of shared memory. Also
>> monitor the used_size and real_used_size  if they are increasing in a
>> `weird` way. The real_used_size might increase because of fragmentation
>> during cr_reload_route commands, but it should not fail, as chunks will be
>> reallocated to other usages of the command.
>>
>>
>>
>> Cheers,
>> Marius
>>
>> --
>> Zbihlei Marius
>>
>> Head of
>> Linux Development Services Romania
>>
>> 1&1 Internet Development srl    Tel KA: 754-9152
>> Str Mircea Eliade 18            Tel RO: +40-31-223-9152
>> Sect 1, Bucuresti               mailto: marius.zbihlei at 1and1.ro
>> 71295, Romania
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw