[SR-Users] Shared memory fragmentation

[Marius Zbihlei]

On 06/08/2012 03:09 PM, Morten Isaksen wrote:
> Hi,
Hello,

> We have 2 Kamailio 3.0.3 servers that has been running with
> carrierroute for about 2 years without any problems. They have 128 MB
> shared memory and
The shared memory is too little. Keep in mind that this is shared by all 
children, so it will not increase with the creating of more children. 
Also keep in mind that the Linux memory alocator (like for mmap) is 
lazy, so allocating 1 GB by the application will not use 1 GB of RAM, 
but instead pages will be delivered to the application as it needs it. 
To force allocation there is the shm_force_alloc core parameter to 
really force allocation of memory by the system bye writing 0's to the 
memory reagion. Also there is the mlock_pages parameter that will force 
the allocated memory pages to not be swappable.
> modparam("carrierroute", "config_source", "db")
> modparam("carrierroute", "db_url", "<DBURL>")
> modparam("carrierroute", "fetch_rows", 500)

Fetch_rows is affects the private memory used when loading data from a 
data source that has the fetch result capability. This decreases the 
private memory requirements for reloading the table.
>
> The carrierroute table is about 91K lines and have been growing slowly.

 From experience this takes around 60-70 MB of shared memory to hold 
it.  This depends a lot based on system (32 vs 64 for example), 
structure etc.

>
> Suddenly we get this ERROR: carrierroute [cr_data.c:585]: could not
> allocate shared memory from available pool after a few "kamctl cr
> reload".
>
> I increased the shared memory to 256 MB but with the same result. I
> have now increased it to 512 MB and it seems to work better now.
>
> I have noticed this. After a restart the shmem counters is like this:
>
> shmem:total_size = 536870912
> shmem:used_size = 28486752
> shmem:real_used_size = 40147128
> shmem:max_used_size = 41135424
> shmem:free_size = 496723784
> shmem:fragments = 555
>
> And after the first "kamctl cr reload" it is like this:
>
> shmem:total_size = 536870912
> shmem:used_size = 28619016
> shmem:real_used_size = 51842768
> shmem:max_used_size = 76993616
> shmem:free_size = 485028144
> shmem:fragments = 722063
>
I would worry if the used_size or real_used_size would have changed a 
lot. The difference from used_size to real_used_size is due to 
fragmentation but this will be manageable if the number of fragments is 
kept constant. As Daniel suggested, for carrierroute reload we need 
double the amount of memory as it holds the old and the new tables until 
it swaps the pointers (so that the applications doesn't block when 
reloading the table).

> Notice the increase in fragments. Sequentials "kamctl cr reload" does
> not change the fragments allot.
>
> Any ideas?

I would recommend using kamailio with at least 1 GB of shared memory. 
Also monitor the used_size and real_used_size  if they are increasing in 
a `weird` way. The real_used_size might increase because of 
fragmentation during cr_reload_route commands, but it should not fail, 
as chunks will be reallocated to other usages of the command.



Cheers,
Marius

-- 
Zbihlei Marius

Head of
Linux Development Services Romania

1&1 Internet Development srl    Tel KA: 754-9152
Str Mircea Eliade 18            Tel RO: +40-31-223-9152
Sect 1, Bucuresti               mailto: marius.zbihlei at 1and1.ro
71295, Romania