[SR-Users] Shared memory fragmentation

[Daniel-Constantin Mierla]

Hello,

On 6/8/12 2:09 PM, Morten Isaksen wrote:
> Hi,
>
> We have 2 Kamailio 3.0.3 servers that has been running with
> carrierroute for about 2 years without any problems. They have 128 MB
> shared memory and
>
> modparam("carrierroute", "config_source", "db")
> modparam("carrierroute", "db_url", "<DBURL>")
> modparam("carrierroute", "fetch_rows", 500)
>
> The carrierroute table is about 91K lines and have been growing slowly.
>
> Suddenly we get this ERROR: carrierroute [cr_data.c:585]: could not
> allocate shared memory from available pool after a few "kamctl cr
> reload".
>
> I increased the shared memory to 256 MB but with the same result. I
> have now increased it to 512 MB and it seems to work better now.
>
> I have noticed this. After a restart the shmem counters is like this:
>
> shmem:total_size = 536870912
> shmem:used_size = 28486752
> shmem:real_used_size = 40147128
> shmem:max_used_size = 41135424
> shmem:free_size = 496723784
> shmem:fragments = 555
>
> And after the first "kamctl cr reload" it is like this:
>
> shmem:total_size = 536870912
> shmem:used_size = 28619016
> shmem:real_used_size = 51842768
> shmem:max_used_size = 76993616
> shmem:free_size = 485028144
> shmem:fragments = 722063
>
>
> Notice the increase in fragments. Sequentials "kamctl cr reload" does
> not change the fragments allot.
>
> Any ideas?
>
be aware that a reload command usually requires at least twice memory 
size than the size needed to store the records. That's because the old 
structure in memory is kept until the database records are all loaded 
and packed in shared memory in a new structure. Then the old and new 
structures are swapped for routing usage and the old one is deallocated. 
I havent really checked for carrier route module, but it is how happens 
with the other modules that do caching (dispatcher, mtree, permissions, 
...).

So if you are at the edge of available memory for storing cr rules, then 
could happen to get out of memory easy, as shared memory can be used for 
other tasks at that time. If number of fragments don't increase even 
with 512MB, then this is likely what happens.

Regarding the fragmentation, new release has options for doing 
defragmentation automatically:

https://www.kamailio.org/wiki/cookbooks/devel/core#mem_join

The code was actually pretty much all there, so I guess backporting will 
not be big task if you want to do it on your local copy.

Also, testing the devel with this feature and giving feedback will be 
really appreciated, I had no time to play with it and large number of 
records. The feature is off by default, you have to set mem_join=1 in 
config to get it on.

Cheers,
Daniel

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw