Hi guys,
Since 1.3.0 (now running 1.4.4) I'm seeing a very slow uptake of SHM memory on our low traffic setup (less than 5 cps per machine). I'm looking for some basis to go further on in my research to the cause. :)
I compiled Kamailio 1.4.4-notls with #define SHM_MEM_SIZE 4*32 in config.h in production. For my testing setup I'm running on the standard 32 there.
After about 3 weeks uptime I start top, sort on memory size and find the kamailio processes (I'm running with 16 children) to all have about 40mb in the SHR column. My understanding is that this should also go down, but it only goes up, slowly. More CPS (for example a benchmark using sipp) makes this go up faster, but it never seems to go down this figure. I think this is wrong, but I could be wrong myself. :)
On a seperate machine with no traffic I compiled the memory debugging according to the "memory troubleshooting" page on the wiki. LOTS of info in the logs. Also ran with valgrind, didn't find anything interesting (but I'm no dev myself really).
My plan now is to take away our acc module (compiled with radius support) and see if it's maybe that module that's causing this. My test on this traffic-less machine is as follows: start, run 20cps for a while (we do no registers, just routing and auth) and note the SHR data from top. Then according to my understanding this figure should drop down after a period of 20 minutes with no traffic. Is this a right assumption?
On the test setup the top data looks like this after about 10 calls:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15975 root 18 0 69240 1536 568 S 0.0 0.6 0:00.00 kamailio 15972 root 18 0 69240 1568 588 S 0.0 0.6 0:00.00 kamailio 15970 root 18 0 69240 1568 588 S 0.0 0.6 0:00.00 kamailio 15969 root 19 0 69244 1488 568 S 0.0 0.6 0:00.00 kamailio 15968 root 15 0 69240 1860 936 S 0.0 0.7 0:00.00 kamailio 15967 root 15 0 71436 3376 2268 S 0.0 1.3 0:00.00 kamailio 15966 root 15 0 71436 3396 2288 S 0.0 1.3 0:00.00 kamailio 15965 root 23 0 69240 5552 4640 S 0.0 2.1 0:00.02 kamailio
As far as I know, it never goes down, the SHR entries. When running with very little SHM i config.h, the process goes out of shm memory and complains, as expected.
Are my assumptions about all of this correct?