Hi Daniel,

Thanks for the reply again. Looking at the email history, I'm not sure how we decided it was definitely a pkg memory problem. What we see is the output of "ps aux" as follows:

root@sip0-test:~# ps aux | egrep -i "kamailio|mem"
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      6794  0.0  0.0  22480  1868 ?        Ss   Oct02   0:12 /opt/testuser/current/sbin/testuser_safe_kamailio
testuser  6822  0.0  0.2 556528  4580 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6824  0.3  8.7 825552 180244 ?       S    Oct02  56:40 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6825  0.3  8.7 825536 180776 ?       S    Oct02  56:20 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6826  0.3  8.7 825912 180296 ?       S    Oct02  55:54 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6827  0.3  8.7 825744 180580 ?       S    Oct02  56:19 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6828  0.3  8.7 825536 180092 ?       S    Oct02  56:25 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6829  0.3  8.7 825536 180632 ?       S    Oct02  56:21 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6830  0.3  8.7 825472 180968 ?       S    Oct02  56:37 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6831  0.3  8.7 825276 180272 ?       S    Oct02  56:41 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6832  0.0  0.0 556528  1324 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6833  0.0  0.0 556528  1324 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6834  0.0  0.0 556528  1324 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6835  0.0  0.0 556528  1324 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6836  0.0  0.0 556528  1324 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6837  0.0  0.0 556528  1324 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6838  0.0  0.0 556528  1324 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6839  0.0  0.0 556528  1324 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6840  0.0  0.0 556528  1776 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6841  0.0  0.0 556528  1780 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6842  0.0  0.0 556528  1780 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6843  0.0  0.0 556528  1328 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6844  0.0  0.0 556528  1780 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6845  0.0  0.0 556528  1328 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6846  0.0  0.0 556528  1328 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6847  0.0  0.0 556528  1328 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6848  0.0  0.0 556528  1676 ?        S    Oct02   0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6849  0.0  0.1 556528  3568 ?        S    Oct02   5:28 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6850  0.0  0.0 556612  1600 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6851  0.0  0.0 556532  1188 ?        S    Oct02   0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser  6852  0.0  0.0 556528  1360 ?        S    Oct02   0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid

You'll see for example that process 6824 is using 8.7% of memory, which is much more than it was using 5 days ago. Yet if I run the same sercmd again I get (exactly!) the same numbers:

root@sip0-test:~# sercmd pkg.stats pid 6824
{
    entry: 1
    pid: 6824
    rank: 1
    used: 209836
    free: 3704080
    real_used: 490224
}

Any ideas?



On 12 October 2013 00:23, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hi David,


On 10/10/13 11:36 PM, David Cunningham wrote:
Hi Daniel,

Thanks for the reply. Perhaps what we're seeing is normal, and the memory use is meant to increase as time progresses. Would you expect to see an ongoing memory use increase, or when should it stop increasing?


private memory (pkg) should stay rather constant. It increases when there is a sip message processed, but once is sent out, it should come back around the average.

There are couple of functions that can fill the private memory and keep it up, such as doing an sql_query() that returns a big data and the result is not freed (sql_result_free()). It is not actually a leak as the next sql_query() will free previous result, but in case you have such query for some corner case that is not executed frequently, then the memory can stay filled in. Another example will be storing very large value in a $var(...) (e.g., $var(x) ).

This is private memory, per process, which is meant for temporary operations. Shared memory (shm) can increase over the time, being the place where the dynamic data required at runtime is stored (e.g., location records, hash tables, transactions) - so as you get more traffic or more phones using kamailio, more shm is used. But your problem was reported for pkg.

Anyhow, keep an eye on the pkg.stats and if you see constant increase which is substantial, then get a mem log dump.

Cheers,
Daniel


--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
  - more details about Kamailio trainings at http://www.asipto.com -




--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019