[SR-Users] memory is not freed in system, while kamailio's statistic shows OK

[Vasiliy Ganchev]

Hi folk!
Have a production server
- used as webrtc to sip proxy
- it is built on kamailio v 4.4.2
- Server handles connections from ~4000 incoming SIP trunks (transport=TLS).
Kamailio provides auth for this trunks.
- Kamailio receives also connections from Webrtc app. Calls from Webrtc
routed to one of the 4000 sip trunks.

Server is running on Debian GNU/Linux 7

The number of calls on the system is small. Most of the load is generated by
registration/re-registrations of SIP trunks

The problem is following:
- output of kamcmd core.shmmem and kamcmd pkg.stats shows constant using of
memory
- output of htop shows constantly increasing memory usage, e.g. after one
week working:
 PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 .....
6013 kamailio   20   0  744M  134M  130M S  0.0  3.6  2:28.80
/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P
/var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio
6002 kamailio   20   0  744M  134M  130M S  0.0  3.6  2:30.93
/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P
/var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio
6012 kamailio   20   0  744M  134M  130M S  0.0  3.5  2:35.00
/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P
/var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio
5991 kamailio   20   0  744M  133M  129M S  0.0  3.5  2:05.88
/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P
/var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio
6018 kamailio   20   0  744M  133M  129M S  0.0  3.5  2:31.90
/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P
/var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio
.....

Despite running with: PKG_MEMORY=32, Amount of used memory per process is
higher.
Finishes all with monit killing kamailio, because of memory usage more than
allowed

studied this tutorial:
https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
but did not find a way how to dig up the problem's root.

Tried also to reproduce the issue in lab environment (having clone of
production server). Generated load with incoming registration using PJSIP,
but with no success - memory usage is constant.

kamailio -v
version: kamailio 4.4.2 (x86_64/linux) 
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled with gcc 4.7.2

Any suggestion/idea how to troubleshoot deeper would be highly apprecieted

Cheers!



--
View this message in context: http://sip-router.1086192.n5.nabble.com/memory-is-not-freed-in-system-while-kamailio-s-statistic-shows-OK-tp152604.html
Sent from the Users mailing list archive at Nabble.com.