Experiencing an issue with QMalloc when Kamailio receives a lot of subscriptions (dialog-info) in a short period of time. The following is reported in logs and the TCP (TLS) connection fails:
```Apr 4 18:06:04 ua-proxy-01 /usr/local/sbin/kamailio[28456]: ERROR: <core> [tcp_main.c:959]: tcpconn_new(): mem. allocation failure Apr 4 18:06:04 ua-proxy-01 /usr/local/sbin/kamailio[28456]: ERROR: <core> [tcp_main.c:3977]: handle_new_connect(): tcpconn_new failed, closing socket```
A snapshot of SHM usage from around the time of the connection is as follows:
```shmem:fragments = 39604 shmem:free_size = 49473400 shmem:max_used_size = 67108696 shmem:real_used_size = 17635464 shmem:total_size = 67108864 shmem:used_size = 9423760```
Kamailio `-I` output is as follows:
```Print out of kamailio internals Version: kamailio 4.3.4 (x86_64/linux) b2843e Default config: /usr/local/etc/kamailio/kamailio.cfg Default paths to modules: /usr/local/lib64/kamailio/modules Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES MAX_RECV_BUFFER_SIZE=262144 MAX_LISTEN=16 MAX_URI_SIZE=1024 BUF_SIZE=65535 DEFAULT PKG_SIZE=8MB DEFAULT SHM_SIZE=64MB ADAPTIVE_WAIT_LOOPS=1024 TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select Source code revision ID: b2843e Compiled with: gcc 4.7.2 Compiled on: 17:56:04 Apr 4 2016 Thank you for flying kamailio!```
Once it has gotten in this state, TCP connections will no longer work, regardless of how low the SHM gets. Other connections (UDP) and subscriptions (dialog-info) work correctly at this time. There are no other OOM errors in the logs. The only thing that will resolve the issue is to restart Kamailio.
Compiling to use FMalloc (`MEMMNG=0 MEMDBG=1 make cfg`) appears to resolve the issue. Kamailio `-I` output as follows:
```Print out of kamailio internals Version: kamailio 4.3.4 (x86_64/linux) b2843e Default config: /usr/local/etc/kamailio/kamailio.cfg Default paths to modules: /usr/local/lib64/kamailio/modules Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES MAX_RECV_BUFFER_SIZE=262144 MAX_LISTEN=16 MAX_URI_SIZE=1024 BUF_SIZE=65535 DEFAULT PKG_SIZE=8MB DEFAULT SHM_SIZE=64MB ADAPTIVE_WAIT_LOOPS=1024 TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select Source code revision ID: b2843e Compiled with: gcc 4.7.2 Compiled on: 18:07:19 Apr 4 2016 Thank you for flying kamailio!```
At the time the TCP connection works correctly, the SHM usage is:
```shmem:fragments = 40669 shmem:free_size = 51558136 shmem:max_used_size = 67107816 shmem:real_used_size = 15550728 shmem:total_size = 67108864 shmem:used_size = 10045520```
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/562
Do you have memjoin parameter set to 1?
If you test with 4.4, then you can change the memory manager with -x command line parameter, you no longer need to recompile kamailio.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/562#issuecomment-205661238
Adding `mem_join=1` to the config also appears to resolve the issue. Is this something that is enabled by default on 4.4? These two links contradict each other:
https://www.kamailio.org/wiki/tutorials/troubleshooting/memory https://www.kamailio.org/wiki/cookbooks/4.4.x/core#mem_join
The take-home from this issue report may well just be that I need to PR an update to the docs to make it more explicit why one might want mem_join. Happy to do that.
I saw in the docs the `-x` parameter was added to 4.4 - this is good news.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/562#issuecomment-205721689
Do you have a suggestion of the best `malloc` method to use in a production environment? Some Kamailio blog posts suggest fmalloc is more suited to production?
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/562#issuecomment-205758091
Closed #562.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/562#event-621639754
mem_join is not set to 1 by default. Could be considered for the future if people find it useful. Note that you can set it to 0 or 1 at runtime, without restart of kamailio, via kamcmd.
fmalloc is the one with less overhead, so it is good for production. qmalloc has more overhead, but it is not noticeable if you don't have really huge traffic. The new one tlsf malloc, seems to work better when dealing with a lot of large chunks of memory (e.g., a lot of tls connections, that may get closed all at once).
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/562#issuecomment-208054773
-
Daniel-Constantin Mierla -
Phil Lavin