[sr-dev] [kamailio/kamailio] QMalloc: TCP Memory Allocation Failures (#562)

Phil Lavin notifications at github.com
Mon Apr 4 19:25:02 CEST 2016


Experiencing an issue with QMalloc when Kamailio receives a lot of subscriptions (dialog-info) in a short period of time. The following is reported in logs and the TCP (TLS) connection fails:

```Apr  4 18:06:04 ua-proxy-01 /usr/local/sbin/kamailio[28456]: ERROR: <core> [tcp_main.c:959]: tcpconn_new(): mem. allocation failure
Apr  4 18:06:04 ua-proxy-01 /usr/local/sbin/kamailio[28456]: ERROR: <core> [tcp_main.c:3977]: handle_new_connect(): tcpconn_new failed, closing socket```

A snapshot of SHM usage from around the time of the connection is as follows:

```shmem:fragments = 39604
shmem:free_size = 49473400
shmem:max_used_size = 67108696
shmem:real_used_size = 17635464
shmem:total_size = 67108864
shmem:used_size = 9423760```

Kamailio `-I` output is as follows:

```Print out of kamailio internals
  Version: kamailio 4.3.4 (x86_64/linux) b2843e
  Default config: /usr/local/etc/kamailio/kamailio.cfg
  Default paths to modules: /usr/local/lib64/kamailio/modules
  Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
  MAX_RECV_BUFFER_SIZE=262144
  MAX_LISTEN=16
  MAX_URI_SIZE=1024
  BUF_SIZE=65535
  DEFAULT PKG_SIZE=8MB
  DEFAULT SHM_SIZE=64MB
  ADAPTIVE_WAIT_LOOPS=1024
  TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select
  Source code revision ID: b2843e
  Compiled with: gcc 4.7.2
  Compiled on: 17:56:04 Apr  4 2016
Thank you for flying kamailio!```

Once it has gotten in this state, TCP connections will no longer work, regardless of how low the SHM gets. Other connections (UDP) and subscriptions (dialog-info) work correctly at this time. There are no other OOM errors in the logs. The only thing that will resolve the issue is to restart Kamailio.

Compiling to use FMalloc (`MEMMNG=0 MEMDBG=1 make cfg`) appears to resolve the issue. Kamailio `-I` output as follows:

```Print out of kamailio internals
  Version: kamailio 4.3.4 (x86_64/linux) b2843e
  Default config: /usr/local/etc/kamailio/kamailio.cfg
  Default paths to modules: /usr/local/lib64/kamailio/modules
  Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
  MAX_RECV_BUFFER_SIZE=262144
  MAX_LISTEN=16
  MAX_URI_SIZE=1024
  BUF_SIZE=65535
  DEFAULT PKG_SIZE=8MB
  DEFAULT SHM_SIZE=64MB
  ADAPTIVE_WAIT_LOOPS=1024
  TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select
  Source code revision ID: b2843e
  Compiled with: gcc 4.7.2
  Compiled on: 18:07:19 Apr  4 2016
Thank you for flying kamailio!```

At the time the TCP connection works correctly, the SHM usage is:

```shmem:fragments = 40669
shmem:free_size = 51558136
shmem:max_used_size = 67107816
shmem:real_used_size = 15550728
shmem:total_size = 67108864
shmem:used_size = 10045520```

---
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/562
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-dev/attachments/20160404/42040bc3/attachment.html>


More information about the sr-dev mailing list