alexbakker created an issue (kamailio/kamailio#4231)alexbakker created an issue (kamailio/kamailio#4231)Performing an HTTPS request using the http_client module results in a shared memory leak, seemingly related to TLS.
Our use case for http_client is to update an htable with the latest banned IP's from APIBAN as described here: https://github.com/apiban/apiban?tab=readme-ov-file#integration-into-kamailio. Normally, we refresh the list of banned IP's once every 15 minutes, so the shared memory usage grows fairly slowly.
The issue can be reproduced using the Kamailio configuration file below. To demonstrate the issue and make the shared memory usage grow faster, we set up a timer that calls http_client_query in a tight loop. To prevent unnecessary load on the APIBAN infrastructure, we request an example file from my personal server instead.
#!KAMAILIO
debug=2
log_stderror=yes
fork=yes
enable_tls=1
loadmodule "tls.so"
loadmodule "cfg_rpc.so"
loadmodule "pv.so"
loadmodule "xlog.so"
loadmodule "ctl.so"
loadmodule "rtimer.so"
loadmodule "http_client.so"
modparam("tls", "certificate", "")
modparam("tls", "private_key", "")
modparam("rtimer", "timer", "name=apiban;interval=100u;mode=1;")
modparam("rtimer", "exec", "timer=apiban;route=APIBAN;")
route[APIBAN] {
xinfo("running apiban refresh\n");
http_client_query("https://alexbakker.me/u/7bvjn9jfas.txt", "$var(banned)");
}
Start Kamailio:
kamailio -f kamailio.cfg -DD
And then watch the shared memory usage grow rapidly:
watch -n1 -- kamcmd core.shmmem
Eventually, Kamailio will run out of shared memory and print the following messages to the log:
21(184184) INFO: <script>: running apiban refresh
21(184184) ERROR: http_client [functions.c:471]: curL_request_url(): failed to perform curl (56) (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) ERROR: http_client [functions.c:471]: curL_request_url(): failed to perform curl (56) (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) ERROR: <core> [core/mem/q_malloc.c:758]: qm_realloc(): qm_realloc(0x7585c73c4000, 1024) called from tls: tls_init.c: ser_realloc(372), module: tls; qm_malloc() failed!
21(184184) INFO: <script>: running apiban refresh
21(184184) INFO: <script>: running apiban refresh
21(184184) ERROR: <core> [core/mem/q_malloc.c:758]: qm_realloc(): qm_realloc(0x7585c73c4000, 1024) called from tls: tls_init.c: ser_realloc(372), module: tls; qm_malloc() failed!
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:463]: curL_request_url(): TLS CA certificate read error (url: https://alexbakker.me/u/7bvjn9jfas.txt)
mem_dump_shm reports a large list of TLS-related allocations. Just sharing the last few lines here, as they're seemingly all from the same location.
20(204914) ALERT: qm_status: qm_status(): 28500. N address=0x705ab3178710 frag=0x705ab31786d0 size=112 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28501. N address=0x705ab31787f0 frag=0x705ab31787b0 size=48 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28502. N address=0x705ab3178890 frag=0x705ab3178850 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28503. N address=0x705ab3178920 frag=0x705ab31788e0 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28504. N address=0x705ab31789b0 frag=0x705ab3178970 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28505. N address=0x705ab3178a40 frag=0x705ab3178a00 size=48 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28506. N address=0x705ab3178ae0 frag=0x705ab3178aa0 size=48 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28507. N address=0x705ab3178b80 frag=0x705ab3178b40 size=144 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28509. N address=0x705ab3178d00 frag=0x705ab3178cc0 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28510. N address=0x705ab3178d90 frag=0x705ab3178d50 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_realloc(372)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28512. N address=0x705ab3178ea0 frag=0x705ab3178e60 size=16 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28513. N address=0x705ab3178f20 frag=0x705ab3178ee0 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28515. N address=0x705ab317a3c0 frag=0x705ab317a380 size=112 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28516. N address=0x705ab317a4a0 frag=0x705ab317a460 size=512 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28517. N address=0x705ab317a710 frag=0x705ab317a6d0 size=1504 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28518. N address=0x705ab317ad60 frag=0x705ab317ad20 size=1040 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
I'm not aware of a workaround for this issue. For our specific use case, I'll move the APIBAN polling out of Kamailio to a separate process for now.
I've verified that this issue is reproducible on 5.5.7, 5.6.6, 5.7.6, 5.8.6 and 6.0.1.
kamailio -vversion: kamailio 6.0.1 (x86_64/linux) fce50d
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: fce50d
compiled on 06:42:29 May 6 2025 with gcc 12.2.0
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm
$ uname -a
Linux ip-10-146-128-121 6.1.0-32-cloud-arm64 #1 SMP Debian 6.1.129-1 (2025-03-06) aarch64 GNU/Linux
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.