alexbakker created an issue (kamailio/kamailio#4231)
### Description
Performing an HTTPS request using the ``http_client`` module results in a shared memory leak, seemingly related to TLS.
### Troubleshooting
#### Reproduction
Our use case for ``http_client`` is to update an htable with the latest banned IP's from APIBAN as described here: https://github.com/apiban/apiban?tab=readme-ov-file#integration-into-kamaili.... Normally, we refresh the list of banned IP's once every 15 minutes, so the shared memory usage grows fairly slowly.
The issue can be reproduced using the Kamailio configuration file below. To demonstrate the issue and make the shared memory usage grow faster, we set up a timer that calls ``http_client_query`` in a tight loop. To prevent unnecessary load on the APIBAN infrastructure, we request an example file from my personal server instead.
``` #!KAMAILIO
debug=2 log_stderror=yes fork=yes enable_tls=1
loadmodule "tls.so" loadmodule "cfg_rpc.so" loadmodule "pv.so" loadmodule "xlog.so" loadmodule "ctl.so" loadmodule "rtimer.so" loadmodule "http_client.so"
modparam("tls", "certificate", "") modparam("tls", "private_key", "")
modparam("rtimer", "timer", "name=apiban;interval=100u;mode=1;") modparam("rtimer", "exec", "timer=apiban;route=APIBAN;")
route[APIBAN] { xinfo("running apiban refresh\n"); http_client_query("https://alexbakker.me/u/7bvjn9jfas.txt", "$var(banned)"); } ```
Start Kamailio:
``` kamailio -f kamailio.cfg -DD ```
And then watch the shared memory usage grow rapidly:
``` watch -n1 -- kamcmd core.shmmem ```
#### Log Messages
Eventually, Kamailio will run out of shared memory and print the following messages to the log:
``` 21(184184) INFO: <script>: running apiban refresh 21(184184) ERROR: http_client [functions.c:471]: curL_request_url(): failed to perform curl (56) (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) ERROR: http_client [functions.c:471]: curL_request_url(): failed to perform curl (56) (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) ERROR: <core> [core/mem/q_malloc.c:758]: qm_realloc(): qm_realloc(0x7585c73c4000, 1024) called from tls: tls_init.c: ser_realloc(372), module: tls; qm_malloc() failed! 21(184184) INFO: <script>: running apiban refresh 21(184184) INFO: <script>: running apiban refresh 21(184184) ERROR: <core> [core/mem/q_malloc.c:758]: qm_realloc(): qm_realloc(0x7585c73c4000, 1024) called from tls: tls_init.c: ser_realloc(372), module: tls; qm_malloc() failed! 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:463]: curL_request_url(): TLS CA certificate read error (url: https://alexbakker.me/u/7bvjn9jfas.txt) ```
``mem_dump_shm`` reports a large list of TLS-related allocations. Just sharing the last few lines here, as they're seemingly all from the same location.
``` 20(204914) ALERT: qm_status: qm_status(): 28500. N address=0x705ab3178710 frag=0x705ab31786d0 size=112 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28501. N address=0x705ab31787f0 frag=0x705ab31787b0 size=48 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28502. N address=0x705ab3178890 frag=0x705ab3178850 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28503. N address=0x705ab3178920 frag=0x705ab31788e0 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28504. N address=0x705ab31789b0 frag=0x705ab3178970 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28505. N address=0x705ab3178a40 frag=0x705ab3178a00 size=48 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28506. N address=0x705ab3178ae0 frag=0x705ab3178aa0 size=48 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28507. N address=0x705ab3178b80 frag=0x705ab3178b40 size=144 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28509. N address=0x705ab3178d00 frag=0x705ab3178cc0 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28510. N address=0x705ab3178d90 frag=0x705ab3178d50 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_realloc(372) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28512. N address=0x705ab3178ea0 frag=0x705ab3178e60 size=16 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28513. N address=0x705ab3178f20 frag=0x705ab3178ee0 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28515. N address=0x705ab317a3c0 frag=0x705ab317a380 size=112 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28516. N address=0x705ab317a4a0 frag=0x705ab317a460 size=512 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28517. N address=0x705ab317a710 frag=0x705ab317a6d0 size=1504 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28518. N address=0x705ab317ad60 frag=0x705ab317ad20 size=1040 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed ```
### Possible Solutions
I'm not aware of a workaround for this issue. For our specific use case, I'll move the APIBAN polling out of Kamailio to a separate process for now.
### Additional Information
I've verified that this issue is reproducible on ``5.5.7``, ``5.6.6``, ``5.7.6``, ``5.8.6`` and ``6.0.1``.
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 6.0.1 (x86_64/linux) fce50d flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: fce50d compiled on 06:42:29 May 6 2025 with gcc 12.2.0 ```
* **Operating System**:
``` $ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm
$ uname -a Linux ip-10-146-128-121 6.1.0-32-cloud-arm64 #1 SMP Debian 6.1.129-1 (2025-03-06) aarch64 GNU/Linux ```