alexbakker created an issue (kamailio/kamailio#4231)
### Description
Performing an HTTPS request using the ``http_client`` module results in a shared memory leak, seemingly related to TLS.
### Troubleshooting
#### Reproduction
Our use case for ``http_client`` is to update an htable with the latest banned IP's from APIBAN as described here: https://github.com/apiban/apiban?tab=readme-ov-file#integration-into-kamaili.... Normally, we refresh the list of banned IP's once every 15 minutes, so the shared memory usage grows fairly slowly.
The issue can be reproduced using the Kamailio configuration file below. To demonstrate the issue and make the shared memory usage grow faster, we set up a timer that calls ``http_client_query`` in a tight loop. To prevent unnecessary load on the APIBAN infrastructure, we request an example file from my personal server instead.
``` #!KAMAILIO
debug=2 log_stderror=yes fork=yes enable_tls=1
loadmodule "tls.so" loadmodule "cfg_rpc.so" loadmodule "pv.so" loadmodule "xlog.so" loadmodule "ctl.so" loadmodule "rtimer.so" loadmodule "http_client.so"
modparam("tls", "certificate", "") modparam("tls", "private_key", "")
modparam("rtimer", "timer", "name=apiban;interval=100u;mode=1;") modparam("rtimer", "exec", "timer=apiban;route=APIBAN;")
route[APIBAN] { xinfo("running apiban refresh\n"); http_client_query("https://alexbakker.me/u/7bvjn9jfas.txt", "$var(banned)"); } ```
Start Kamailio:
``` kamailio -f kamailio.cfg -DD ```
And then watch the shared memory usage grow rapidly:
``` watch -n1 -- kamcmd core.shmmem ```
#### Log Messages
Eventually, Kamailio will run out of shared memory and print the following messages to the log:
``` 21(184184) INFO: <script>: running apiban refresh 21(184184) ERROR: http_client [functions.c:471]: curL_request_url(): failed to perform curl (56) (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) ERROR: http_client [functions.c:471]: curL_request_url(): failed to perform curl (56) (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) ERROR: <core> [core/mem/q_malloc.c:758]: qm_realloc(): qm_realloc(0x7585c73c4000, 1024) called from tls: tls_init.c: ser_realloc(372), module: tls; qm_malloc() failed! 21(184184) INFO: <script>: running apiban refresh 21(184184) INFO: <script>: running apiban refresh 21(184184) ERROR: <core> [core/mem/q_malloc.c:758]: qm_realloc(): qm_realloc(0x7585c73c4000, 1024) called from tls: tls_init.c: ser_realloc(372), module: tls; qm_malloc() failed! 21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt) 21(184184) INFO: <script>: running apiban refresh 21(184184) INFO: <script>: running apiban refresh 21(184184) WARNING: http_client [functions.c:463]: curL_request_url(): TLS CA certificate read error (url: https://alexbakker.me/u/7bvjn9jfas.txt) ```
``mem_dump_shm`` reports a large list of TLS-related allocations. Just sharing the last few lines here, as they're seemingly all from the same location.
``` 20(204914) ALERT: qm_status: qm_status(): 28500. N address=0x705ab3178710 frag=0x705ab31786d0 size=112 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28501. N address=0x705ab31787f0 frag=0x705ab31787b0 size=48 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28502. N address=0x705ab3178890 frag=0x705ab3178850 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28503. N address=0x705ab3178920 frag=0x705ab31788e0 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28504. N address=0x705ab31789b0 frag=0x705ab3178970 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28505. N address=0x705ab3178a40 frag=0x705ab3178a00 size=48 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28506. N address=0x705ab3178ae0 frag=0x705ab3178aa0 size=48 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28507. N address=0x705ab3178b80 frag=0x705ab3178b40 size=144 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28509. N address=0x705ab3178d00 frag=0x705ab3178cc0 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28510. N address=0x705ab3178d90 frag=0x705ab3178d50 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_realloc(372) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28512. N address=0x705ab3178ea0 frag=0x705ab3178e60 size=16 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28513. N address=0x705ab3178f20 frag=0x705ab3178ee0 size=32 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28515. N address=0x705ab317a3c0 frag=0x705ab317a380 size=112 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28516. N address=0x705ab317a4a0 frag=0x705ab317a460 size=512 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28517. N address=0x705ab317a710 frag=0x705ab317a6d0 size=1504 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed 20(204914) ALERT: qm_status: qm_status(): 28518. N address=0x705ab317ad60 frag=0x705ab317ad20 size=1040 used=1 20(204914) ALERT: qm_status: qm_status(): alloc'd from tls: tls_init.c: ser_malloc(364) 20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end check= c0c0c0c0, abcdefed ```
### Possible Solutions
I'm not aware of a workaround for this issue. For our specific use case, I'll move the APIBAN polling out of Kamailio to a separate process for now.
### Additional Information
I've verified that this issue is reproducible on ``5.5.7``, ``5.6.6``, ``5.7.6``, ``5.8.6`` and ``6.0.1``.
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 6.0.1 (x86_64/linux) fce50d flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: fce50d compiled on 06:42:29 May 6 2025 with gcc 12.2.0 ```
* **Operating System**:
``` $ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm
$ uname -a Linux ip-10-146-128-121 6.1.0-32-cloud-arm64 #1 SMP Debian 6.1.129-1 (2025-03-06) aarch64 GNU/Linux ```
miconda left a comment (kamailio/kamailio#4231)
I see that Kamailio reports being compiled for x86_64, but the OS reports is arm64. Is this the combination you use? Is the OS on arm64? If yes, would it be possible that you test on an amd64/x86_64, over the past I encountered issues with various external libraries on arm that were not revealed on amd64/x86_64.
Then, for the moment, if you compile from sources, there are alternative http-client get functions in the secsipid and ruxc modules, maybe you can test with them and see if works better.
alexbakker left a comment (kamailio/kamailio#4231)
Apologies for the confusion. No, the architecture Kamailio is compiled for is the same as the architecture it runs on. We run Kamailio on arm64 in production. But our workstation machines are x86_64. I must have run ``uname -a`` on the wrong machine
I've updated the issue to fix this. Either way, the issue is reproducible on both arm64 and x86_64.
github-actions[bot] left a comment (kamailio/kamailio#4231)
This issue is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks.
xkaraman left a comment (kamailio/kamailio#4231)
@alexbakker
Are you compiling these from sources? If yes can you try and use the `sbin/kamcmd` installed from source as well? I have the suspicion that system `kamcmd` if available does some wrong or reads something else.
I can't reproduce the leak on my system when builded from source and latest master.
What i can see though are negative number when i try to use system kamcmd instead of source installed kamcmd.
alexbakker left a comment (kamailio/kamailio#4231)
@xkaraman Yes, this is all compiled from source, including ``kamcmd``.
xkaraman left a comment (kamailio/kamailio#4231)
@alexbakker
I have tried to reproduce again with `6.0.1` and `5.5.7` tags. Unfortunately, i coudln't get the memory used to increase uncontrollably. Even with `100u` interval, memory increase for a bit (normal for so many calls though) and the went back down to normal values when kamailio first started.
Are you using any other modules that might interact with this? or any outdated or uncompatible versions of `kamailio` and `kamcmd`?
As i said before my system `kamcmd` part of kamailio 5.3.x, indeed shows some negative values when trying to see the `core.shmmem` when running 6.0.1 tag.
It does not exhibit the same behaviour though when running the 5.5.7 tag.
alexbakker left a comment (kamailio/kamailio#4231)
Are you using any other modules that might interact with this? or any outdated or uncompatible versions of kamailio and kamcmd?
No. Kamailio and kamcmd were compiled from source and were tested together with matching versions. And the issue is reproducible with the exact configuration file I provided.
To make sure we're running the same thing, here's a Dockerfile to help reproduce the issue:
```Dockerfile FROM debian:bookworm
RUN DEBIAN_FRONTEND=noninteractive apt-get update && \ apt-get upgrade -y && \ apt-get install -y gpg wget watch
RUN wget -O- https://deb.kamailio.org/kamailiodebkey.gpg | gpg --dearmor | tee /usr/share/keyrings/kamailio.gpg
RUN printf "deb [signed-by=/usr/share/keyrings/kamailio.gpg] http://deb.kamailio.org/kamailio60 bookworm main\ndeb-src [signed-by=/usr/share/keyrings/kamailio.gpg] http://deb.kamailio.org/kamailio60 bookworm main" > /etc/apt/sources.list
RUN DEBIAN_FRONTEND=noninteractive apt-get update && \ apt-get install -y kamailio kamailio-tls-modules kamailio-utils-modules
RUN cat <<'EOF' > /etc/kamailio/kamailio.cfg #!KAMAILIO
debug=2 log_stderror=yes fork=yes enable_tls=1
loadmodule "tls.so" loadmodule "cfg_rpc.so" loadmodule "pv.so" loadmodule "xlog.so" loadmodule "ctl.so" loadmodule "rtimer.so" loadmodule "http_client.so"
modparam("tls", "certificate", "") modparam("tls", "private_key", "")
modparam("rtimer", "timer", "name=apiban;interval=100u;mode=1;") modparam("rtimer", "exec", "timer=apiban;route=APIBAN;")
route[APIBAN] { xinfo("running apiban refresh\n"); http_client_query("https://alexbakker.me/u/7bvjn9jfas.txt", "$var(banned)"); }
EOF
CMD ["kamailio", "-DD"] ```
To start it:
```sh podman run --name kamailio_issue_4231 -it $(podman build -q .) ```
To monitor the memory usage increase:
```sh podman exec -it kamailio_issue_4231 watch -n1 -- kamcmd core.shmmem ```
xkaraman left a comment (kamailio/kamailio#4231)
Hey @alexbakker ,
Thanks for the dockerfile. indeed i can reproduce this on docker.
Just to clarify, could you reproduce it without docker as well?
On my system with:
``` lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal
uname -a Linux app01.dev.nbg.gilawa.net 5.4.0-216-generic #236-Ubuntu SMP Fri Apr 11 19:53:21 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux ```
i couldn't reproduce it with same config and with/without `-DD` .
xkaraman left a comment (kamailio/kamailio#4231)
After some more testing and OS changing thanks to docker, we could see that the problem does not happen in `focal` and `bullseye` where `openssl <3.0`. **Bullseye:** ``` curl (7.74.0-1.3+deb11u15 openssl (1.1.1w-0+deb11u3 ``` **Focal:** ``` [curl 7.68.0-1ubuntu2.25](https://launchpad.net/ubuntu/+source/curl/7.68.0-1ubuntu2.25) (main) [openssl 1.1.1f-1ubuntu2.24](https://launchpad.net/ubuntu/+source/openssl/1.1.1f-1ubuntu2.24) (main) ```
Letting it both run with your exact config, they stayed stable for the whole duration.
Other OS, like `bookworm`, `jammy (22.04)` and `noble (24.04)` exhibit the same behaviour you described with increasing memory and ultimately running out of memory. They all have `openssl > 3.0` with curl versions varying.
Bookworm: ``` curl (7.88.1-10+deb12u12) openssl (3.0.16-1~deb12u1) ```
Jammy: ``` [curl 7.81.0-1ubuntu1.20](https://launchpad.net/ubuntu/+source/curl/7.81.0-1ubuntu1.20) (main) [openssl 3.0.2-0ubuntu1.19](https://launchpad.net/ubuntu/+source/openssl/3.0.2-0ubuntu1.19) (main) ```
Noble: ``` [curl 8.5.0-2ubuntu10.6](https://launchpad.net/ubuntu/+source/curl/8.5.0-2ubuntu10.6) (main) [openssl 3.0.13-0ubuntu3.5](https://launchpad.net/ubuntu/+source/openssl/3.0.13-0ubuntu3.5) (main) ```
xkaraman left a comment (kamailio/kamailio#4231)
More info thay may or may no be related:
https://github.com/curl/curl/issues/7683#issuecomment-920906872. `curl 7.78` seem tos introduced some kind of this bug BUT
https://github.com/curl/curl/commit/60738f398cdd66312bce6ce92a87f19e71feacf4 fixed it in `curl 7.80`.
alexbakker left a comment (kamailio/kamailio#4231)
Just to clarify, could you reproduce it without docker as well?
Yes.
xkaraman left a comment (kamailio/kamailio#4231)
Hey @alexbakker,
I did some more testing to verify this on `bookworm` os.
So, i tried to changed shared memory manager for the kamailio.
It seems that `fm` and `tslf` memory managers does not exhibit this leak but `qm` does, can you confrim it as well?
You can use them by providing `-x` argument to `kamailio` like
`kamailio -DD -x tlsf|fm|qm`
-
Alexander Bakker -
Daniel-Constantin Mierla
-
github-actions[bot]
-
Xenofon Karamanos