[sr-dev] [kamailio/kamailio] Memory usage increases everytime tls.reload is executed (Issue #3823)

### Description We are using Kamailio 5.7.4 on Debian 12 (from http://deb.kamailio.org/kamailio57) with rtpengine as an Edgeproxy for our clients. The instance terminates SIP/TLS (with Cliencertificates) and forwards the SIP Traffic to internal systems. After some days we are getting errors like this `tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: shm=7318616 threshold1=8912896` First we thought Kamailio just doesnt have enough memory, so we doubled it.. But after some days the Logmessage (and Userissues) occured again. So we monitored the shmmem statistics and found that used and max_used are constantly growing til it reaches the limit. As i mentioned we are using client-certificates and so we are also using the CRL feature. We do have a systemd-timer which fetches the CRL every hour and runs 'kamcmd tls.reload' when finished. Our tls.cfg looks like this: ``` [server:default] method = TLSv1.2+ private_key = /etc/letsencrypt/live/hostname.de/privkey.pem certificate = /etc/letsencrypt/live/hostname.de/fullchain.pem ca_list = /etc/kamailio/ca_list.pem ca_path = /etc/kamailio/ca_list.pem crl = /etc/kamailio/combined.crl.pem verify_certificate = yes require_certificate = yes [client:default] verify_certificate = yes require_certificate = yes ``` After testing a bit we found that every time tls.reload is executed Kamailio consumes a bit more memory which eventually leads to all the memory being consumed which leads to issues for our users. See following example: ``` [0][root@edgar-dev:~]# while true ; do /usr/sbin/kamcmd tls.reload ; /usr/sbin/kamcmd core.shmmem ; sleep 1 ; done Ok. TLS configuration reloaded. { total: 268435456 free: 223001520 used: 41352552 real_used: 45433936 max_used: 45445968 fragments: 73 } Ok. TLS configuration reloaded. { total: 268435456 free: 222377960 used: 41975592 real_used: 46057496 max_used: 46069232 fragments: 78 } Ok. TLS configuration reloaded. { total: 268435456 free: 221748664 used: 42604992 real_used: 46686792 max_used: 46698080 fragments: 77 } Ok. TLS configuration reloaded. { total: 268435456 free: 221110832 used: 43242408 real_used: 47324624 max_used: 47335608 fragments: 81 } ^C [130][root@edgar-dev:~]# ``` ### Troubleshooting #### Reproduction Everytime tls.reload is called the memory consumptions grows.. #### Debugging Data <!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location. gdb /path/to/kamailio /path/to/corefile bt full info locals list If you are familiar with gdb, feel free to attach more of what you consider to be relevant. --> ``` If you let me know what would be interesting for tracking this down, i am happy to provide logs/debugging data! ``` #### Log Messages <!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> ``` If you let me know what would be interesting for tracking this down, i am happy to provide logs/debugging data! ``` #### SIP Traffic SIP doesnt seem to be relevant here ### Possible Solutions Calling tls.reload less often or restart kamailio before memory is consumed ;) ### Additional Information ``` version: kamailio 5.7.4 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 12.2.0 ``` * **Operating System**: ``` * Debian GNU/Linux 12 (bookworm) * Linux edgar-dev 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3823 You are receiving this because you are subscribed to this thread. Message ID: &lt;kamailio/kamailio/issues/3823(a)github.com&gt;