[kamailio/kamailio] Recurring crash: freeing already freed pointer called from tls: tls_init.c: ser_free (Issue #4055)
5 Dec
2024
5 Dec
'24
6:15 p.m.
### Description
We have 9 SIP proxy servers. 5 of them are running kamailio 5.5.4, and 4 of them are
running 5.8.3.
The 4 kamailio instances running 5.8.3 are experiencing a periodic crash, seemingly
related to TLS.
### Troubleshooting
#### Reproduction
Unknown at this time, it appears to be random so far.
#### Log Messages
```
ERROR: http_client [functions.c:352]: curL_request_url(): failed to perform curl (52)
(url: http://127.0.0.1:8082/flushchain)
tls: {IP_REDACTED} [{IP_REDACTED}]:{TLS_PORT_REDACTED}
ALERT: <core> [main.c:809]: handle_sigs(): core was not generated
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {IP_REDACTED}:*
NOTICE: tls [tls_domain.c:1176]: ksr_tls_fix_domain(): registered server_name callback
handler for socket [0.0.0.0:0], server_name='<default>' ...
tcp: 127.0.0.1 [127.0.0.1]:8080
ERROR: tls [tls_server.c:1316]: tls_h_read_f(): src addr: 159.65.167.207:29749
Listening on
udp: {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
NOTICE: tls [tls_domain.c:1176]: ksr_tls_fix_domain(): registered server_name callback
handler for socket [0.0.0.0:0], server_name='<default>' ...
CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message
processing timeout on connection id: 579785 (state: 0) - closing
ERROR: tls [tls_server.c:1312]: tls_h_read_f(): protocol level error
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:030000D6:digital envelope
routines::generate error (sni: unknown)
ERROR: tls [tls_server.c:1316]: tls_h_read_f(): src addr: 159.65.167.207:19625
CRITICAL: <core> [core/mem/q_malloc.c:136]: qm_debug_check_frag(): BUG: qm: fragm.
0x7fd8721da060 (address 0x7fd8721da0a0) end overwritten (3438203331736c74, 0)! Memory
allocator was called from tls: tls_init.c:397. Fragment marked by (null):0. Exec from
core/mem/q_malloc.c:546.
Listening on
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
WARNING: {f96acebe-204b-405f-88c2-e58732f5317d 1 2 BYE} dialog [dlg_handlers.c:1343]:
dlg_onroute(): unable to find dialog for BYE with route param '195.3a31'
[1425:5027] and call-id 'f96acebe-204b-405f-88c2-e58732f5317d'
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed
pointer (0x7f19e531d080), called from tls: tls_init.c: ser_free(397), first free tls:
tls_init.c: ser_free(397) - ignoring
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed
pointer (0x7f19e512edd0), called from tls: tls_init.c: ser_free(397), first free tls:
tls_init.c: ser_free(397) - ignoring
ERROR: tls [tls_server.c:1312]: tls_h_read_f(): protocol level error
ERROR: tls [tls_server.c:1319]: tls_h_read_f(): dst addr:
{IP_REDACTED}:{TLS_PORT_REDACTED}
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0800006B:elliptic curve
routines::point is not on curve (sni: {FQDN_REDACTED})
udp: {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED}
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed
pointer (0x7f19e6489600), called from tls: tls_init.c: ser_free(397), first free tls:
tls_init.c: ser_free(397) - ignoring
Aliases:
*: r{FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0A00012D:SSL routines::ssl
session id callback failed (sni: unknown)
tcp: {IP_REDACTED} [ {IP_REDACTED}]:{TLS_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
Aliases:
*: {FQDN_REDACTED}:*
WARNING: {0_3602603144(a)192.168.0.173 1 3 BYE} dialog [dlg_handlers.c:1343]: dlg_onroute():
unable to find dialog for BYE with route param 'e17.a901' [1822:4250] and call-id
'0_3602603144(a)192.168.0.173'
CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message
processing timeout on connection id: 579807 (state: 0) - closing
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0A080010:SSL routines::EC lib
(sni: mobile-srv.primevox.net)
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed
pointer (0x7f19e52d9200), called from tls: tls_init.c: ser_free(397), first free tls:
tls_init.c: ser_free(397) - ignoring
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed
pointer (0x7f19e4f878a0), called from tls: tls_init.c: ser_free(397), first free tls:
tls_init.c: ser_free(397) - ignoring
CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 69
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed
pointer (0x7f19e46597b0), called from tls: tls_init.c: ser_free(397), first free tls:
tls_init.c: ser_free(397) - ignoring
tcp: {IP_REDACTED} [{IP_REDACTED}]: {TCP_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
tcp: 127.0.0.1 [127.0.0.1]:8080
*: {FQDN_REDACTED}:*
tcp: {IP_REDACTED} [ {IP_REDACTED}]:{TCP_PORT_REDACTED}
udp: {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED}
udp: {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED}
tcp: {IP_REDACTED} [ {IP_REDACTED}]:{TCP_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
ALERT: <core> [main.c:805]: handle_sigs(): child process 2969636 exited by a signal
6
*: {FQDN_REDACTED}:*
CRITICAL: <core> [core/pass_fd.c:85]: recv_all(): 1st recv on 68 failed: Connection
reset by peer
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
WARNING: http_client [functions.c:318]: curL_request_url(): failed to connect() to host
(url: http://127.0.0.1:8082/flushchain)
WARNING: tls [tls_init.c:980]: tls_h_mod_init_f(): openssl bug #1491 (crash/mem leaks on
low memory) workaround enabled (on low memory tls operations will fail preemptively) with
free memory thresholds 19922944 and 9961472 bytes
ALERT: <core> [main.c:805]: handle_sigs(): child process 262943 exited by a signal
11
ALERT: <core> [main.c:809]: handle_sigs(): core was not generated
CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message
processing timeout on connection id: 579806 (state: 0) - closing
tls: {IP_REDACTED} [{IP_REDACTED}]:{TLS_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {IP_REDACTED}:*
WARNING: tls [tls_init.c:980]: tls_h_mod_init_f(): openssl bug #1491 (crash/mem leaks on
low memory) workaround enabled (on low memory tls operations will fail preemptively) with
free memory thresholds 19922944 and 9961472 bytes
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:1C8000C5:Provider
routines::reseed error (sni: unknown)
ERROR: tls [tls_server.c:1319]: tls_h_read_f(): dst addr: 104.225.96.168:8647
```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.8.3 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 17:00:40 Nov 11 2024 with gcc 12.2.0
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04,
CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->
```
Debian GNU/Linux 12 (bookworm)
Linux {FQDN} 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64
GNU/Linux
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4055
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4055(a)github.com>
5 Dec
5 Dec
6:28 p.m.
If you are using openssl 3.x, please check the value of the tls_threads_mode parameter in
your kamailio.cfg. It should be set to 2 (which will be the default in later versions).
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4055#issuecomment-2520837112
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4055/2520837112(a)github.com>
6:29 p.m.
Closed #4055 as completed.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4055#event-15546932686
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issue/4055/issue_event/15546932686(a)github.com>
6:29 p.m.
This seems like the known issue with openssl. Best handled via sr-users as isn't a
bug. See (https://github.com/kamailio/kamailio/issues/3791) `tls_threads_mode = 1`
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4055#issuecomment-2520837829
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4055/2520837829(a)github.com>
18
days inactive
18
days old
3 comments
3 participants
participants (3)
-
Fred Posner -
Henning Westerholt
-
Luke Escudé