### Description
<!-- Explain what you did, what you expected to happen, and what actually happened. --> Im getting frequent log entries like this: `CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f19a3edec68), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring`
This on a fresh VM running Ubuntu 22.04 LTS on the 5.6 branch (building latest release tag)
As well as that log message being printed on a frequent basis (every 2 to 3 mins) i am experiencing random crashes/restarts of kamailio. Looking at the core dumps tls_init seems to precipitate the core dump.
I have tried using both the TLS module, and TLSA, in the case of TLSA to rule out OpenSSL3 being an issue i have tried linking against locally built OpenSSL 3.0.5 and OpenSSL 1.1.1q with identical results
I am in the process of repeating the build on Ubuntu 20.04 LTS to rule out weirdness with packages and similar issues i have experienced whilst evaluating 22.04 LTS ### Troubleshooting
#### Reproduction
<!-- If the issue can be reproduced, describe how it can be done. -->
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` (gdb) bt full #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140425701996352) at ./nptl/pthread_kill.c:44 tid = <optimized out> ret = 0 pd = 0x7fb76815b740 old_mask = {__val = {523986010119, 910533066753, 1, 140425395047168, 94230980435360, 0, 5677797456, 140425395047168, 140725986283600, 140425393986269, 94230963279192, 140425395047168, 94230963279402, 7970739754, 140725986283664, 140425394028321}} ret = <optimized out> pd = <optimized out> old_mask = <optimized out> ret = <optimized out> tid = <optimized out> ret = <optimized out> resultvar = <optimized out> resultvar = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> __futex = <optimized out> resultvar = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> __futex = <optimized out> __private = <optimized out> __oldval = <optimized out> result = <optimized out> #1 __pthread_kill_internal (signo=6, threadid=140425701996352) at ./nptl/pthread_kill.c:78 No locals. #2 __GI___pthread_kill (threadid=140425701996352, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 No locals. #3 0x00007fb7681a0476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 ret = <optimized out> #4 0x00007fb7681867f3 in __GI_abort () at ./stdlib/abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x100000000, sa_sigaction = 0x100000000}, sa_mask = {__val = {140425395047168, 140725986283888, 140425393986269, 140425394365984, 140425394130944, 140425394801736, 5734026312, 140725986283952, 140425394028321, 15624006419995803439, 94230961557227, 94230963279168, 94230963279576, 140425422497024, 13020282480951897856, 72}}, sa_flags = -619194838, sa_restorer = 0x0} sigs = {__val = {32, 13020282480951897856, 140725986284000, 94230961610857, 13546569962021497653, 14189927350396752214, 94230962920532, 0, 94230963279402, 140425394130944, 10, 38654707763, 12, 140425395047168, 32, 1438623432}} #5 0x000055b3dafd8f12 in qm_debug_check_frag (qm=0x7fb755bc1000, f=0x7fb7576ce500, file=0x7fb766123154 "tls: tls_init.c", line=323, efile=0x55b3db17d559 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:129 p = 0x7fb755ca0b00 __func__ = "qm_debug_check_frag" #6 0x000055b3dafdd788 in qm_free (qmp=0x7fb755bc1000, p=0x7fb7576ce538, file=0x7fb766123154 "tls: tls_init.c", func=0x7fb766124990 <__func__.0> "ser_free", line=323, mname=0x7fb766123150 "tls") at core/mem/q_malloc.c:511 qm = 0x7fb755bc1000 f = 0x7fb7576ce500 size = 24 next = 0x7fb7576ce500 prev = 0x7fb7576ce3f8 __func__ = "qm_free" #7 0x000055b3dafe954e in qm_shm_free (qmp=0x7fb755bc1000, p=0x7fb7576ce538, file=0x7fb766123154 "tls: tls_init.c", func=0x7fb766124990 <__func__.0> "ser_free", line=323, mname=0x7fb766123150 "tls") at core/mem/q_malloc.c:1350 No locals. #8 0x00007fb7660db698 in ser_free (ptr=0x7fb7576ce538, fname=0x7fb765efa6e5 "../crypto/err/err_local.h", fline=88) at tls_init.c:323 __func__ = "ser_free" #9 0x00007fb765d2e907 in ERR_pop_to_mark () from /lib/x86_64-linux-gnu/libcrypto.so.3 No symbol table info available. #10 0x00007fb765d76a42 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3 No symbol table info available. #11 0x00007fb765d669f5 in EVP_PKEY_set_type_by_keymgmt () from /lib/x86_64-linux-gnu/libcrypto.so.3 No symbol table info available. #12 0x00007fb765d5ef9a in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3 No symbol table info available. #13 0x00007fb765d69973 in EVP_PKEY_generate () from /lib/x86_64-linux-gnu/libcrypto.so.3 No symbol table info available. #14 0x00007fb766029279 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3 No symbol table info available. #15 0x00007fb76607a621 in ?? () from /lib/x86_64-linux-gnu/libssl.so.3 No symbol table info available. --Type <RET> for more, q to quit, c to continue without paging--c #16 0x00007fb7660687fd in ?? () from /lib/x86_64-linux-gnu/libssl.so.3 No symbol table info available. #17 0x00007fb766104216 in tls_accept (c=0x7fb7576db4e8, error=0x7ffd526c58a4) at tls_server.c:468 ret = 1382832224 ssl = 0x7fb75755bef0 cert = 0x7fb75742dbb0 tls_c = 0x7fb7574429a8 tls_log = 1712337076 __func__ = "tls_accept" pkey = 0x0 #18 0x00007fb76610f86d in tls_h_read_f (c=0x7fb7576db4e8, flags=0x7ffd526e5dac) at tls_server.c:1173 r = 0x7fb7576db610 bytes_free = 16383 bytes_read = 150 read_size = 16383 ssl_error = 0 ssl_read = 0 ssl = 0x7fb75755bef0 rd_buf = "\026\003\003\000\221\001\000\000\215\003\003b\347\222\177\213T\226\b\b\214\273\355\255[l\354\211\371rJ;<\033\356B\374y\024cg\352\067\000\000\020\300\060\300/\300(\300'\300,\300+\300$\300#\001\000\000T\000\000\000\022\000\020\000\000\rdr.teamsdr.uk\000\005\000\005\001\000\000\000\000\000\n\000\006\000\004\000\030\000\027\000\v\000\002\001\000\000\r\000\024\000\022\004\001\005\001\002\001\004\003\005\003\002\003\002\002\006\001\006\003\000#\000\000\000\027\000\000\377\001\000\001\000G\225 E\030j\223\313Hb\244@\271\323\030GQ\254\272\024\032z=H-\277\360\226\035qw\244\270(\004D\004\241\023\062\064\026\226\376\321\067\274\370\376?"... wr_buf = "\026\003\003\000E\002\000\000A\003\003OHu\016\240\363=\252\356\260#\221\a\310ò&\237\000:FɆ\351DOWNGRD\001\000\300\060\000\000\031\377\001\000\001\000\000\000\000\000\000\v\000\004\003\000\001\002\000#\000\000\000\027\000\000\026\003\003\020\006\v\000\020\002\000\017\377\000\005x0\202\005t0\202\004\\240\003\002\001\002\002\022\004\062_\v\313\031\021 \240[\376N\227\346\310\004e\r0\r\006\t*\206H\206\367\r\001\001\v\005\000\060\062\061\v0\t\006\003U\004\006\023\002US1\026\060\024\006\003U\004\n\023\rLet's Encrypt1\v0\t\006\003U\004\003\023\002R30\036\027\r22070519555"... rd = {buf = 0x7ffd526c5a00 "\026\003\003", pos = 150, used = 150, size = 65536} wr = {buf = 0x7ffd526d5a00 "\026\003\003", pos = 0, used = 4096, size = 65536} tls_c = 0x7fb7574429a8 enc_rd_buf = 0x0 n = 0 flush_flags = 1 err_src = 0x7fb76612c9b0 "TLS read:" ip_buf = "172.16.51.200", '\000' <repeats 50 times> x = 0 tls_dbg = 0 __func__ = "tls_h_read_f" #19 0x000055b3daf819f3 in tcp_read_headers (c=0x7fb7576db4e8, read_flags=0x7ffd526e5dac) at core/tcp_read.c:441 bytes = 4 remaining = 0 p = 0x7fb75736b870 "2" r = 0x7fb7576db610 mc = 32695 body_len = 22326 mfline = 0x7fb755b9dadd <futex_release+29> "\211E\374\203}\374\002\017\224\300\017\266\300H\205\300t6H\213E\350H\203\354\bj" mtransid = {s = 0x55b3db16d884 "", len = 1439304448} __func__ = "tcp_read_headers" #20 0x000055b3daf8ac9d in tcp_read_req (con=0x7fb7576db4e8, bytes_read=0x7ffd526e5da8, read_flags=0x7ffd526e5dac) at core/tcp_read.c:1469 bytes = -1 total_bytes = 0 resp = 1 size = 154618822664 req = 0x7fb7576db610 dst = {send_sock = 0x1, to = {s = {sa_family = 2816, sa_data = "\312U\267\177\000\000\320\nR\375\177\000"}, sin = {sin_family = 2816, sin_port = 21962, sin_addr = {s_addr = 32695}, sin_zero = "\320\nR\375\177\000"}, sin6 = {sin6_family = 2816, sin6_port = 21962, sin6_flowinfo = 32695, sin6_addr = {__in6_u = {__u6_addr8 = "\320\nR\375\177\000\000\030\000\000\000\000\000\000", __u6_addr16 = {23760, 21102, 32765, 0, 24, 0, 0, 0}, __u6_addr32 = {1382964432, 32765, 24, 0}}}, sin6_scope_id = 1382964400}, sas = {ss_family = 2816, __ss_padding = "\312U\267\177\000\000\320\nR\375\177\000\000\030\000\000\000\000\000\000\000\260\nR\001\000\000\000\000\v\312U\267\177\000\000\260\nR\375\177\000\000\335ڹU\267\177\000\000\204\330\026۳U\000\000\000\v\312U\267\177\000\000\002\000\000\000\000\000\000\000\002\000\000\000\001\000\000\000\360\nR\375\177\000\000!\177\272U\267\177\000\000\360]nR\375\177\000\000\024]nR\375\177\000", __ss_align = 17179869204}}, id = -619259776, send_flags = {f = 21939, blst_imask = 0}, proto = 110 'n', proto_pad0 = -11 '\365', proto_pad1 = 28435} c = 0 '\000' ret = 1438243549 __func__ = "tcp_read_req" #21 0x000055b3daf9072c in handle_io (fm=0x7fb766755bb8, events=1, idx=-1) at core/tcp_read.c:1780 ret = 8 n = 8 read_flags = RD_CONN_SHORT_READ con = 0x7fb7576db4e8 s = 12 resp = 140425703179696 t = 1467216201 ee = 0x0 __func__ = "handle_io" #22 0x000055b3daf7bd70 in io_wait_loop_epoll (h=0x55b3db270960 <io_w>, t=2, repeat=0) at core/io_wait.h:1070 n = 1 r = 0 fm = 0x7fb766755bb8 revents = 1 __func__ = "io_wait_loop_epoll" #23 0x000055b3daf93760 in tcp_receive_loop (unix_sock=36) at core/tcp_read.c:1976 __func__ = "tcp_receive_loop" #24 0x000055b3daf6fdde in tcp_init_children (woneinit=0x7ffd526e6198) at core/tcp_main.c:5236 r = 0 i = 7 reader_fd_1 = 36 pid = 0 si_desc = "tcp receiver (generic)\000\000\356\376\357ڳU\000\000panR\375\177\000\000\036\350\371\332\000\000\000\000\070\n\037f\000\000\000\000\313\307\025۳U\000\000\241\000nR\375\177\000\000\fj\f۳U\000\000\000\000\000\000\000\000\000\000ز\277U\267\177\000\000\036\000\000\000\000\000\000\000\020\316\363U\001\000\000\000`anR\375\177\000\000\036l\f۳U\000" si = 0x0 __func__ = "tcp_init_children" #25 0x000055b3dacbe65c in main_loop () at main.c:1851 i = 4 pid = 7714 si = 0x0 si_desc = "udp receiver child=3 sock=127.0.0.1:5050\000\060\066\060\000\000\000\000\003\000\000\000\036\000\000\000\000/a\230\356U\261\264X\307\036۳U\000\000\210\360\020۳U\000\000\000\000\000\000\000\000\000\000s\\022۳U\000\000\036\000\000\000\000\000\000\000\260\305'h\267\177\000\000\060cnR\375\177\000\000zI\347ڳU\000" nrprocs = 4 woneinit = 1 __func__ = "main_loop" #26 0x000055b3dacc95b0 in main (argc=11, argv=0x7ffd526e68a8) at main.c:3085 cfg_stream = 0x55b3dc09c380 c = -1 r = 0 tmp = 0x7ffd526e7e37 "" tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x55b3db112278 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1456913526 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7fb768491cee <_dl_runtime_resolve_xsave+142> p = 0x7ffd526e68a8 "\334}nR\375\177" st = {st_dev = 25, st_ino = 1712, st_nlink = 2, st_mode = 16892, st_uid = 9002, st_gid = 9002, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1659341338, tv_nsec = 499520451}, st_mtim = {tv_sec = 1659341338, tv_nsec = 499520451}, st_ctim = {tv_sec = 1659341338, tv_nsec = 499520451}, __glibc_reserved = {0, 0, 0}} tbuf = '\000' <repeats 64 times>, "\377\000\000\000\000\000\000\000\000\377", '\000' <repeats 18 times>, "\377", '\000' <repeats 19 times>, "`", '\000' <repeats 15 times>, "\001", '\000' <repeats 145 times>... option_index = 0 long_options = {{name = 0x55b3db1146b6 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x55b3db10f521 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x55b3db1146bb "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x55b3db1146c1 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x55b3db1146c7 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x55b3db1146d0 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x55b3db1146da "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x55b3db1146e4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x55b3db1146ef "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x55b3db1146f8 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x55b3db114703 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x55b3db114709 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x55b3db114713 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x55b3db11471a "all-errors", has_arg = 0, flag = 0x0, val = 1035}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" (gdb) info locals tid = <optimized out> ret = 0 pd = 0x7fb76815b740 old_mask = {__val = {523986010119, 910533066753, 1, 140425395047168, 94230980435360, 0, 5677797456, 140425395047168, 140725986283600, 140425393986269, 94230963279192, 140425395047168, 94230963279402, 7970739754, 140725986283664, 140425394028321}} ret = <optimized out> pd = <optimized out> old_mask = <optimized out> ret = <optimized out> tid = <optimized out> ret = <optimized out> resultvar = <optimized out> resultvar = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> __futex = <optimized out> resultvar = <optimized out> __arg3 = <optimized out> __arg2 = <optimized out> __arg1 = <optimized out> _a3 = <optimized out> _a2 = <optimized out> _a1 = <optimized out> __futex = <optimized out> __private = <optimized out> __oldval = <optimized out> result = <optimized out> (gdb) list 39 in ./nptl/pthread_kill.cq ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` Aug 1 12:21:45 kam-test01 /usr/local/sbin/kamailio[25442]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized Aug 1 12:21:47 kam-test01 kamailio: INFO: <core> [core/sctp_core.c:74]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module Aug 1 12:21:47 kam-test01 kamailio: INFO: <core> [core/tcp_main.c:5080]: init_tcp(): using epoll_lt as the io watch method (auto detected) Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_mod.c:399]: mod_init(): use OpenSSL version: 30000020 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_mod.c:401]: mod_init(): With ECDH-Support! Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_mod.c:404]: mod_init(): With Diffie Hellman Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: auth [auth_mod.c:345]: mod_init(): qop set, but nonce-count (nonce_count) support disabled Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: rr [rr_mod.c:188]: mod_init(): outbound module not available Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: db_text [dbt_lib.c:146]: dbt_cache_get_db(): using database at: /etc/kamailio/dbtext Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: WARNING: tls [tls_init.c:804]: tls_h_mod_init_f(): tls: openssl bug #1491 (crash/mem leaks on low memory) workaround disabled Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: <core> [main.c:3062]: main(): processes (at least): 30 - shm size: 134217728 - pkg size: 33554432 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 for fd 9 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: <core> [core/udp_server.c:207]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 on fd 9 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 for fd 10 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: <core> [core/udp_server.c:207]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 on fd 10 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 for fd 11 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: <core> [core/udp_server.c:207]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 on fd 11 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:328]: ksr_tls_fill_missing(): TLSs<default>: tls_method=20 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:340]: ksr_tls_fill_missing(): TLSs<default>: certificate='/etc/kamailio/livecert/fullchain.pem' Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:347]: ksr_tls_fill_missing(): TLSs<default>: ca_list='/etc/kamailio/livecert/chain.pem' Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:354]: ksr_tls_fill_missing(): TLSs<default>: ca_path='(null)' Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:361]: ksr_tls_fill_missing(): TLSs<default>: crl='(null)' Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:364]: ksr_tls_fill_missing(): TLSs<default>: require_certificate=0 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:372]: ksr_tls_fill_missing(): TLSs<default>: cipher_list='AES256-SHA' Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:379]: ksr_tls_fill_missing(): TLSs<default>: private_key='/etc/kamailio/livecert/privkey.pem' Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:382]: ksr_tls_fill_missing(): TLSs<default>: verify_certificate=0 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:386]: ksr_tls_fill_missing(): TLSs<default>: verify_depth=9 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:389]: ksr_tls_fill_missing(): TLSs<default>: verify_client=0 Aug 1 12:21:47 kam-test01 /usr/local/sbin/kamailio[25502]: NOTICE: tls [tls_domain.c:1144]: ksr_tls_fix_domain(): registered server_name callback handler for socket [:0], server_name='<default>' ... Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:750]: set_verification(): TLSs<default>: No client certificate required and no checks performed Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:328]: ksr_tls_fill_missing(): TLSc<default>: tls_method=18 Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:340]: ksr_tls_fill_missing(): TLSc<default>: certificate='/etc/kamailio/livecert/fullchain.pem' Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:347]: ksr_tls_fill_missing(): TLSc<default>: ca_list='/etc/kamailio/livecert/chain.pem' Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:354]: ksr_tls_fill_missing(): TLSc<default>: ca_path='(null)' Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:361]: ksr_tls_fill_missing(): TLSc<default>: crl='(null)' Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:364]: ksr_tls_fill_missing(): TLSc<default>: require_certificate=0 Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:372]: ksr_tls_fill_missing(): TLSc<default>: cipher_list='ECDHE-RSA-AES256-GCM-SHA384' Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:379]: ksr_tls_fill_missing(): TLSc<default>: private_key='/etc/kamailio/livecert/privkey.pem' Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:382]: ksr_tls_fill_missing(): TLSc<default>: verify_certificate=0 Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:386]: ksr_tls_fill_missing(): TLSc<default>: verify_depth=9 Aug 1 12:21:49 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:389]: ksr_tls_fill_missing(): TLSc<default>: verify_client=0 Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:753]: set_verification(): TLSc<default>: Server MAY present invalid certificate Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:328]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: tls_method=22 Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:340]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: certificate='/etc/kamailio/livecert/fullchain.pem' Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:347]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: ca_list='/etc/kamailio/livecert/chain.pem' Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:354]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: ca_path='(null)' Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:361]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: crl='(null)' Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:364]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: require_certificate=0 Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:372]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: cipher_list='ECDHE-RSA-AES256-GCM-SHA384' Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:379]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: private_key='/etc/kamailio/livecert/privkey.pem' Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:382]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_certificate=0 Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:386]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_depth=9 Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:389]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_client=0 Aug 1 12:21:50 kam-test01 /usr/local/sbin/kamailio[25502]: NOTICE: tls [tls_domain.c:1144]: ksr_tls_fix_domain(): registered server_name callback handler for socket [172.16.51.200:5067], server_name='test2.test.com' ... Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:750]: set_verification(): TLSs<172.16.51.200:5067>: No client certificate required and no checks performed Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:328]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: tls_method=22 Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:340]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: certificate='/etc/kamailio/livecert/fullchain.pem' Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:347]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: ca_list='/etc/kamailio/livecert/chain.pem' Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:354]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: ca_path='(null)' Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:361]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: crl='(null)' Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:364]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: require_certificate=0 Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:372]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: cipher_list='ECDHE-RSA-AES256-GCM-SHA384' Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:379]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: private_key='/etc/kamailio/livecert/privkey.pem' Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:382]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_certificate=0 Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:386]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_depth=9 Aug 1 12:21:51 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:389]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_client=0 Aug 1 12:21:52 kam-test01 /usr/local/sbin/kamailio[25502]: NOTICE: tls [tls_domain.c:1144]: ksr_tls_fix_domain(): registered server_name callback handler for socket [172.16.51.200:5067], server_name='test.test.com' ... Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:750]: set_verification(): TLSs<172.16.51.200:5067>: No client certificate required and no checks performed Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:328]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: tls_method=22 Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:340]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: certificate='/etc/kamailio/livecert/fullchain.pem' Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:347]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: ca_list='/etc/kamailio/livecert/chain.pem' Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:354]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: ca_path='(null)' Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:361]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: crl='(null)' Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:364]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: require_certificate=0 Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:372]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: cipher_list='ECDHE-RSA-AES256-GCM-SHA384' Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:379]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: private_key='/etc/kamailio/livecert/privkey.pem' Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:382]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_certificate=0 Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:386]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_depth=9 Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:389]: ksr_tls_fill_missing(): TLSs<172.16.51.200:5067>: verify_client=0 Aug 1 12:21:53 kam-test01 /usr/local/sbin/kamailio[25502]: NOTICE: tls [tls_domain.c:1144]: ksr_tls_fix_domain(): registered server_name callback handler for socket [172.16.51.200:5067], server_name='test3.test.com' ... Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:750]: set_verification(): TLSs<172.16.51.200:5067>: No client certificate required and no checks performed Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:328]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: tls_method=18 Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:340]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: certificate='/etc/kamailio/livecert/fullchain.pem' Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:347]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: ca_list='/etc/kamailio/livecert/chain.pem' Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:354]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: ca_path='(null)' Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:361]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: crl='(null)' Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:364]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: require_certificate=0 Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:372]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: cipher_list='ECDHE-RSA-AES256-GCM-SHA384' Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:379]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: private_key='/etc/kamailio/livecert/privkey.pem' Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:382]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: verify_certificate=0 Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:386]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: verify_depth=9 Aug 1 12:21:54 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:389]: ksr_tls_fill_missing(): TLSc<172.16.51.200:5067>: verify_client=0 Aug 1 12:21:56 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: tls [tls_domain.c:753]: set_verification(): TLSc<172.16.51.200:5067>: Server MAY present invalid certificate Aug 1 12:21:56 kam-test01 /usr/local/sbin/kamailio[25504]: INFO: rtpengine [rtpengine.c:2929]: rtpp_test(): rtpengine instance udp:127.0.0.1:2223 found, support for it enabled Aug 1 12:21:56 kam-test01 /usr/local/sbin/kamailio[25520]: INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/25520 Aug 1 12:21:56 kam-test01 /usr/local/sbin/kamailio[25523]: INFO: ctl [io_listener.c:213]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) Aug 1 12:21:57 kam-test01 /usr/local/sbin/kamailio[25525]: ERROR: <script>: 758:No dispatcher sets up yet Aug 1 12:21:57 kam-test01 /usr/local/sbin/kamailio[25525]: ERROR: <script>: 758:No dispatcher sets up yet Aug 1 12:21:57 kam-test01 /usr/local/sbin/kamailio[25525]: WARNING: <script>: 548:[LOCATION] Location not found rc: -1 sip:lPBKm2s70BBd6u@sipuat.voicesimplified.com;transport=TLS Aug 1 12:21:57 kam-test01 /usr/local/sbin/kamailio[25525]: ERROR: <script>: 758:No dispatcher sets up yet Aug 1 12:21:59 kam-test01 /usr/local/sbin/kamailio[25532]: ERROR: <core> [core/tcp_main.c:4672]: tcpconn_main_timeout(): connect 185.29.208.21:5061 failed (timeout) Aug 1 12:22:06 kam-test01 /usr/local/sbin/kamailio[25508]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:06 kam-test01 /usr/local/sbin/kamailio[25510]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:06 kam-test01 /usr/local/sbin/kamailio[25511]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:06 kam-test01 /usr/local/sbin/kamailio[25508]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:06 kam-test01 /usr/local/sbin/kamailio[25504]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:06 kam-test01 /usr/local/sbin/kamailio[25505]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:06 kam-test01 /usr/local/sbin/kamailio[25504]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:07 kam-test01 /usr/local/sbin/kamailio[25506]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:07 kam-test01 /usr/local/sbin/kamailio[25505]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:07 kam-test01 /usr/local/sbin/kamailio[25506]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:07 kam-test01 /usr/local/sbin/kamailio[25507]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:07 kam-test01 /usr/local/sbin/kamailio[25506]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:22:07 kam-test01 /usr/local/sbin/kamailio[25506]: NOTICE: <script>: Destination up: sip:redacted Aug 1 12:24:48 kam-test01 /usr/local/sbin/kamailio[25525]: INFO: <script>: R-URI Alias Found and processed Aug 1 12:24:48 kam-test01 /usr/local/sbin/kamailio[25510]: INFO: <script>: R-URI Alias Found and processed Aug 1 12:26:06 kam-test01 /usr/local/sbin/kamailio[25513]: WARNING: tm [t_reply.c:1677]: t_retransmit_reply(): no resolved dst to retransmit Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25526]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f7a8a860d80), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25532]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 36 Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25502]: ALERT: <core> [main.c:776]: handle_sigs(): child process 25526 exited by a signal 11 Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25502]: ALERT: <core> [main.c:779]: handle_sigs(): core was generated Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25502]: INFO: <core> [main.c:801]: handle_sigs(): terminating due to SIGCHLD Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25532]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25513]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25531]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25522]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25521]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25525]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25520]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25514]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25529]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25511]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25515]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25509]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25519]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25516]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25517]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25512]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25510]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25506]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25504]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25524]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25518]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25523]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25508]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25528]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25530]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25505]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25507]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received Aug 1 12:27:17 kam-test01 /usr/local/sbin/kamailio[25527]: INFO: <core> [main.c:856]: sig_usr(): signal 15 received ```
### Additional Information Tested against kamailio 5.6 and devel branches with a distclean between builds
``` (paste your output here) ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) -->
``` No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy
Linux kam-test01 5.15.0-1014-azure #17-Ubuntu SMP Thu Jun 23 19:04:16 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ```
Seems to only effect Ubuntu 22, 20lts doesn't exhibit same behaviour using identical build process
I have the same problem on Ubuntu 22.04.1 LTS
@nakchak just to understand it correctly, you are saying that you observe the same problem with TLS module and also with TLSA module on ubuntu 22.04? You are seeing also the same issue with OpenSSL 3.0.5 and OpenSSL 1.1.1q locally compiled on the ubuntu 22.04 base system?
Ubuntu 22.04.1 LTS clean install. openssl 3.0.2-0ubuntu1.6 kamailio 5.5.4-1
frequent log entries: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f2314082e68), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring
@henningw Yes that is correct, i have tested with clean installs of Ubuntu 22.04 (deleting the virtual disk, running the installer iso, then running an `apt full-upgrade` on the system and rebooting) with the following tested configurations having the same result:
- Distribution default OpenSSL packages and builds of the 5.5,5.6 and master branches with both TLS and TLSA modules
- OpenSSL1.1.1q compiled from source on vm, built 5.6 branch using TLSA configured to use the build folder of OpenSSL as the lib search folder
- OpenSSL3.0.5 compiled from source on vm, built 5.6 branch using TLSA configured to use the build folder of OpenSSL as the lib search folder
No issues what so over when using Ubuntu 20.04 as the base system.
Its not the end of the world for us, but a surprising result none the less given its an LTS release of the OS and could cause chaos for anyone presumably running `apt dist-upgrade`, have had numerous problems with 22.04 and builds failing due to issues with the packeges in the official repositories (such as this one with rtp engine https://github.com/sipwise/rtpengine/issues/1497#event-6977101165)
Thanks for the update. Its a bit surprising indeed. Thanks for the link for the rtpengine issue as well.
Just guessing, but this issue might be some specific build options on the new ubuntu with OpenSSL, maybe caused from upgrades in the compiler/library or other user-land software. At least it seems to be easily reproducable, given the frequency that its happens.
@henningw im thinking similar things, i havnt tried trying using the previous releases versions of gcc, g++ make or autoconf i might try that and see what effect (if any) that has to the build. Other than the known OpenSSL API Deprecation warnings in the build output nothing seems amiss when building... I dont think its anything to do with pkg-config (had issues originally with tlsa when it was first available in devel to do with pkg-config supplied paths [https://github.com/kamailio/kamailio/issues/2815] as i get the same results when using a locally compiled build of openssl)
Running into the same issue. Did a dist-upgrade from Ubuntu 20.04.3 with Kamailio 5.5.2 to 22.04.1 with Kamailio 5.5.4. TLS module enabled.
Have you tried with `--atexit=no`?
- https://www.kamailio.org/docs/modules/stable/modules/tls.html#tls.overview
I haven’t. Thanks for the hint. I can’t do that test this week but will do it next week and come back with the result.
I will try that in next day or so and report back
Any updates on this one?
Finally got to test this again. Did the upgrade from Ubuntu 20.04 to 22.04 but took a slightly different approach this time. I upgraded Kamailio to 5.5.4 while still on Ubuntu 20.04 and saw that it got started with --atexit=no. After upgrading to Ubuntu 22.04 --atexit=no is still there and I don't see anymore log entries about "freeing already freed pointer". Also Kamailio is not crashing anymore.
I'm not running that system in production yet, but it looks like the error is gone for me.
Closed #3205 as completed.
I am using Kamailio 5.7.6 and keep seeing this issue on my system where the server crashes with either signal 11 or signal 6 immediately after seeing this error:
``` xxx xx 18:51:15 xxxx kamailio[28091]: 28(29) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x73d5733c4f08), called from tls: tls_init.c: ser_free(399), first free tls: tls_init.c: ser_free(399) - ignoring ```
I tried running the server with `--atexit=no` and that did not seem to solve the problem.
I have added the TLS parameter `init_mode` to my configuration:
``` modparam("tls", "init_mode", 3) ```
So far this seems to get better behavior.
If it works stable, its good. Another suggestion is to not use the (older) tls module init_mode parameter facility, but give the newer core parameter tls_threads_mode a try. Recommended for production is the value 2. This is the recommended way to go especially if you are running on OpenSSL 3.x library versions on newer distributions.
Setting `init_mode` did not solve the problem for me. During production load today, we still got many `ser_free` double free reports. Kamailio does not crash everytime there is this error in the logs, sometimes after 7 or 8 occurrences (over 15 minutes or so).
I'll try to set `tls_threads_mode`.
So far, 24 hours in, it looks like `tls_threads_mode=2` solve the problem for me.
Thanks for the feedback, hopefully its solved then. One reason the tls_thread_mode is a global parameter is that OpenSSL is also used from other modules (database drivers, HTTP API etc..). So it needs to apply also to them.
> init_mode
> So far, 24 hours in, it looks like `tls_threads_mode=2` solve the problem for me.
I found that `tls_threads_mode` is a parameter introduced in Kamailio 5.8.x. Have you upgraded Kamailio?