[kamailio/kamailio] Kamailio 5.4.1 - Crash on startup when TLS is enabled after upgrading from stretch to buster (#2466)
<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description
After upgrading Debian from stretch to buster, kamailio service keeps crashing on service stat. Same kamailio configuration file worked fine on Debian stretch. If I disable TLS the service starts successfully.
### Troubleshooting
Looks like a TLS issue
#### Reproduction
Debian buster TLS enabled
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` root@kamailio11:/# gdb /usr/sbin/kamailio /core GNU gdb (Debian 8.2.1-2+b3) 8.2.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/ed/15dbbb24fb309311e7a6a16cf9222a82afb102.debug...done. done. [New LWP 7259] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f17c0877e26 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (gdb) bt full #0 0x00007f17c0877e26 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #1 0x00007f17c087812f in OPENSSL_cleanup () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #2 0x00007f17c46aad8c in __run_exit_handlers (status=1, listp=0x7f17c482c718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 atfct = <optimized out> onfct = <optimized out> cxafct = <optimized out> f = <optimized out> new_exitfn_called = 3 cur = 0x7f17c482dd80 <initial> #3 0x00007f17c46aaeba in __GI_exit (status=<optimized out>) at exit.c:139 No locals. #4 0x000055c479a9c502 in handle_sigs () at main.c:808 chld = 0 chld_status = 139 any_chld_stopped = 1 memlog = 0 __func__ = "handle_sigs" #5 0x000055c479aa8053 in main_loop () at main.c:1817 i = 8 pid = 7301 si = 0x0 si_desc = "udp receiver child=7 sock=172.19.140.101:5060\000\000\000\003\000\000\000+\000\000\000\000\253pP\333\351\322a\320-\221z\304U\000\000\210`\354y\304U\000\000\000\000\000\000\000\000\000\000\260Lv\304\027\177\000\000\363N\361y\304U\000\000\000\000\000\200\000\000\000\000\060P\254$\375\177\000\000hm\322y\304U\000" nrprocs = 8 woneinit = 1 __func__ = "main_loop" #6 0x000055c479ab1516 in main (argc=9, argv=0x7ffd24ac5578) at main.c:2856 cfg_stream = 0x55c47a82b260 c = -1 r = 0 tmp = 0x7ffd24ac6de3 "" tmp_len = 32535 port = -995964432 proto = 32535 ahost = 0x0 aport = 0 options = 0x55c479ec8d38 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" --Type <RET> for more, q to quit, c to continue without paging-- ret = -1 seed = 1032197731 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0x0 st = {st_dev = 21, st_ino = 79137, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1599027572, tv_nsec = 735090396}, st_mtim = {tv_sec = 1599027608, tv_nsec = 820311152}, st_ctim = {tv_sec = 1599027608, tv_nsec = 820311152}, __glibc_reserved = {0, 0, 0}} tbuf = "\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\060ע\304\027\177\000\000\000\000\000\000\000\000\000\000\240A\271$\375\177\000\000\364ڠ\304\027\177\000\000\b\000\000\000\000\000\000\000\230B\271$\375\177\000\000\060ע\304\027\177\000\000\030S\254$\375\177\000\000\024S\254$\375\177", '\000' <repeats 18 times>, "\230B\271$\375\177\000\000\250A\271$\375\177\000\000'\036\177\304\027\177\000\000&\260be\000\000\000\000\200T\254$\375\177\000\000\300\212\225\001\000\000\000\000\340S\254$\375\177\000\000\320S\254$\375\177\000\000\030S\254$\375\177\000\000Xע\304\027\177\000\000\001", '\000' <repeats 31 times>... option_index = 0 long_options = {{name = 0x55c479ecadc6 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x55c479ec650c "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x55c479ecadcb "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x55c479ecadd1 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x55c479ecadd7 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x55c479ecade0 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x55c479ecadea "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x55c479ecadf4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x55c479ecadff "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x55c479ecae08 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x55c479ecae13 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` Sep 2 06:23:10 kamailio11 kamailio: INFO: <core> [core/tcp_main.c:4982]: init_tcp(): using epoll_lt as the io watch method (auto detected) Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: rr [rr_mod.c:185]: mod_init(): outbound module not available Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: dmq_usrloc [dmq_usrloc.c:81]: mod_init(): dmq usrloc replication mode = 1 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: cfgutils [cfgutils.c:869]: mod_init(): no hash_file given, disable hash functionality Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [http_async_client_mod.c:222]: mod_init(): Initializing Http Async module Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: auth [auth_mod.c:345]: mod_init(): qop set, but nonce-count (nonce_count) support disabled Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: permissions [permissions.c:601]: mod_init(): default allow file (/etc/kamailio/permissions.allow) not found => empty rule set Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: permissions [permissions.c:615]: mod_init(): default deny file (/etc/kamailio/permissions.deny) not found => empty rule set Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: pike [pike.c:97]: pike_init(): PIKE - initializing Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: pike [ip_tree.c:81]: init_lock_set(): probing 256 set size Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_mod.c:389]: mod_init(): With ECDH-Support! Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_mod.c:392]: mod_init(): With Diffie Hellman Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: WARNING: tls [tls_init.c:792]: tls_h_mod_init_f(): openssl bug #1491 (crash/mem leaks on low memory) workaround enabled (on low memory tls operations will fail preemptively) with free memory thresholds 22544384 and 11272192 bytes Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: <core> [core/cfg/cfg_ctx.c:595]: cfg_set_now(): tls.low_mem_threshold1 has been changed to 22544384 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: <core> [core/cfg/cfg_ctx.c:595]: cfg_set_now(): tls.low_mem_threshold2 has been changed to 11272192 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: <core> [main.c:2833]: main(): processes (at least): 43 - shm size: 2147483648 - pkg size: 33554432 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [async_http.c:101]: async_http_init_sockets(): inter-process event notification sockets initialized Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [async_http.c:101]: async_http_init_sockets(): inter-process event notification sockets initialized Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [async_http.c:101]: async_http_init_sockets(): inter-process event notification sockets initialized Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [async_http.c:101]: async_http_init_sockets(): inter-process event notification sockets initialized Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:309]: ksr_tls_fill_missing(): TLSs<default>: tls_method=22 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:321]: ksr_tls_fill_missing(): TLSs<default>: certificate='/etc/kamailio/ssl/widebridge.pem' Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:328]: ksr_tls_fill_missing(): TLSs<default>: ca_list='/etc/kamailio/ssl/cacert.pem' Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:335]: ksr_tls_fill_missing(): TLSs<default>: crl='(null)' Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:339]: ksr_tls_fill_missing(): TLSs<default>: require_certificate=0 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:346]: ksr_tls_fill_missing(): TLSs<default>: cipher_list='HIGH' Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:353]: ksr_tls_fill_missing(): TLSs<default>: private_key='/etc/kamailio/ssl/widebridge.key' Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:357]: ksr_tls_fill_missing(): TLSs<default>: verify_certificate=0 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:360]: ksr_tls_fill_missing(): TLSs<default>: verify_depth=9 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:363]: ksr_tls_fill_missing(): TLSs<default>: verify_client=0 Sep 2 06:23:10 kamailio11 /usr/sbin/kamailio[7354]: NOTICE: tls [tls_domain.c:1109]: ksr_tls_fix_domain(): registered server_name callback handler for socket [:0], server_name='<default>' ... Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:716]: set_verification(): TLSs<default>: No client certificate required and no checks performed Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:309]: ksr_tls_fill_missing(): TLSc<default>: tls_method=22 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:321]: ksr_tls_fill_missing(): TLSc<default>: certificate='(null)' Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:328]: ksr_tls_fill_missing(): TLSc<default>: ca_list='(null)' Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:335]: ksr_tls_fill_missing(): TLSc<default>: crl='(null)' Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:339]: ksr_tls_fill_missing(): TLSc<default>: require_certificate=0 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:346]: ksr_tls_fill_missing(): TLSc<default>: cipher_list='(null)' Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:353]: ksr_tls_fill_missing(): TLSc<default>: private_key='(null)' Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:357]: ksr_tls_fill_missing(): TLSc<default>: verify_certificate=0 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:360]: ksr_tls_fill_missing(): TLSc<default>: verify_depth=9 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:363]: ksr_tls_fill_missing(): TLSc<default>: verify_client=0 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: tls [tls_domain.c:719]: set_verification(): TLSc<default>: Server MAY present invalid certificate Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7355]: INFO: rtpengine [rtpengine.c:2781]: rtpp_test(): rtpengine instance udp:127.0.0.1:22222 found, support for it enabled Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [async_http.c:84]: async_http_init_worker(): started worker process: 1 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7367]: INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/7367 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [async_http.c:84]: async_http_init_worker(): started worker process: 2 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [async_http.c:84]: async_http_init_worker(): started worker process: 3 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: http_async_client [async_http.c:84]: async_http_init_worker(): started worker process: 4 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7371]: INFO: ctl [io_listener.c:214]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7376]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x645f6d6574737973 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7375]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x645f6d6574737973 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: ALERT: <core> [main.c:776]: handle_sigs(): child process 7375 exited by a signal 11 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: ALERT: <core> [main.c:779]: handle_sigs(): core was not generated Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: ALERT: <core> [main.c:776]: handle_sigs(): child process 7376 exited by a signal 11 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: ALERT: <core> [main.c:779]: handle_sigs(): core was not generated Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7354]: INFO: <core> [main.c:802]: handle_sigs(): terminating due to SIGCHLD Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7382]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7385]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7384]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7377]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7383]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7380]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7381]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7368]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7379]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7366]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7378]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7364]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7374]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7362]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7373]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7360]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7372]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7358]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7371]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7356]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7370]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7387]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7369]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7388]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7367]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7386]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7365]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7390]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7363]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7395]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7361]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7393]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7359]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7357]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7355]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7389]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7394]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7392]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7391]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 23 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 24 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 25 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 26 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 27 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 28 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 29 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 30 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 31 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 32 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 34 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 35 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 5 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 36 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 9 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 40 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 44 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 48 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 52 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 53 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 54 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 55 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 56 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 57 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 59 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 61 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 63 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 65 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 67 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 69 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 71 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 73 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 75 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 77 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 79 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 81 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 83 Sep 2 06:23:11 kamailio11 /usr/sbin/kamailio[7396]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 85 Sep 2 06:23:15 kamailio11 /usr/sbin/kamailio[7396]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 2 06:23:15 kamailio11 /usr/sbin/kamailio[7354]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized ```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` root@kamailio11:/etc/kamailio# kamailio -v version: kamailio 5.4.1 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 8.3.0 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) -->
``` root@kamailio11:/etc/kamailio# cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"
Linux kamailio11.dev.wb.internal 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux ```
I'm having the same issue after upgrading our kamailio to 5.4.1
``` [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.lb.pid -f /etc/kamailio/lb/kamaili'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f00fe4aee26 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (gdb) bt #0 0x00007f00fe4aee26 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #1 0x00007f00fe4af12f in OPENSSL_cleanup () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 #2 0x00007f010235ed8c in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007f010235eeba in exit () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x000055cfea78a745 in handle_sigs () at main.c:810 #5 0x000055cfea790525 in main_loop () at main.c:1817 #6 0x000055cfea78591b in main (argc=<optimized out>, argv=<optimized out>) at main.c:2856 ```
Can you try again after reverting the next commits?
* 60bd3aa284a5d9eb117a9763aa590aecf276ba16 * 99c6a68dea60ca745dfdfa33085bb6acbcee846a * 234b02236b3ad13cdaf5624d11c727ad7d804747
They were related to cleaning up obsoleted libssl api, but maybe there are unexpected side effects.
The commit ids from previous comment were for master branch, for 5.4 branch are:
* e9baba91c6c7e3d1817e8a07bbabbce3a8efbe79 * 67979e07b63b574687b1924c38fabd724d0ad876 * 10af8efa9061c2f52ee1db51c4cf665073c20dcc
reverting those three commits, seems to help. I can't reproduce the crash now.
@linuxmaniac you can revert them in the repo, so we get the git branches (master & 5.4) and (soon) nightly builds ok.
Later I will have another look at those patches and see if I spot what breaks there.
reverted on 5.4 and master
Still crashes for me. I installed Kamailio v5.4.2 nightly build
kamailio-dbg/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-extra-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-json-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-lua-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-mysql-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-redis-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-systemd-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-tls-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-utils-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio-websocket-modules/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed] kamailio/unknown,now 5.4.2~bpo10.20200903013416.27 amd64 [installed]
**Kamailio log:**
Sep 3 07:04:40 kamailio12 kamailio: INFO: <core> [core/tcp_main.c:4982]: init_tcp(): using epoll_lt as the io watch method (auto detected) Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: rr [rr_mod.c:185]: mod_init(): outbound module not available Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: dmq_usrloc [dmq_usrloc.c:81]: mod_init(): dmq usrloc replication mode = 1 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: cfgutils [cfgutils.c:869]: mod_init(): no hash_file given, disable hash functionality Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [http_async_client_mod.c:222]: mod_init(): Initializing Http Async module Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: auth [auth_mod.c:345]: mod_init(): qop set, but nonce-count (nonce_count) support disabled Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: permissions [permissions.c:601]: mod_init(): default allow file (/etc/kamailio/permissions.allow) not found => empty rule set Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: permissions [permissions.c:615]: mod_init(): default deny file (/etc/kamailio/permissions.deny) not found => empty rule set Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: pike [pike.c:97]: pike_init(): PIKE - initializing Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: pike [ip_tree.c:81]: init_lock_set(): probing 256 set size Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_mod.c:389]: mod_init(): With ECDH-Support! Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_mod.c:392]: mod_init(): With Diffie Hellman Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: WARNING: tls [tls_init.c:784]: tls_h_mod_init_f(): openssl bug #1491 (crash/mem leaks on low memory) workaround enabled (on low memory tls operations will fail preemptively) with free memory thresholds 22544384 and 11272192 bytes Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: <core> [core/cfg/cfg_ctx.c:595]: cfg_set_now(): tls.low_mem_threshold1 has been changed to 22544384 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: <core> [core/cfg/cfg_ctx.c:595]: cfg_set_now(): tls.low_mem_threshold2 has been changed to 11272192 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: <core> [main.c:2833]: main(): processes (at least): 43 - shm size: 2147483648 - pkg size: 33554432 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [async_http.c:101]: async_http_init_sockets(): inter-process event notification sockets initialized Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [async_http.c:101]: async_http_init_sockets(): inter-process event notification sockets initialized Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [async_http.c:101]: async_http_init_sockets(): inter-process event notification sockets initialized Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [async_http.c:101]: async_http_init_sockets(): inter-process event notification sockets initialized Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:305]: ksr_tls_fill_missing(): TLSs<default>: tls_method=18 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:317]: ksr_tls_fill_missing(): TLSs<default>: certificate='/etc/kamailio/ssl/widebridge.pem' Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:324]: ksr_tls_fill_missing(): TLSs<default>: ca_list='/etc/kamailio/ssl/cacert.pem' Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:331]: ksr_tls_fill_missing(): TLSs<default>: crl='(null)' Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:335]: ksr_tls_fill_missing(): TLSs<default>: require_certificate=0 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:342]: ksr_tls_fill_missing(): TLSs<default>: cipher_list='HIGH' Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:349]: ksr_tls_fill_missing(): TLSs<default>: private_key='/etc/kamailio/ssl/widebridge.key' Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:353]: ksr_tls_fill_missing(): TLSs<default>: verify_certificate=0 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:356]: ksr_tls_fill_missing(): TLSs<default>: verify_depth=9 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:359]: ksr_tls_fill_missing(): TLSs<default>: verify_client=0 Sep 3 07:04:40 kamailio12 /usr/sbin/kamailio[24676]: NOTICE: tls [tls_domain.c:1105]: ksr_tls_fix_domain(): registered server_name callback handler for socket [:0], server_name='<default>' ... Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:712]: set_verification(): TLSs<default>: No client certificate required and no checks performed Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:305]: ksr_tls_fill_missing(): TLSc<default>: tls_method=18 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:317]: ksr_tls_fill_missing(): TLSc<default>: certificate='(null)' Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:324]: ksr_tls_fill_missing(): TLSc<default>: ca_list='(null)' Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:331]: ksr_tls_fill_missing(): TLSc<default>: crl='(null)' Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:335]: ksr_tls_fill_missing(): TLSc<default>: require_certificate=0 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:342]: ksr_tls_fill_missing(): TLSc<default>: cipher_list='(null)' Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:349]: ksr_tls_fill_missing(): TLSc<default>: private_key='(null)' Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:353]: ksr_tls_fill_missing(): TLSc<default>: verify_certificate=0 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:356]: ksr_tls_fill_missing(): TLSc<default>: verify_depth=9 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:359]: ksr_tls_fill_missing(): TLSc<default>: verify_client=0 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: tls [tls_domain.c:715]: set_verification(): TLSc<default>: Server MAY present invalid certificate Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24677]: ERROR: rtpengine [rtpengine.c:2870]: send_rtpp_command(): can't send command "ping" to RTPEngine udp:127.0.0.1:22222 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24677]: ERROR: rtpengine [rtpengine.c:2771]: rtpp_test(): proxy did not respond to ping Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24689]: INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/24689 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [async_http.c:84]: async_http_init_worker(): started worker process: 1 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [async_http.c:84]: async_http_init_worker(): started worker process: 2 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [async_http.c:84]: async_http_init_worker(): started worker process: 3 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24676]: INFO: http_async_client [async_http.c:84]: async_http_init_worker(): started worker process: 4 Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24693]: INFO: ctl [io_listener.c:214]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x322e3176534c54 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0xc0c0c0c0 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0xabcdefed (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f0237b6de90), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7f0237a7a9a0 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x8 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x38 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x28 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7f02b7aef9c4 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7f02b7af0f30 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7f02b7aef9c0 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x125 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0xf0f0f0f0 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7453726568706943 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:41 kamailio12 /usr/sbin/kamailio[24698]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x676e6972 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:55 kamailio12 /usr/sbin/kamailio[24718]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 53 Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: ALERT: <core> [main.c:776]: handle_sigs(): child process 24698 exited by a signal 11 Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: ALERT: <core> [main.c:779]: handle_sigs(): core was generated Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: INFO: <core> [main.c:802]: handle_sigs(): terminating due to SIGCHLD Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24716]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24718]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24715]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24717]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24714]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24706]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24713]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24707]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24712]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24705]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24711]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24701]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24710]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24700]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24709]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24699]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24708]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24696]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24697]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24704]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24694]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24703]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24693]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24702]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24691]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24695]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24690]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24692]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24689]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24686]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24687]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24688]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24677]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24681]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24682]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24680]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24683]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24679]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24684]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24678]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24685]: INFO: <core> [main.c:857]: sig_usr(): signal 15 received Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x322e3176534c54 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0xc0c0c0c0 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0xabcdefed (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f0237b6de90), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7f0237a7a9a0 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x8 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x38 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x28 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7f02b7aef9c4 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7f02b7af0f30 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7f02b7aef9c0 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x125 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0xf0f0f0f0 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x7453726568706943 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring Sep 3 07:04:56 kamailio12 /usr/sbin/kamailio[24676]: CRITICAL: <core> [core/mem/q_malloc.c:501]: qm_free(): BUG: bad pointer 0x676e6972 (out of memory block!) called from tls: tls_init.c: ser_free(323) - ignoring
**bt full:**
root@kamailio12:/# gdb /usr/sbin/kamailio /core GNU gdb (Debian 8.2.1-2+b3) 8.2.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/ed/15dbbb24fb309311e7a6a16cf9222a82afb102.debug...done. done. [New LWP 24676] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65 65 ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory. (gdb) bt full #0 __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65 No locals. #1 0x00007f02bbfa99ef in _IO_vfprintf_internal (s=s@entry=0x55b0eed25ab0, format=format@entry=0x55b0ee3d12d0 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n", ap=ap@entry=0x7ffccf26c860) at vfprintf.c:1638 len = <optimized out> string_malloced = 0 step0_jumps = {0, 3637, 3213, 3109, 4653, 2997, 4437, 4037, 3717, 4869, 4773, 3397, 4557, 4549, 3589, 4981, 3701, 4757, 3317, 2021, 1429, 1221, 2261, 1701, 1653, 797, 1773, 437, 437, 4333} space = 0 is_short = 0 use_outdigits = 0 outc = <optimized out> step1_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 4869, 4773, 3397, 4557, 4549, 3589, 4981, 3701, 4757, 3317, 2021, 1429, 1221, 2261, 1701, 1653, 797, 1773, 437, 437, 0} group = 0 prec = <optimized out> step2_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4773, 3397, 4557, 4549, 3589, 4981, 3701, 4757, 3317, 2021, 1429, 1221, 2261, 1701, 1653, 797, 1773, 437, 437, 0} string = 0x298 <error: Cannot access memory at address 0x298> left = 0 is_long_double = <optimized out> width = 0 signed_number = <optimized out> step3a_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3493, 0, 0, 0, 3589, 4981, 3701, 4757, 3317, 0, 0, 0, 0, 1701, 0, 0, 0, 0, 0, 0} alt = 0 showsign = 0 is_long = <optimized out> is_char = <optimized out> pad = 32 ' ' step3b_jumps = {0 <repeats 11 times>, 4557, 0, 0, 3589, 4981, 3701, 4757, 3317, 2021, 1429, 1221, 2261, 1701, 1653, 797, 1773, 0, 0, 0} --Type <RET> for more, q to quit, c to continue without paging-- step4_jumps = {0 <repeats 14 times>, 3589, 4981, 3701, 4757, 3317, 2021, 1429, 1221, 2261, 1701, 1653, 797, 1773, 0, 0, 0} args_value = <optimized out> is_negative = <optimized out> number = <optimized out> base = <optimized out> the_arg = {pa_wchar = 0 L'\000', pa_int = 0, pa_long_int = 0, pa_long_long_int = 0, pa_u_int = 0, pa_u_long_int = 0, pa_u_long_long_int = 0, pa_double = 0, pa_long_double = <invalid float value>, pa_string = 0x0, pa_wstring = 0x0, pa_pointer = 0x0, pa_user = 0x0} spec = 115 's' _buffer = {__routine = 0x7ffccf26c870, __arg = 0x93, __canceltype = -819541616, __prev = 0x55b0ee3d2556} _avail = <optimized out> thousands_sep = 0x0 grouping = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff> done = 234 f = 0x55b0ee3d1357 "s:%lu. Exec from %s:%u.\n" lead_str_end = 0x55b0ee3d12d0 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n" end_of_spec = <optimized out> work_buffer = '\000' <repeats 16 times>, "P\304&\317\374\177\000\000\377\377\377\377\377\377\377\377\000\000\000\000\000\000\000\000\t", '\000' <repeats 11 times>, "\374\177\000\000\n\000\000\000\260U\000\000 \000\000\000\000\000\000\000\070\310&\317\374\177\000\000\003\000\000\000\000\000\000\000\003", '\000' <repeats 15 times>, "`\367\020\274\002\177\000\000h\r\000\000\000\000\000\000\b%=\356\260U\000\000\v\000\000\000\000\000\000\000\b%=\356\260U\000\000x\364>\271\002\177\000\000c%=\356\260U\000\000\a(=\356\260U\000\000\000\250\207~[\241>5\020\304&\317\374\177\000\000\020\304&\317\374\177\000\000@\305&\317\374\177\000\000"... workstart = 0x0 workend = <optimized out> ap_save = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7ffccf26c940, reg_save_area = 0x7ffccf26c880}} nspecs_done = 10 save_errno = 11 readonly_format = 0 __PRETTY_FUNCTION__ = "_IO_vfprintf_internal" __result = <optimized out> #2 0x00007f02bc04bc68 in __GI___vsyslog_chk (pri=<optimized out>, flag=-1, fmt=<optimized out>, ap=0x7ffccf26c860) at ../misc/syslog.c:220 --Type <RET> for more, q to quit, c to continue without paging-- now_tm = {tm_sec = 56, tm_min = 4, tm_hour = 7, tm_mday = 3, tm_mon = 8, tm_year = 120, tm_wday = 4, tm_yday = 246, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x55b0eec5e670 "UTC"} now = 1599116696 fd = <optimized out> f = 0x55b0eed25ab0 buf = 0x0 bufsize = 0 msgoff = 21 saved_errno = 11 failbuf = "\230\225P_\000\000\000\000@<\021\274\002\177\000\000\022\000\000\000\000\000\000\000\260Z\322\356\260" clarg = <optimized out> #3 0x00007f02bc04bd4c in __syslog (pri=<optimized out>, fmt=<optimized out>) at ../misc/syslog.c:117 ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7ffccf26c980, reg_save_area = 0x7ffccf26c880}} #4 0x000055b0ee24c627 in qm_debug_check_frag (qm=0x7f0237a79000, f=0x7f0237ff81b8, file=0x7f02b7aef9c4 "tls: tls_init.c", line=323, efile=0x55b0ee3d1149 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:123 __llevel = -3 p = 0x7f0237a79000 __func__ = "qm_debug_check_frag" #5 0x000055b0ee250e37 in qm_free (qmp=0x7f0237a79000, p=0x7f0237ff81f0, file=0x7f02b7aef9c4 "tls: tls_init.c", func=0x7f02b7af0f50 <__func__.23424> "ser_free", line=323, mname=0x7f02b7aef9c0 "tls") at core/mem/q_malloc.c:511 qm = 0x7f0237a79000 f = 0x7f0237ff81b8 size = 24 next = 0x7f0237b6ded8 prev = 0x7f0237b6dde0 __func__ = "qm_free" #6 0x000055b0ee25c909 in qm_shm_free (qmp=0x7f0237a79000, p=0x7f0237ff81f0, file=0x7f02b7aef9c4 "tls: tls_init.c", func=0x7f02b7af0f50 <__func__.23424> "ser_free", line=323, mname=0x7f02b7aef9c0 "tls") at core/mem/q_malloc.c:1275 No locals. #7 0x00007f02b7a9b088 in ser_free (ptr=0x7f0237ff81f0, fname=0x7f02b820993c "../crypto/conf/conf_ssl.c", fline=51) at tls_init.c:323 __func__ = "ser_free" --Type <RET> for more, q to quit, c to continue without paging-- #8 0x00007f02b80cb628 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #9 0x00007f02b80cafb5 in CONF_modules_finish () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #10 0x00007f02b80cb139 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #11 0x00007f02b81601e0 in OPENSSL_cleanup () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #12 0x00007f02b7a9ff45 in tls_h_mod_destroy_f () at tls_init.c:870 __func__ = "tls_h_mod_destroy_f" #13 0x000055b0ee08f4d1 in destroy_tls () at core/tls_hooks.c:75 No locals. #14 0x000055b0edf36358 in cleanup (show_status=1) at main.c:584 memlog = 0 __func__ = "cleanup" #15 0x000055b0edf37ef3 in shutdown_children (sig=15, show_status=1) at main.c:706 __func__ = "shutdown_children" #16 0x000055b0edf3b4e6 in handle_sigs () at main.c:806 chld = 0 chld_status = 139 any_chld_stopped = 1 memlog = 0 __func__ = "handle_sigs" #17 0x000055b0edf47053 in main_loop () at main.c:1817 i = 8 pid = 24718 si = 0x0 --Type <RET> for more, q to quit, c to continue without paging-- si_desc = "udp receiver child=7 sock=172.19.140.102:5060\000\000\000\003\000\000\000+\000\000\000\000\250\207~[\241>5\320=\322\356\260U\000\000\210P6\356\260U\000\000\000\000\000\000\000\000\000\000\260\274\004\274\002\177\000\000\363>;\356\260U\000\000\000\000\000\200\000\000\000\000p\321&\317\374\177\000\000h]\034\356\260U\000" nrprocs = 8 woneinit = 1 __func__ = "main_loop" #18 0x000055b0edf50516 in main (argc=9, argv=0x7ffccf26d6b8) at main.c:2856 cfg_stream = 0x55b0eec3c260 c = -1 r = 0 tmp = 0x7ffccf26edf8 "" tmp_len = 32514 port = -1137628688 proto = 32514 ahost = 0x0 aport = 0 options = 0x55b0ee367d38 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 626764136 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0x0 st = {st_dev = 21, st_ino = 142159, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1599116672, tv_nsec = 487768574}, st_mtim = {tv_sec = 1599116673, tv_nsec = 103787828}, st_ctim = {tv_sec = 1599116673, tv_nsec = 103787828}, __glibc_reserved = {0, 0, 0}} tbuf = "\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\060\067\061\274\002\177\000\000\000\000\000\000\000\000\000\000\240\301\061\317\374\177\000\000\364:/\274\002\177\000\000\b\000\000\000\000\000\000\000\230\302\061\317\374\177\000\000\060\067\0--Type <RET> for more, q to quit, c to continue without paging-- 61\274\002\177\000\000X\324&\317\374\177\000\000T\324&\317\374\177", '\000' <repeats 18 times>, "\230\302\061\317\374\177\000\000\250\301\061\317\374\177\000\000'\216\r\274\002\177\000\000&\260be\000\000\000\000\300\325&\317\374\177\000\000\300\212\225\001\000\000\000\000 \325&\317\374\177\000\000\020\325&\317\374\177\000\000X\324&\317\374\177\000\000X71\274\002\177\000\000\001", '\000' <repeats 31 times>... option_index = 0 long_options = {{name = 0x55b0ee369dc6 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x55b0ee36550c "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x55b0ee369dcb "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x55b0ee369dd1 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x55b0ee369dd7 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x55b0ee369de0 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x55b0ee369dea "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x55b0ee369df4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x55b0ee369dff "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x55b0ee369e08 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x55b0ee369e13 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main"
You have to enable core files per pid, because the backtrace indicates is the core file dumped by shutting down process, which can be just a side effect, because likely the crash happens somewhere else. You can see how to enable core file per process at:
* https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile#how_do_...
The the core files should be suffixed by pid number, grab the gdb `bt full` for each of the core files.
As I tried to reproduce today, with the reverts done by @linuxmaniac, I could not get it (on a Debian stable), so it could be specific to your config. Maybe you can create a minimal kamailio config that reproduces the case. Being about starting up, you should try to keep global parameters, loading the modules and modparams, with a request_route having just exit. To make it smaller, if you have time to play with it, you can start removing from modues (with the corresponding modparams).
Also, load tls module first, because it needs to initialize the libssl with some options needed by Kamailio, before other modules that may use libssl.
After some investigation I found out that the cause for the problem is modparam("http_async_client", "workers", **4**). Starting kamailio with modparam("http_async_client", "workers", **1**) works fine. I also tested it on v5.4.1 before @linuxmaniac reverts and it works fine with modparam("http_async_client", "workers", **1**)
Here are the bt full of all the core files:
root@kamailio12:/# gdb /usr/sbin/kamailio /core.32155 GNU gdb (Debian 8.2.1-2+b3) 8.2.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. stopType "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/ed/15dbbb24fb309311e7a6a16cf9222a82afb102.debug...done. done. [New LWP 32155] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -m 2048 -M 32'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fb49968e61f in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (gdb) bt full #0 0x00007fb49968e61f in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #1 0x00007fb49968dfb5 in CONF_modules_finish () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #2 0x00007fb49968e139 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #3 0x00007fb4997231e0 in OPENSSL_cleanup () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #4 0x00007fb49993af45 in tls_h_mod_destroy_f () at tls_init.c:870 __func__ = "tls_h_mod_destroy_f" #5 0x0000559822cbf4d1 in destroy_tls () at core/tls_hooks.c:75 No locals. #6 0x0000559822b66358 in cleanup (show_status=1) at main.c:584 memlog = 0 __func__ = "cleanup" #7 0x0000559822b67ef3 in shutdown_children (sig=15, show_status=1) at main.c:706 __func__ = "shutdown_children" #8 0x0000559822b6b4e6 in handle_sigs () at main.c:806 chld = 0 chld_status = 139 any_chld_stopped = 1 memlog = 0 __func__ = "handle_sigs" #9 0x0000559822b77053 in main_loop () at main.c:1817 i = 8 pid = 32195 si = 0x0 si_desc = "udp receiver child=7 sock=172.19.140.102:5060\000\000\000\003\000\000\000)\000\000\000\000\274[\351\352ZP\207`\027\v%\230U\000\000\210P\371"\230U\000\000\000\000\000\000\000\000\000\000\260|\255\233\264\177\000\000\363>\376"\230U\000\000\000\000\000\200\000\000\000\000\320\177N\204\377\177\000\000h]\337"\230U\000" nrprocs = 8 woneinit = 1 --Type <RET> for more, q to quit, c to continue without paging-- __func__ = "main_loop" #10 0x0000559822b80516 in main (argc=7, argv=0x7fff844e8518) at main.c:2856 cfg_stream = 0x559824fcb260 c = -1 r = 0 tmp = 0x7fff844e8def "" tmp_len = 0 port = 0 proto = 1 ahost = 0x0 aport = 0 options = 0x559822f97d38 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3411128981 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xffff00001f80 p = 0x0 st = {st_dev = 21, st_ino = 428223, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1599374984, tv_nsec = 763308205}, st_mtim = {tv_sec = 1599374995, tv_nsec = 555647093}, st_ctim = {tv_sec = 1599374995, tv_nsec = 555647093}, __glibc_reserved = {0, 0, 0}} tbuf = "\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\060\367\331\233\264\177\000\000\000\000\000\000\000\000\000\000\240\301U\204\377\177\000\000\364\372\327\233\264\177\000\000\b\000\000\000\000\000\000\000\230\302U\204\377\177\000\000\060\367\331\233\264\177\000\000\270\202N\204\377\177\000\000\264\202N\204\377\177", '\000' <repeats 18 times>, "\230\302U\204\377\177\000\000\250\301U\204\377\177\000\000'N\266\233\264\177\000\000&\260be\000\000\000\000 \204N\204\377\177\000\000\300\212\225\001\000\000\000\000\200\203N\204\377\177\000\000p\203N\204\377\177\000\000\270\202N\204\377\177\000\000X\367\331\233\264\177\000\000\001", '\000' <repeats 31 times>... option_index = 0 long_options = {{name = 0x559822f99dc6 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x559822f9550c "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x559822f99dcb "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x559822f99dd1 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x559822f99dd7 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x559822f99de0 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x559822f99dea "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x559822f99df4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x559822f99dff "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x559822f99e08 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x559822f99e13 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" (gdb) (gdb) quit
root@kamailio12:/# gdb /usr/sbin/kamailio /core.32172 GNU gdb (Debian 8.2.1-2+b3) 8.2.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/ed/15dbbb24fb309311e7a6a16cf9222a82afb102.debug...done. done. [New LWP 32172] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -m 2048 -M 32'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fb49968e61f in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (gdb) bt full #0 0x00007fb49968e61f in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #1 0x00007fb49968e72d in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #2 0x00007fb49968dd05 in CONF_modules_load () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #3 0x00007fb49968e2cb in CONF_modules_load_file () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #4 0x00007fb417fa717e in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 No symbol table info available. #5 0x00007fb417f6d1cf in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 No symbol table info available. #6 0x00007fb417fec560 in set_curl_mem_callbacks () at http_multi.c:367 rc = CURLE_OK __func__ = "set_curl_mem_callbacks" #7 0x00007fb417fed407 in init_http_multi (evbase=0x5598251278e0, wg=0x7fb419b347c0) at http_multi.c:396 __func__ = "init_http_multi" #8 0x00007fb417fd65f8 in async_http_run_worker (worker=0x7fb4198d45e0) at async_http.c:91 No locals. #9 0x00007fb417ffa3a6 in child_init (rank=0) at http_async_client_mod.c:366 pid = 0 i = 2 __func__ = "child_init" #10 0x0000559822cca246 in init_mod_child (m=0x7fb499a93d70, rank=0) at core/sr_module.c:780 __func__ = "init_mod_child" #11 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a949f0, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #12 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a94e90, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #13 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a95330, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" --Type <RET> for more, q to quit, c to continue without paging-- #14 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a95b30, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #15 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a966d0, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #16 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a96d18, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #17 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a97250, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #18 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a978d0, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #19 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a97c80, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #20 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a985f0, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #21 0x0000559822ccab71 in init_child (rank=0) at core/sr_module.c:825 ret = 32692 type = 0x559822fc88eb "PROC_MAIN" __func__ = "init_child" #22 0x0000559822b76101 in main_loop () at main.c:1763 i = 8 pid = 32165 si = 0x0 si_desc = "udp receiver child=7 sock=172.19.140.102:5060\000\000\000\003\000\000\000)\000\000\000\000\274[\351\352ZP\207`\027\v%\230U\000\000\210P\371"\230U\000\000\000\000\000\000\000\000\000\000\260|\255\233\264\177\000\000\363>\376"\230U\000\000\000\000\000\200\000\000\000\000\320\177N\204\377\177\000\000h]\337"\230U\000" nrprocs = 8 woneinit = 1 __func__ = "main_loop" #23 0x0000559822b80516 in main (argc=7, argv=0x7fff844e8518) at main.c:2856 cfg_stream = 0x559824fcb260 c = -1 r = 0 --Type <RET> for more, q to quit, c to continue without paging-- tmp = 0x7fff844e8def "" tmp_len = 0 port = 0 proto = 1 ahost = 0x0 aport = 0 options = 0x559822f97d38 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3411128981 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xffff00001f80 p = 0x0 st = {st_dev = 21, st_ino = 428223, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1599374984, tv_nsec = 763308205}, st_mtim = {tv_sec = 1599374995, tv_nsec = 555647093}, st_ctim = {tv_sec = 1599374995, tv_nsec = 555647093}, __glibc_reserved = {0, 0, 0}} tbuf = "\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\060\367\331\233\264\177\000\000\000\000\000\000\000\000\000\000\240\301U\204\377\177\000\000\364\372\327\233\264\177\000\000\b\000\000\000\000\000\000\000\230\302U\204\377\177\000\000\060\367\331\233\264\177\000\000\270\202N\204\377\177\000\000\264\202N\204\377\177", '\000' <repeats 18 times>, "\230\302U\204\377\177\000\000\250\301U\204\377\177\000\000'N\266\233\264\177\000\000&\260be\000\000\000\000 \204N\204\377\177\000\000\300\212\225\001\000\000\000\000\200\203N\204\377\177\000\000p\203N\204\377\177\000\000\270\202N\204\377\177\000\000X\367\331\233\264\177\000\000\001", '\000' <repeats 31 times>... option_index = 0 long_options = {{name = 0x559822f99dc6 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x559822f9550c "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x559822f99dcb "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x559822f99dd1 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x559822f99dd7 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x559822f99de0 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x559822f99dea "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x559822f99df4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x559822f99dff "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x559822f99e08 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x559822f99e13 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" (gdb) (gdb) quit
root@kamailio12:/# gdb /usr/sbin/kamailio /core.32173 GNU gdb (Debian 8.2.1-2+b3) 8.2.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/ed/15dbbb24fb309311e7a6a16cf9222a82afb102.debug...done. done. [New LWP 32173] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -m 2048 -M 32'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fb49968e61f in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (gdb) bt full #0 0x00007fb49968e61f in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #1 0x00007fb49968e72d in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #2 0x00007fb49968dd05 in CONF_modules_load () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #3 0x00007fb49968e2cb in CONF_modules_load_file () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #4 0x00007fb417fa717e in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 No symbol table info available. #5 0x00007fb417f6d1cf in ?? () from /usr/lib/x86_64-linux-gnu/libcurl.so.4 No symbol table info available. #6 0x00007fb417fec560 in set_curl_mem_callbacks () at http_multi.c:367 rc = CURLE_OK __func__ = "set_curl_mem_callbacks" #7 0x00007fb417fed407 in init_http_multi (evbase=0x559825127df0, wg=0x7fb419b34848) at http_multi.c:396 __func__ = "init_http_multi" #8 0x00007fb417fd65f8 in async_http_run_worker (worker=0x7fb4198d4600) at async_http.c:91 No locals. #9 0x00007fb417ffa3a6 in child_init (rank=0) at http_async_client_mod.c:366 pid = 0 i = 3 __func__ = "child_init" #10 0x0000559822cca246 in init_mod_child (m=0x7fb499a93d70, rank=0) at core/sr_module.c:780 __func__ = "init_mod_child" #11 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a949f0, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #12 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a94e90, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #13 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a95330, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" --Type <RET> for more, q to quit, c to continue without paging-- #14 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a95b30, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #15 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a966d0, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #16 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a96d18, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #17 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a97250, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #18 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a978d0, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #19 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a97c80, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #20 0x0000559822cc9e61 in init_mod_child (m=0x7fb499a985f0, rank=0) at core/sr_module.c:776 __func__ = "init_mod_child" #21 0x0000559822ccab71 in init_child (rank=0) at core/sr_module.c:825 ret = 32692 type = 0x559822fc88eb "PROC_MAIN" __func__ = "init_child" #22 0x0000559822b76101 in main_loop () at main.c:1763 i = 8 pid = 32165 si = 0x0 si_desc = "udp receiver child=7 sock=172.19.140.102:5060\000\000\000\003\000\000\000)\000\000\000\000\274[\351\352ZP\207`\027\v%\230U\000\000\210P\371"\230U\000\000\000\000\000\000\000\000\000\000\260|\255\233\264\177\000\000\363>\376"\230U\000\000\000\000\000\200\000\000\000\000\320\177N\204\377\177\000\000h]\337"\230U\000" nrprocs = 8 woneinit = 1 __func__ = "main_loop" #23 0x0000559822b80516 in main (argc=7, argv=0x7fff844e8518) at main.c:2856 cfg_stream = 0x559824fcb260 c = -1 r = 0 --Type <RET> for more, q to quit, c to continue without paging-- tmp = 0x7fff844e8def "" tmp_len = 0 port = 0 proto = 1 ahost = 0x0 aport = 0 options = 0x559822f97d38 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3411128981 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xffff00001f80 p = 0x0 st = {st_dev = 21, st_ino = 428223, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1599374984, tv_nsec = 763308205}, st_mtim = {tv_sec = 1599374995, tv_nsec = 555647093}, st_ctim = {tv_sec = 1599374995, tv_nsec = 555647093}, __glibc_reserved = {0, 0, 0}} tbuf = "\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\060\367\331\233\264\177\000\000\000\000\000\000\000\000\000\000\240\301U\204\377\177\000\000\364\372\327\233\264\177\000\000\b\000\000\000\000\000\000\000\230\302U\204\377\177\000\000\060\367\331\233\264\177\000\000\270\202N\204\377\177\000\000\264\202N\204\377\177", '\000' <repeats 18 times>, "\230\302U\204\377\177\000\000\250\301U\204\377\177\000\000'N\266\233\264\177\000\000&\260be\000\000\000\000 \204N\204\377\177\000\000\300\212\225\001\000\000\000\000\200\203N\204\377\177\000\000p\203N\204\377\177\000\000\270\202N\204\377\177\000\000X\367\331\233\264\177\000\000\001", '\000' <repeats 31 times>... option_index = 0 long_options = {{name = 0x559822f99dc6 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x559822f9550c "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x559822f99dcb "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x559822f99dd1 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x559822f99dd7 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x559822f99de0 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x559822f99dea "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x559822f99df4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x559822f99dff "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x559822f99e08 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x559822f99e13 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" (gdb) (gdb) quit
Did you load `tls` module before `http_async_client`? Also, this module has to be loaded before any other module using libcurl (see: https://www.kamailio.org/docs/modules/stable/modules/http_async_client.html#...)
For further troubleshooting, install the debug symbols packages for libcurl and libssl to get more details in the frames referencing them. Once installed, just grab again the `bt full` from all core files (no need to generate another crash).
Yes, tls module loaded first. I created a minimal config file that reproduces the problem. The crash is because of this line: modparam("http_async_client", "workers", 4) comment out this line makes the crash disappear.
#!KAMAILIO
####### Global Parameters #########
/* LOG Levels: 3=DBG, 2=INFO, 1=NOTICE, 0=WARN, -1=ERR, ... */ debug=2 log_stderror=no memdbg=5 memlog=5
log_facility=LOG_LOCAL0 log_prefix="{$mt $hdr(CSeq) $ci} "
children=8 tcp_children=16 auto_aliases=no alias="example.com"
listen=udp:eth0 server_signature=no tcp_connection_lifetime=3605 tcp_max_connections=40960 tcp_accept_no_cl=yes enable_tls=yes listen=tls:eth0 tls_max_connections=40000 enable_sctp=no
####### Modules Section ########
loadmodule "kex.so" loadmodule "corex.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "tls.so" loadmodule "http_async_client.so"
# ----------------- setting module-specific parameters ---------------
# ----- tls params ----- modparam("tls", "config", "/etc/kamailio/tls.cfg")
# ----- http client ---- modparam("http_async_client", "workers", 4)
####### Routing Logic ########
request_route { exit; }
Hi Joey, I've tried to investigate the issue with http_async_client initialization, but I've not been able to reproduce it in buster. Anyway looking at the logs I've found something that might be not good, the place(s) where curl memory callbacks are set. I've created a branch with a change on this https://github.com/kamailio/kamailio/tree/hac_curl_mem_callbacks, do you have any chance to try it?
Closed #2466.
Hi @grumvalski, I have tested the branch with your changes and it looks like it fixes the problem. Very well.. Thank you!
Thanks for the feedback. Re-open this as it still needs to be merged into main git branch and probably stable branches.
Reopened #2466.
Thanks for testing Joey, the branch is now merged in master. I'll also backport it to stable branches.
On Wed, Sep 9, 2020 at 11:57 AM Henning Westerholt notifications@github.com wrote:
Reopened #2466 https://github.com/kamailio/kamailio/issues/2466.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kamailio/kamailio/issues/2466#event-3745003813, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABO7UZIIAKQQHD7P3PE2HZDSE5GKRANCNFSM4QSUPPMA . _______________________________________________ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Closed #2466.
Thanks @grumvalski for handling this one!
-
Daniel-Constantin Mierla -
Federico Cabiddu
-
Federico Cabiddu -
Henning Westerholt
-
Joey Golan
-
Victor Seva