<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports. If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list: * http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list: * http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue. If there is no content to be filled in a section, the entire section can be removed. You can delete the comments from the template sections when filling. You can delete next line and everything above before submitting (it is a comment). --> ### Description I have a cluster of 2 Kamailios nodes on debian 9 + Kamailio 5.5.4. We have installed a 3rd server on debian 11 + Kamailio 5.6.1 but it keeps crashing due to an issue with DMQ USRLOC ### Troubleshooting #### Reproduction We don't know how to reproduce but this is currently happening constantly on our new server #### Debugging Data <!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location. gdb /path/to/kamailio /path/to/corefile bt full info locals list If you are familiar with gdb, feel free to attach more of what you consider to be relevant. --> Kamailio generates a core on each segfault: ``` -rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7507.csbc03.example.com.1663227725 -rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7576.csbc03.example.com.1663227733 -rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7642.csbc03.example.com.1663227739 -rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7708.csbc03.example.com.1663227746 -rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7772.csbc03.example.com.1663227753 ``` But they seem to be empty? ``` root@csbc03:/var/tmp# gdb core.kamailio.7772.csbc03.example.com.1663227753 GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... "0x7ffec5273ce0s": not in executable format: file format not recognized (gdb) bt full No stack. (gdb) quit root@csbc03:/var/tmp# ``` Very similar output for any of the other core files when In run `bt full`. I confirmed debug symbols were installed: ``` root@csbc03:/var/tmp# dpkg -l | grep kam | grep dbg ii kamailio-dbg:amd64 5.6.1+bpo11 amd64 very fast and configurable SIP server [debug symbols] root@csbc03:/var/tmp# ``` #### Log Messages <!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> /var/log/syslog ``` [ 501.251476] kamailio[7772]: segfault at f8 ip 00007f86ded9bf72 sp 00007ffd892eb7c0 error 4 in dmq_usrloc.so[7f86ded89000+17000] [ 501.251496] Code: 40 38 01 d0 89 05 12 95 00 00 48 8b 05 cf 8f 00 00 8b 00 83 f8 01 0f 85 80 00 00 00 48 8b 85 48 ff ff ff 48 8b 80 a0 00 00 00 <8b> 90 f8 00 00 00 48 8b 85 48 ff ff ff 48 8b 80 a0 00 00 00 48 8b ``` /var/log/kamailio/kamailio.lorg ``` Sep 15 02:42:34 csbc03 csbc[7792]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 59 Sep 15 02:42:36 csbc03 csbc[7736]: ALERT: <core> [main.c:774]: handle_sigs(): child process 7772 exited by a signal 11 Sep 15 02:42:36 csbc03 csbc[7736]: ALERT: <core> [main.c:777]: handle_sigs(): core was generated ``` #### SIP Traffic <!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> ``` (paste your sip traffic here) ``` ### Possible Solutions <!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. --> ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` root@csbc03:/var/tmp# kamailio -v version: kamailio 5.6.1 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 10.2.1 root@csbc03:/var/tmp# ``` * **Operating System**: <!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) --> ``` Debian 11 Bullseye ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3238 You are receiving this because you are subscribed to this thread. Message ID: &lt;kamailio/kamailio/issues/3238(a)github.com&gt;