[sr-dev] [kamailio/kamailio] segfault at f8 ip 00007fc13901af72 sp 00007ffdb457ef80 error 4 in dmq_usrloc.so[7fc139008000+17000] (Issue #3238)

Thu Sep 15 10:13:05 CEST 2022

<!--
Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.

If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:

  * http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:

  * http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.

If there is no content to be filled in a section, the entire section can be removed.

You can delete the comments from the template sections when filling.

You can delete next line and everything above before submitting (it is a comment).
-->

### Description

I have a cluster of 2 Kamailios nodes on debian 9 + Kamailio 5.5.4.
We have installed a 3rd server on debian 11 + Kamailio 5.6.1 but it keeps crashing due to an issue with DMQ USRLOC

### Troubleshooting

#### Reproduction

We don't know how to reproduce but this is currently happening constantly on our new server

#### Debugging Data

<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.

  gdb /path/to/kamailio /path/to/corefile
  bt full
  info locals
  list

If you are familiar with gdb, feel free to attach more of what you consider to
be relevant.
-->

Kamailio generates a core on each segfault:
```
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7507.csbc03.example.com.1663227725
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7576.csbc03.example.com.1663227733
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7642.csbc03.example.com.1663227739
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7708.csbc03.example.com.1663227746
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7772.csbc03.example.com.1663227753
```

But they seem to be empty?
```
root at csbc03:/var/tmp# gdb core.kamailio.7772.csbc03.example.com.1663227753
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
"0x7ffec5273ce0s": not in executable format: file format not recognized
(gdb) bt full
No stack.
(gdb) quit
root at csbc03:/var/tmp#
```
Very similar output for any of the other core files when In run `bt full`.

I confirmed debug symbols were installed:

```
root at csbc03:/var/tmp# dpkg -l | grep kam | grep dbg
ii  kamailio-dbg:amd64              5.6.1+bpo11                    amd64        very fast and configurable SIP server [debug symbols]
root at csbc03:/var/tmp#
```

#### Log Messages

<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
/var/log/syslog
```
[  501.251476] kamailio[7772]: segfault at f8 ip 00007f86ded9bf72 sp 00007ffd892eb7c0 error 4 in dmq_usrloc.so[7f86ded89000+17000]
[  501.251496] Code: 40 38 01 d0 89 05 12 95 00 00 48 8b 05 cf 8f 00 00 8b 00 83 f8 01 0f 85 80 00 00 00 48 8b 85 48 ff ff ff 48 8b 80 a0 00 00 00 <8b> 90 f8 00 00 00 48 8b 85 48 ff ff ff 48 8b 80 a0 00 00 00 48 8b
```

/var/log/kamailio/kamailio.lorg
```
Sep 15 02:42:34 csbc03 csbc[7792]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 59
Sep 15 02:42:36 csbc03 csbc[7736]: ALERT: <core> [main.c:774]: handle_sigs(): child process 7772 exited by a signal 11
Sep 15 02:42:36 csbc03 csbc[7736]: ALERT: <core> [main.c:777]: handle_sigs(): core was generated
```

#### SIP Traffic

<!--
If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->

```
(paste your sip traffic here)
```

### Possible Solutions

<!--
If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix.
-->

### Additional Information

  * **Kamailio Version** - output of `kamailio -v`

```
root at csbc03:/var/tmp# kamailio -v
version: kamailio 5.6.1 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 10.2.1
root at csbc03:/var/tmp#
```

* **Operating System**:

<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->

```
Debian 11 Bullseye
```

-- 
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3238
You are receiving this because you are subscribed to this thread.

Message ID: <kamailio/kamailio/issues/3238 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20220915/7afc6349/attachment-0001.htm>