<p></p>
<h3 dir="auto">Description</h3>
<p dir="auto">I have a cluster of 2 Kamailios nodes on debian 9 + Kamailio 5.5.4.<br>
We have installed a 3rd server on debian 11 + Kamailio 5.6.1 but it keeps crashing due to an issue with DMQ USRLOC</p>
<h3 dir="auto">Troubleshooting</h3>
<h4 dir="auto">Reproduction</h4>
<p dir="auto">We don't know how to reproduce but this is currently happening constantly on our new server</p>
<h4 dir="auto">Debugging Data</h4>
<p dir="auto">Kamailio generates a core on each segfault:</p>
<pre class="notranslate"><code class="notranslate">-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7507.csbc03.example.com.1663227725
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7576.csbc03.example.com.1663227733
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7642.csbc03.example.com.1663227739
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7708.csbc03.example.com.1663227746
-rw------- 1 kamailio kamailio 556M Sep 15 02:42 core.kamailio.7772.csbc03.example.com.1663227753
</code></pre>
<p dir="auto">But they seem to be empty?</p>
<pre class="notranslate"><code class="notranslate">root@csbc03:/var/tmp# gdb core.kamailio.7772.csbc03.example.com.1663227753
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
"0x7ffec5273ce0s": not in executable format: file format not recognized
(gdb) bt full
No stack.
(gdb) quit
root@csbc03:/var/tmp#
</code></pre>
<p dir="auto">Very similar output for any of the other core files when In run <code class="notranslate">bt full</code>.</p>
<p dir="auto">I confirmed debug symbols were installed:</p>
<pre class="notranslate"><code class="notranslate">root@csbc03:/var/tmp# dpkg -l | grep kam | grep dbg
ii kamailio-dbg:amd64 5.6.1+bpo11 amd64 very fast and configurable SIP server [debug symbols]
root@csbc03:/var/tmp#
</code></pre>
<h4 dir="auto">Log Messages</h4>
<p dir="auto">/var/log/syslog</p>
<pre class="notranslate"><code class="notranslate">[ 501.251476] kamailio[7772]: segfault at f8 ip 00007f86ded9bf72 sp 00007ffd892eb7c0 error 4 in dmq_usrloc.so[7f86ded89000+17000]
[ 501.251496] Code: 40 38 01 d0 89 05 12 95 00 00 48 8b 05 cf 8f 00 00 8b 00 83 f8 01 0f 85 80 00 00 00 48 8b 85 48 ff ff ff 48 8b 80 a0 00 00 00 <8b> 90 f8 00 00 00 48 8b 85 48 ff ff ff 48 8b 80 a0 00 00 00 48 8b
</code></pre>
<p dir="auto">/var/log/kamailio/kamailio.lorg</p>
<pre class="notranslate"><code class="notranslate">Sep 15 02:42:34 csbc03 csbc[7792]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 59
Sep 15 02:42:36 csbc03 csbc[7736]: ALERT: <core> [main.c:774]: handle_sigs(): child process 7772 exited by a signal 11
Sep 15 02:42:36 csbc03 csbc[7736]: ALERT: <core> [main.c:777]: handle_sigs(): core was generated
</code></pre>
<h4 dir="auto">SIP Traffic</h4>
<pre class="notranslate"><code class="notranslate">(paste your sip traffic here)
</code></pre>
<h3 dir="auto">Possible Solutions</h3>
<h3 dir="auto">Additional Information</h3>
<ul dir="auto">
<li><strong>Kamailio Version</strong> - output of <code class="notranslate">kamailio -v</code></li>
</ul>
<pre class="notranslate"><code class="notranslate">root@csbc03:/var/tmp# kamailio -v
version: kamailio 5.6.1 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 10.2.1
root@csbc03:/var/tmp#
</code></pre>
<ul dir="auto">
<li><strong>Operating System</strong>:</li>
</ul>
<pre class="notranslate"><code class="notranslate">Debian 11 Bullseye
</code></pre>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/3238">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/ABO7UZKQZZY7IYVPJOUHREDV6LLBDANCNFSM6AAAAAAQNEH5WM">unsubscribe</a>.<br />You are receiving this because you are subscribed to this thread.<img src="https://github.com/notifications/beacon/ABO7UZJNCQQGPWMHDQ2PQO3V6LLBDA5CNFSM6AAAAAAQNEH5WOWGG33NNVSW45C7OR4XAZNFJFZXG5LFVJRW63LNMVXHIX3JMTHFDZ25JU.gif" height="1" width="1" alt="" /><span style="color: transparent; font-size: 0; display: none; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span><kamailio/kamailio/issues/3238</span><span>@</span><span>github</span><span>.</span><span>com></span></span></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/kamailio/kamailio/issues/3238",
"url": "https://github.com/kamailio/kamailio/issues/3238",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>