<h3>Description</h3>
<p>After upgrading to 5.3.0 from 5.2.2 (standard packages on FreeBSD 12.0), I am experiencing intermittent crashes related to handling of BYE messages.</p>
<h3>Troubleshooting</h3>
<h4>Reproduction</h4>
<p>This happens ~ weekly and I have not found a good way to reproduce it.</p>
<h4>Debugging Data</h4>
<p>An example backtrace is below from the last dumped core (the SIGSEGV one); unfortunately it overwrote the earlier one:</p>
<pre><code>* thread #1, name = 'kamailio', stop reason = signal SIGSEGV
* frame #0: 0x00000008009c5b79 libc.so.7`___lldb_unnamed_symbol403$$libc.so.7 + 41
frame #1: 0x00000008009ed63e libc.so.7`__free + 990
frame #2: 0x000000080271562b libthr.so.3`pthread_rwlock_destroy + 59
frame #3: 0x0000000802bedbf6 libcrypto.so.111`CRYPTO_THREAD_lock_free + 22
frame #4: 0x0000000802aef3c4 libcrypto.so.111`RSA_free + 100
frame #5: 0x0000000802b10c32 libcrypto.so.111`EVP_PKEY_free + 66
frame #6: 0x000000080296ed86 libssl.so.111`___lldb_unnamed_symbol646$$libssl.so.111 + 134
frame #7: 0x000000080295f93c libssl.so.111`SSL_CTX_free + 236
frame #8: 0x00000008028aee42 tls.so`tls_free_domain + 114
frame #9: 0x00000008028af1d7 tls.so`tls_free_cfg + 199
frame #10: 0x00000008028af2df tls.so`tls_destroy_cfg + 191
frame #11: 0x00000008028ad1f1 tls.so`destroy_tls_h + 1185
frame #12: 0x000000000041adea kamailio`destroy_tls + 26
frame #13: 0x00000000002e36fd kamailio`cleanup + 269
frame #14: 0x00000000002eb5b7 kamailio`___lldb_unnamed_symbol5$$kamailio + 1351
frame #15: 0x00000000002ea5e5 kamailio`handle_sigs + 21669
frame #16: 0x00000000002fb83e kamailio`main_loop + 40014
frame #17: 0x0000000000307d2b kamailio`main + 50267
frame #18: 0x00000000002e311b kamailio`_start + 283
</code></pre>
<p>This is with OpenSSL 1.1 With the LD_PRELOAD hack to 5.2.2, things were completely stable; I am trying to use kamailio without the LD_PRELOAD'ed mutex wrapper now, which I believe is no longer required. It looks like the SSL-related stuff in the TLS crash (which was 5 minutes later!) is unrelated to the initial problem and may just be an artifact of one of the kamailio processes crashing earlier.</p>
<h4>Log Messages</h4>
<pre><code>Nov 2 08:11:31 home /usr/local/sbin/kamailio[94702]: CRITICAL: {1 527440 BYE 973470944-5061-16392@BA.A.B.I} <core> [core/mem/q_malloc.c:149]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(c0c0c000, abcdefed)[0x801544c58:0x801544c90]! Memory allocator was called from core: core/action.c:754. Fragment marked by core: core/dset.c:733. Exec from core/mem/q_malloc.c:504.
Nov 2 08:13:41 home /usr/local/sbin/kamailio[94703]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 22
Nov 2 08:13:41 home kernel: pid 94702 (kamailio), uid 0: exited on signal 6 (core dumped)
Nov 2 08:13:41 home /usr/local/sbin/kamailio[94692]: ALERT: <core> [main.c:767]: handle_sigs(): child process 94702 exited by a signal 6
Nov 2 08:13:41 home /usr/local/sbin/kamailio[94692]: ALERT: <core> [main.c:770]: handle_sigs(): core was generated
Nov 2 08:14:56 home login[8284]: ROOT LOGIN (root) ON ttyu0
Nov 2 08:16:26 home kernel: pid 94692 (kamailio), uid 0: exited on signal 11 (core dumped)
</code></pre>
<p>I had an identical problem a week ago, also with a crash on a BYE for an active call:</p>
<pre><code>Oct 27 13:00:02 home /usr/local/sbin/kamailio[79819]: CRITICAL: {1 598425 BYE 649761149-5061-291@BA.A.B.I} <core> [core/mem/q_malloc.c:149]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(c0c0c000, abcdefed)[0x801551808:0x801551840]! Memory allocator was called from core: core/action.c:754. Fragment marked by core: core/dset.c:733. Exec from core/mem/q_malloc.c:504.
Oct 27 13:02:09 home /usr/local/sbin/kamailio[79820]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 22
Oct 27 13:02:09 home kernel: pid 79819 (kamailio), uid 0: exited on signal 6 (core dumped)
Oct 27 13:02:09 home /usr/local/sbin/kamailio[79809]: ALERT: <core> [main.c:767]: handle_sigs(): child process 79819 exited by a signal 6
Oct 27 13:02:09 home /usr/local/sbin/kamailio[79809]: ALERT: <core> [main.c:770]: handle_sigs(): core was generated
Oct 27 13:04:55 home kernel: pid 79809 (kamailio), uid 0: exited on signal 11 (core dumped)
</code></pre>
<h3>Possible Solutions</h3>
<h3>Additional Information</h3>
<ul>
<li><strong>Kamailio Version</strong> - output of <code>kamailio -v</code></li>
</ul>
<pre><code>version: kamailio 5.3.0 (x86_64/freebsd) 4cc67a
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, select, kqueue.
id: 4cc67a
compiled on 18:51:34 Oct 25 2019 with cc 6.0
</code></pre>
<ul>
<li><strong>Operating System</strong>:</li>
</ul>
<p>FreeBSD 12.0</p>
<pre><code>FreeBSD home.XXX 12.0-RELEASE-p10 FreeBSD 12.0-RELEASE-p10 GENERIC amd64
</code></pre>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/2121?email_source=notifications&email_token=ABO7UZONMGJKU54H4MG65Z3QRWRYZA5CNFSM4JIGOUV2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HWLLP2A">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/ABO7UZMNHBWDHZDKJ45AUOLQRWRYZANCNFSM4JIGOUVQ">unsubscribe</a>.<img src="https://github.com/notifications/beacon/ABO7UZOOK45VOMNGVDKTUGTQRWRYZA5CNFSM4JIGOUV2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HWLLP2A.gif" height="1" width="1" alt="" /></p>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/kamailio/kamailio/issues/2121?email_source=notifications\u0026email_token=ABO7UZONMGJKU54H4MG65Z3QRWRYZA5CNFSM4JIGOUV2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HWLLP2A",
"url": "https://github.com/kamailio/kamailio/issues/2121?email_source=notifications\u0026email_token=ABO7UZONMGJKU54H4MG65Z3QRWRYZA5CNFSM4JIGOUV2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HWLLP2A",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>