[sr-dev] kamailio 4.1.6 (x86_64/linux) 010d57 Crash

[Kristian Kielhofner]

Hello,

  We recently upgraded several of our Kamailio instances to 4.1.6 and
have experienced this crash on several of them (full backtrace
attached).

#0  0x00000031e9e30265 in raise () from /lib64/libc.so.6
#1  0x00000031e9e31d10 in abort () from /lib64/libc.so.6
#2  0x00000000004687de in sig_alarm_abort (signo=14) at main.c:687
#3  <signal handler called>
#4  0x00000031e9ed07c9 in syscall () from /lib64/libc.so.6
#5  0x00002acddbc4ff24 in futex_get (lock=0x2acdddcad6d8) at
../../mem/../futexlock.h:110
#6  0x00002acddbc50555 in remove_dialog_timer (tl=0x2acde2bcb708) at
dlg_timer.c:168
#7  0x00002acddbc3edb8 in destroy_dlg (dlg=0x2acde2bcb6b0) at dlg_hash.c:357
#8  0x00002acddbc3f70a in destroy_dlg_table () at dlg_hash.c:438
#9  0x00002acddbc25c21 in mod_destroy () at dialog.c:776
#10 0x00000000004f7ebb in destroy_modules () at sr_module.c:817
#11 0x00000000004672c4 in cleanup (show_status=1) at main.c:562
#12 0x0000000000468952 in shutdown_children (sig=15, show_status=1) at
main.c:704
#13 0x000000000046a012 in handle_sigs () at main.c:795
#14 0x000000000046e33c in main_loop () at main.c:1748
#15 0x0000000000470ac9 in main (argc=7, argv=0x7fff100e4068) at main.c:2547

  Any thoughts? Thanks!

-- 
Kristian Kielhofner
-------------- next part --------------
GNU gdb (GDB) CentOS (7.0.1-45.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/sbin/kamailio...done.
[New Thread 12588]
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib64/kamailio/modules/db_mysql.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/db_mysql.so
Reading symbols from /usr/lib64/mysql/libmysqlclient.so.15...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/mysql/libmysqlclient.so.15
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.6
Reading symbols from /lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.6
Reading symbols from /usr/local/lib64/kamailio/libsrdb2.so.1...done.
Loaded symbols for /usr/local/lib64/kamailio/libsrdb2.so.1
Reading symbols from /usr/local/lib64/kamailio/libsrdb1.so.1...done.
Loaded symbols for /usr/local/lib64/kamailio/libsrdb1.so.1
Reading symbols from /usr/lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libgssapi_krb5.so.2
Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /usr/local/lib64/kamailio/modules/db_flatstore.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/db_flatstore.so
Reading symbols from /usr/local/lib64/kamailio/modules/tm.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/tm.so
Reading symbols from /usr/local/lib64/kamailio/modules/tmx.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/tmx.so
Reading symbols from /usr/local/lib64/kamailio/libkmi.so.1...done.
Loaded symbols for /usr/local/lib64/kamailio/libkmi.so.1
Reading symbols from /usr/local/lib64/kamailio/libkcore.so.1...done.
Loaded symbols for /usr/local/lib64/kamailio/libkcore.so.1
Reading symbols from /usr/local/lib64/kamailio/modules/corex.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/corex.so
Reading symbols from /usr/local/lib64/kamailio/modules/sl.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/sl.so
Reading symbols from /usr/local/lib64/kamailio/modules/kex.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/kex.so
Reading symbols from /usr/local/lib64/kamailio/libsrutils.so.1...done.
Loaded symbols for /usr/local/lib64/kamailio/libsrutils.so.1
Reading symbols from /usr/local/lib64/kamailio/modules/rr.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/rr.so
Reading symbols from /usr/local/lib64/kamailio/modules/maxfwd.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/maxfwd.so
Reading symbols from /usr/local/lib64/kamailio/modules/usrloc.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/usrloc.so
Reading symbols from /usr/local/lib64/kamailio/modules/registrar.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/registrar.so
Reading symbols from /usr/local/lib64/kamailio/modules/auth.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/auth.so
Reading symbols from /usr/local/lib64/kamailio/modules/auth_db.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/auth_db.so
Reading symbols from /usr/local/lib64/kamailio/modules/uac.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/uac.so
Reading symbols from /usr/local/lib64/kamailio/modules/uri_db.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/uri_db.so
Reading symbols from /usr/local/lib64/kamailio/modules/alias_db.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/alias_db.so
Reading symbols from /usr/local/lib64/kamailio/modules/domain.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/domain.so
Reading symbols from /usr/local/lib64/kamailio/modules/avpops.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/avpops.so
Reading symbols from /usr/local/lib64/kamailio/modules/textops.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/textops.so
Reading symbols from /usr/local/lib64/kamailio/modules/textopsx.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/textopsx.so
Reading symbols from /usr/local/lib64/kamailio/modules/xlog.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/xlog.so
Reading symbols from /usr/local/lib64/kamailio/modules/permissions.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/permissions.so
Reading symbols from /usr/local/lib64/kamailio/modules/acc.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/acc.so
Reading symbols from /usr/local/lib64/kamailio/modules/mi_fifo.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/mi_fifo.so
Reading symbols from /usr/local/lib64/kamailio/modules/mi_rpc.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/mi_rpc.so
Reading symbols from /usr/local/lib64/kamailio/libbinrpc.so.0...done.
Loaded symbols for /usr/local/lib64/kamailio/libbinrpc.so.0
Reading symbols from /usr/local/lib64/kamailio/modules/dialog.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/dialog.so
Reading symbols from /usr/local/lib64/kamailio/modules/exec.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/exec.so
Reading symbols from /usr/local/lib64/kamailio/modules/sanity.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/sanity.so
Reading symbols from /usr/local/lib64/kamailio/modules/ctl.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/ctl.so
Reading symbols from /usr/local/lib64/kamailio/modules/pv.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/pv.so
Reading symbols from /usr/local/lib64/kamailio/modules/regex.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/regex.so
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /usr/local/lib64/kamailio/modules/cfg_rpc.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/cfg_rpc.so
Reading symbols from /usr/local/lib64/kamailio/modules/uac_redirect.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/uac_redirect.so
Reading symbols from /usr/local/lib64/kamailio/modules/nathelper.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/nathelper.so
Reading symbols from /usr/local/lib64/kamailio/modules/rtpproxy.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/rtpproxy.so
Reading symbols from /usr/local/lib64/kamailio/modules/tls.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/tls.so
Reading symbols from /opt/openssl/lib/libssl.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /opt/openssl/lib/libssl.so.1.0.0
Reading symbols from /opt/openssl/lib/libcrypto.so.1.0.0...(no debugging symbols found)...done.
Loaded symbols for /opt/openssl/lib/libcrypto.so.1.0.0
Reading symbols from /usr/local/lib64/kamailio/modules/drouting.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/drouting.so
Reading symbols from /usr/local/lib64/kamailio/modules/siptrace.so...done.
Loaded symbols for /usr/local/lib64/kamailio/modules/siptrace.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -f /etc/kamailio/kamailio.cfg'.
Program terminated with signal 6, Aborted.
#0  0x00000031e9e30265 in raise () from /lib64/libc.so.6
(gdb) set pagination off
(gdb) info threads
* 1 Thread 12588  0x00000031e9e30265 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00000031e9e30265 in raise () from /lib64/libc.so.6
#1  0x00000031e9e31d10 in abort () from /lib64/libc.so.6
#2  0x00000000004687de in sig_alarm_abort (signo=14) at main.c:687
#3  <signal handler called>
#4  0x00000031e9ed07c9 in syscall () from /lib64/libc.so.6
#5  0x00002acddbc4ff24 in futex_get (lock=0x2acdddcad6d8) at ../../mem/../futexlock.h:110
#6  0x00002acddbc50555 in remove_dialog_timer (tl=0x2acde2bcb708) at dlg_timer.c:168
#7  0x00002acddbc3edb8 in destroy_dlg (dlg=0x2acde2bcb6b0) at dlg_hash.c:357
#8  0x00002acddbc3f70a in destroy_dlg_table () at dlg_hash.c:438
#9  0x00002acddbc25c21 in mod_destroy () at dialog.c:776
#10 0x00000000004f7ebb in destroy_modules () at sr_module.c:817
#11 0x00000000004672c4 in cleanup (show_status=1) at main.c:562
#12 0x0000000000468952 in shutdown_children (sig=15, show_status=1) at main.c:704
#13 0x000000000046a012 in handle_sigs () at main.c:795
#14 0x000000000046e33c in main_loop () at main.c:1748
#15 0x0000000000470ac9 in main (argc=7, argv=0x7fff100e4068) at main.c:2547
(gdb) quit