[sr-dev] [kamailio] Crash in startup when using DMQ_USRLOC (#236)

00Asgaroth00 notifications at github.com
Tue Jul 7 12:47:24 CEST 2015 

Hi,

This is a copy/paste from the user's mailing list, thought I would log a bug to keep track of it.

Original message follows:

--------------------------------------------------------------------------------------------------------------------
Hi All,

Another thing I noticed while playing with kamailio and dmq/dmq_usrloc is that 7 out of 10 times the kamailio process crashes when I initially start it up.

Any tips/requests for further debugging would be greatly appreciated.

Kamailio version:

version: kamailio 4.3.0 (x86_64/linux) c6aa95
flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: c6aa95
compiled on 16:14:27 Jun 23 2015 with gcc 4.4.7

I start kamailio as follows, from commandline:

/usr/sbin/kamailio -P /var/run/kamailio_registrar01/kamailio.pid -f /shared/kamailio/etc/kamailio_registrar.cfg -l "udp:(10.6.0.173, 10.7.0.173)" -m 1024 -M64 -A RESOURCE_REGISTRAR01 -l tcp:10.6.0.173:80 -u root -g root -ddd

DMQ related settings are as follows:

# grep -i dmq module_definitions.cfg
loadmodule "dmq.so"
loadmodule "dmq_usrloc.so"
# ----- dmq params -----
modparam("dmq", "server_address", DMQ_ADDRESS)
modparam("dmq", "notification_address", "sip:sip-registrars.internal_domain.com:5060")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
# ----- dmq_usrloc params -----
modparam("dmq_usrloc", "enable", 1)

DMQ_ADDRESS is defined as "sip:10.6.0.173:5060" in this servers case.
sip-registrars.internal_domain.com resolves to the following address list 10.6.0.173, 10.6.0.174, 10.6.0.175

Here is the debug log of the crash:

Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [worker.c:82]: worker_loop(): dmq_worker [0 27445] getting lock
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [worker.c:84]: worker_loop(): dmq_worker [0 27445] lock acquired
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param: tag=66fabe782d71b7a9842d4a6a5eb93c4d-1de2
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header reached, state=29
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:454]: dmq_notification_callback(): dmq triggered from dmq_notification_callback
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [dmqnode.c:153]: build_dmq_node(): build_dmq_node sip:10.6.0.173:5060;status=active with private memory
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [dmqnode.c:153]: build_dmq_node(): build_dmq_node sip:10.6.0.175:5060;status=active with private memory
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [dmqnode.c:153]: build_dmq_node(): build_dmq_node sip:10.6.0.174:5060;status=active with private memory
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:468]: dmq_notification_callback(): received 0 new or changed nodes
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:528]: build_notification_body(): body_len = 1024 - clen = 0
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:528]: build_notification_body(): body_len = 1024 - clen = 35
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:528]: build_notification_body(): body_len = 1024 - clen = 70
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:388]: usrloc_dmq_request_sync(): requesting sync from dmq peers
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:404]: usrloc_dmq_request_sync(): sending serialized data {"action":3}
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:226]: usrloc_dmq_send(): sending dmq broadcast...
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 && [10.6.0.173] == [10.6.0.173]
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [socket_info.c:566]: grep_sock_info(): checking if port 5060 (advertise 0) matches port 5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac: next_hop=<sip:usrloc at 10.6.0.174:5060>
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24685
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 && [10.6.0.173] == [10.6.0.173]
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [socket_info.c:566]: grep_sock_info(): checking if port 5060 (advertise 0) matches port 5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac: next_hop=<sip:usrloc at 10.6.0.175:5060>
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24686
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [dmq_funcs.c:163]: bcast_dmq_message(): skipping node sip:10.6.0.173:5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: sl [sl.c:280]: send_reply(): reply in stateless mode (sl)
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [msg_translator.c:158]: check_via_address(): (10.6.0.174, 10.6.0.174, 0)
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [worker.c:134]: worker_loop(): sent reply
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param: tag=66fabe782d71b7a9842d4a6a5eb93c4d-1ac2
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header reached, state=29
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:250]: usrloc_dmq_handle_msg(): dmq message received from sip:usrloc at 10.6.0.174:5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:360]: usrloc_dmq_handle_msg(): Received DMQ_SYNC. Sending all contacts...

And here is the backtrace from the core file:

# gdb /usr/sbin/kamailio /core.27445
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/kamailio...(no debugging symbols found)...done.
[New Thread 27445]
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/kamailio/modules/db_unixodbc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/db_unixodbc.so
Reading symbols from /usr/lib64/libodbc.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libodbc.so.2
Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
Reading symbols from /usr/lib64/kamailio/libkcore.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/libkcore.so.1
Reading symbols from /usr/lib64/libltdl.so.7...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libltdl.so.7
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/kamailio/modules/mi_fifo.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/mi_fifo.so
Reading symbols from /usr/lib64/kamailio/libkmi.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/libkmi.so.1
Reading symbols from /usr/lib64/kamailio/modules/tm.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/tm.so
Reading symbols from /usr/lib64/kamailio/modules/sl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/sl.so
Reading symbols from /usr/lib64/kamailio/modules/pv.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/pv.so
Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/kamailio/modules/kex.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/kex.so
Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
Reading symbols from /usr/lib64/kamailio/modules/registrar.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
Reading symbols from /usr/lib64/kamailio/modules/xlog.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
Reading symbols from /usr/lib64/kamailio/modules/auth.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/auth.so
Reading symbols from /usr/lib64/kamailio/modules/auth_db.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/auth_db.so
Reading symbols from /usr/lib64/kamailio/modules/rr.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/rr.so
Reading symbols from /usr/lib64/kamailio/modules/path.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/path.so
Reading symbols from /usr/lib64/kamailio/modules/nathelper.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/nathelper.so
Reading symbols from /usr/lib64/kamailio/modules/siputils.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
Reading symbols from /usr/lib64/kamailio/modules/textops.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/textops.so
Reading symbols from /usr/lib64/kamailio/modules/xmlrpc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/xmlrpc.so
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/kamailio/modules/ctl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
Reading symbols from /usr/lib64/kamailio/modules/dmq.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/gconv/ISO8859-1.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/gconv/ISO8859-1.so
Reading symbols from /usr/lib64/psqlodbc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/psqlodbc.so
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from /usr/lib64/libodbcinst.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libodbcinst.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio_registrar01/kamailio.pid -f /shared/kam'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install kamailio-4.3.0-0.el6.bf.x86_64
(gdb) bt
#0  0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6
#1  0x00007fdacfeff03d in usrloc_get_all_ucontact () from /usr/lib64/kamailio/modules/dmq_usrloc.so
#2  0x00007fdacff02dfd in usrloc_dmq_handle_msg () from /usr/lib64/kamailio/modules/dmq_usrloc.so
#3  0x00007fdad012d4a7 in worker_loop () from /usr/lib64/kamailio/modules/dmq.so
#4  0x00007fdad011232c in ?? () from /usr/lib64/kamailio/modules/dmq.so
#5  0x0000000000592675 in ?? ()
#6  0x00000000005923b0 in ?? ()
#7  0x000000000059297f in init_child ()
#8  0x00000000004a7680 in main_loop ()
#9  0x00000000004acd36 in main ()
(gdb) ^C(gdb) Quit



---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/236
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-dev/attachments/20150707/95901d37/attachment-0001.html>

More information about the sr-dev mailing list