[SR-Users] dmq/dmq_userloc crash on startup (kamailio 4.3.0)

Asgaroth 00asgaroth00 at gmail.com
Thu Jul 2 09:02:15 CEST 2015 

Thanks for the quick look Daniel.

If anyone requires any further info, dont hesitate to ask, I still have 
the core file.

On 02/07/2015 07:33, Daniel-Constantin Mierla wrote:
> Hello,
>
> at the quick look it seems that dmq_usrloc was not updated with the
> latest format of the serialized structure returned by usrloc when
> fetching the contacts. It seems to use the same function as nathelper
> (which use it for sending keepalives) and that is returning both contact
> and received addresses, but dmq_usrloc seem to take only one of them
> (like in the older versions of the function).
>
> I am traveling for a while, but I guess some other devs that use dmq can
> look at it.
>
> Cheers,
> Daniel
>
> On 01/07/15 23:33, Asgaroth wrote:
>> Hi All,
>>
>> Another thing I noticed while playing with kamailio and dmq/dmq_usrloc
>> is that 7 out of 10 times the kamailio process crashes when I
>> initially start it up.
>>
>> Any tips/requests for further debugging would be greatly appreciated.
>>
>> Kamailio version:
>>
>> version: kamailio 4.3.0 (x86_64/linux) c6aa95
>> flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS,
>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>> F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
>> HAVE_RESOLV_RES
>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>> id: c6aa95
>> compiled on 16:14:27 Jun 23 2015 with gcc 4.4.7
>>
>> I start kamailio as follows, from commandline:
>>
>> /usr/sbin/kamailio -P /var/run/kamailio_registrar01/kamailio.pid -f
>> /shared/kamailio/etc/kamailio_registrar.cfg -l "udp:(10.6.0.173,
>> 10.7.0.173)" -m 1024 -M64 -A RESOURCE_REGISTRAR01 -l tcp:10.6.0.173:80
>> -u root -g root -ddd
>>
>> DMQ related settings are as follows:
>>
>> # grep -i dmq module_definitions.cfg
>> loadmodule "dmq.so"
>> loadmodule "dmq_usrloc.so"
>> # ----- dmq params -----
>> modparam("dmq", "server_address", DMQ_ADDRESS)
>> modparam("dmq", "notification_address",
>> "sip:sip-registrars.internal_domain.com:5060")
>> modparam("dmq", "multi_notify", 1)
>> modparam("dmq", "num_workers", 4)
>> # ----- dmq_usrloc params -----
>> modparam("dmq_usrloc", "enable", 1)
>>
>> DMQ_ADDRESS is defined as "sip:10.6.0.173:5060" in this servers case.
>> sip-registrars.internal_domain.com resolves to the following address
>> list 10.6.0.173, 10.6.0.174, 10.6.0.175
>>
>> Here is the debug log of the crash:
>>
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [worker.c:82]: worker_loop(): dmq_worker [0 27445] getting lock
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [worker.c:84]: worker_loop(): dmq_worker [0 27445] lock acquired
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param:
>> tag=66fabe782d71b7a9842d4a6a5eb93c4d-1de2
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header
>> reached, state=29
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [notification_peer.c:454]: dmq_notification_callback(): dmq triggered
>> from dmq_notification_callback
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [dmqnode.c:153]: build_dmq_node(): build_dmq_node
>> sip:10.6.0.173:5060;status=active with private memory
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [dmqnode.c:153]: build_dmq_node(): build_dmq_node
>> sip:10.6.0.175:5060;status=active with private memory
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [dmqnode.c:153]: build_dmq_node(): build_dmq_node
>> sip:10.6.0.174:5060;status=active with private memory
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [notification_peer.c:468]: dmq_notification_callback(): received 0 new
>> or changed nodes
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [notification_peer.c:528]: build_notification_body(): body_len = 1024
>> - clen = 0
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [notification_peer.c:528]: build_notification_body(): body_len = 1024
>> - clen = 35
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [notification_peer.c:528]: build_notification_body(): body_len = 1024
>> - clen = 70
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>> dmq_usrloc [usrloc_sync.c:388]: usrloc_dmq_request_sync(): requesting
>> sync from dmq peers
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>> dmq_usrloc [usrloc_sync.c:404]: usrloc_dmq_request_sync(): sending
>> serialized data {"action":3}
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>> dmq_usrloc [usrloc_sync.c:226]: usrloc_dmq_send(): sending dmq
>> broadcast...
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 &&
>> [10.6.0.173] == [10.6.0.173]
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [socket_info.c:566]: grep_sock_info(): checking if port 5060
>> (advertise 0) matches port 5060
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm
>> [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac:
>> next_hop=<sip:usrloc at 10.6.0.174:5060>
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm
>> [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24685
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 &&
>> [10.6.0.173] == [10.6.0.173]
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [socket_info.c:566]: grep_sock_info(): checking if port 5060
>> (advertise 0) matches port 5060
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm
>> [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac:
>> next_hop=<sip:usrloc at 10.6.0.175:5060>
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm
>> [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24686
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [dmq_funcs.c:163]: bcast_dmq_message(): skipping node sip:10.6.0.173:5060
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: sl
>> [sl.c:280]: send_reply(): reply in stateless mode (sl)
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [msg_translator.c:158]: check_via_address(): (10.6.0.174, 10.6.0.174, 0)
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>> [worker.c:134]: worker_loop(): sent reply
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param:
>> tag=66fabe782d71b7a9842d4a6a5eb93c4d-1ac2
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>> [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header
>> reached, state=29
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>> dmq_usrloc [usrloc_sync.c:250]: usrloc_dmq_handle_msg(): dmq message
>> received from sip:usrloc at 10.6.0.174:5060
>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>> dmq_usrloc [usrloc_sync.c:360]: usrloc_dmq_handle_msg(): Received
>> DMQ_SYNC. Sending all contacts...
>>
>> And here is the backtrace from the core file:
>>
>> # gdb /usr/sbin/kamailio /core.27445
>> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
>> Copyright (C) 2010 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>> copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from /usr/sbin/kamailio...(no debugging symbols
>> found)...done.
>> [New Thread 27445]
>> Reading symbols from /lib64/libdl.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libdl.so.2
>> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libresolv.so.2
>> Reading symbols from /lib64/libc.so.6...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libc.so.6
>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>> Reading symbols from /usr/lib64/kamailio/modules/db_unixodbc.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/db_unixodbc.so
>> Reading symbols from /usr/lib64/libodbc.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libodbc.so.2
>> Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
>> Reading symbols from /usr/lib64/kamailio/libkcore.so.1...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/libkcore.so.1
>> Reading symbols from /usr/lib64/libltdl.so.7...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libltdl.so.7
>> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
>> found)...done.
>> [Thread debugging using libthread_db enabled]
>> Loaded symbols for /lib64/libpthread.so.0
>> Reading symbols from /usr/lib64/kamailio/modules/mi_fifo.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/mi_fifo.so
>> Reading symbols from /usr/lib64/kamailio/libkmi.so.1...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/libkmi.so.1
>> Reading symbols from /usr/lib64/kamailio/modules/tm.so...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/tm.so
>> Reading symbols from /usr/lib64/kamailio/modules/sl.so...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/sl.so
>> Reading symbols from /usr/lib64/kamailio/modules/pv.so...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/pv.so
>> Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
>> Reading symbols from /lib64/libm.so.6...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libm.so.6
>> Reading symbols from /usr/lib64/kamailio/modules/kex.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/kex.so
>> Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
>> Reading symbols from /usr/lib64/kamailio/modules/registrar.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
>> Reading symbols from /usr/lib64/kamailio/modules/xlog.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
>> Reading symbols from /usr/lib64/kamailio/modules/auth.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/auth.so
>> Reading symbols from /usr/lib64/kamailio/modules/auth_db.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/auth_db.so
>> Reading symbols from /usr/lib64/kamailio/modules/rr.so...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/rr.so
>> Reading symbols from /usr/lib64/kamailio/modules/path.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/path.so
>> Reading symbols from /usr/lib64/kamailio/modules/nathelper.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/nathelper.so
>> Reading symbols from /usr/lib64/kamailio/modules/siputils.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
>> Reading symbols from /usr/lib64/kamailio/modules/textops.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/textops.so
>> Reading symbols from /usr/lib64/kamailio/modules/xmlrpc.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/xmlrpc.so
>> Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libxml2.so.2
>> Reading symbols from /lib64/libz.so.1...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libz.so.1
>> Reading symbols from /usr/lib64/kamailio/modules/ctl.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
>> Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
>> Reading symbols from /usr/lib64/kamailio/modules/dmq.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
>> Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...(no
>> debugging symbols found)...done.
>> Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
>> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libnss_files.so.2
>> Reading symbols from /usr/lib64/gconv/ISO8859-1.so...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/gconv/ISO8859-1.so
>> Reading symbols from /usr/lib64/psqlodbc.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/psqlodbc.so
>> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libssl.so.10
>> Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libpq.so.5
>> Reading symbols from /usr/lib64/libodbcinst.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/libodbcinst.so.2
>> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/libgssapi_krb5.so.2
>> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libkrb5.so.3
>> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libcom_err.so.2
>> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libk5crypto.so.3
>> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
>> symbols found)...done.
>> Loaded symbols for /usr/lib64/libcrypto.so.10
>> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libcrypt.so.1
>> Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libldap_r-2.4.so.2
>> Reading symbols from /lib64/libkrb5support.so.0...(no debugging
>> symbols found)...done.
>> Loaded symbols for /lib64/libkrb5support.so.0
>> Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libkeyutils.so.1
>> Reading symbols from /lib64/libfreebl3.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libfreebl3.so
>> Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/liblber-2.4.so.2
>> Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libsasl2.so.2
>> Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libssl3.so
>> Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libsmime3.so
>> Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libnss3.so
>> Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /usr/lib64/libnssutil3.so
>> Reading symbols from /lib64/libplds4.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libplds4.so
>> Reading symbols from /lib64/libplc4.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libplc4.so
>> Reading symbols from /lib64/libnspr4.so...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libnspr4.so
>> Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libselinux.so.1
>> Reading symbols from /lib64/librt.so.1...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/librt.so.1
>> Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
>> found)...done.
>> Loaded symbols for /lib64/libnss_dns.so.2
>> Core was generated by `/usr/sbin/kamailio -P
>> /var/run/kamailio_registrar01/kamailio.pid -f /shared/kam'.
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6
>> Missing separate debuginfos, use: debuginfo-install
>> kamailio-4.3.0-0.el6.bf.x86_64
>> (gdb) bt
>> #0  0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6
>> #1  0x00007fdacfeff03d in usrloc_get_all_ucontact () from
>> /usr/lib64/kamailio/modules/dmq_usrloc.so
>> #2  0x00007fdacff02dfd in usrloc_dmq_handle_msg () from
>> /usr/lib64/kamailio/modules/dmq_usrloc.so
>> #3  0x00007fdad012d4a7 in worker_loop () from
>> /usr/lib64/kamailio/modules/dmq.so
>> #4  0x00007fdad011232c in ?? () from /usr/lib64/kamailio/modules/dmq.so
>> #5  0x0000000000592675 in ?? ()
>> #6  0x00000000005923b0 in ?? ()
>> #7  0x000000000059297f in init_child ()
>> #8  0x00000000004a7680 in main_loop ()
>> #9  0x00000000004acd36 in main ()
>> (gdb) ^C(gdb) Quit
>>
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>> sr-users at lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

More information about the sr-users mailing list