[SR-Users] dmq/dmq_userloc crash on startup (kamailio 4.3.0)

[Asgaroth]

Hi All,

I created issue report #236 for this bug just to keep track of it :)

Thanks

On 02/07/2015 08:02, Asgaroth wrote:
> Thanks for the quick look Daniel.
>
> If anyone requires any further info, dont hesitate to ask, I still 
> have the core file.
>
> On 02/07/2015 07:33, Daniel-Constantin Mierla wrote:
>> Hello,
>>
>> at the quick look it seems that dmq_usrloc was not updated with the
>> latest format of the serialized structure returned by usrloc when
>> fetching the contacts. It seems to use the same function as nathelper
>> (which use it for sending keepalives) and that is returning both contact
>> and received addresses, but dmq_usrloc seem to take only one of them
>> (like in the older versions of the function).
>>
>> I am traveling for a while, but I guess some other devs that use dmq can
>> look at it.
>>
>> Cheers,
>> Daniel
>>
>> On 01/07/15 23:33, Asgaroth wrote:
>>> Hi All,
>>>
>>> Another thing I noticed while playing with kamailio and dmq/dmq_usrloc
>>> is that 7 out of 10 times the kamailio process crashes when I
>>> initially start it up.
>>>
>>> Any tips/requests for further debugging would be greatly appreciated.
>>>
>>> Kamailio version:
>>>
>>> version: kamailio 4.3.0 (x86_64/linux) c6aa95
>>> flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS,
>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>>> F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
>>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
>>> HAVE_RESOLV_RES
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>> id: c6aa95
>>> compiled on 16:14:27 Jun 23 2015 with gcc 4.4.7
>>>
>>> I start kamailio as follows, from commandline:
>>>
>>> /usr/sbin/kamailio -P /var/run/kamailio_registrar01/kamailio.pid -f
>>> /shared/kamailio/etc/kamailio_registrar.cfg -l "udp:(10.6.0.173,
>>> 10.7.0.173)" -m 1024 -M64 -A RESOURCE_REGISTRAR01 -l tcp:10.6.0.173:80
>>> -u root -g root -ddd
>>>
>>> DMQ related settings are as follows:
>>>
>>> # grep -i dmq module_definitions.cfg
>>> loadmodule "dmq.so"
>>> loadmodule "dmq_usrloc.so"
>>> # ----- dmq params -----
>>> modparam("dmq", "server_address", DMQ_ADDRESS)
>>> modparam("dmq", "notification_address",
>>> "sip:sip-registrars.internal_domain.com:5060")
>>> modparam("dmq", "multi_notify", 1)
>>> modparam("dmq", "num_workers", 4)
>>> # ----- dmq_usrloc params -----
>>> modparam("dmq_usrloc", "enable", 1)
>>>
>>> DMQ_ADDRESS is defined as "sip:10.6.0.173:5060" in this servers case.
>>> sip-registrars.internal_domain.com resolves to the following address
>>> list 10.6.0.173, 10.6.0.174, 10.6.0.175
>>>
>>> Here is the debug log of the crash:
>>>
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [worker.c:82]: worker_loop(): dmq_worker [0 27445] getting lock
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [worker.c:84]: worker_loop(): dmq_worker [0 27445] lock acquired
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param:
>>> tag=66fabe782d71b7a9842d4a6a5eb93c4d-1de2
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header
>>> reached, state=29
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [notification_peer.c:454]: dmq_notification_callback(): dmq triggered
>>> from dmq_notification_callback
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [dmqnode.c:153]: build_dmq_node(): build_dmq_node
>>> sip:10.6.0.173:5060;status=active with private memory
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [dmqnode.c:153]: build_dmq_node(): build_dmq_node
>>> sip:10.6.0.175:5060;status=active with private memory
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [dmqnode.c:153]: build_dmq_node(): build_dmq_node
>>> sip:10.6.0.174:5060;status=active with private memory
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [notification_peer.c:468]: dmq_notification_callback(): received 0 new
>>> or changed nodes
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [notification_peer.c:528]: build_notification_body(): body_len = 1024
>>> - clen = 0
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [notification_peer.c:528]: build_notification_body(): body_len = 1024
>>> - clen = 35
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [notification_peer.c:528]: build_notification_body(): body_len = 1024
>>> - clen = 70
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>>> dmq_usrloc [usrloc_sync.c:388]: usrloc_dmq_request_sync(): requesting
>>> sync from dmq peers
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>>> dmq_usrloc [usrloc_sync.c:404]: usrloc_dmq_request_sync(): sending
>>> serialized data {"action":3}
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>>> dmq_usrloc [usrloc_sync.c:226]: usrloc_dmq_send(): sending dmq
>>> broadcast...
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 &&
>>> [10.6.0.173] == [10.6.0.173]
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [socket_info.c:566]: grep_sock_info(): checking if port 5060
>>> (advertise 0) matches port 5060
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm
>>> [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac:
>>> next_hop=<sip:usrloc at 10.6.0.174:5060>
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm
>>> [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24685
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 &&
>>> [10.6.0.173] == [10.6.0.173]
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [socket_info.c:566]: grep_sock_info(): checking if port 5060
>>> (advertise 0) matches port 5060
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm
>>> [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac:
>>> next_hop=<sip:usrloc at 10.6.0.175:5060>
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm
>>> [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24686
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [dmq_funcs.c:163]: bcast_dmq_message(): skipping node 
>>> sip:10.6.0.173:5060
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: sl
>>> [sl.c:280]: send_reply(): reply in stateless mode (sl)
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [msg_translator.c:158]: check_via_address(): (10.6.0.174, 
>>> 10.6.0.174, 0)
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq
>>> [worker.c:134]: worker_loop(): sent reply
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param:
>>> tag=66fabe782d71b7a9842d4a6a5eb93c4d-1ac2
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core>
>>> [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header
>>> reached, state=29
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>>> dmq_usrloc [usrloc_sync.c:250]: usrloc_dmq_handle_msg(): dmq message
>>> received from sip:usrloc at 10.6.0.174:5060
>>> Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG:
>>> dmq_usrloc [usrloc_sync.c:360]: usrloc_dmq_handle_msg(): Received
>>> DMQ_SYNC. Sending all contacts...
>>>
>>> And here is the backtrace from the core file:
>>>
>>> # gdb /usr/sbin/kamailio /core.27445
>>> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
>>> Copyright (C) 2010 Free Software Foundation, Inc.
>>> License GPLv3+: GNU GPL version 3 or later
>>> <http://gnu.org/licenses/gpl.html>
>>> This is free software: you are free to change and redistribute it.
>>> There is NO WARRANTY, to the extent permitted by law.  Type "show
>>> copying"
>>> and "show warranty" for details.
>>> This GDB was configured as "x86_64-redhat-linux-gnu".
>>> For bug reporting instructions, please see:
>>> <http://www.gnu.org/software/gdb/bugs/>...
>>> Reading symbols from /usr/sbin/kamailio...(no debugging symbols
>>> found)...done.
>>> [New Thread 27445]
>>> Reading symbols from /lib64/libdl.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libdl.so.2
>>> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libresolv.so.2
>>> Reading symbols from /lib64/libc.so.6...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libc.so.6
>>> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /lib64/ld-linux-x86-64.so.2
>>> Reading symbols from /usr/lib64/kamailio/modules/db_unixodbc.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/db_unixodbc.so
>>> Reading symbols from /usr/lib64/libodbc.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libodbc.so.2
>>> Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
>>> Reading symbols from /usr/lib64/kamailio/libkcore.so.1...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/libkcore.so.1
>>> Reading symbols from /usr/lib64/libltdl.so.7...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libltdl.so.7
>>> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
>>> found)...done.
>>> [Thread debugging using libthread_db enabled]
>>> Loaded symbols for /lib64/libpthread.so.0
>>> Reading symbols from /usr/lib64/kamailio/modules/mi_fifo.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/mi_fifo.so
>>> Reading symbols from /usr/lib64/kamailio/libkmi.so.1...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/libkmi.so.1
>>> Reading symbols from /usr/lib64/kamailio/modules/tm.so...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/tm.so
>>> Reading symbols from /usr/lib64/kamailio/modules/sl.so...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/sl.so
>>> Reading symbols from /usr/lib64/kamailio/modules/pv.so...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/pv.so
>>> Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
>>> Reading symbols from /lib64/libm.so.6...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libm.so.6
>>> Reading symbols from /usr/lib64/kamailio/modules/kex.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/kex.so
>>> Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
>>> Reading symbols from /usr/lib64/kamailio/modules/registrar.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
>>> Reading symbols from /usr/lib64/kamailio/modules/xlog.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
>>> Reading symbols from /usr/lib64/kamailio/modules/auth.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/auth.so
>>> Reading symbols from /usr/lib64/kamailio/modules/auth_db.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/auth_db.so
>>> Reading symbols from /usr/lib64/kamailio/modules/rr.so...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/rr.so
>>> Reading symbols from /usr/lib64/kamailio/modules/path.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/path.so
>>> Reading symbols from /usr/lib64/kamailio/modules/nathelper.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/nathelper.so
>>> Reading symbols from /usr/lib64/kamailio/modules/siputils.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
>>> Reading symbols from /usr/lib64/kamailio/modules/textops.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/textops.so
>>> Reading symbols from /usr/lib64/kamailio/modules/xmlrpc.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/xmlrpc.so
>>> Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libxml2.so.2
>>> Reading symbols from /lib64/libz.so.1...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libz.so.1
>>> Reading symbols from /usr/lib64/kamailio/modules/ctl.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
>>> Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
>>> Reading symbols from /usr/lib64/kamailio/modules/dmq.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
>>> Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...(no
>>> debugging symbols found)...done.
>>> Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
>>> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libnss_files.so.2
>>> Reading symbols from /usr/lib64/gconv/ISO8859-1.so...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/gconv/ISO8859-1.so
>>> Reading symbols from /usr/lib64/psqlodbc.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/psqlodbc.so
>>> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libssl.so.10
>>> Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libpq.so.5
>>> Reading symbols from /usr/lib64/libodbcinst.so.2...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/libodbcinst.so.2
>>> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /lib64/libgssapi_krb5.so.2
>>> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libkrb5.so.3
>>> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libcom_err.so.2
>>> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libk5crypto.so.3
>>> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /usr/lib64/libcrypto.so.10
>>> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libcrypt.so.1
>>> Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libldap_r-2.4.so.2
>>> Reading symbols from /lib64/libkrb5support.so.0...(no debugging
>>> symbols found)...done.
>>> Loaded symbols for /lib64/libkrb5support.so.0
>>> Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libkeyutils.so.1
>>> Reading symbols from /lib64/libfreebl3.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libfreebl3.so
>>> Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/liblber-2.4.so.2
>>> Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libsasl2.so.2
>>> Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libssl3.so
>>> Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libsmime3.so
>>> Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libnss3.so
>>> Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /usr/lib64/libnssutil3.so
>>> Reading symbols from /lib64/libplds4.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libplds4.so
>>> Reading symbols from /lib64/libplc4.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libplc4.so
>>> Reading symbols from /lib64/libnspr4.so...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libnspr4.so
>>> Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libselinux.so.1
>>> Reading symbols from /lib64/librt.so.1...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/librt.so.1
>>> Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
>>> found)...done.
>>> Loaded symbols for /lib64/libnss_dns.so.2
>>> Core was generated by `/usr/sbin/kamailio -P
>>> /var/run/kamailio_registrar01/kamailio.pid -f /shared/kam'.
>>> Program terminated with signal 11, Segmentation fault.
>>> #0  0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6
>>> Missing separate debuginfos, use: debuginfo-install
>>> kamailio-4.3.0-0.el6.bf.x86_64
>>> (gdb) bt
>>> #0  0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6
>>> #1  0x00007fdacfeff03d in usrloc_get_all_ucontact () from
>>> /usr/lib64/kamailio/modules/dmq_usrloc.so
>>> #2  0x00007fdacff02dfd in usrloc_dmq_handle_msg () from
>>> /usr/lib64/kamailio/modules/dmq_usrloc.so
>>> #3  0x00007fdad012d4a7 in worker_loop () from
>>> /usr/lib64/kamailio/modules/dmq.so
>>> #4  0x00007fdad011232c in ?? () from /usr/lib64/kamailio/modules/dmq.so
>>> #5  0x0000000000592675 in ?? ()
>>> #6  0x00000000005923b0 in ?? ()
>>> #7  0x000000000059297f in init_child ()
>>> #8  0x00000000004a7680 in main_loop ()
>>> #9  0x00000000004acd36 in main ()
>>> (gdb) ^C(gdb) Quit
>>>
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users at lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>