[SR-Users] Segmentation fault in DMQ module

Daniel-Constantin Mierla miconda at gmail.com
Fri Mar 27 08:39:32 CET 2020


Thanks for testing and the feedback!

Check also if this one can affect you:

  * https://github.com/kamailio/kamailio/issues/2224#issuecomment-602730307

I just backported all of them to 5.3, so if you want to test, you can
use directly the latest branch 5.3.

Cheers,
Daniel

On 26.03.20 13:20, Virgílio Cunha wrote:
> Hello Daniel,
>
> I followed your recommendation and upgraded to kamailio-5.3.3 but the
> problem remains with this version.
> So I applied this
> patch: https://github.com/kamailio/kamailio/commit/21bdbff049571ca22f1205a041c8b1162005d319 and
> I run the tests again. 
> After applying the patch, I was no longer able to replicate the problem :)
>
> Thank you for your great support.
>
> Best Regards,
> Virgílio Cunha
>
> Em qua., 18 de mar. de 2020 às 18:22, Virgílio Cunha
> <virgilio18vsc at gmail.com <mailto:virgilio18vsc at gmail.com>> escreveu:
>
>     Hi,
>
>     Sorry for using this topic for this question, but where can I find
>     Kamailio's release policy? 
>     I needed to know how long the kamailio 5.3.x branch will maintain
>     its support.
>
>     Best Regards,
>     Virgílio Cunha
>
>     Em ter., 17 de mar. de 2020 às 11:03, Daniel-Constantin Mierla
>     <miconda at gmail.com <mailto:miconda at gmail.com>> escreveu:
>
>         Hello,
>
>         if it is a runtime issue, then it is better to upgrade to a
>         supported release series and try to reproduce it.
>
>         Looking at startup/shutdown code is easier than the rest used
>         during runtime by a module, but it is not the case here. So we
>         may waste a lot of time to dig into it and discover it was
>         fixed or the code in an older release was refactored not
>         matching any longer what is now in stable releases.
>
>         Cheers,
>         Daniel
>
>         On 17.03.20 11:35, Virgílio Cunha wrote:
>>         Hi,
>>
>>         The problem does not just occur when kamailio is restarted.
>>         The restart was the way I found to reproduce the problem,
>>         because when a proxy is restarted, there is more traffic from
>>         DMQ messages, so the problem is more likely to occur.
>>
>>         The full backtrace is following:
>>
>>         GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
>>         Copyright (C) 2010 Free Software Foundation, Inc.
>>         License GPLv3+: GNU GPL version 3 or later
>>         <http://gnu.org/licenses/gpl.html>
>>         This is free software: you are free to change and
>>         redistribute it.
>>         There is NO WARRANTY, to the extent permitted by law.  Type
>>         "show copying"
>>         and "show warranty" for details.
>>         This GDB was configured as "x86_64-redhat-linux-gnu".
>>         For bug reporting instructions, please see:
>>         <http://www.gnu.org/software/gdb/bugs/>...
>>         Reading symbols from
>>         /opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy...Reading
>>         symbols from /usr/lib/debug/usr/sbin/kamailio.debug...done.
>>         done.
>>         [New Thread 506]
>>         Reading symbols from /lib64/libdl.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libdl.so.2
>>         Reading symbols from /lib64/libresolv.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libresolv.so.2
>>         Reading symbols from /lib64/libc.so.6...(no debugging symbols
>>         found)...done.
>>         Loaded symbols for /lib64/libc.so.6
>>         Reading symbols from /lib64/ld-linux-x86-64.so.2...(no
>>         debugging symbols found)...done.
>>         Loaded symbols for /lib64/ld-linux-x86-64.so.2
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/db_postgres.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/db_postgres.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/db_postgres.so
>>         Reading symbols from /usr/lib64/libpq.so.5...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libpq.so.5
>>         Reading symbols from
>>         /usr/lib64/kamailio/libsrdb2.so.1...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/libsrdb2.so.1.0.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/libsrdb2.so.1
>>         Reading symbols from
>>         /usr/lib64/kamailio/libsrdb1.so.1...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/libsrdb1.so.1.0.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
>>         Reading symbols from /usr/lib64/libssl.so.10...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libssl.so.10
>>         Reading symbols from /usr/lib64/libcrypto.so.10...(no
>>         debugging symbols found)...done.
>>         Loaded symbols for /usr/lib64/libcrypto.so.10
>>         Reading symbols from /lib64/libgssapi_krb5.so.2...(no
>>         debugging symbols found)...done.
>>         Loaded symbols for /lib64/libgssapi_krb5.so.2
>>         Reading symbols from /lib64/libldap_r-2.4.so.2...(no
>>         debugging symbols found)...done.
>>         Loaded symbols for /lib64/libldap_r-2.4.so.2
>>         Reading symbols from /lib64/libpthread.so.0...(no debugging
>>         symbols found)...done.
>>         [Thread debugging using libthread_db enabled]
>>         Loaded symbols for /lib64/libpthread.so.0
>>         Reading symbols from /lib64/libkrb5.so.3...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libkrb5.so.3
>>         Reading symbols from /lib64/libcom_err.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libcom_err.so.2
>>         Reading symbols from /lib64/libk5crypto.so.3...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libk5crypto.so.3
>>         Reading symbols from /lib64/libz.so.1...(no debugging symbols
>>         found)...done.
>>         Loaded symbols for /lib64/libz.so.1
>>         Reading symbols from /lib64/libkrb5support.so.0...(no
>>         debugging symbols found)...done.
>>         Loaded symbols for /lib64/libkrb5support.so.0
>>         Reading symbols from /lib64/libkeyutils.so.1...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libkeyutils.so.1
>>         Reading symbols from /lib64/liblber-2.4.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/liblber-2.4.so.2
>>         Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libsasl2.so.2
>>         Reading symbols from /usr/lib64/libssl3.so...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libssl3.so
>>         Reading symbols from /usr/lib64/libsmime3.so...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libsmime3.so
>>         Reading symbols from /usr/lib64/libnss3.so...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libnss3.so
>>         Reading symbols from /usr/lib64/libnssutil3.so...(no
>>         debugging symbols found)...done.
>>         Loaded symbols for /usr/lib64/libnssutil3.so
>>         Reading symbols from /lib64/libplds4.so...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libplds4.so
>>         Reading symbols from /lib64/libplc4.so...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libplc4.so
>>         Reading symbols from /lib64/libnspr4.so...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libnspr4.so
>>         Reading symbols from /lib64/libselinux.so.1...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libselinux.so.1
>>         Reading symbols from /lib64/libcrypt.so.1...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libcrypt.so.1
>>         Reading symbols from /lib64/librt.so.1...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/librt.so.1
>>         Reading symbols from /lib64/libfreebl3.so...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libfreebl3.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/ctl.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/ctl.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
>>         Reading symbols from /lib64/libnss_files.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libnss_files.so.2
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/kex.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/kex.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/kex.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/libsrutils.so.1...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/libsrutils.so.1.0.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
>>         Reading symbols from /lib64/libm.so.6...(no debugging symbols
>>         found)...done.
>>         Loaded symbols for /lib64/libm.so.6
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/tm.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/tm.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/tm.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/tmx.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/tmx.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/tmx.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/corex.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/corex.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/corex.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/cfg_rpc.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/cfg_rpc.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/sl.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/sl.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/sl.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/rr.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/rr.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/rr.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/maxfwd.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/maxfwd.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/maxfwd.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/siputils.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/siputils.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/sanity.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/sanity.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/sanity.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/textops.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/textops.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/textops.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/textopsx.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/textopsx.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/textopsx.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/dmq.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/dmq.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/htable.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/htable.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/htable.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/pv.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/pv.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/pv.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/xlog.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/xlog.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/jsonrpcs.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/jsonrpcs.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/jsonrpcs.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/uac.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/uac.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/uac.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/uac_redirect.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/uac_redirect.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/uac_redirect.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/db_text.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/db_text.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/db_text.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/dispatcher.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/dispatcher.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/dispatcher.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/sqlops.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/sqlops.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/sqlops.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/regex.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/regex.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/regex.so
>>         Reading symbols from /lib64/libpcre.so.0...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libpcre.so.0
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/permissions.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/permissions.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/permissions.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/dialplan.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/dialplan.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/dialplan.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/mtree.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/mtree.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/mtree.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/cfgutils.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/cfgutils.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/cfgutils.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/statsd.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/statsd.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/statsd.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/app_lua.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/app_lua.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/app_lua.so
>>         Reading symbols from /usr/lib64/liblua-5.1.so...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/liblua-5.1.so
>>         <http://liblua-5.1.so>
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/rtimer.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/rtimer.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/rtimer.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/mqueue.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/mqueue.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/mqueue.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/auth.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/auth.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/auth.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/usrloc.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/usrloc.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/registrar.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/registrar.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/dmq_usrloc.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/dmq_usrloc.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/xhttp.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/xhttp.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/xhttp.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/exec.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/exec.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/exec.so
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/http_client.so...Reading symbols
>>         from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/http_client.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/http_client.so
>>         Reading symbols from /usr/lib64/libxml2.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libxml2.so.2
>>         Reading symbols from /usr/lib64/libcurl.so.4...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libcurl.so.4
>>         Reading symbols from /lib64/libidn.so.11...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libidn.so.11
>>         Reading symbols from /lib64/libldap-2.4.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libldap-2.4.so.2
>>         Reading symbols from /usr/lib64/libssh2.so.1...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /usr/lib64/libssh2.so.1
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/json.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/json.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/json.so
>>         Reading symbols from /lib64/libjson-c.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libjson-c.so.2
>>         Reading symbols from
>>         /usr/lib64/kamailio/modules/ipops.so...Reading symbols from
>>         /usr/lib/debug/usr/lib64/kamailio/modules/ipops.so.debug...done.
>>         done.
>>         Loaded symbols for /usr/lib64/kamailio/modules/ipops.so
>>         Reading symbols from /lib64/libnss_dns.so.2...(no debugging
>>         symbols found)...done.
>>         Loaded symbols for /lib64/libnss_dns.so.2
>>         Core was generated by
>>         `/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy -f
>>         /opt/ptin/kamailio-ksipadp/prox'.
>>         Program terminated with signal 11, Segmentation fault.
>>         #0  0x00007fefe58ea7c4 in atomic_get (v=0x0) at
>>         ../../core/atomic/atomic_common.h:66
>>         66              return atomic_get_int(&(v->val));
>>         Missing separate debuginfos, use: debuginfo-install
>>         kamailio-ksipadp-2.8.0-1.ptin.el6.x86_64
>>         (gdb) bt full
>>         #0  0x00007fefe58ea7c4 in atomic_get (v=0x0) at
>>         ../../core/atomic/atomic_common.h:66
>>         No locals.
>>         #1  0x00007fefe58ee246 in job_queue_size (queue=0x0) at
>>         worker.c:254
>>         No locals.
>>         #2  0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
>>         peer=0x7fefebcc4df8) at worker.c:184
>>                 i = 1
>>                 found_available = 0
>>                 new_job = {f = 0x7fefe2613797
>>         <usrloc_dmq_handle_msg>, msg = 0x7fefebce67c8, orig_peer =
>>         0x7fefebcc4df8, next = 0x0, prev = 0x0}
>>                 worker = 0x7fefe7a1d890
>>                 cloned_msg = 0x7fefebce67c8
>>                 cloned_msg_len = 3888
>>                 __FUNCTION__ = "add_dmq_job"
>>         #3  0x00007fefe58dfec1 in dmq_handle_message
>>         (msg=0x7ff028b453f8, str1=0x0, str2=0x0) at message.c:66
>>                 peer = 0x7fefebcc4df8
>>                 __FUNCTION__ = "dmq_handle_message"
>>         #4  0x0000000000433729 in do_action (h=0x7ffc89e35670,
>>         a=0x7ff0285a9340, msg=0x7ff028b453f8) at core/action.c:1054
>>                 ret = -5
>>                 v = 1
>>                 dst = {send_sock = 0x7ff0285a8810, to = {s =
>>         {sa_family = 21376, sa_data =
>>         "\343\211\374\177\000\000\370S\264(\360\177\000"}, sin = {
>>                       sin_family = 21376, sin_port = 35299, sin_addr
>>         = {s_addr = 32764}, sin_zero = "\370S\264(\360\177\000"},
>>         sin6 = {sin6_family = 21376,
>>                       sin6_port = 35299, sin6_flowinfo = 32764,
>>         sin6_addr = {__in6_u = {__u6_addr8 =
>>         "\370S\264(\360\177\000\000pV\343\211\374\177\000",
>>                           __u6_addr16 = {21496, 10420, 32752, 0,
>>         22128, 35299, 32764, 0}, __u6_addr32 = {682906616, 32752,
>>         2313377392, 32764}}},
>>                       sin6_scope_id = 1}}, id = 32764, proto = 0
>>         '\000', send_flags = {f = 0, blst_imask = 0}}
>>                 tmp = 0x389c422190 ""
>>                 new_uri = 0x7c00000000 <Address 0x7c00000000 out of
>>         bounds>
>>                 end = 0x6e00000077 <Address 0x6e00000077 out of bounds>
>>                 crt = 0x389c98e120 ""
>>                 cmd = 0x7ff0283219b0
>>                 len = 0
>>                 user = 0
>>                 uri = {user = {s = 0x7ffc89e34a10
>>         "@K\343\211\374\177", len = 4459394}, passwd = {s =
>>         0x7ffc89e34e80 "\356Q\343\211\374\177",
>>                     len = 682906616}, host = {s = 0x7ff0285a9020
>>         "\177\001", len = -1981589904}, port = {s = 0x0, len = 1},
>>         params = {
>>                     s = 0x7ff028b453f8 "\017", len = 295169284},
>>         sip_params = {s = 0x32 <Address 0x32 out of bounds>, len =
>>         1073741824}, headers = {
>>                     s = 0x2000000 <Address 0x2000000 out of bounds>,
>>         len = 4289104}, port_no = 60676, proto = 5047, type =
>>         2312043322, flags = 2168712452,
>>                   transport = {s = 0x0, len = -1675583518}, ttl = {s
>>         = 0x89e34e90 <Address 0x89e34e90 out of bounds>, len =
>>         35715777}, user_param = {
>>                     s = 0xffffffff0000003c <Address
>>         0xffffffff0000003c out of bounds>, len = -1671419584}, maddr
>>         = {s = 0x0, len = -1981592704}, method = {
>>                     s = 0x389c603c48 "", len = -1671391960}, lr = {s
>>         = 0x0, len = -1981592544}, r2 = {s = 0x1dde9a0 " 9\336\001",
>>         len = 0}, gr = {
>>                     s = 0x4 <Address 0x4 out of bounds>, len = 2479},
>>         transport_val = {s = 0x389c422190 "", len = 1}, ttl_val = {
>>                     s = 0x7ffc89e34b40 "pM\343\211\374\177", len =
>>         5683491}, user_param_val = {s = 0x40e7e8
>>         "\370\377\377\377\377\377\377\377@ݘ\234\070",
>>                     len = -1671362728}, maddr_val = {s = 0x404130 "",
>>         len = 0}, method_val = {s = 0x1000003b4 <Address 0x1000003b4
>>         out of bounds>, len = 0},
>>                   lr_val = {s = 0x7fefe7374b9a "%x%c%x%c", len =
>>         -1673386776}, r2_val = {s = 0x7ffc89e34bd0 "", len =
>>         -1981592584}, gr_val = {
>>         ---Type <return> to continue, or q <return> to quit---
>>                     s = 0x389c422190 "", len = 0}}
>>                 next_hop = {user = {s = 0x1 <Address 0x1 out of
>>         bounds>, len = 40}, passwd = {s = 0x0, len = -1407107317},
>>         host = {
>>                     s = 0x2000000 <Address 0x2000000 out of bounds>,
>>         len = 31398688}, port = {s = 0x1dde9a0 " 9\336\001", len =
>>         31320480}, params = {
>>                     s = 0x7ffc89e34c20 "", len = -1981593152},
>>         sip_params = {s = 0x7ffc89e34910 "\200N\343\211\374\177", len
>>         = 682906616}, headers = {
>>                     s = 0x32 <Address 0x32 out of bounds>, len =
>>         1073741824}, port_no = 0, proto = 512, type = ERROR_URI_T,
>>         flags = 4289104, transport = {
>>                     s = 0x7ffc89e34900 "\020J\343\211\374\177", len =
>>         4457737}, ttl = {s = 0x7ffc89e348f0 "", len = -1981593520},
>>         user_param = {
>>                     s = 0x7ffc89e34858 "\r", len = -1981593264},
>>         maddr = {s = 0x7ffc89e34870 "", len = -1981593456}, method =
>>         {s = 0x7ffc89e348b0 "\r",
>>                     len = -1981593392}, lr = {s = 0x8cd2d0 <Address
>>         0x8cd2d0 out of bounds>, len = 13}, r2 = {
>>                     s = 0x10100000002 <Address 0x10100000002 out of
>>         bounds>, len = -1}, gr = {s = 0x0, len = 0}, transport_val =
>>         {s = 0x0, len = 682906616},
>>                   ttl_val = {s = 0x7ff0285a9020 "\177\001", len =
>>         -1981593296}, user_param_val = {s = 0x7ffc89e34fa0
>>         "\001\200\255\373\374\177", len = 0},
>>                   maddr_val = {s = 0x10000000d <Address 0x10000000d
>>         out of bounds>, len = -1981591296}, method_val = {
>>                     s = 0x7fef00000000 <Address 0x7fef00000000 out of
>>         bounds>, len = -1671151657}, lr_val = {s = 0x7fefe7374ba2 "",
>>         len = 682906616},
>>                   r2_val = {s = 0x32 <Address 0x32 out of bounds>,
>>         len = 1073741824}, gr_val = {s = 0x2000000 <Address 0x2000000
>>         out of bounds>,
>>                     len = 4289104}}
>>                 u = 0x389c4224e8
>>                 port = 0
>>                 dst_host = 0x89e34bd0
>>                 i = 32764
>>                 flags = 52
>>                 avp = 0x189e35040
>>                 st = {flags = 2313374752, id = 32764, name = {n =
>>         -1981593512, s = {s = 0x7ffc89e34858 "\r", len =
>>         -1981593152}, re = 0x7ffc89e34858},
>>                   avp = 0x7ffc89e347a0}
>>                 sct = 0x0
>>                 sjt = 0x7ff028b453f8
>>                 rve = 0x32
>>                 mct = 0x40000000
>>                 rv = 0x7ffc89e34c20
>>                 rv1 = 0x2000000
>>                 c1 = {cache_type = 2313374144, val_type = 32764, c =
>>         {avp_val = {n = 1, s = {s = 0x1 <Address 0x1 out of bounds>,
>>         len = 1}, re = 0x1}, pval = {
>>                       rs = {s = 0x1 <Address 0x1 out of bounds>, len
>>         = 1}, ri = -1407115050, flags = 52}},
>>                   i2s = "\000\000\000\000\000\000\000\000\274\224
>>         \254\064\000\000\000\240G\343\211\374\177"}
>>                 s = {s = 0x1 <Address 0x1 out of bounds>, len =
>>         -1981593512}
>>                 srevp = {0x7ffc89e34960, 0x34ac208a97}
>>                 mod_f_params = {{type = NOSUBTYPE, u = {number = 0,
>>         string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
>>         0x0, select = 0x0}}, {
>>                     type = NOSUBTYPE, u = {number = 0, string = 0x0,
>>         str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
>>         0x0}}, {type = NOSUBTYPE, u = {
>>                       number = 0, string = 0x0, str = {s = 0x0, len =
>>         0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
>>         NOSUBTYPE, u = {number = 0,
>>                       string = 0x0, str = {s = 0x0, len = 0}, data =
>>         0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u =
>>         {number = 0, string = 0x0, str = {
>>                         s = 0x0, len = 0}, data = 0x0, attr = 0x0,
>>         select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
>>         0x0, str = {s = 0x0, len = 0},
>>                       data = 0x0, attr = 0x0, select = 0x0}}, {type =
>>         NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
>>         = 0}, data = 0x0,
>>                       attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
>>         u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data
>>         = 0x0, attr = 0x0,
>>                       select = 0x0}}}
>>         ---Type <return> to continue, or q <return> to quit---
>>                 __FUNCTION__ = "do_action"
>>         #5  0x00000000004404b8 in run_actions (h=0x7ffc89e35670,
>>         a=0x7ff0285a9340, msg=0x7ff028b453f8) at core/action.c:1552
>>                 t = 0x7ff0285a9340
>>                 ret = -1
>>                 mod = 0x1dde9a0
>>                 ms = 176570657
>>                 __FUNCTION__ = "run_actions"
>>         #6  0x0000000000433692 in do_action (h=0x7ffc89e35670,
>>         a=0x7ff0285a9668, msg=0x7ff028b453f8) at core/action.c:1045
>>                 ret = 1
>>                 v = 1
>>                 dst = {send_sock = 0x7ffc89e35380, to = {s =
>>         {sa_family = 61042, sa_data = "I", '\000' <repeats 12
>>         times>}, sin = {sin_family = 61042,
>>                       sin_port = 73, sin_addr = {s_addr = 0},
>>         sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
>>         {sin6_family = 61042, sin6_port = 73,
>>                       sin6_flowinfo = 0, sin6_addr = {__in6_u =
>>         {__u6_addr8 =
>>         "\000\000\000\000\000\000\000\000\020\000\000\000\005\000\000",
>>         __u6_addr16 = {
>>                             0, 0, 0, 0, 16, 0, 5, 0}, __u6_addr32 =
>>         {0, 0, 16, 5}}}, sin6_scope_id = 673678288}}, id = 32752,
>>         proto = 0 '\000', send_flags = {
>>                     f = 0, blst_imask = 32752}}
>>                 tmp = 0x66307b "H\213E\270H\203\300\070\351O\003"
>>                 new_uri = 0x7ff028a93048 ""
>>                 end = 0x7ff028a93038 "\303l\256"
>>                 crt = 0x2000000 <Address 0x2000000 out of bounds>
>>                 cmd = 0x1773508
>>                 len = 6791248
>>                 user = 0
>>                 uri = {user = {s = 0xae6ecd "", len = 11430897},
>>         passwd = {s = 0x7c95f8 "core: core/parser/msg_parser.c", len
>>         = 682507016}, host = {
>>                     s = 0xae6c43 "To: <sip:usrloc at 10.112.103.64:5080
>>         <http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
>>         <sip:usrloc at 10.112.101.128:5060
>>         <http://sip:usrloc@10.112.101.128:5060>>;tag=8a3534eb92149bbe647f599d354f2511-090e\r\nCSeq:
>>         10 KDMQ\r\nCall-ID: 409f4ac5788a8cc4-24333 at 10.112.101.128
>>         <mailto:409f4ac5788a8cc4-24333 at 10.112.101.128>\r\nContent-Length:
>>         39"..., len = 11430924}, port = {s = 0x0, len = 682507072},
>>         params = {
>>                     s = 0x7ff028ae3c90  <incomplete sequence \350>,
>>         len = 0}, sip_params = {s = 0x7ffc89e35330 "\246", len =
>>         6697083}, headers = {
>>                     s = 0x7c9170 "core", len = -1981590832}, port_no
>>         = 8, proto = 0, type = ERROR_URI_T, flags = 30415600,
>>         transport = {
>>                     s = 0xa8428197 <Address 0xa8428197 out of
>>         bounds>, len = -1675583518}, ttl = {s = 0x3000000030 <Address
>>         0x3000000030 out of bounds>,
>>                     len = 44108294}, user_param = {s = 0x400000017
>>         <Address 0x400000017 out of bounds>, len = 8176032}, maddr =
>>         {s = 0x0, len = -1981590448},
>>                   method = {s = 0x389c603c48 "", len = 11431629}, lr
>>         = {s = 0x7ffc89e35450 "\220T\343\211\374\177", len =
>>         6869803}, r2 = {
>>                     s = 0x7cc190 "get_hdr_field", len = 6730416}, gr
>>         = {s = 0x7ffc89e35300 "", len = 673677328}, transport_val =
>>         {s = 0x389c422190 "",
>>                     len = 673677328}, ttl_val = {s = 0x7ffc89e35350
>>         "\200S\343\211\374\177", len = -413652166}, user_param_val =
>>         {s = 0x40e7f9 "", len = 0},
>>                   maddr_val = {s = 0x2000000 <Address 0x2000000 out
>>         of bounds>, len = 24588552}, method_val = {
>>                     s = 0x67a450
>>         "\030A\211\321A)\301D\211ȉ\302H\213\205(\377\377\377\211P
>>         \276\004", len = 8965880}, lr_val = {
>>                     s = 0x8cd2d0 <Address 0x8cd2d0 out of bounds>,
>>         len = 8}, r2_val = {s = 0xa6 <Address 0xa6 out of bounds>,
>>         len = 8136007}, gr_val = {
>>                     s = 0x7c95f8 "core: core/parser/msg_parser.c",
>>         len = 682176488}}
>>                 next_hop = {user = {s = 0x1e7374b9a <Address
>>         0x1e7374b9a out of bounds>, len = 674472240}, passwd = {
>>                     s = 0xffffffff <Address 0xffffffff out of
>>         bounds>, len = 4289104}, host = {s = 0x4e2cb1825 <Address
>>         0x4e2cb1825 out of bounds>,
>>                     len = -1667705792}, port = {s = 0x0, len =
>>         1073741824}, params = {s = 0x2000000 <Address 0x2000000 out
>>         of bounds>, len = 4289104},
>>                   sip_params = {s = 0x7ffc89e35180
>>         "\360Q\343\211\374\177", len = 6031673}, headers = {s =
>>         0x7fefe2cb183f "", len = -1981591296},
>>                   port_no = 23013, proto = 58022, type = 32751, flags
>>         = 2313376240, transport = {s = 0x7fefe7374b9a "%x%c%x%c", len
>>         = -1981590944}, ttl = {
>>         ---Type <return> to continue, or q <return> to quit---
>>                     s = 0x1a <Address 0x1a out of bounds>, len =
>>         4289104}, user_param = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-",
>>         len = -1670974014}, maddr = {
>>                     s = 0x30fbad8001 <Address 0x30fbad8001 out of
>>         bounds>, len = -492414491}, method = {s = 0x7fefe2a659e5
>>         "5e6a5a14-1fa-", len = -492414491},
>>                   lr = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len =
>>         -492414478}, r2 = {s = 0x7fefe2a659ff "", len = -492414491},
>>         gr = {s = 0x7fefe2a659ff "",
>>                     len = 0}, transport_val = {s = 0x0, len =
>>         6730416}, ttl_val = {s = 0x7ffc89e351a0 "", len = 673677328},
>>         user_param_val = {
>>                     s = 0x2000000 <Address 0x2000000 out of bounds>,
>>         len = 673677328}, maddr_val = {s = 0x7ffc89e351f0 "Cl\256",
>>         len = -413652166},
>>                   method_val = {s = 0x7ff028278010 "\001", len = 0},
>>         lr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>,
>>         len = 24589344}, r2_val = {
>>                     s = 0x67a138 "", len = 8965088}, gr_val = {s =
>>         0x8cd2d0 <Address 0x8cd2d0 out of bounds>, len = 682507072}}
>>                 u = 0x66273d
>>                 port = 0
>>                 dst_host = 0x40000000
>>                 i = 28
>>                 flags = 0
>>                 avp = 0xae6cc7
>>                 st = {flags = 31129600, id = 0, name = {n = 0, s = {s
>>         = 0x0, len = 93}, re = 0x0}, avp = 0x7ffc89e35080}
>>                 sct = 0x7ffc89e35490
>>                 sjt = 0x6943d3
>>                 rve = 0x7ff0285a8808
>>                 mct = 0x6a283b
>>                 rv = 0x0
>>                 rv1 = 0x7c9170
>>                 c1 = {cache_type = 24, val_type = 48, c = {avp_val =
>>         {n = -1981590720, s = {s = 0x7ffc89e35340 "\370\225|", len =
>>         -1981590912},
>>                       re = 0x7ffc89e35340}, pval = {rs = {s =
>>         0x7ffc89e35340 "\370\225|", len = -1981590912}, ri = 0, flags
>>         = 0}},
>>                   i2s = '\000' <repeats 21 times>}
>>                 s = {s = 0x7fefe2cb183f "", len = -490006491}
>>                 srevp = {0x7fefe2cb1825, 0x7fefe2cb1832}
>>                 mod_f_params = {{type = NOSUBTYPE, u = {number = 0,
>>         string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
>>         0x0, select = 0x0}}, {
>>                     type = NOSUBTYPE, u = {number = 0, string = 0x0,
>>         str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
>>         0x0}}, {type = NOSUBTYPE, u = {
>>                       number = 0, string = 0x0, str = {s = 0x0, len =
>>         0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
>>         NOSUBTYPE, u = {number = 0,
>>                       string = 0x0, str = {s = 0x0, len = 0}, data =
>>         0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u =
>>         {number = 0, string = 0x0, str = {
>>                         s = 0x0, len = 0}, data = 0x0, attr = 0x0,
>>         select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
>>         0x0, str = {s = 0x0, len = 0},
>>                       data = 0x0, attr = 0x0, select = 0x0}}, {type =
>>         NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
>>         = 0}, data = 0x0,
>>                       attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
>>         u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data
>>         = 0x0, attr = 0x0,
>>                       select = 0x0}}}
>>                 __FUNCTION__ = "do_action"
>>         #7  0x00000000004404b8 in run_actions (h=0x7ffc89e35670,
>>         a=0x7ff0285a85d0, msg=0x7ff028b453f8) at core/action.c:1552
>>                 t = 0x7ff0285a9668
>>                 ret = 1
>>                 mod = 0x417250
>>                 ms = 176570657
>>                 __FUNCTION__ = "run_actions"
>>         #8  0x0000000000440c4a in run_top_route (a=0x7ff0285a85d0,
>>         msg=0x7ff028b453f8, c=0x0) at core/action.c:1641
>>         ---Type <return> to continue, or q <return> to quit---
>>                 ctx = {rec_lev = 2, run_flags = 0, last_retcode = 1,
>>         jmp_env = {{__jmpbuf = {140669451785720,
>>         -8516593617994257148, 50, 1073741824, 33554432,
>>                         4289104, -8516593618092823292,
>>         8518532807277145348}, __mask_was_saved = 0, __saved_mask =
>>         {__val = {140668355198191, 4302791168, 48,
>>                           48, 18446744073709551615, 140722621863712,
>>         140668355198919, 50, 0, 4611686022755909632, 140669451785208,
>>         140722621863776, 5814046,
>>                           6608344928, 140669451785208, 0}}}}}
>>                 p = 0x7ffc89e35670
>>                 ret = 682907128
>>                 sfbk = 0
>>         #9  0x000000000052e6b9 in receive_msg (
>>             buf=0xae6bc0 "KDMQ sip:usrloc at 10.112.103.64:5080
>>         <http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia:
>>         SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0'
>>         <repeats 24 times>, ".0\r\nTo: <sip:usrloc at 10.112.103.64:5080
>>         <http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
>>         <sip:usrloc at 10.112.101.128
>>         <mailto:sip%3Ausrloc at 10.112.101.128>"..., len=781,
>>         rcv_info=0x7ffc89e35a70) at core/receive.c:264
>>                 msg = 0x7ff028b453f8
>>                 ctx = {rec_lev = -1981589208, run_flags = 32764,
>>         last_retcode = 8, jmp_env = {{__jmpbuf = {163, 4253893,
>>         8100093, 140669451683568,
>>                         140722621864176, 4845170, 0, 21474836480},
>>         __mask_was_saved = 673677840, __saved_mask = {__val =
>>         {140668768878592, 140722621864448,
>>                           6697083, 8094080, 5501958, 243141751568,
>>         140669476122624, 140722621864896, 4294967295, 88212724672, 1,
>>         140668357584160, 0, 33554432,
>>                           1024, 6608345520}}}}}
>>                 bctx = 0x7ffc89e359b0
>>                 ret = 0
>>                 stats_on = 0
>>                 tvb = {tv_sec = 140722621864208, tv_usec = 8961768}
>>                 tve = {tv_sec = 0, tv_usec = 243139749096}
>>                 tz = {tz_minuteswest = 50, tz_dsttime = 0}
>>                 diff = 0
>>                 inb = {
>>                   s = 0xae6bc0 "KDMQ sip:usrloc at 10.112.103.64:5080
>>         <http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia:
>>         SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0'
>>         <repeats 24 times>, ".0\r\nTo: <sip:usrloc at 10.112.103.64:5080
>>         <http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
>>         <sip:usrloc at 10.112.101.128
>>         <mailto:sip%3Ausrloc at 10.112.101.128>"..., len = 781}
>>                 netinfo = {data = {s = 0x7ffc00000001 <Address
>>         0x7ffc00000001 out of bounds>, len = 0}, dst = 0x1, rcv =
>>         0x389c422190}
>>                 keng = 0x0
>>                 __FUNCTION__ = "receive_msg"
>>         #10 0x00000000006456a3 in udp_rcv_loop () at
>>         core/udp_server.c:515
>>                 len = 781
>>                 buf = "KDMQ sip:usrloc at 10.112.103.64:5080
>>         <http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia:
>>         SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0'
>>         <repeats 24 times>, ".0\r\nTo: <sip:usrloc at 10.112.103.64:5080
>>         <http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
>>         <sip:usrloc at 10.112.101.128:5060
>>         <http://sip:usrloc@10.112.101.128:5060>>;tag=8a3"...
>>                 tmp = 0x100000000 <Address 0x100000000 out of bounds>
>>                 from = 0x7ff028b2c728
>>                 fromlen = 16
>>                 ri = {src_ip = {af = 2, len = 4, u = {addrl =
>>         {140722462617610, 140722621864592}, addr32 = {2154131466,
>>         32764, 2313378448, 32764}, addr16 = {
>>                         28682, 32869, 32764, 0, 23184, 35299, 32764,
>>         0}, addr =
>>         "\npe\200\374\177\000\000\220Z\343\211\374\177\000"}}, dst_ip
>>         = {af = 2,
>>                     len = 4, u = {addrl = {1080520714, 0}, addr32 =
>>         {1080520714, 0, 0, 0}, addr16 = {28682, 16487, 0, 0, 0, 0, 0, 0},
>>                       addr = "\npg@", '\000' <repeats 11 times>}},
>>         src_port = 5060, dst_port = 5080, proto_reserved1 = 0,
>>         proto_reserved2 = 0, src_su = {s = {
>>                       sa_family = 2, sa_data =
>>         "\023\304\npe\200\000\000\000\000\000\000\000"}, sin =
>>         {sin_family = 2, sin_port = 50195, sin_addr = {
>>                         s_addr = 2154131466}, sin_zero =
>>         "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2,
>>         sin6_port = 50195,
>>         ---Type <return> to continue, or q <return> to quit---
>>                       sin6_flowinfo = 2154131466, sin6_addr =
>>         {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
>>         __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
>>                           __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id
>>         = 0}}, bind_address = 0x7ff02830fff0, proto = 1 '\001'}
>>                 __FUNCTION__ = "udp_rcv_loop"
>>         #11 0x0000000000423c7d in main_loop () at main.c:1623
>>                 i = 0
>>                 pid = 0
>>                 si = 0x7ff02830fff0
>>                 si_desc = "udp receiver child=0
>>         sock=10.112.103.64:5080
>>         <http://10.112.103.64:5080>\000\000\000\000
>>         _\343\211\374\177\000\000\062\000\000\000\000\000\000\000\000\000\000@\000\000\000\000\000\000\000\002\000\000\000\000PrA\000\000\000\000\000PSn\234\070\000\000\000\060\000\000\000\060\000\000\000\020]\343\211\374\177\000\000@\\\343\211\374\177\000\000\000px\347\357\177\000"
>>                 nrprocs = 4
>>                 woneinit = 0
>>                 __FUNCTION__ = "main_loop"
>>         #12 0x000000000042b0ce in main (argc=13, argv=0x7ffc89e36008)
>>         at main.c:2646
>>                 cfg_stream = 0x1bb5010
>>                 c = -1
>>                 r = 0
>>                 tmp = 0x7ffc89e36ef7 ""
>>                 tmp_len = 32764
>>                 port = -1981587768
>>                 proto = 56
>>                 options = 0x7434f8
>>         ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
>>                 ret = -1
>>                 seed = 4012862799
>>                 rfd = 4
>>                 debug_save = 0
>>                 debug_flag = 0
>>                 dont_fork_cnt = 0
>>                 n_lst = 0x7ffc89e35f08
>>                 p = 0x100000000 <Address 0x100000000 out of bounds>
>>                 st = {st_dev = 64769, st_ino = 148592, st_nlink = 2,
>>         st_mode = 16877, st_uid = 498, st_gid = 498, __pad0 = 0,
>>         st_rdev = 0, st_size = 4096,
>>                   st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec
>>         = 1576522467, tv_nsec = 972000117}, st_mtim = {tv_sec =
>>         1519927576,
>>                     tv_nsec = 668342155}, st_ctim = {tv_sec =
>>         1584026268, tv_nsec = 165523404}, __unused = {0, 0, 0}}
>>                 __FUNCTION__ = "main"
>>         (gdb)
>>
>>         Em ter., 17 de mar. de 2020 às 09:52, Daniel-Constantin
>>         Mierla <miconda at gmail.com <mailto:miconda at gmail.com>> escreveu:
>>
>>             Hello,
>>
>>             does it happen only when doing restart of kamailio? Can
>>             you send the full backtrace from gdb?
>>
>>             Also, 5.0.x is no longer a supported release for quite
>>             some time, try to upgrade to 5.2.x or even better to the
>>             latest one 5.3.x and see if you can reproduce it with one
>>             of the supported releases, maybe it is something that was
>>             already fixed.
>>
>>             Cheers,
>>             Daniel
>>
>>             On 17.03.20 10:32, Virgílio Cunha wrote:
>>>             Hello,
>>>
>>>             I have in production environment with 4 proxies using
>>>             DMQ to sync terminal registers and the DMQ configuration
>>>             parameters is following:
>>>             loadmodule "dmq.so"
>>>             #modparam("dmq", "server_address",
>>>             "sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
>>>             modparam("dmq", "notification_address", "DMQ_HOSTS")
>>>             modparam("dmq", "multi_notify", 1)
>>>             modparam("dmq", "num_workers", 4)
>>>
>>>
>>>             In the lab environment with the same configuration that
>>>             in prod, I can simulate this problem with 12000 terminal
>>>             REGISTERS and, when doing several restarts on a proxy,
>>>             one of them is generated a coredump. 
>>>             Analyzing the core, I can see that the problem is
>>>             concurrency in the dmq synchronization. 
>>>
>>>             Job_queue_size (...) is called with a valid /queue /and,
>>>             but inside this function, the /queue /is null.
>>>
>>>
>>>             (gdb) up
>>>             #1  0x00007fefe58ee246 in job_queue_size (queue=0x0) at
>>>             worker.c:254
>>>             254             return atomic_get(&queue->count);
>>>             (gdb) print queue
>>>             $1 = (job_queue_t *) 0x0
>>>             (gdb) up
>>>             #2  0x00007fefe58ed920 in add_dmq_job
>>>             (msg=0x7ff028b453f8, peer=0x7fefebcc4df8) at worker.c:184
>>>             184                    
>>>             if(job_queue_size(workers[i].queue) == 0) {
>>>             (gdb) print i
>>>             $2 = 1
>>>             (gdb) print workers[i].queue
>>>             $3 = (job_queue_t *) 0x7fefebce49b0
>>>
>>>             If I configure DMQ with  num_workers=1 I can't reproduce
>>>             this problem.
>>>             I'm using the kamailio 5.0.8 release.
>>>
>>>             Is this problem known to you? What is the right way to
>>>             solve it?
>>>
>>>             Best Regards,
>>>             Virgílio Cunha
>>>
>>>             _______________________________________________
>>>             Kamailio (SER) - Users Mailing List
>>>             sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>>             https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>>             -- 
>>             Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>             www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>
>         -- 
>         Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>         www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>
-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200327/9fabb99d/attachment.html>


More information about the sr-users mailing list