17 Mar
2020
17 Mar
'20
10:32 a.m.
Hello,
I have in production environment with 4 proxies using DMQ to sync terminal
registers and the DMQ configuration parameters is following:
loadmodule "dmq.so"
#modparam("dmq", "server_address",
"sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
modparam("dmq", "notification_address", "DMQ_HOSTS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
In the lab environment with the same configuration that in prod, I can
simulate this problem with 12000 terminal REGISTERS and, when doing several
restarts on a proxy, one of them is generated a coredump.
Analyzing the core, I can see that the problem is concurrency in the dmq
synchronization.
Job_queue_size (...) is called with a valid *queue *and, but inside this
function, the *queue *is null.
(gdb) up
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
254 return atomic_get(&queue->count);
(gdb) print queue
$1 = (job_queue_t *) 0x0
(gdb) up
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
peer=0x7fefebcc4df8) at worker.c:184
184 if(job_queue_size(workers[i].queue) == 0) {
(gdb) print i
$2 = 1
(gdb) print workers[i].queue
$3 = (job_queue_t *) 0x7fefebce49b0
If I configure DMQ with num_workers=1 I can't reproduce this problem.
I'm using the kamailio 5.0.8 release.
Is this problem known to you? What is the right way to solve it?
Best Regards,
Virgílio Cunha
17 Mar
17 Mar
10:48 a.m.
Hello Virgilio,
if you can reproduce it in your lab, please try to update Kamailio to a supported version
(5.3.3 or the latest 5.2.x). Please open an issue ticket on our tracker if the problem can
be also reproduced there.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: sr-users <sr-users-bounces(a)lists.kamailio.org> On Behalf Of Virgílio Cunha
Sent: Tuesday, March 17, 2020 10:33 AM
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Subject: [SR-Users] Segmentation fault in DMQ module
Hello,
I have in production environment with 4 proxies using DMQ to sync terminal registers and
the DMQ configuration parameters is following:
loadmodule "dmq.so"
#modparam("dmq", "server_address",
"sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
modparam("dmq", "notification_address", "DMQ_HOSTS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
In the lab environment with the same configuration that in prod, I can simulate this
problem with 12000 terminal REGISTERS and, when doing several restarts on a proxy, one of
them is generated a coredump.
Analyzing the core, I can see that the problem is concurrency in the dmq
synchronization.
Job_queue_size (...) is called with a valid queue and, but inside this function, the queue
is null.
(gdb) up
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
254 return atomic_get(&queue->count);
(gdb) print queue
$1 = (job_queue_t *) 0x0
(gdb) up
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8, peer=0x7fefebcc4df8) at
worker.c:184
184 if(job_queue_size(workers[i].queue) == 0) {
(gdb) print i
$2 = 1
(gdb) print workers[i].queue
$3 = (job_queue_t *) 0x7fefebce49b0
If I configure DMQ with num_workers=1 I can't reproduce this problem.
I'm using the kamailio 5.0.8 release.
Is this problem known to you? What is the right way to solve it?
Best Regards,
Virgílio Cunha
10:52 a.m.
Hello,
does it happen only when doing restart of kamailio? Can you send the
full backtrace from gdb?
Also, 5.0.x is no longer a supported release for quite some time, try to
upgrade to 5.2.x or even better to the latest one 5.3.x and see if you
can reproduce it with one of the supported releases, maybe it is
something that was already fixed.
Cheers,
Daniel
On 17.03.20 10:32, Virgílio Cunha wrote:
Hello,
I have in production environment with 4 proxies using DMQ to sync
terminal registers and the DMQ configuration parameters is following:
loadmodule "dmq.so"
#modparam("dmq", "server_address",
"sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
modparam("dmq", "notification_address", "DMQ_HOSTS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
In the lab environment with the same configuration that in prod, I can
simulate this problem with 12000 terminal REGISTERS and, when doing
several restarts on a proxy, one of them is generated a coredump.
Analyzing the core, I can see that the problem is concurrency in the
dmq synchronization.
Job_queue_size (...) is called with a valid /queue /and, but inside
this function, the /queue /is null.
(gdb) up
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
254 return atomic_get(&queue->count);
(gdb) print queue
$1 = (job_queue_t *) 0x0
(gdb) up
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
peer=0x7fefebcc4df8) at worker.c:184
184 if(job_queue_size(workers[i].queue) == 0) {
(gdb) print i
$2 = 1
(gdb) print workers[i].queue
$3 = (job_queue_t *) 0x7fefebce49b0
If I configure DMQ with num_workers=1 I can't reproduce this problem.
I'm using the kamailio 5.0.8 release.
Is this problem known to you? What is the right way to solve it?
Best Regards,
Virgílio Cunha
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
11:35 a.m.
Hi,
The problem does not just occur when kamailio is restarted. The restart was
the way I found to reproduce the problem, because when a proxy is
restarted, there is more traffic from DMQ messages, so the problem is more
likely to occur.
The full backtrace is following:
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change
and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy...Reading
symbols from /usr/lib/debug/usr/sbin/kamailio.debug...done.
done.
[New Thread 506]
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/kamailio/modules/db_postgres.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/db_postgres.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/db_postgres.so
Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from /usr/lib64/kamailio/libsrdb2.so.1...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/libsrdb2.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrdb2.so.1
Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/libsrdb1.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /usr/lib64/kamailio/modules/ctl.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/ctl.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/kamailio/modules/kex.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/kex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/kex.so
Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/libsrutils.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/kamailio/modules/tm.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/tm.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/tm.so
Reading symbols from /usr/lib64/kamailio/modules/tmx.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/tmx.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/tmx.so
Reading symbols from /usr/lib64/kamailio/modules/corex.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/corex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/corex.so
Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/cfg_rpc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
Reading symbols from /usr/lib64/kamailio/modules/sl.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/sl.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sl.so
Reading symbols from /usr/lib64/kamailio/modules/rr.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/rr.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/rr.so
Reading symbols from /usr/lib64/kamailio/modules/maxfwd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/maxfwd.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/maxfwd.so
Reading symbols from /usr/lib64/kamailio/modules/siputils.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/siputils.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
Reading symbols from /usr/lib64/kamailio/modules/sanity.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sanity.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sanity.so
Reading symbols from /usr/lib64/kamailio/modules/textops.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/textops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/textops.so
Reading symbols from /usr/lib64/kamailio/modules/textopsx.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/textopsx.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/textopsx.so
Reading symbols from /usr/lib64/kamailio/modules/dmq.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/dmq.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
Reading symbols from /usr/lib64/kamailio/modules/htable.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/htable.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/htable.so
Reading symbols from /usr/lib64/kamailio/modules/pv.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/pv.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/pv.so
Reading symbols from /usr/lib64/kamailio/modules/xlog.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/xlog.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
Reading symbols from /usr/lib64/kamailio/modules/jsonrpcs.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/jsonrpcs.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/jsonrpcs.so
Reading symbols from /usr/lib64/kamailio/modules/uac.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/uac.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/uac.so
Reading symbols from /usr/lib64/kamailio/modules/uac_redirect.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/uac_redirect.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/uac_redirect.so
Reading symbols from /usr/lib64/kamailio/modules/db_text.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/db_text.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/db_text.so
Reading symbols from /usr/lib64/kamailio/modules/dispatcher.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dispatcher.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dispatcher.so
Reading symbols from /usr/lib64/kamailio/modules/sqlops.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sqlops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sqlops.so
Reading symbols from /usr/lib64/kamailio/modules/regex.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/regex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/regex.so
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /usr/lib64/kamailio/modules/permissions.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/permissions.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/permissions.so
Reading symbols from /usr/lib64/kamailio/modules/dialplan.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dialplan.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dialplan.so
Reading symbols from /usr/lib64/kamailio/modules/mtree.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/mtree.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/mtree.so
Reading symbols from /usr/lib64/kamailio/modules/cfgutils.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/cfgutils.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/cfgutils.so
Reading symbols from /usr/lib64/kamailio/modules/statsd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/statsd.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/statsd.so
Reading symbols from /usr/lib64/kamailio/modules/app_lua.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/app_lua.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/app_lua.so
Reading symbols from /usr/lib64/liblua-5.1.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/liblua-5.1.so
Reading symbols from /usr/lib64/kamailio/modules/rtimer.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/rtimer.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/rtimer.so
Reading symbols from /usr/lib64/kamailio/modules/mqueue.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/mqueue.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/mqueue.so
Reading symbols from /usr/lib64/kamailio/modules/auth.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/auth.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/auth.so
Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/usrloc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
Reading symbols from /usr/lib64/kamailio/modules/registrar.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/registrar.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dmq_usrloc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
Reading symbols from /usr/lib64/kamailio/modules/xhttp.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/xhttp.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/xhttp.so
Reading symbols from /usr/lib64/kamailio/modules/exec.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/exec.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/exec.so
Reading symbols from /usr/lib64/kamailio/modules/http_client.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/http_client.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/http_client.so
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/libcurl.so.4...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libcurl.so.4
Reading symbols from /lib64/libidn.so.11...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libidn.so.11
Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /usr/lib64/libssh2.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssh2.so.1
Reading symbols from /usr/lib64/kamailio/modules/json.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/json.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/json.so
Reading symbols from /lib64/libjson-c.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libjson-c.so.2
Reading symbols from /usr/lib64/kamailio/modules/ipops.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/ipops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/ipops.so
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Core was generated by `/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy -f
/opt/ptin/kamailio-ksipadp/prox'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
../../core/atomic/atomic_common.h:66
66 return atomic_get_int(&(v->val));
Missing separate debuginfos, use: debuginfo-install
kamailio-ksipadp-2.8.0-1.ptin.el6.x86_64
(gdb) bt full
#0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
../../core/atomic/atomic_common.h:66
No locals.
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
No locals.
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
peer=0x7fefebcc4df8) at worker.c:184
i = 1
found_available = 0
new_job = {f = 0x7fefe2613797 <usrloc_dmq_handle_msg>, msg =
0x7fefebce67c8, orig_peer = 0x7fefebcc4df8, next = 0x0, prev = 0x0}
worker = 0x7fefe7a1d890
cloned_msg = 0x7fefebce67c8
cloned_msg_len = 3888
__FUNCTION__ = "add_dmq_job"
#3 0x00007fefe58dfec1 in dmq_handle_message (msg=0x7ff028b453f8, str1=0x0,
str2=0x0) at message.c:66
peer = 0x7fefebcc4df8
__FUNCTION__ = "dmq_handle_message"
#4 0x0000000000433729 in do_action (h=0x7ffc89e35670, a=0x7ff0285a9340,
msg=0x7ff028b453f8) at core/action.c:1054
ret = -5
v = 1
dst = {send_sock = 0x7ff0285a8810, to = {s = {sa_family = 21376,
sa_data = "\343\211\374\177\000\000\370S\264(\360\177\000"}, sin = {
sin_family = 21376, sin_port = 35299, sin_addr = {s_addr =
32764}, sin_zero = "\370S\264(\360\177\000"}, sin6 = {sin6_family = 21376,
sin6_port = 35299, sin6_flowinfo = 32764, sin6_addr =
{__in6_u = {__u6_addr8 = "\370S\264(\360\177\000\000pV\343\211\374\177\000",
__u6_addr16 = {21496, 10420, 32752, 0, 22128, 35299,
32764, 0}, __u6_addr32 = {682906616, 32752, 2313377392, 32764}}},
sin6_scope_id = 1}}, id = 32764, proto = 0 '\000', send_flags
= {f = 0, blst_imask = 0}}
tmp = 0x389c422190 ""
new_uri = 0x7c00000000 <Address 0x7c00000000 out of bounds end = 0x6e00000077 <Address
0x6e00000077 out of bounds crt =
0x389c98e120 ""
cmd = 0x7ff0283219b0
len = 0
user = 0
uri = {user = {s = 0x7ffc89e34a10 "@K\343\211\374\177", len =
4459394}, passwd = {s = 0x7ffc89e34e80 "\356Q\343\211\374\177",
len = 682906616}, host = {s = 0x7ff0285a9020 "\177\001", len =
-1981589904}, port = {s = 0x0, len = 1}, params = {
s = 0x7ff028b453f8 "\017", len = 295169284}, sip_params = {s =
0x32 <Address 0x32 out of bounds>, len = 1073741824}, headers = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
4289104}, port_no = 60676, proto = 5047, type = 2312043322, flags =
2168712452,
transport = {s = 0x0, len = -1675583518}, ttl = {s = 0x89e34e90
<Address 0x89e34e90 out of bounds>, len = 35715777}, user_param = {
s = 0xffffffff0000003c <Address 0xffffffff0000003c out of
bounds>, len = -1671419584}, maddr = {s = 0x0, len = -1981592704}, method =
{
s = 0x389c603c48 "", len = -1671391960}, lr = {s = 0x0, len =
-1981592544}, r2 = {s = 0x1dde9a0 " 9\336\001", len = 0}, gr = {
s = 0x4 <Address 0x4 out of bounds>, len = 2479}, transport_val
= {s = 0x389c422190 "", len = 1}, ttl_val = {
s = 0x7ffc89e34b40 "pM\343\211\374\177", len = 5683491},
user_param_val = {s = 0x40e7e8 "\370\377\377\377\377\377\377\377@ݘ\234\070",
len = -1671362728}, maddr_val = {s = 0x404130 "", len = 0},
method_val = {s = 0x1000003b4 <Address 0x1000003b4 out of bounds>, len = 0},
lr_val = {s = 0x7fefe7374b9a "%x%c%x%c", len = -1673386776},
r2_val = {s = 0x7ffc89e34bd0 "", len = -1981592584}, gr_val = {
---Type <return> to continue, or q <return> to quit---
s = 0x389c422190 "", len = 0}}
next_hop = {user = {s = 0x1 <Address 0x1 out of bounds>, len = 40},
passwd = {s = 0x0, len = -1407107317}, host = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
31398688}, port = {s = 0x1dde9a0 " 9\336\001", len = 31320480}, params = {
s = 0x7ffc89e34c20 "", len = -1981593152}, sip_params = {s =
0x7ffc89e34910 "\200N\343\211\374\177", len = 682906616}, headers = {
s = 0x32 <Address 0x32 out of bounds>, len = 1073741824},
port_no = 0, proto = 512, type = ERROR_URI_T, flags = 4289104, transport = {
s = 0x7ffc89e34900 "\020J\343\211\374\177", len = 4457737}, ttl
= {s = 0x7ffc89e348f0 "", len = -1981593520}, user_param = {
s = 0x7ffc89e34858 "\r", len = -1981593264}, maddr = {s =
0x7ffc89e34870 "", len = -1981593456}, method = {s = 0x7ffc89e348b0
"\r",
len = -1981593392}, lr = {s = 0x8cd2d0 <Address 0x8cd2d0 out of
bounds>, len = 13}, r2 = {
s = 0x10100000002 <Address 0x10100000002 out of bounds>, len =
-1}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 682906616},
ttl_val = {s = 0x7ff0285a9020 "\177\001", len = -1981593296},
user_param_val = {s = 0x7ffc89e34fa0 "\001\200\255\373\374\177", len = 0},
maddr_val = {s = 0x10000000d <Address 0x10000000d out of bounds>,
len = -1981591296}, method_val = {
s = 0x7fef00000000 <Address 0x7fef00000000 out of bounds>, len
= -1671151657}, lr_val = {s = 0x7fefe7374ba2 "", len = 682906616},
r2_val = {s = 0x32 <Address 0x32 out of bounds>, len =
1073741824}, gr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>,
len = 4289104}}
u = 0x389c4224e8
port = 0
dst_host = 0x89e34bd0
i = 32764
flags = 52
avp = 0x189e35040
st = {flags = 2313374752, id = 32764, name = {n = -1981593512, s =
{s = 0x7ffc89e34858 "\r", len = -1981593152}, re = 0x7ffc89e34858},
avp = 0x7ffc89e347a0}
sct = 0x0
sjt = 0x7ff028b453f8
rve = 0x32
mct = 0x40000000
rv = 0x7ffc89e34c20
rv1 = 0x2000000
c1 = {cache_type = 2313374144, val_type = 32764, c = {avp_val = {n
= 1, s = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, re = 0x1}, pval = {
rs = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, ri =
-1407115050, flags = 52}},
i2s = "\000\000\000\000\000\000\000\000\274\224
\254\064\000\000\000\240G\343\211\374\177"}
s = {s = 0x1 <Address 0x1 out of bounds>, len = -1981593512}
srevp = {0x7ffc89e34960, 0x34ac208a97}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
= {
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}}
---Type <return> to continue, or q <return> to quit---
__FUNCTION__ = "do_action"
#5 0x00000000004404b8 in run_actions (h=0x7ffc89e35670, a=0x7ff0285a9340,
msg=0x7ff028b453f8) at core/action.c:1552
t = 0x7ff0285a9340
ret = -1
mod = 0x1dde9a0
ms = 176570657
__FUNCTION__ = "run_actions"
#6 0x0000000000433692 in do_action (h=0x7ffc89e35670, a=0x7ff0285a9668,
msg=0x7ff028b453f8) at core/action.c:1045
ret = 1
v = 1
dst = {send_sock = 0x7ffc89e35380, to = {s = {sa_family = 61042,
sa_data = "I", '\000' <repeats 12 times>}, sin = {sin_family =
61042,
sin_port = 73, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 61042, sin6_port =
73,
sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 =
"\000\000\000\000\000\000\000\000\020\000\000\000\005\000\000", __u6_addr16
= {
0, 0, 0, 0, 16, 0, 5, 0}, __u6_addr32 = {0, 0, 16,
5}}}, sin6_scope_id = 673678288}}, id = 32752, proto = 0 '\000', send_flags
= {
f = 0, blst_imask = 32752}}
tmp = 0x66307b "H\213E\270H\203\300\070\351O\003"
new_uri = 0x7ff028a93048 ""
end = 0x7ff028a93038 "\303l\256"
crt = 0x2000000 <Address 0x2000000 out of bounds
cmd = 0x1773508
len = 6791248
user = 0
uri = {user = {s = 0xae6ecd "", len = 11430897}, passwd = {s =
0x7c95f8 "core: core/parser/msg_parser.c", len = 682507016}, host = {
s = 0xae6c43 "To: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
sip:usrloc@10.112.101.128:5060>;tag=8a3534eb92149bbe647f599d354f2511-090e\r\nCSeq:
10 KDMQ\r\nCall-ID: 409f4ac5788a8cc4-24333(a)10.112.101.128\r\nContent-Length:
39"..., len = 11430924}, port = {s = 0x0, len = 682507072}, params = {
s = 0x7ff028ae3c90 <incomplete sequence \350>, len = 0},
sip_params = {s = 0x7ffc89e35330 "\246", len = 6697083}, headers = {
s = 0x7c9170 "core", len = -1981590832}, port_no = 8, proto =
0, type = ERROR_URI_T, flags = 30415600, transport = {
s = 0xa8428197 <Address 0xa8428197 out of bounds>, len =
-1675583518}, ttl = {s = 0x3000000030 <Address 0x3000000030 out of bounds>,
len = 44108294}, user_param = {s = 0x400000017 <Address
0x400000017 out of bounds>, len = 8176032}, maddr = {s = 0x0, len =
-1981590448},
method = {s = 0x389c603c48 "", len = 11431629}, lr = {s =
0x7ffc89e35450 "\220T\343\211\374\177", len = 6869803}, r2 = {
s = 0x7cc190 "get_hdr_field", len = 6730416}, gr = {s =
0x7ffc89e35300 "", len = 673677328}, transport_val = {s = 0x389c422190
"",
len = 673677328}, ttl_val = {s = 0x7ffc89e35350
"\200S\343\211\374\177", len = -413652166}, user_param_val = {s = 0x40e7f9
"", len = 0},
maddr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>, len
= 24588552}, method_val = {
s = 0x67a450
"\030A\211\321A)\301D\211ȉ\302H\213\205(\377\377\377\211P \276\004", len =
8965880}, lr_val = {
s = 0x8cd2d0 <Address 0x8cd2d0 out of bounds>, len = 8}, r2_val
= {s = 0xa6 <Address 0xa6 out of bounds>, len = 8136007}, gr_val = {
s = 0x7c95f8 "core: core/parser/msg_parser.c", len = 682176488}}
next_hop = {user = {s = 0x1e7374b9a <Address 0x1e7374b9a out of
bounds>, len = 674472240}, passwd = {
s = 0xffffffff <Address 0xffffffff out of bounds>, len =
4289104}, host = {s = 0x4e2cb1825 <Address 0x4e2cb1825 out of bounds>,
len = -1667705792}, port = {s = 0x0, len = 1073741824}, params
= {s = 0x2000000 <Address 0x2000000 out of bounds>, len = 4289104},
sip_params = {s = 0x7ffc89e35180 "\360Q\343\211\374\177", len =
6031673}, headers = {s = 0x7fefe2cb183f "", len = -1981591296},
port_no = 23013, proto = 58022, type = 32751, flags = 2313376240,
transport = {s = 0x7fefe7374b9a "%x%c%x%c", len = -1981590944}, ttl = {
---Type <return> to continue, or q <return> to quit---
s = 0x1a <Address 0x1a out of bounds>, len = 4289104},
user_param = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len = -1670974014}, maddr
= {
s = 0x30fbad8001 <Address 0x30fbad8001 out of bounds>, len =
-492414491}, method = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len =
-492414491},
lr = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len = -492414478}, r2 =
{s = 0x7fefe2a659ff "", len = -492414491}, gr = {s = 0x7fefe2a659ff
"",
len = 0}, transport_val = {s = 0x0, len = 6730416}, ttl_val =
{s = 0x7ffc89e351a0 "", len = 673677328}, user_param_val = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
673677328}, maddr_val = {s = 0x7ffc89e351f0 "Cl\256", len = -413652166},
method_val = {s = 0x7ff028278010 "\001", len = 0}, lr_val = {s =
0x2000000 <Address 0x2000000 out of bounds>, len = 24589344}, r2_val = {
s = 0x67a138 "", len = 8965088}, gr_val = {s = 0x8cd2d0
<Address 0x8cd2d0 out of bounds>, len = 682507072}}
u = 0x66273d
port = 0
dst_host = 0x40000000
i = 28
flags = 0
avp = 0xae6cc7
st = {flags = 31129600, id = 0, name = {n = 0, s = {s = 0x0, len =
93}, re = 0x0}, avp = 0x7ffc89e35080}
sct = 0x7ffc89e35490
sjt = 0x6943d3
rve = 0x7ff0285a8808
mct = 0x6a283b
rv = 0x0
rv1 = 0x7c9170
c1 = {cache_type = 24, val_type = 48, c = {avp_val = {n =
-1981590720, s = {s = 0x7ffc89e35340 "\370\225|", len = -1981590912},
re = 0x7ffc89e35340}, pval = {rs = {s = 0x7ffc89e35340
"\370\225|", len = -1981590912}, ri = 0, flags = 0}},
i2s = '\000' <repeats 21 times>}
s = {s = 0x7fefe2cb183f "", len = -490006491}
srevp = {0x7fefe2cb1825, 0x7fefe2cb1832}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
= {
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}}
__FUNCTION__ = "do_action"
#7 0x00000000004404b8 in run_actions (h=0x7ffc89e35670, a=0x7ff0285a85d0,
msg=0x7ff028b453f8) at core/action.c:1552
t = 0x7ff0285a9668
ret = 1
mod = 0x417250
ms = 176570657
__FUNCTION__ = "run_actions"
#8 0x0000000000440c4a in run_top_route (a=0x7ff0285a85d0,
msg=0x7ff028b453f8, c=0x0) at core/action.c:1641
---Type <return> to continue, or q <return> to quit---
ctx = {rec_lev = 2, run_flags = 0, last_retcode = 1, jmp_env =
{{__jmpbuf = {140669451785720, -8516593617994257148, 50, 1073741824,
33554432,
4289104, -8516593618092823292, 8518532807277145348},
__mask_was_saved = 0, __saved_mask = {__val = {140668355198191, 4302791168,
48,
48, 18446744073709551615, 140722621863712,
140668355198919, 50, 0, 4611686022755909632, 140669451785208,
140722621863776, 5814046,
6608344928, 140669451785208, 0}}}}}
p = 0x7ffc89e35670
ret = 682907128
sfbk = 0
#9 0x000000000052e6b9 in receive_msg (
buf=0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080 SIP/2.0\r\nVia:
SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats 24
times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
sip:usrloc@10.112.101.128"..., len=781, rcv_info=0x7ffc89e35a70) at
core/receive.c:264
msg = 0x7ff028b453f8
ctx = {rec_lev = -1981589208, run_flags = 32764, last_retcode = 8,
jmp_env = {{__jmpbuf = {163, 4253893, 8100093, 140669451683568,
140722621864176, 4845170, 0, 21474836480}, __mask_was_saved
= 673677840, __saved_mask = {__val = {140668768878592, 140722621864448,
6697083, 8094080, 5501958, 243141751568, 140669476122624,
140722621864896, 4294967295, 88212724672, 1, 140668357584160, 0, 33554432,
1024, 6608345520}}}}}
bctx = 0x7ffc89e359b0
ret = 0
stats_on = 0
tvb = {tv_sec = 140722621864208, tv_usec = 8961768}
tve = {tv_sec = 0, tv_usec = 243139749096}
tz = {tz_minuteswest = 50, tz_dsttime = 0}
diff = 0
inb = {
s = 0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080 SIP/2.0\r\nVia:
SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats 24
times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
sip:usrloc@10.112.101.128"..., len = 781}
netinfo = {data = {s = 0x7ffc00000001 <Address 0x7ffc00000001 out
of bounds>, len = 0}, dst = 0x1, rcv = 0x389c422190}
keng = 0x0
__FUNCTION__ = "receive_msg"
#10 0x00000000006456a3 in udp_rcv_loop () at core/udp_server.c:515
len = 781
buf = "KDMQ sip:usrloc@10.112.103.64:5080 SIP/2.0\r\nVia:
SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats 24
times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
sip:usrloc@10.112.101.128:5060>;tag=8a3"...
tmp = 0x100000000 <Address 0x100000000 out of bounds from = 0x7ff028b2c728
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {140722462617610,
140722621864592}, addr32 = {2154131466, 32764, 2313378448, 32764}, addr16 =
{
28682, 32869, 32764, 0, 23184, 35299, 32764, 0}, addr =
"\npe\200\374\177\000\000\220Z\343\211\374\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {1080520714, 0}, addr32 = {1080520714, 0,
0, 0}, addr16 = {28682, 16487, 0, 0, 0, 0, 0, 0},
addr = "\npg@", '\000' <repeats 11 times>}},
src_port = 5060,
dst_port = 5080, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023\304\npe\200\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {
s_addr = 2154131466}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,
---Type <return> to continue, or q <return> to quit---
sin6_flowinfo = 2154131466, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7ff02830fff0, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#11 0x0000000000423c7d in main_loop () at main.c:1623
i = 0
pid = 0
si = 0x7ff02830fff0
si_desc = "udp receiver child=0 sock=10.112.103.64:5080\000\000\000\000
_\343\211\374\177\000\000\062\000\000\000\000\000\000\000\000\000\000@
\000\000\000\000\000\000\000\002\000\000\000\000PrA\000\000\000\000\000PSn\234\070\000\000\000\060\000\000\000\060\000\000\000\020]\343\211\374\177\000\000@
\\\343\211\374\177\000\000\000px\347\357\177\000"
nrprocs = 4
woneinit = 0
__FUNCTION__ = "main_loop"
#12 0x000000000042b0ce in main (argc=13, argv=0x7ffc89e36008) at main.c:2646
cfg_stream = 0x1bb5010
c = -1
r = 0
tmp = 0x7ffc89e36ef7 ""
tmp_len = 32764
port = -1981587768
proto = 56
options = 0x7434f8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 4012862799
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x7ffc89e35f08
p = 0x100000000 <Address 0x100000000 out of bounds
st = {st_dev = 64769, st_ino = 148592, st_nlink = 2, st_mode =
16877, st_uid = 498, st_gid = 498, __pad0 = 0, st_rdev = 0, st_size = 4096,
st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec = 1576522467,
tv_nsec = 972000117}, st_mtim = {tv_sec = 1519927576,
tv_nsec = 668342155}, st_ctim = {tv_sec = 1584026268, tv_nsec =
165523404}, __unused = {0, 0, 0}}
__FUNCTION__ = "main"
(gdb)
Em ter., 17 de mar. de 2020 às 09:52, Daniel-Constantin Mierla <
miconda(a)gmail.com> escreveu:
> Hello,
> does it happen only when doing restart of
kamailio? Can you send the full
> backtrace from gdb?
> Also, 5.0.x is no longer a supported
release for quite some time, try to
> upgrade to 5.2.x or even better to the latest one 5.3.x and see if you can
> reproduce it with one of the supported releases, maybe it is something that
> was already fixed.
> Cheers,
> Daniel
> On 17.03.20 10:32, Virgílio Cunha wrote:
> Hello,
> I have in production environment with 4
proxies using DMQ to sync terminal
> registers and the DMQ configuration parameters is following:
> loadmodule "dmq.so"
> #modparam("dmq", "server_address",
"sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
> modparam("dmq", "notification_address", "DMQ_HOSTS")
> modparam("dmq", "multi_notify", 1)
> modparam("dmq", "num_workers", 4)
> In the lab environment with the same configuration that in prod, I can
> simulate this problem with 12000 terminal REGISTERS and, when doing several
> restarts on a proxy, one of them is generated a coredump.
> Analyzing the core, I can see that the problem is concurrency in the dmq
> synchronization.
> Job_queue_size (...) is called with a valid
*queue *and, but inside this
> function, the *queue *is null.
> (gdb) up
> #1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
> 254 return atomic_get(&queue->count);
> (gdb) print queue
> $1 = (job_queue_t *) 0x0
> (gdb) up
> #2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
> peer=0x7fefebcc4df8) at worker.c:184
> 184 if(job_queue_size(workers[i].queue) == 0) {
> (gdb) print i
> $2 = 1
> (gdb) print workers[i].queue
> $3 = (job_queue_t *) 0x7fefebce49b0
> If I configure DMQ with num_workers=1 I
can't reproduce this problem.
> I'm using the kamailio 5.0.8 release.
> Is this problem known to you? What is the
right way to solve it?
> Best Regards,
> Virgílio Cunha
>
_______________________________________________
> Kamailio (SER) - Users Mailing
Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
12:03 p.m.
Hello,
if it is a runtime issue, then it is better to upgrade to a supported
release series and try to reproduce it.
Looking at startup/shutdown code is easier than the rest used during
runtime by a module, but it is not the case here. So we may waste a lot
of time to dig into it and discover it was fixed or the code in an older
release was refactored not matching any longer what is now in stable
releases.
Cheers,
Daniel
On 17.03.20 11:35, Virgílio Cunha wrote:
Hi,
The problem does not just occur when kamailio is restarted. The
restart was the way I found to reproduce the problem, because when a
proxy is restarted, there is more traffic from DMQ messages, so the
problem is more likely to occur.
The full backtrace is following:
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy...Reading symbols from
/usr/lib/debug/usr/sbin/kamailio.debug...done.
done.
[New Thread 506]
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from
/usr/lib64/kamailio/modules/db_postgres.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/db_postgres.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/db_postgres.so
Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from /usr/lib64/kamailio/libsrdb2.so.1...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/libsrdb2.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrdb2.so.1
Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/libsrdb1.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /usr/lib64/kamailio/modules/ctl.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/ctl.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/kamailio/modules/kex.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/kex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/kex.so
Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/libsrutils.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/kamailio/modules/tm.so...Reading
symbols from /usr/lib/debug/usr/lib64/kamailio/modules/tm.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/tm.so
Reading symbols from /usr/lib64/kamailio/modules/tmx.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/tmx.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/tmx.so
Reading symbols from /usr/lib64/kamailio/modules/corex.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/corex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/corex.so
Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/cfg_rpc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
Reading symbols from /usr/lib64/kamailio/modules/sl.so...Reading
symbols from /usr/lib/debug/usr/lib64/kamailio/modules/sl.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sl.so
Reading symbols from /usr/lib64/kamailio/modules/rr.so...Reading
symbols from /usr/lib/debug/usr/lib64/kamailio/modules/rr.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/rr.so
Reading symbols from /usr/lib64/kamailio/modules/maxfwd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/maxfwd.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/maxfwd.so
Reading symbols from /usr/lib64/kamailio/modules/siputils.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/siputils.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
Reading symbols from /usr/lib64/kamailio/modules/sanity.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sanity.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sanity.so
Reading symbols from /usr/lib64/kamailio/modules/textops.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/textops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/textops.so
Reading symbols from /usr/lib64/kamailio/modules/textopsx.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/textopsx.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/textopsx.so
Reading symbols from /usr/lib64/kamailio/modules/dmq.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dmq.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
Reading symbols from /usr/lib64/kamailio/modules/htable.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/htable.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/htable.so
Reading symbols from /usr/lib64/kamailio/modules/pv.so...Reading
symbols from /usr/lib/debug/usr/lib64/kamailio/modules/pv.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/pv.so
Reading symbols from /usr/lib64/kamailio/modules/xlog.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/xlog.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
Reading symbols from /usr/lib64/kamailio/modules/jsonrpcs.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/jsonrpcs.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/jsonrpcs.so
Reading symbols from /usr/lib64/kamailio/modules/uac.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/uac.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/uac.so
Reading symbols from
/usr/lib64/kamailio/modules/uac_redirect.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/uac_redirect.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/uac_redirect.so
Reading symbols from /usr/lib64/kamailio/modules/db_text.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/db_text.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/db_text.so
Reading symbols from
/usr/lib64/kamailio/modules/dispatcher.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dispatcher.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dispatcher.so
Reading symbols from /usr/lib64/kamailio/modules/sqlops.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sqlops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sqlops.so
Reading symbols from /usr/lib64/kamailio/modules/regex.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/regex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/regex.so
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from
/usr/lib64/kamailio/modules/permissions.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/permissions.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/permissions.so
Reading symbols from /usr/lib64/kamailio/modules/dialplan.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dialplan.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dialplan.so
Reading symbols from /usr/lib64/kamailio/modules/mtree.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/mtree.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/mtree.so
Reading symbols from /usr/lib64/kamailio/modules/cfgutils.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/cfgutils.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/cfgutils.so
Reading symbols from /usr/lib64/kamailio/modules/statsd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/statsd.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/statsd.so
Reading symbols from /usr/lib64/kamailio/modules/app_lua.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/app_lua.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/app_lua.so
Reading symbols from /usr/lib64/liblua-5.1.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/liblua-5.1.so <http://liblua-5.1.so>
Reading symbols from /usr/lib64/kamailio/modules/rtimer.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/rtimer.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/rtimer.so
Reading symbols from /usr/lib64/kamailio/modules/mqueue.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/mqueue.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/mqueue.so
Reading symbols from /usr/lib64/kamailio/modules/auth.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/auth.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/auth.so
Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/usrloc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
Reading symbols from
/usr/lib64/kamailio/modules/registrar.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/registrar.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
Reading symbols from
/usr/lib64/kamailio/modules/dmq_usrloc.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dmq_usrloc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
Reading symbols from /usr/lib64/kamailio/modules/xhttp.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/xhttp.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/xhttp.so
Reading symbols from /usr/lib64/kamailio/modules/exec.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/exec.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/exec.so
Reading symbols from
/usr/lib64/kamailio/modules/http_client.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/http_client.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/http_client.so
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/libcurl.so.4...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libcurl.so.4
Reading symbols from /lib64/libidn.so.11...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libidn.so.11
Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /usr/lib64/libssh2.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssh2.so.1
Reading symbols from /usr/lib64/kamailio/modules/json.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/json.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/json.so
Reading symbols from /lib64/libjson-c.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libjson-c.so.2
Reading symbols from /usr/lib64/kamailio/modules/ipops.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/ipops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/ipops.so
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Core was generated by `/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy -f
/opt/ptin/kamailio-ksipadp/prox'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
../../core/atomic/atomic_common.h:66
66 return atomic_get_int(&(v->val));
Missing separate debuginfos, use: debuginfo-install
kamailio-ksipadp-2.8.0-1.ptin.el6.x86_64
(gdb) bt full
#0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
../../core/atomic/atomic_common.h:66
No locals.
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
No locals.
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
peer=0x7fefebcc4df8) at worker.c:184
i = 1
found_available = 0
new_job = {f = 0x7fefe2613797 <usrloc_dmq_handle_msg>, msg =
0x7fefebce67c8, orig_peer = 0x7fefebcc4df8, next = 0x0, prev = 0x0}
worker = 0x7fefe7a1d890
cloned_msg = 0x7fefebce67c8
cloned_msg_len = 3888
__FUNCTION__ = "add_dmq_job"
#3 0x00007fefe58dfec1 in dmq_handle_message (msg=0x7ff028b453f8,
str1=0x0, str2=0x0) at message.c:66
peer = 0x7fefebcc4df8
__FUNCTION__ = "dmq_handle_message"
#4 0x0000000000433729 in do_action (h=0x7ffc89e35670,
a=0x7ff0285a9340, msg=0x7ff028b453f8) at core/action.c:1054
ret = -5
v = 1
dst = {send_sock = 0x7ff0285a8810, to = {s = {sa_family =
21376, sa_data = "\343\211\374\177\000\000\370S\264(\360\177\000"},
sin = {
sin_family = 21376, sin_port = 35299, sin_addr = {s_addr
= 32764}, sin_zero = "\370S\264(\360\177\000"}, sin6 = {sin6_family =
21376,
sin6_port = 35299, sin6_flowinfo = 32764, sin6_addr =
{__in6_u = {__u6_addr8 =
"\370S\264(\360\177\000\000pV\343\211\374\177\000",
__u6_addr16 = {21496, 10420, 32752, 0, 22128, 35299,
32764, 0}, __u6_addr32 = {682906616, 32752, 2313377392, 32764}}},
sin6_scope_id = 1}}, id = 32764, proto = 0 '\000',
send_flags = {f = 0, blst_imask = 0}}
tmp = 0x389c422190 ""
new_uri = 0x7c00000000 <Address 0x7c00000000 out of bounds>
end = 0x6e00000077 <Address 0x6e00000077 out of bounds>
crt = 0x389c98e120 ""
cmd = 0x7ff0283219b0
len = 0
user = 0
uri = {user = {s = 0x7ffc89e34a10 "@K\343\211\374\177", len =
4459394}, passwd = {s = 0x7ffc89e34e80 "\356Q\343\211\374\177",
len = 682906616}, host = {s = 0x7ff0285a9020 "\177\001",
len = -1981589904}, port = {s = 0x0, len = 1}, params = {
s = 0x7ff028b453f8 "\017", len = 295169284}, sip_params =
{s = 0x32 <Address 0x32 out of bounds>, len = 1073741824}, headers = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
4289104}, port_no = 60676, proto = 5047, type = 2312043322, flags =
2168712452,
transport = {s = 0x0, len = -1675583518}, ttl = {s =
0x89e34e90 <Address 0x89e34e90 out of bounds>, len = 35715777},
user_param = {
s = 0xffffffff0000003c <Address 0xffffffff0000003c out of
bounds>, len = -1671419584}, maddr = {s = 0x0, len = -1981592704},
method = {
s = 0x389c603c48 "", len = -1671391960}, lr = {s = 0x0,
len = -1981592544}, r2 = {s = 0x1dde9a0 " 9\336\001", len = 0}, gr = {
s = 0x4 <Address 0x4 out of bounds>, len = 2479},
transport_val = {s = 0x389c422190 "", len = 1}, ttl_val = {
s = 0x7ffc89e34b40 "pM\343\211\374\177", len = 5683491},
user_param_val = {s = 0x40e7e8
"\370\377\377\377\377\377\377\377@ݘ\234\070",
len = -1671362728}, maddr_val = {s = 0x404130 "", len =
0}, method_val = {s = 0x1000003b4 <Address 0x1000003b4 out of bounds>,
len = 0},
lr_val = {s = 0x7fefe7374b9a "%x%c%x%c", len = -1673386776},
r2_val = {s = 0x7ffc89e34bd0 "", len = -1981592584}, gr_val = {
---Type <return> to continue, or q <return> to quit---
s = 0x389c422190 "", len = 0}}
next_hop = {user = {s = 0x1 <Address 0x1 out of bounds>, len =
40}, passwd = {s = 0x0, len = -1407107317}, host = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
31398688}, port = {s = 0x1dde9a0 " 9\336\001", len = 31320480}, params = {
s = 0x7ffc89e34c20 "", len = -1981593152}, sip_params = {s
= 0x7ffc89e34910 "\200N\343\211\374\177", len = 682906616}, headers = {
s = 0x32 <Address 0x32 out of bounds>, len = 1073741824},
port_no = 0, proto = 512, type = ERROR_URI_T, flags = 4289104,
transport = {
s = 0x7ffc89e34900 "\020J\343\211\374\177", len =
4457737}, ttl = {s = 0x7ffc89e348f0 "", len = -1981593520}, user_param = {
s = 0x7ffc89e34858 "\r", len = -1981593264}, maddr = {s =
0x7ffc89e34870 "", len = -1981593456}, method = {s = 0x7ffc89e348b0
"\r",
len = -1981593392}, lr = {s = 0x8cd2d0 <Address 0x8cd2d0
out of bounds>, len = 13}, r2 = {
s = 0x10100000002 <Address 0x10100000002 out of bounds>,
len = -1}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len =
682906616},
ttl_val = {s = 0x7ff0285a9020 "\177\001", len =
-1981593296}, user_param_val = {s = 0x7ffc89e34fa0
"\001\200\255\373\374\177", len = 0},
maddr_val = {s = 0x10000000d <Address 0x10000000d out of
bounds>, len = -1981591296}, method_val = {
s = 0x7fef00000000 <Address 0x7fef00000000 out of bounds>,
len = -1671151657}, lr_val = {s = 0x7fefe7374ba2 "", len = 682906616},
r2_val = {s = 0x32 <Address 0x32 out of bounds>, len =
1073741824}, gr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>,
len = 4289104}}
u = 0x389c4224e8
port = 0
dst_host = 0x89e34bd0
i = 32764
flags = 52
avp = 0x189e35040
st = {flags = 2313374752, id = 32764, name = {n = -1981593512,
s = {s = 0x7ffc89e34858 "\r", len = -1981593152}, re = 0x7ffc89e34858},
avp = 0x7ffc89e347a0}
sct = 0x0
sjt = 0x7ff028b453f8
rve = 0x32
mct = 0x40000000
rv = 0x7ffc89e34c20
rv1 = 0x2000000
c1 = {cache_type = 2313374144, val_type = 32764, c = {avp_val
= {n = 1, s = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, re =
0x1}, pval = {
rs = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, ri
= -1407115050, flags = 52}},
i2s = "\000\000\000\000\000\000\000\000\274\224
\254\064\000\000\000\240G\343\211\374\177"}
s = {s = 0x1 <Address 0x1 out of bounds>, len = -1981593512}
srevp = {0x7ffc89e34960, 0x34ac208a97}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s
= 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data
= 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr
= 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u =
{number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr
= 0x0,
select = 0x0}}}
---Type <return> to continue, or q <return> to quit---
__FUNCTION__ = "do_action"
#5 0x00000000004404b8 in run_actions (h=0x7ffc89e35670,
a=0x7ff0285a9340, msg=0x7ff028b453f8) at core/action.c:1552
t = 0x7ff0285a9340
ret = -1
mod = 0x1dde9a0
ms = 176570657
__FUNCTION__ = "run_actions"
#6 0x0000000000433692 in do_action (h=0x7ffc89e35670,
a=0x7ff0285a9668, msg=0x7ff028b453f8) at core/action.c:1045
ret = 1
v = 1
dst = {send_sock = 0x7ffc89e35380, to = {s = {sa_family =
61042, sa_data = "I", '\000' <repeats 12 times>}, sin =
{sin_family =
61042,
sin_port = 73, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 61042,
sin6_port = 73,
sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 =
"\000\000\000\000\000\000\000\000\020\000\000\000\005\000\000",
__u6_addr16 = {
0, 0, 0, 0, 16, 0, 5, 0}, __u6_addr32 = {0, 0, 16,
5}}}, sin6_scope_id = 673678288}}, id = 32752, proto = 0 '\000',
send_flags = {
f = 0, blst_imask = 32752}}
tmp = 0x66307b "H\213E\270H\203\300\070\351O\003"
new_uri = 0x7ff028a93048 ""
end = 0x7ff028a93038 "\303l\256"
crt = 0x2000000 <Address 0x2000000 out of bounds>
cmd = 0x1773508
len = 6791248
user = 0
uri = {user = {s = 0xae6ecd "", len = 11430897}, passwd = {s =
0x7c95f8 "core: core/parser/msg_parser.c", len = 682507016}, host = {
s = 0xae6c43 "To: <sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
<sip:usrloc@10.112.101.128:5060
<http://sip:usrloc@10.112.101.128:5060>>;tag=8a3534eb92149bbe647f599d354f2511-090e\r\nCSeq:
10 KDMQ\r\nCall-ID: 409f4ac5788a8cc4-24333(a)10.112.101.128
<mailto:409f4ac5788a8cc4-24333@10.112.101.128>\r\nContent-Length:
39"..., len = 11430924}, port = {s = 0x0, len = 682507072}, params = {
s = 0x7ff028ae3c90 <incomplete sequence \350>, len = 0},
sip_params = {s = 0x7ffc89e35330 "\246", len = 6697083}, headers = {
s = 0x7c9170 "core", len = -1981590832}, port_no = 8,
proto = 0, type = ERROR_URI_T, flags = 30415600, transport = {
s = 0xa8428197 <Address 0xa8428197 out of bounds>, len =
-1675583518}, ttl = {s = 0x3000000030 <Address 0x3000000030 out of
bounds>,
len = 44108294}, user_param = {s = 0x400000017 <Address
0x400000017 out of bounds>, len = 8176032}, maddr = {s = 0x0, len =
-1981590448},
method = {s = 0x389c603c48 "", len = 11431629}, lr = {s =
0x7ffc89e35450 "\220T\343\211\374\177", len = 6869803}, r2 = {
s = 0x7cc190 "get_hdr_field", len = 6730416}, gr = {s =
0x7ffc89e35300 "", len = 673677328}, transport_val = {s = 0x389c422190
"",
len = 673677328}, ttl_val = {s = 0x7ffc89e35350
"\200S\343\211\374\177", len = -413652166}, user_param_val = {s =
0x40e7f9 "", len = 0},
maddr_val = {s = 0x2000000 <Address 0x2000000 out of
bounds>, len = 24588552}, method_val = {
s = 0x67a450
"\030A\211\321A)\301D\211ȉ\302H\213\205(\377\377\377\211P \276\004",
len = 8965880}, lr_val = {
s = 0x8cd2d0 <Address 0x8cd2d0 out of bounds>, len = 8},
r2_val = {s = 0xa6 <Address 0xa6 out of bounds>, len = 8136007},
gr_val = {
s = 0x7c95f8 "core: core/parser/msg_parser.c", len =
682176488}}
next_hop = {user = {s = 0x1e7374b9a <Address 0x1e7374b9a out
of bounds>, len = 674472240}, passwd = {
s = 0xffffffff <Address 0xffffffff out of bounds>, len =
4289104}, host = {s = 0x4e2cb1825 <Address 0x4e2cb1825 out of bounds>,
len = -1667705792}, port = {s = 0x0, len = 1073741824},
params = {s = 0x2000000 <Address 0x2000000 out of bounds>, len = 4289104},
sip_params = {s = 0x7ffc89e35180 "\360Q\343\211\374\177",
len = 6031673}, headers = {s = 0x7fefe2cb183f "", len = -1981591296},
port_no = 23013, proto = 58022, type = 32751, flags =
2313376240, transport = {s = 0x7fefe7374b9a "%x%c%x%c", len =
-1981590944}, ttl = {
---Type <return> to continue, or q <return> to quit---
s = 0x1a <Address 0x1a out of bounds>, len = 4289104},
user_param = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len = -1670974014},
maddr = {
s = 0x30fbad8001 <Address 0x30fbad8001 out of bounds>, len
= -492414491}, method = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len =
-492414491},
lr = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len = -492414478},
r2 = {s = 0x7fefe2a659ff "", len = -492414491}, gr = {s =
0x7fefe2a659ff "",
len = 0}, transport_val = {s = 0x0, len = 6730416},
ttl_val = {s = 0x7ffc89e351a0 "", len = 673677328}, user_param_val = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
673677328}, maddr_val = {s = 0x7ffc89e351f0 "Cl\256", len = -413652166},
method_val = {s = 0x7ff028278010 "\001", len = 0}, lr_val =
{s = 0x2000000 <Address 0x2000000 out of bounds>, len = 24589344},
r2_val = {
s = 0x67a138 "", len = 8965088}, gr_val = {s = 0x8cd2d0
<Address 0x8cd2d0 out of bounds>, len = 682507072}}
u = 0x66273d
port = 0
dst_host = 0x40000000
i = 28
flags = 0
avp = 0xae6cc7
st = {flags = 31129600, id = 0, name = {n = 0, s = {s = 0x0,
len = 93}, re = 0x0}, avp = 0x7ffc89e35080}
sct = 0x7ffc89e35490
sjt = 0x6943d3
rve = 0x7ff0285a8808
mct = 0x6a283b
rv = 0x0
rv1 = 0x7c9170
c1 = {cache_type = 24, val_type = 48, c = {avp_val = {n =
-1981590720, s = {s = 0x7ffc89e35340 "\370\225|", len = -1981590912},
re = 0x7ffc89e35340}, pval = {rs = {s = 0x7ffc89e35340
"\370\225|", len = -1981590912}, ri = 0, flags = 0}},
i2s = '\000' <repeats 21 times>}
s = {s = 0x7fefe2cb183f "", len = -490006491}
srevp = {0x7fefe2cb1825, 0x7fefe2cb1832}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s
= 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data
= 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr
= 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u =
{number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr
= 0x0,
select = 0x0}}}
__FUNCTION__ = "do_action"
#7 0x00000000004404b8 in run_actions (h=0x7ffc89e35670,
a=0x7ff0285a85d0, msg=0x7ff028b453f8) at core/action.c:1552
t = 0x7ff0285a9668
ret = 1
mod = 0x417250
ms = 176570657
__FUNCTION__ = "run_actions"
#8 0x0000000000440c4a in run_top_route (a=0x7ff0285a85d0,
msg=0x7ff028b453f8, c=0x0) at core/action.c:1641
---Type <return> to continue, or q <return> to quit---
ctx = {rec_lev = 2, run_flags = 0, last_retcode = 1, jmp_env =
{{__jmpbuf = {140669451785720, -8516593617994257148, 50, 1073741824,
33554432,
4289104, -8516593618092823292, 8518532807277145348},
__mask_was_saved = 0, __saved_mask = {__val = {140668355198191,
4302791168, 48,
48, 18446744073709551615, 140722621863712,
140668355198919, 50, 0, 4611686022755909632, 140669451785208,
140722621863776, 5814046,
6608344928, 140669451785208, 0}}}}}
p = 0x7ffc89e35670
ret = 682907128
sfbk = 0
#9 0x000000000052e6b9 in receive_msg (
buf=0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia: SIP/2.0/UDP
10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats 24 times>,
".0\r\nTo: <sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
<sip:usrloc@10.112.101.128 <mailto:sip%3Ausrloc@10.112.101.128>"...,
len=781, rcv_info=0x7ffc89e35a70) at core/receive.c:264
msg = 0x7ff028b453f8
ctx = {rec_lev = -1981589208, run_flags = 32764, last_retcode
= 8, jmp_env = {{__jmpbuf = {163, 4253893, 8100093, 140669451683568,
140722621864176, 4845170, 0, 21474836480},
__mask_was_saved = 673677840, __saved_mask = {__val =
{140668768878592, 140722621864448,
6697083, 8094080, 5501958, 243141751568,
140669476122624, 140722621864896, 4294967295, 88212724672, 1,
140668357584160, 0, 33554432,
1024, 6608345520}}}}}
bctx = 0x7ffc89e359b0
ret = 0
stats_on = 0
tvb = {tv_sec = 140722621864208, tv_usec = 8961768}
tve = {tv_sec = 0, tv_usec = 243139749096}
tz = {tz_minuteswest = 50, tz_dsttime = 0}
diff = 0
inb = {
s = 0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia: SIP/2.0/UDP
10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats 24 times>,
".0\r\nTo: <sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
<sip:usrloc@10.112.101.128 <mailto:sip%3Ausrloc@10.112.101.128>"...,
len = 781}
netinfo = {data = {s = 0x7ffc00000001 <Address 0x7ffc00000001
out of bounds>, len = 0}, dst = 0x1, rcv = 0x389c422190}
keng = 0x0
__FUNCTION__ = "receive_msg"
#10 0x00000000006456a3 in udp_rcv_loop () at core/udp_server.c:515
len = 781
buf = "KDMQ sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia: SIP/2.0/UDP
10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats 24 times>,
".0\r\nTo: <sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
<sip:usrloc@10.112.101.128:5060
<http://sip:usrloc@10.112.101.128:5060>>;tag=8a3"...
tmp = 0x100000000 <Address 0x100000000 out of bounds>
from = 0x7ff028b2c728
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl =
{140722462617610, 140722621864592}, addr32 = {2154131466, 32764,
2313378448, 32764}, addr16 = {
28682, 32869, 32764, 0, 23184, 35299, 32764, 0}, addr
= "\npe\200\374\177\000\000\220Z\343\211\374\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {1080520714, 0}, addr32 =
{1080520714, 0, 0, 0}, addr16 = {28682, 16487, 0, 0, 0, 0, 0, 0},
addr = "\npg@", '\000' <repeats 11 times>}},
src_port =
5060, dst_port = 5080, proto_reserved1 = 0, proto_reserved2 = 0,
src_su = {s = {
sa_family = 2, sa_data =
"\023\304\npe\200\000\000\000\000\000\000\000"}, sin = {sin_family =
2, sin_port = 50195, sin_addr = {
s_addr = 2154131466}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
---Type <return> to continue, or q <return> to quit---
sin6_flowinfo = 2154131466, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0,
0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7ff02830fff0, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#11 0x0000000000423c7d in main_loop () at main.c:1623
i = 0
pid = 0
si = 0x7ff02830fff0
si_desc = "udp receiver child=0 sock=10.112.103.64:5080
<http://10.112.103.64:5080>\000\000\000\000
_\343\211\374\177\000\000\062\000\000\000\000\000\000\000\000\000\000@\000\000\000\000\000\000\000\002\000\000\000\000PrA\000\000\000\000\000PSn\234\070\000\000\000\060\000\000\000\060\000\000\000\020]\343\211\374\177\000\000@\\\343\211\374\177\000\000\000px\347\357\177\000"
nrprocs = 4
woneinit = 0
__FUNCTION__ = "main_loop"
#12 0x000000000042b0ce in main (argc=13, argv=0x7ffc89e36008) at
main.c:2646
cfg_stream = 0x1bb5010
c = -1
r = 0
tmp = 0x7ffc89e36ef7 ""
tmp_len = 32764
port = -1981587768
proto = 56
options = 0x7434f8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 4012862799
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x7ffc89e35f08
p = 0x100000000 <Address 0x100000000 out of bounds>
st = {st_dev = 64769, st_ino = 148592, st_nlink = 2, st_mode =
16877, st_uid = 498, st_gid = 498, __pad0 = 0, st_rdev = 0, st_size =
4096,
st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec =
1576522467, tv_nsec = 972000117}, st_mtim = {tv_sec = 1519927576,
tv_nsec = 668342155}, st_ctim = {tv_sec = 1584026268,
tv_nsec = 165523404}, __unused = {0, 0, 0}}
__FUNCTION__ = "main"
(gdb)
Em ter., 17 de mar. de 2020 às 09:52, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> escreveu:
Hello,
does it happen only when doing restart of kamailio? Can you send
the full backtrace from gdb?
Also, 5.0.x is no longer a supported release for quite some time,
try to upgrade to 5.2.x or even better to the latest one 5.3.x and
see if you can reproduce it with one of the supported releases,
maybe it is something that was already fixed.
Cheers,
Daniel
On 17.03.20 10:32, Virgílio Cunha wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Hello,
I have in production environment with 4 proxies using DMQ to sync
terminal registers and the DMQ configuration parameters is following:
loadmodule "dmq.so"
#modparam("dmq", "server_address",
"sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
modparam("dmq", "notification_address", "DMQ_HOSTS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
In the lab environment with the same configuration that in prod,
I can simulate this problem with 12000 terminal REGISTERS and,
when doing several restarts on a proxy, one of them is generated
a coredump.
Analyzing the core, I can see that the problem is concurrency in
the dmq synchronization.
Job_queue_size (...) is called with a valid /queue /and, but
inside this function, the /queue /is null.
(gdb) up
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
254 return atomic_get(&queue->count);
(gdb) print queue
$1 = (job_queue_t *) 0x0
(gdb) up
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
peer=0x7fefebcc4df8) at worker.c:184
184 if(job_queue_size(workers[i].queue) == 0) {
(gdb) print i
$2 = 1
(gdb) print workers[i].queue
$3 = (job_queue_t *) 0x7fefebce49b0
If I configure DMQ with num_workers=1 I can't reproduce this
problem.
I'm using the kamailio 5.0.8 release.
Is this problem known to you? What is the right way to solve it?
Best Regards,
Virgílio Cunha
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
18 Mar
18 Mar
7:22 p.m.
Hi,
Sorry for using this topic for this question, but where can I find
Kamailio's release policy?
I needed to know how long the kamailio 5.3.x branch will maintain its
support.
Best Regards,
Virgílio Cunha
Em ter., 17 de mar. de 2020 às 11:03, Daniel-Constantin Mierla <
miconda(a)gmail.com> escreveu:
Hello,
if it is a runtime issue, then it is better to upgrade to a supported
release series and try to reproduce it.
Looking at startup/shutdown code is easier than the rest used during
runtime by a module, but it is not the case here. So we may waste a lot of
time to dig into it and discover it was fixed or the code in an older
release was refactored not matching any longer what is now in stable
releases.
Cheers,
Daniel
On 17.03.20 11:35, Virgílio Cunha wrote:
Hi,
The problem does not just occur when kamailio is restarted. The restart
was the way I found to reproduce the problem, because when a proxy is
restarted, there is more traffic from DMQ messages, so the problem is more
likely to occur.
The full backtrace is following:
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <
http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy...Reading symbols from
/usr/lib/debug/usr/sbin/kamailio.debug...done.
done.
[New Thread 506]
Reading symbols from /lib64/libdl.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/kamailio/modules/db_postgres.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/db_postgres.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/db_postgres.so
Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from /usr/lib64/kamailio/libsrdb2.so.1...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/libsrdb2.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrdb2.so.1
Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/libsrdb1.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /usr/lib64/kamailio/modules/ctl.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/ctl.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/kamailio/modules/kex.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/kex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/kex.so
Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/libsrutils.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/kamailio/modules/tm.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/tm.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/tm.so
Reading symbols from /usr/lib64/kamailio/modules/tmx.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/tmx.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/tmx.so
Reading symbols from /usr/lib64/kamailio/modules/corex.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/corex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/corex.so
Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/cfg_rpc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
Reading symbols from /usr/lib64/kamailio/modules/sl.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/sl.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sl.so
Reading symbols from /usr/lib64/kamailio/modules/rr.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/rr.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/rr.so
Reading symbols from /usr/lib64/kamailio/modules/maxfwd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/maxfwd.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/maxfwd.so
Reading symbols from /usr/lib64/kamailio/modules/siputils.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/siputils.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
Reading symbols from /usr/lib64/kamailio/modules/sanity.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sanity.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sanity.so
Reading symbols from /usr/lib64/kamailio/modules/textops.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/textops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/textops.so
Reading symbols from /usr/lib64/kamailio/modules/textopsx.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/textopsx.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/textopsx.so
Reading symbols from /usr/lib64/kamailio/modules/dmq.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/dmq.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
Reading symbols from /usr/lib64/kamailio/modules/htable.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/htable.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/htable.so
Reading symbols from /usr/lib64/kamailio/modules/pv.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/pv.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/pv.so
Reading symbols from /usr/lib64/kamailio/modules/xlog.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/xlog.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
Reading symbols from /usr/lib64/kamailio/modules/jsonrpcs.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/jsonrpcs.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/jsonrpcs.so
Reading symbols from /usr/lib64/kamailio/modules/uac.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/uac.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/uac.so
Reading symbols from /usr/lib64/kamailio/modules/uac_redirect.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/uac_redirect.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/uac_redirect.so
Reading symbols from /usr/lib64/kamailio/modules/db_text.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/db_text.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/db_text.so
Reading symbols from /usr/lib64/kamailio/modules/dispatcher.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dispatcher.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dispatcher.so
Reading symbols from /usr/lib64/kamailio/modules/sqlops.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sqlops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sqlops.so
Reading symbols from /usr/lib64/kamailio/modules/regex.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/regex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/regex.so
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /usr/lib64/kamailio/modules/permissions.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/permissions.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/permissions.so
Reading symbols from /usr/lib64/kamailio/modules/dialplan.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dialplan.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dialplan.so
Reading symbols from /usr/lib64/kamailio/modules/mtree.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/mtree.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/mtree.so
Reading symbols from /usr/lib64/kamailio/modules/cfgutils.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/cfgutils.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/cfgutils.so
Reading symbols from /usr/lib64/kamailio/modules/statsd.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/statsd.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/statsd.so
Reading symbols from /usr/lib64/kamailio/modules/app_lua.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/app_lua.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/app_lua.so
Reading symbols from /usr/lib64/liblua-5.1.so...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/liblua-5.1.so
Reading symbols from /usr/lib64/kamailio/modules/rtimer.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/rtimer.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/rtimer.so
Reading symbols from /usr/lib64/kamailio/modules/mqueue.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/mqueue.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/mqueue.so
Reading symbols from /usr/lib64/kamailio/modules/auth.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/auth.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/auth.so
Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/usrloc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
Reading symbols from /usr/lib64/kamailio/modules/registrar.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/registrar.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dmq_usrloc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
Reading symbols from /usr/lib64/kamailio/modules/xhttp.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/xhttp.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/xhttp.so
Reading symbols from /usr/lib64/kamailio/modules/exec.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/exec.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/exec.so
Reading symbols from /usr/lib64/kamailio/modules/http_client.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/http_client.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/http_client.so
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/libcurl.so.4...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libcurl.so.4
Reading symbols from /lib64/libidn.so.11...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libidn.so.11
Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /usr/lib64/libssh2.so.1...(no debugging symbols
found)...done.
Loaded symbols for /usr/lib64/libssh2.so.1
Reading symbols from /usr/lib64/kamailio/modules/json.so...Reading symbols
from /usr/lib/debug/usr/lib64/kamailio/modules/json.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/json.so
Reading symbols from /lib64/libjson-c.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libjson-c.so.2
Reading symbols from /usr/lib64/kamailio/modules/ipops.so...Reading
symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/ipops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/ipops.so
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Core was generated by `/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy -f
/opt/ptin/kamailio-ksipadp/prox'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
../../core/atomic/atomic_common.h:66
66 return atomic_get_int(&(v->val));
Missing separate debuginfos, use: debuginfo-install
kamailio-ksipadp-2.8.0-1.ptin.el6.x86_64
(gdb) bt full
#0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
../../core/atomic/atomic_common.h:66
No locals.
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
No locals.
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
peer=0x7fefebcc4df8) at worker.c:184
i = 1
found_available = 0
new_job = {f = 0x7fefe2613797 <usrloc_dmq_handle_msg>, msg =
0x7fefebce67c8, orig_peer = 0x7fefebcc4df8, next = 0x0, prev = 0x0}
worker = 0x7fefe7a1d890
cloned_msg = 0x7fefebce67c8
cloned_msg_len = 3888
__FUNCTION__ = "add_dmq_job"
#3 0x00007fefe58dfec1 in dmq_handle_message (msg=0x7ff028b453f8,
str1=0x0, str2=0x0) at message.c:66
peer = 0x7fefebcc4df8
__FUNCTION__ = "dmq_handle_message"
#4 0x0000000000433729 in do_action (h=0x7ffc89e35670, a=0x7ff0285a9340,
msg=0x7ff028b453f8) at core/action.c:1054
ret = -5
v = 1
dst = {send_sock = 0x7ff0285a8810, to = {s = {sa_family = 21376,
sa_data = "\343\211\374\177\000\000\370S\264(\360\177\000"}, sin = {
sin_family = 21376, sin_port = 35299, sin_addr = {s_addr =
32764}, sin_zero = "\370S\264(\360\177\000"}, sin6 = {sin6_family = 21376,
sin6_port = 35299, sin6_flowinfo = 32764, sin6_addr =
{__in6_u = {__u6_addr8 = "\370S\264(\360\177\000\000pV\343\211\374\177\000",
__u6_addr16 = {21496, 10420, 32752, 0, 22128, 35299,
32764, 0}, __u6_addr32 = {682906616, 32752, 2313377392, 32764}}},
sin6_scope_id = 1}}, id = 32764, proto = 0 '\000',
send_flags = {f = 0, blst_imask = 0}}
tmp = 0x389c422190 ""
new_uri = 0x7c00000000 <Address 0x7c00000000 out of bounds>
end = 0x6e00000077 <Address 0x6e00000077 out of bounds>
crt = 0x389c98e120 ""
cmd = 0x7ff0283219b0
len = 0
user = 0
uri = {user = {s = 0x7ffc89e34a10 "@K\343\211\374\177", len =
4459394}, passwd = {s = 0x7ffc89e34e80 "\356Q\343\211\374\177",
len = 682906616}, host = {s = 0x7ff0285a9020 "\177\001", len =
-1981589904}, port = {s = 0x0, len = 1}, params = {
s = 0x7ff028b453f8 "\017", len = 295169284}, sip_params = {s =
0x32 <Address 0x32 out of bounds>, len = 1073741824}, headers = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
4289104}, port_no = 60676, proto = 5047, type = 2312043322, flags =
2168712452,
transport = {s = 0x0, len = -1675583518}, ttl = {s = 0x89e34e90
<Address 0x89e34e90 out of bounds>, len = 35715777}, user_param = {
s = 0xffffffff0000003c <Address 0xffffffff0000003c out of
bounds>, len = -1671419584}, maddr = {s = 0x0, len = -1981592704}, method =
{
s = 0x389c603c48 "", len = -1671391960}, lr = {s = 0x0, len =
-1981592544}, r2 = {s = 0x1dde9a0 " 9\336\001", len = 0}, gr = {
s = 0x4 <Address 0x4 out of bounds>, len = 2479},
transport_val = {s = 0x389c422190 "", len = 1}, ttl_val = {
s = 0x7ffc89e34b40 "pM\343\211\374\177", len = 5683491},
user_param_val = {s = 0x40e7e8 "\370\377\377\377\377\377\377\377@ݘ
\234\070",
len = -1671362728}, maddr_val = {s = 0x404130 "", len = 0},
method_val = {s = 0x1000003b4 <Address 0x1000003b4 out of bounds>, len = 0},
lr_val = {s = 0x7fefe7374b9a "%x%c%x%c", len = -1673386776},
r2_val = {s = 0x7ffc89e34bd0 "", len = -1981592584}, gr_val = {
---Type <return> to continue, or q <return> to quit---
s = 0x389c422190 "", len = 0}}
next_hop = {user = {s = 0x1 <Address 0x1 out of bounds>, len =
40}, passwd = {s = 0x0, len = -1407107317}, host = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
31398688}, port = {s = 0x1dde9a0 " 9\336\001", len = 31320480}, params = {
s = 0x7ffc89e34c20 "", len = -1981593152}, sip_params = {s =
0x7ffc89e34910 "\200N\343\211\374\177", len = 682906616}, headers = {
s = 0x32 <Address 0x32 out of bounds>, len = 1073741824},
port_no = 0, proto = 512, type = ERROR_URI_T, flags = 4289104, transport = {
s = 0x7ffc89e34900 "\020J\343\211\374\177", len = 4457737},
ttl = {s = 0x7ffc89e348f0 "", len = -1981593520}, user_param = {
s = 0x7ffc89e34858 "\r", len = -1981593264}, maddr = {s =
0x7ffc89e34870 "", len = -1981593456}, method = {s = 0x7ffc89e348b0
"\r",
len = -1981593392}, lr = {s = 0x8cd2d0 <Address 0x8cd2d0 out
of bounds>, len = 13}, r2 = {
s = 0x10100000002 <Address 0x10100000002 out of bounds>, len =
-1}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 682906616},
ttl_val = {s = 0x7ff0285a9020 "\177\001", len = -1981593296},
user_param_val = {s = 0x7ffc89e34fa0 "\001\200\255\373\374\177", len = 0},
maddr_val = {s = 0x10000000d <Address 0x10000000d out of
bounds>, len = -1981591296}, method_val = {
s = 0x7fef00000000 <Address 0x7fef00000000 out of bounds>, len
= -1671151657}, lr_val = {s = 0x7fefe7374ba2 "", len = 682906616},
r2_val = {s = 0x32 <Address 0x32 out of bounds>, len =
1073741824}, gr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>,
len = 4289104}}
u = 0x389c4224e8
port = 0
dst_host = 0x89e34bd0
i = 32764
flags = 52
avp = 0x189e35040
st = {flags = 2313374752, id = 32764, name = {n = -1981593512, s =
{s = 0x7ffc89e34858 "\r", len = -1981593152}, re = 0x7ffc89e34858},
avp = 0x7ffc89e347a0}
sct = 0x0
sjt = 0x7ff028b453f8
rve = 0x32
mct = 0x40000000
rv = 0x7ffc89e34c20
rv1 = 0x2000000
c1 = {cache_type = 2313374144, val_type = 32764, c = {avp_val = {n
= 1, s = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, re = 0x1}, pval = {
rs = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, ri =
-1407115050, flags = 52}},
i2s = "\000\000\000\000\000\000\000\000\274\224
\254\064\000\000\000\240G\343\211\374\177"}
s = {s = 0x1 <Address 0x1 out of bounds>, len = -1981593512}
srevp = {0x7ffc89e34960, 0x34ac208a97}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
= {
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}}
---Type <return> to continue, or q <return> to quit---
__FUNCTION__ = "do_action"
#5 0x00000000004404b8 in run_actions (h=0x7ffc89e35670, a=0x7ff0285a9340,
msg=0x7ff028b453f8) at core/action.c:1552
t = 0x7ff0285a9340
ret = -1
mod = 0x1dde9a0
ms = 176570657
__FUNCTION__ = "run_actions"
#6 0x0000000000433692 in do_action (h=0x7ffc89e35670, a=0x7ff0285a9668,
msg=0x7ff028b453f8) at core/action.c:1045
ret = 1
v = 1
dst = {send_sock = 0x7ffc89e35380, to = {s = {sa_family = 61042,
sa_data = "I", '\000' <repeats 12 times>}, sin = {sin_family =
61042,
sin_port = 73, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 61042, sin6_port =
73,
sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 =
"\000\000\000\000\000\000\000\000\020\000\000\000\005\000\000", __u6_addr16
= {
0, 0, 0, 0, 16, 0, 5, 0}, __u6_addr32 = {0, 0, 16,
5}}}, sin6_scope_id = 673678288}}, id = 32752, proto = 0 '\000', send_flags
= {
f = 0, blst_imask = 32752}}
tmp = 0x66307b "H\213E\270H\203\300\070\351O\003"
new_uri = 0x7ff028a93048 ""
end = 0x7ff028a93038 "\303l\256"
crt = 0x2000000 <Address 0x2000000 out of bounds>
cmd = 0x1773508
len = 6791248
user = 0
uri = {user = {s = 0xae6ecd "", len = 11430897}, passwd = {s =
0x7c95f8 "core: core/parser/msg_parser.c", len = 682507016}, host = {
s = 0xae6c43 "To: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
sip:usrloc@10.112.101.128:5060>;tag=8a3534eb92149bbe647f599d354f2511-090e\r\nCSeq:
10 KDMQ\r\nCall-ID: 409f4ac5788a8cc4-24333(a)10.112.101.128\r\nContent-Length:
39"..., len = 11430924}, port = {s = 0x0, len = 682507072}, params = {
s = 0x7ff028ae3c90 <incomplete sequence \350>, len = 0},
sip_params = {s = 0x7ffc89e35330 "\246", len = 6697083}, headers = {
s = 0x7c9170 "core", len = -1981590832}, port_no = 8, proto =
0, type = ERROR_URI_T, flags = 30415600, transport = {
s = 0xa8428197 <Address 0xa8428197 out of bounds>, len =
-1675583518}, ttl = {s = 0x3000000030 <Address 0x3000000030 out of bounds>,
len = 44108294}, user_param = {s = 0x400000017 <Address
0x400000017 out of bounds>, len = 8176032}, maddr = {s = 0x0, len =
-1981590448},
method = {s = 0x389c603c48 "", len = 11431629}, lr = {s =
0x7ffc89e35450 "\220T\343\211\374\177", len = 6869803}, r2 = {
s = 0x7cc190 "get_hdr_field", len = 6730416}, gr = {s =
0x7ffc89e35300 "", len = 673677328}, transport_val = {s = 0x389c422190
"",
len = 673677328}, ttl_val = {s = 0x7ffc89e35350
"\200S\343\211\374\177", len = -413652166}, user_param_val = {s = 0x40e7f9
"", len = 0},
maddr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>,
len = 24588552}, method_val = {
s = 0x67a450
"\030A\211\321A)\301D\211ȉ\302H\213\205(\377\377\377\211P \276\004", len =
8965880}, lr_val = {
s = 0x8cd2d0 <Address 0x8cd2d0 out of bounds>, len = 8},
r2_val = {s = 0xa6 <Address 0xa6 out of bounds>, len = 8136007}, gr_val = {
s = 0x7c95f8 "core: core/parser/msg_parser.c", len =
682176488}}
next_hop = {user = {s = 0x1e7374b9a <Address 0x1e7374b9a out of
bounds>, len = 674472240}, passwd = {
s = 0xffffffff <Address 0xffffffff out of bounds>, len =
4289104}, host = {s = 0x4e2cb1825 <Address 0x4e2cb1825 out of bounds>,
len = -1667705792}, port = {s = 0x0, len = 1073741824}, params
= {s = 0x2000000 <Address 0x2000000 out of bounds>, len = 4289104},
sip_params = {s = 0x7ffc89e35180 "\360Q\343\211\374\177", len =
6031673}, headers = {s = 0x7fefe2cb183f "", len = -1981591296},
port_no = 23013, proto = 58022, type = 32751, flags =
2313376240, transport = {s = 0x7fefe7374b9a "%x%c%x%c", len = -1981590944},
ttl = {
---Type <return> to continue, or q <return> to quit---
s = 0x1a <Address 0x1a out of bounds>, len = 4289104},
user_param = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len = -1670974014}, maddr
= {
s = 0x30fbad8001 <Address 0x30fbad8001 out of bounds>, len =
-492414491}, method = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len =
-492414491},
lr = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len = -492414478}, r2
= {s = 0x7fefe2a659ff "", len = -492414491}, gr = {s = 0x7fefe2a659ff
"",
len = 0}, transport_val = {s = 0x0, len = 6730416}, ttl_val =
{s = 0x7ffc89e351a0 "", len = 673677328}, user_param_val = {
s = 0x2000000 <Address 0x2000000 out of bounds>, len =
673677328}, maddr_val = {s = 0x7ffc89e351f0 "Cl\256", len = -413652166},
method_val = {s = 0x7ff028278010 "\001", len = 0}, lr_val = {s =
0x2000000 <Address 0x2000000 out of bounds>, len = 24589344}, r2_val = {
s = 0x67a138 "", len = 8965088}, gr_val = {s = 0x8cd2d0
<Address 0x8cd2d0 out of bounds>, len = 682507072}}
u = 0x66273d
port = 0
dst_host = 0x40000000
i = 28
flags = 0
avp = 0xae6cc7
st = {flags = 31129600, id = 0, name = {n = 0, s = {s = 0x0, len =
93}, re = 0x0}, avp = 0x7ffc89e35080}
sct = 0x7ffc89e35490
sjt = 0x6943d3
rve = 0x7ff0285a8808
mct = 0x6a283b
rv = 0x0
rv1 = 0x7c9170
c1 = {cache_type = 24, val_type = 48, c = {avp_val = {n =
-1981590720, s = {s = 0x7ffc89e35340 "\370\225|", len = -1981590912},
re = 0x7ffc89e35340}, pval = {rs = {s = 0x7ffc89e35340
"\370\225|", len = -1981590912}, ri = 0, flags = 0}},
i2s = '\000' <repeats 21 times>}
s = {s = 0x7fefe2cb183f "", len = -490006491}
srevp = {0x7fefe2cb1825, 0x7fefe2cb1832}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
= {
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}}
__FUNCTION__ = "do_action"
#7 0x00000000004404b8 in run_actions (h=0x7ffc89e35670, a=0x7ff0285a85d0,
msg=0x7ff028b453f8) at core/action.c:1552
t = 0x7ff0285a9668
ret = 1
mod = 0x417250
ms = 176570657
__FUNCTION__ = "run_actions"
#8 0x0000000000440c4a in run_top_route (a=0x7ff0285a85d0,
msg=0x7ff028b453f8, c=0x0) at core/action.c:1641
---Type <return> to continue, or q <return> to quit---
ctx = {rec_lev = 2, run_flags = 0, last_retcode = 1, jmp_env =
{{__jmpbuf = {140669451785720, -8516593617994257148, 50, 1073741824,
33554432,
4289104, -8516593618092823292, 8518532807277145348},
__mask_was_saved = 0, __saved_mask = {__val = {140668355198191, 4302791168,
48,
48, 18446744073709551615, 140722621863712,
140668355198919, 50, 0, 4611686022755909632, 140669451785208,
140722621863776, 5814046,
6608344928, 140669451785208, 0}}}}}
p = 0x7ffc89e35670
ret = 682907128
sfbk = 0
#9 0x000000000052e6b9 in receive_msg (
buf=0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080 SIP/2.0\r\nVia:
SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats 24
times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
sip:usrloc@10.112.101.128"..., len=781, rcv_info=0x7ffc89e35a70) at
core/receive.c:264
msg = 0x7ff028b453f8
ctx = {rec_lev = -1981589208, run_flags = 32764, last_retcode = 8,
jmp_env = {{__jmpbuf = {163, 4253893, 8100093, 140669451683568,
140722621864176, 4845170, 0, 21474836480},
__mask_was_saved = 673677840, __saved_mask = {__val = {140668768878592,
140722621864448,
6697083, 8094080, 5501958, 243141751568,
140669476122624, 140722621864896, 4294967295, 88212724672, 1,
140668357584160, 0, 33554432,
1024, 6608345520}}}}}
bctx = 0x7ffc89e359b0
ret = 0
stats_on = 0
tvb = {tv_sec = 140722621864208, tv_usec = 8961768}
tve = {tv_sec = 0, tv_usec = 243139749096}
tz = {tz_minuteswest = 50, tz_dsttime = 0}
diff = 0
inb = {
s = 0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080
SIP/2.0\r\nVia: SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3",
'0' <repeats 24 times>, ".0\r\nTo:
<sip:usrloc@10.112.103.64:5080>\r\nFrom:
<sip:usrloc@10.112.101.128"..., len = 781}
netinfo = {data = {s = 0x7ffc00000001 <Address 0x7ffc00000001 out
of bounds>, len = 0}, dst = 0x1, rcv = 0x389c422190}
keng = 0x0
__FUNCTION__ = "receive_msg"
#10 0x00000000006456a3 in udp_rcv_loop () at core/udp_server.c:515
len = 781
buf = "KDMQ sip:usrloc@10.112.103.64:5080 SIP/2.0\r\nVia:
SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats 24
times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
sip:usrloc@10.112.101.128:5060>;tag=8a3"...
tmp = 0x100000000 <Address 0x100000000 out of bounds>
from = 0x7ff028b2c728
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {140722462617610,
140722621864592}, addr32 = {2154131466, 32764, 2313378448, 32764}, addr16 =
{
28682, 32869, 32764, 0, 23184, 35299, 32764, 0}, addr =
"\npe\200\374\177\000\000\220Z\343\211\374\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {1080520714, 0}, addr32 = {1080520714,
0, 0, 0}, addr16 = {28682, 16487, 0, 0, 0, 0, 0, 0},
addr = "\npg@", '\000' <repeats 11 times>}},
src_port =
5060, dst_port = 5080, proto_reserved1 = 0, proto_reserved2 = 0, src_su =
{s = {
sa_family = 2, sa_data =
"\023\304\npe\200\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {
s_addr = 2154131466}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,
---Type <return> to continue, or q <return> to quit---
sin6_flowinfo = 2154131466, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7ff02830fff0, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#11 0x0000000000423c7d in main_loop () at main.c:1623
i = 0
pid = 0
si = 0x7ff02830fff0
si_desc = "udp receiver child=0 sock=10.112.103.64:5080\000\000\000\000
_\343\211\374\177\000\000\062\000\000\000\000\000\000\000\000\000\000@
\000\000\000\000\000\000\000\002\000\000\000\000PrA\000\000\000\000\000PSn\234\070\000\000\000\060\000\000\000\060\000\000\000\020]\343\211\374\177\000\000@
\\\343\211\374\177\000\000\000px\347\357\177\000"
nrprocs = 4
woneinit = 0
__FUNCTION__ = "main_loop"
#12 0x000000000042b0ce in main (argc=13, argv=0x7ffc89e36008) at
main.c:2646
cfg_stream = 0x1bb5010
c = -1
r = 0
tmp = 0x7ffc89e36ef7 ""
tmp_len = 32764
port = -1981587768
proto = 56
options = 0x7434f8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 4012862799
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x7ffc89e35f08
p = 0x100000000 <Address 0x100000000 out of bounds>
st = {st_dev = 64769, st_ino = 148592, st_nlink = 2, st_mode =
16877, st_uid = 498, st_gid = 498, __pad0 = 0, st_rdev = 0, st_size = 4096,
st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec =
1576522467, tv_nsec = 972000117}, st_mtim = {tv_sec = 1519927576,
tv_nsec = 668342155}, st_ctim = {tv_sec = 1584026268, tv_nsec
= 165523404}, __unused = {0, 0, 0}}
__FUNCTION__ = "main"
(gdb)
Em ter., 17 de mar. de 2020 às 09:52, Daniel-Constantin Mierla <
miconda(a)gmail.com> escreveu:
Hello,
does it happen only when doing restart of kamailio? Can you send the full
backtrace from gdb?
Also, 5.0.x is no longer a supported release for quite some time, try to
upgrade to 5.2.x or even better to the latest one 5.3.x and see if you can
reproduce it with one of the supported releases, maybe it is something that
was already fixed.
Cheers,
Daniel
On 17.03.20 10:32, Virgílio Cunha wrote:
Hello,
I have in production environment with 4 proxies using DMQ to sync
terminal registers and the DMQ configuration parameters is following:
loadmodule "dmq.so"
#modparam("dmq", "server_address",
"sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
modparam("dmq", "notification_address", "DMQ_HOSTS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
In the lab environment with the same configuration that in prod, I can
simulate this problem with 12000 terminal REGISTERS and, when doing several
restarts on a proxy, one of them is generated a coredump.
Analyzing the core, I can see that the problem is concurrency in the dmq
synchronization.
Job_queue_size (...) is called with a valid *queue *and, but inside this
function, the *queue *is null.
(gdb) up
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
254 return atomic_get(&queue->count);
(gdb) print queue
$1 = (job_queue_t *) 0x0
(gdb) up
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
peer=0x7fefebcc4df8) at worker.c:184
184 if(job_queue_size(workers[i].queue) == 0) {
(gdb) print i
$2 = 1
(gdb) print workers[i].queue
$3 = (job_queue_t *) 0x7fefebce49b0
If I configure DMQ with num_workers=1 I can't reproduce this problem.
I'm using the kamailio 5.0.8 release.
Is this problem known to you? What is the right way to solve it?
Best Regards,
Virgílio Cunha
_______________________________________________
Kamailio (SER) - Users Mailing
Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
26 Mar
26 Mar
1:20 p.m.
Hello Daniel,
I followed your recommendation and upgraded to kamailio-5.3.3 but the
problem remains with this version.
So I applied this patch:
https://github.com/kamailio/kamailio/commit/21bdbff049571ca22f1205a041c8b11…
and
I run the tests again.
After applying the patch, I was no longer able to replicate the problem :)
Thank you for your great support.
Best Regards,
Virgílio Cunha
Em qua., 18 de mar. de 2020 às 18:22, Virgílio Cunha <
virgilio18vsc(a)gmail.com> escreveu:
Hi,
Sorry for using this topic for this question, but where can I find
Kamailio's release policy?
I needed to know how long the kamailio 5.3.x branch will maintain its
support.
Best Regards,
Virgílio Cunha
Em ter., 17 de mar. de 2020 às 11:03, Daniel-Constantin Mierla <
miconda(a)gmail.com> escreveu:
> Hello,
>
> if it is a runtime issue, then it is better to upgrade to a supported
> release series and try to reproduce it.
>
> Looking at startup/shutdown code is easier than the rest used during
> runtime by a module, but it is not the case here. So we may waste a lot of
> time to dig into it and discover it was fixed or the code in an older
> release was refactored not matching any longer what is now in stable
> releases.
>
> Cheers,
> Daniel
> On 17.03.20 11:35, Virgílio Cunha wrote:
>
> Hi,
>
> The problem does not just occur when kamailio is restarted. The restart
> was the way I found to reproduce the problem, because when a proxy is
> restarted, there is more traffic from DMQ messages, so the problem is more
> likely to occur.
>
> The full backtrace is following:
>
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from
> /opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy...Reading symbols from
> /usr/lib/debug/usr/sbin/kamailio.debug...done.
> done.
> [New Thread 506]
> Reading symbols from /lib64/libdl.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libresolv.so.2
> Reading symbols from /lib64/libc.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /usr/lib64/kamailio/modules/db_postgres.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/db_postgres.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/db_postgres.so
> Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libpq.so.5
> Reading symbols from /usr/lib64/kamailio/libsrdb2.so.1...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/libsrdb2.so.1.0.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/libsrdb2.so.1
> Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/libsrdb1.so.1.0.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
> Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libssl.so.10
> Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libcrypto.so.10
> Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libgssapi_krb5.so.2
> Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libldap_r-2.4.so.2
> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> found)...done.
> [Thread debugging using libthread_db enabled]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libkrb5.so.3
> Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libcom_err.so.2
> Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libk5crypto.so.3
> Reading symbols from /lib64/libz.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libz.so.1
> Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libkrb5support.so.0
> Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libkeyutils.so.1
> Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/liblber-2.4.so.2
> Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libsasl2.so.2
> Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libssl3.so
> Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libsmime3.so
> Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libnss3.so
> Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libnssutil3.so
> Reading symbols from /lib64/libplds4.so...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libplds4.so
> Reading symbols from /lib64/libplc4.so...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libplc4.so
> Reading symbols from /lib64/libnspr4.so...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnspr4.so
> Reading symbols from /lib64/libselinux.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libselinux.so.1
> Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libcrypt.so.1
> Reading symbols from /lib64/librt.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/librt.so.1
> Reading symbols from /lib64/libfreebl3.so...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libfreebl3.so
> Reading symbols from /usr/lib64/kamailio/modules/ctl.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/ctl.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnss_files.so.2
> Reading symbols from /usr/lib64/kamailio/modules/kex.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/kex.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/kex.so
> Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/libsrutils.so.1.0.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
> Reading symbols from /lib64/libm.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /usr/lib64/kamailio/modules/tm.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/tm.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/tm.so
> Reading symbols from /usr/lib64/kamailio/modules/tmx.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/tmx.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/tmx.so
> Reading symbols from /usr/lib64/kamailio/modules/corex.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/corex.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/corex.so
> Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/cfg_rpc.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
> Reading symbols from /usr/lib64/kamailio/modules/sl.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/sl.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/sl.so
> Reading symbols from /usr/lib64/kamailio/modules/rr.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/rr.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/rr.so
> Reading symbols from /usr/lib64/kamailio/modules/maxfwd.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/maxfwd.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/maxfwd.so
> Reading symbols from /usr/lib64/kamailio/modules/siputils.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/siputils.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
> Reading symbols from /usr/lib64/kamailio/modules/sanity.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/sanity.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/sanity.so
> Reading symbols from /usr/lib64/kamailio/modules/textops.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/textops.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/textops.so
> Reading symbols from /usr/lib64/kamailio/modules/textopsx.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/textopsx.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/textopsx.so
> Reading symbols from /usr/lib64/kamailio/modules/dmq.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/dmq.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
> Reading symbols from /usr/lib64/kamailio/modules/htable.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/htable.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/htable.so
> Reading symbols from /usr/lib64/kamailio/modules/pv.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/pv.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/pv.so
> Reading symbols from /usr/lib64/kamailio/modules/xlog.so...Reading
> symbols from /usr/lib/debug/usr/lib64/kamailio/modules/xlog.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
> Reading symbols from /usr/lib64/kamailio/modules/jsonrpcs.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/jsonrpcs.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/jsonrpcs.so
> Reading symbols from /usr/lib64/kamailio/modules/uac.so...Reading symbols
> from /usr/lib/debug/usr/lib64/kamailio/modules/uac.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/uac.so
> Reading symbols from
> /usr/lib64/kamailio/modules/uac_redirect.so...Reading symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/uac_redirect.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/uac_redirect.so
> Reading symbols from /usr/lib64/kamailio/modules/db_text.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/db_text.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/db_text.so
> Reading symbols from /usr/lib64/kamailio/modules/dispatcher.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/dispatcher.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/dispatcher.so
> Reading symbols from /usr/lib64/kamailio/modules/sqlops.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/sqlops.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/sqlops.so
> Reading symbols from /usr/lib64/kamailio/modules/regex.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/regex.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/regex.so
> Reading symbols from /lib64/libpcre.so.0...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libpcre.so.0
> Reading symbols from /usr/lib64/kamailio/modules/permissions.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/permissions.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/permissions.so
> Reading symbols from /usr/lib64/kamailio/modules/dialplan.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/dialplan.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/dialplan.so
> Reading symbols from /usr/lib64/kamailio/modules/mtree.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/mtree.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/mtree.so
> Reading symbols from /usr/lib64/kamailio/modules/cfgutils.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/cfgutils.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/cfgutils.so
> Reading symbols from /usr/lib64/kamailio/modules/statsd.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/statsd.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/statsd.so
> Reading symbols from /usr/lib64/kamailio/modules/app_lua.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/app_lua.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/app_lua.so
> Reading symbols from /usr/lib64/liblua-5.1.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/liblua-5.1.so
> Reading symbols from /usr/lib64/kamailio/modules/rtimer.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/rtimer.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/rtimer.so
> Reading symbols from /usr/lib64/kamailio/modules/mqueue.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/mqueue.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/mqueue.so
> Reading symbols from /usr/lib64/kamailio/modules/auth.so...Reading
> symbols from /usr/lib/debug/usr/lib64/kamailio/modules/auth.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/auth.so
> Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/usrloc.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
> Reading symbols from /usr/lib64/kamailio/modules/registrar.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/registrar.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
> Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/dmq_usrloc.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
> Reading symbols from /usr/lib64/kamailio/modules/xhttp.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/xhttp.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/xhttp.so
> Reading symbols from /usr/lib64/kamailio/modules/exec.so...Reading
> symbols from /usr/lib/debug/usr/lib64/kamailio/modules/exec.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/exec.so
> Reading symbols from /usr/lib64/kamailio/modules/http_client.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/http_client.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/http_client.so
> Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libxml2.so.2
> Reading symbols from /usr/lib64/libcurl.so.4...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libcurl.so.4
> Reading symbols from /lib64/libidn.so.11...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libidn.so.11
> Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libldap-2.4.so.2
> Reading symbols from /usr/lib64/libssh2.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib64/libssh2.so.1
> Reading symbols from /usr/lib64/kamailio/modules/json.so...Reading
> symbols from /usr/lib/debug/usr/lib64/kamailio/modules/json.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/json.so
> Reading symbols from /lib64/libjson-c.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libjson-c.so.2
> Reading symbols from /usr/lib64/kamailio/modules/ipops.so...Reading
> symbols from
> /usr/lib/debug/usr/lib64/kamailio/modules/ipops.so.debug...done.
> done.
> Loaded symbols for /usr/lib64/kamailio/modules/ipops.so
> Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnss_dns.so.2
> Core was generated by `/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy -f
> /opt/ptin/kamailio-ksipadp/prox'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
> ../../core/atomic/atomic_common.h:66
> 66 return atomic_get_int(&(v->val));
> Missing separate debuginfos, use: debuginfo-install
> kamailio-ksipadp-2.8.0-1.ptin.el6.x86_64
> (gdb) bt full
> #0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
> ../../core/atomic/atomic_common.h:66
> No locals.
> #1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
> No locals.
> #2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
> peer=0x7fefebcc4df8) at worker.c:184
> i = 1
> found_available = 0
> new_job = {f = 0x7fefe2613797 <usrloc_dmq_handle_msg>, msg =
> 0x7fefebce67c8, orig_peer = 0x7fefebcc4df8, next = 0x0, prev = 0x0}
> worker = 0x7fefe7a1d890
> cloned_msg = 0x7fefebce67c8
> cloned_msg_len = 3888
> __FUNCTION__ = "add_dmq_job"
> #3 0x00007fefe58dfec1 in dmq_handle_message (msg=0x7ff028b453f8,
> str1=0x0, str2=0x0) at message.c:66
> peer = 0x7fefebcc4df8
> __FUNCTION__ = "dmq_handle_message"
> #4 0x0000000000433729 in do_action (h=0x7ffc89e35670, a=0x7ff0285a9340,
> msg=0x7ff028b453f8) at core/action.c:1054
> ret = -5
> v = 1
> dst = {send_sock = 0x7ff0285a8810, to = {s = {sa_family = 21376,
> sa_data = "\343\211\374\177\000\000\370S\264(\360\177\000"}, sin = {
> sin_family = 21376, sin_port = 35299, sin_addr = {s_addr =
> 32764}, sin_zero = "\370S\264(\360\177\000"}, sin6 = {sin6_family = 21376,
> sin6_port = 35299, sin6_flowinfo = 32764, sin6_addr =
> {__in6_u = {__u6_addr8 =
"\370S\264(\360\177\000\000pV\343\211\374\177\000",
> __u6_addr16 = {21496, 10420, 32752, 0, 22128, 35299,
> 32764, 0}, __u6_addr32 = {682906616, 32752, 2313377392, 32764}}},
> sin6_scope_id = 1}}, id = 32764, proto = 0 '\000',
> send_flags = {f = 0, blst_imask = 0}}
> tmp = 0x389c422190 ""
> new_uri = 0x7c00000000 <Address 0x7c00000000 out of bounds>
> end = 0x6e00000077 <Address 0x6e00000077 out of bounds>
> crt = 0x389c98e120 ""
> cmd = 0x7ff0283219b0
> len = 0
> user = 0
> uri = {user = {s = 0x7ffc89e34a10 "@K\343\211\374\177", len =
> 4459394}, passwd = {s = 0x7ffc89e34e80 "\356Q\343\211\374\177",
> len = 682906616}, host = {s = 0x7ff0285a9020 "\177\001", len
> = -1981589904}, port = {s = 0x0, len = 1}, params = {
> s = 0x7ff028b453f8 "\017", len = 295169284}, sip_params = {s
> = 0x32 <Address 0x32 out of bounds>, len = 1073741824}, headers = {
> s = 0x2000000 <Address 0x2000000 out of bounds>, len =
> 4289104}, port_no = 60676, proto = 5047, type = 2312043322, flags =
> 2168712452,
> transport = {s = 0x0, len = -1675583518}, ttl = {s = 0x89e34e90
> <Address 0x89e34e90 out of bounds>, len = 35715777}, user_param = {
> s = 0xffffffff0000003c <Address 0xffffffff0000003c out of
> bounds>, len = -1671419584}, maddr = {s = 0x0, len = -1981592704}, method =
> {
> s = 0x389c603c48 "", len = -1671391960}, lr = {s = 0x0, len =
> -1981592544}, r2 = {s = 0x1dde9a0 " 9\336\001", len = 0}, gr = {
> s = 0x4 <Address 0x4 out of bounds>, len = 2479},
> transport_val = {s = 0x389c422190 "", len = 1}, ttl_val = {
> s = 0x7ffc89e34b40 "pM\343\211\374\177", len = 5683491},
> user_param_val = {s = 0x40e7e8 "\370\377\377\377\377\377\377\377@ݘ
> \234\070",
> len = -1671362728}, maddr_val = {s = 0x404130 "", len = 0},
> method_val = {s = 0x1000003b4 <Address 0x1000003b4 out of bounds>, len = 0},
> lr_val = {s = 0x7fefe7374b9a "%x%c%x%c", len = -1673386776},
> r2_val = {s = 0x7ffc89e34bd0 "", len = -1981592584}, gr_val = {
> ---Type <return> to continue, or q <return> to quit---
> s = 0x389c422190 "", len = 0}}
> next_hop = {user = {s = 0x1 <Address 0x1 out of bounds>, len =
> 40}, passwd = {s = 0x0, len = -1407107317}, host = {
> s = 0x2000000 <Address 0x2000000 out of bounds>, len =
> 31398688}, port = {s = 0x1dde9a0 " 9\336\001", len = 31320480}, params = {
> s = 0x7ffc89e34c20 "", len = -1981593152}, sip_params = {s =
> 0x7ffc89e34910 "\200N\343\211\374\177", len = 682906616}, headers = {
> s = 0x32 <Address 0x32 out of bounds>, len = 1073741824},
> port_no = 0, proto = 512, type = ERROR_URI_T, flags = 4289104, transport = {
> s = 0x7ffc89e34900 "\020J\343\211\374\177", len = 4457737},
> ttl = {s = 0x7ffc89e348f0 "", len = -1981593520}, user_param = {
> s = 0x7ffc89e34858 "\r", len = -1981593264}, maddr = {s =
> 0x7ffc89e34870 "", len = -1981593456}, method = {s = 0x7ffc89e348b0
"\r",
> len = -1981593392}, lr = {s = 0x8cd2d0 <Address 0x8cd2d0 out
> of bounds>, len = 13}, r2 = {
> s = 0x10100000002 <Address 0x10100000002 out of bounds>, len
> = -1}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 682906616},
> ttl_val = {s = 0x7ff0285a9020 "\177\001", len = -1981593296},
> user_param_val = {s = 0x7ffc89e34fa0 "\001\200\255\373\374\177", len = 0},
> maddr_val = {s = 0x10000000d <Address 0x10000000d out of
> bounds>, len = -1981591296}, method_val = {
> s = 0x7fef00000000 <Address 0x7fef00000000 out of bounds>,
> len = -1671151657}, lr_val = {s = 0x7fefe7374ba2 "", len = 682906616},
> r2_val = {s = 0x32 <Address 0x32 out of bounds>, len =
> 1073741824}, gr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>,
> len = 4289104}}
> u = 0x389c4224e8
> port = 0
> dst_host = 0x89e34bd0
> i = 32764
> flags = 52
> avp = 0x189e35040
> st = {flags = 2313374752, id = 32764, name = {n = -1981593512, s
> = {s = 0x7ffc89e34858 "\r", len = -1981593152}, re = 0x7ffc89e34858},
> avp = 0x7ffc89e347a0}
> sct = 0x0
> sjt = 0x7ff028b453f8
> rve = 0x32
> mct = 0x40000000
> rv = 0x7ffc89e34c20
> rv1 = 0x2000000
> c1 = {cache_type = 2313374144, val_type = 32764, c = {avp_val =
> {n = 1, s = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, re = 0x1}, pval
> = {
> rs = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, ri =
> -1407115050, flags = 52}},
> i2s = "\000\000\000\000\000\000\000\000\274\224
> \254\064\000\000\000\240G\343\211\374\177"}
> s = {s = 0x1 <Address 0x1 out of bounds>, len = -1981593512}
> srevp = {0x7ffc89e34960, 0x34ac208a97}
> mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
> str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
> type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
> 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
> = {
> number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
> 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
> string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
> 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
> = {
> s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
> 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,
> len = 0},
> data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
> u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
> attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
> = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
> select = 0x0}}}
> ---Type <return> to continue, or q <return> to quit---
> __FUNCTION__ = "do_action"
> #5 0x00000000004404b8 in run_actions (h=0x7ffc89e35670,
> a=0x7ff0285a9340, msg=0x7ff028b453f8) at core/action.c:1552
> t = 0x7ff0285a9340
> ret = -1
> mod = 0x1dde9a0
> ms = 176570657
> __FUNCTION__ = "run_actions"
> #6 0x0000000000433692 in do_action (h=0x7ffc89e35670, a=0x7ff0285a9668,
> msg=0x7ff028b453f8) at core/action.c:1045
> ret = 1
> v = 1
> dst = {send_sock = 0x7ffc89e35380, to = {s = {sa_family = 61042,
> sa_data = "I", '\000' <repeats 12 times>}, sin = {sin_family
= 61042,
> sin_port = 73, sin_addr = {s_addr = 0}, sin_zero =
> "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 61042, sin6_port =
> 73,
> sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 =
> "\000\000\000\000\000\000\000\000\020\000\000\000\005\000\000",
__u6_addr16
> = {
> 0, 0, 0, 0, 16, 0, 5, 0}, __u6_addr32 = {0, 0, 16,
> 5}}}, sin6_scope_id = 673678288}}, id = 32752, proto = 0 '\000', send_flags
> = {
> f = 0, blst_imask = 32752}}
> tmp = 0x66307b "H\213E\270H\203\300\070\351O\003"
> new_uri = 0x7ff028a93048 ""
> end = 0x7ff028a93038 "\303l\256"
> crt = 0x2000000 <Address 0x2000000 out of bounds>
> cmd = 0x1773508
> len = 6791248
> user = 0
> uri = {user = {s = 0xae6ecd "", len = 11430897}, passwd = {s =
> 0x7c95f8 "core: core/parser/msg_parser.c", len = 682507016}, host = {
> s = 0xae6c43 "To: <sip:usrloc@10.112.103.64:5080>\r\nFrom:
<
>
sip:usrloc@10.112.101.128:5060>;tag=8a3534eb92149bbe647f599d354f2511-090e\r\nCSeq:
> 10 KDMQ\r\nCall-ID: 409f4ac5788a8cc4-24333(a)10.112.101.128\r\nContent-Length:
> 39"..., len = 11430924}, port = {s = 0x0, len = 682507072}, params = {
> s = 0x7ff028ae3c90 <incomplete sequence \350>, len = 0},
> sip_params = {s = 0x7ffc89e35330 "\246", len = 6697083}, headers = {
> s = 0x7c9170 "core", len = -1981590832}, port_no = 8, proto =
> 0, type = ERROR_URI_T, flags = 30415600, transport = {
> s = 0xa8428197 <Address 0xa8428197 out of bounds>, len =
> -1675583518}, ttl = {s = 0x3000000030 <Address 0x3000000030 out of bounds>,
> len = 44108294}, user_param = {s = 0x400000017 <Address
> 0x400000017 out of bounds>, len = 8176032}, maddr = {s = 0x0, len =
> -1981590448},
> method = {s = 0x389c603c48 "", len = 11431629}, lr = {s =
> 0x7ffc89e35450 "\220T\343\211\374\177", len = 6869803}, r2 = {
> s = 0x7cc190 "get_hdr_field", len = 6730416}, gr = {s =
> 0x7ffc89e35300 "", len = 673677328}, transport_val = {s = 0x389c422190
"",
> len = 673677328}, ttl_val = {s = 0x7ffc89e35350
> "\200S\343\211\374\177", len = -413652166}, user_param_val = {s = 0x40e7f9
> "", len = 0},
> maddr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>,
> len = 24588552}, method_val = {
> s = 0x67a450
> "\030A\211\321A)\301D\211ȉ\302H\213\205(\377\377\377\211P \276\004", len =
> 8965880}, lr_val = {
> s = 0x8cd2d0 <Address 0x8cd2d0 out of bounds>, len = 8},
> r2_val = {s = 0xa6 <Address 0xa6 out of bounds>, len = 8136007}, gr_val = {
> s = 0x7c95f8 "core: core/parser/msg_parser.c", len =
> 682176488}}
> next_hop = {user = {s = 0x1e7374b9a <Address 0x1e7374b9a out of
> bounds>, len = 674472240}, passwd = {
> s = 0xffffffff <Address 0xffffffff out of bounds>, len =
> 4289104}, host = {s = 0x4e2cb1825 <Address 0x4e2cb1825 out of bounds>,
> len = -1667705792}, port = {s = 0x0, len = 1073741824},
> params = {s = 0x2000000 <Address 0x2000000 out of bounds>, len = 4289104},
> sip_params = {s = 0x7ffc89e35180 "\360Q\343\211\374\177", len =
> 6031673}, headers = {s = 0x7fefe2cb183f "", len = -1981591296},
> port_no = 23013, proto = 58022, type = 32751, flags =
> 2313376240, transport = {s = 0x7fefe7374b9a "%x%c%x%c", len =
-1981590944},
> ttl = {
> ---Type <return> to continue, or q <return> to quit---
> s = 0x1a <Address 0x1a out of bounds>, len = 4289104},
> user_param = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len = -1670974014},
maddr
> = {
> s = 0x30fbad8001 <Address 0x30fbad8001 out of bounds>, len =
> -492414491}, method = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len =
> -492414491},
> lr = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len = -492414478}, r2
> = {s = 0x7fefe2a659ff "", len = -492414491}, gr = {s = 0x7fefe2a659ff
"",
> len = 0}, transport_val = {s = 0x0, len = 6730416}, ttl_val =
> {s = 0x7ffc89e351a0 "", len = 673677328}, user_param_val = {
> s = 0x2000000 <Address 0x2000000 out of bounds>, len =
> 673677328}, maddr_val = {s = 0x7ffc89e351f0 "Cl\256", len = -413652166},
> method_val = {s = 0x7ff028278010 "\001", len = 0}, lr_val = {s
> = 0x2000000 <Address 0x2000000 out of bounds>, len = 24589344}, r2_val = {
> s = 0x67a138 "", len = 8965088}, gr_val = {s = 0x8cd2d0
> <Address 0x8cd2d0 out of bounds>, len = 682507072}}
> u = 0x66273d
> port = 0
> dst_host = 0x40000000
> i = 28
> flags = 0
> avp = 0xae6cc7
> st = {flags = 31129600, id = 0, name = {n = 0, s = {s = 0x0, len
> = 93}, re = 0x0}, avp = 0x7ffc89e35080}
> sct = 0x7ffc89e35490
> sjt = 0x6943d3
> rve = 0x7ff0285a8808
> mct = 0x6a283b
> rv = 0x0
> rv1 = 0x7c9170
> c1 = {cache_type = 24, val_type = 48, c = {avp_val = {n =
> -1981590720, s = {s = 0x7ffc89e35340 "\370\225|", len = -1981590912},
> re = 0x7ffc89e35340}, pval = {rs = {s = 0x7ffc89e35340
> "\370\225|", len = -1981590912}, ri = 0, flags = 0}},
> i2s = '\000' <repeats 21 times>}
> s = {s = 0x7fefe2cb183f "", len = -490006491}
> srevp = {0x7fefe2cb1825, 0x7fefe2cb1832}
> mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
> str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
> type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
> 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
> = {
> number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
> 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,
> string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
> 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
> = {
> s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
> 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,
> len = 0},
> data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
> u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,
> attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
> = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0,
> select = 0x0}}}
> __FUNCTION__ = "do_action"
> #7 0x00000000004404b8 in run_actions (h=0x7ffc89e35670,
> a=0x7ff0285a85d0, msg=0x7ff028b453f8) at core/action.c:1552
> t = 0x7ff0285a9668
> ret = 1
> mod = 0x417250
> ms = 176570657
> __FUNCTION__ = "run_actions"
> #8 0x0000000000440c4a in run_top_route (a=0x7ff0285a85d0,
> msg=0x7ff028b453f8, c=0x0) at core/action.c:1641
> ---Type <return> to continue, or q <return> to quit---
> ctx = {rec_lev = 2, run_flags = 0, last_retcode = 1, jmp_env =
> {{__jmpbuf = {140669451785720, -8516593617994257148, 50, 1073741824,
> 33554432,
> 4289104, -8516593618092823292, 8518532807277145348},
> __mask_was_saved = 0, __saved_mask = {__val = {140668355198191, 4302791168,
> 48,
> 48, 18446744073709551615, 140722621863712,
> 140668355198919, 50, 0, 4611686022755909632, 140669451785208,
> 140722621863776, 5814046,
> 6608344928, 140669451785208, 0}}}}}
> p = 0x7ffc89e35670
> ret = 682907128
> sfbk = 0
> #9 0x000000000052e6b9 in receive_msg (
> buf=0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080 SIP/2.0\r\nVia:
> SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats
24
> times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
> sip:usrloc@10.112.101.128"..., len=781, rcv_info=0x7ffc89e35a70) at
> core/receive.c:264
> msg = 0x7ff028b453f8
> ctx = {rec_lev = -1981589208, run_flags = 32764, last_retcode =
> 8, jmp_env = {{__jmpbuf = {163, 4253893, 8100093, 140669451683568,
> 140722621864176, 4845170, 0, 21474836480},
> __mask_was_saved = 673677840, __saved_mask = {__val = {140668768878592,
> 140722621864448,
> 6697083, 8094080, 5501958, 243141751568,
> 140669476122624, 140722621864896, 4294967295, 88212724672, 1,
> 140668357584160, 0, 33554432,
> 1024, 6608345520}}}}}
> bctx = 0x7ffc89e359b0
> ret = 0
> stats_on = 0
> tvb = {tv_sec = 140722621864208, tv_usec = 8961768}
> tve = {tv_sec = 0, tv_usec = 243139749096}
> tz = {tz_minuteswest = 50, tz_dsttime = 0}
> diff = 0
> inb = {
> s = 0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080
> SIP/2.0\r\nVia: SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3",
> '0' <repeats 24 times>, ".0\r\nTo:
<sip:usrloc@10.112.103.64:5080>\r\nFrom:
> <sip:usrloc@10.112.101.128"..., len = 781}
> netinfo = {data = {s = 0x7ffc00000001 <Address 0x7ffc00000001 out
> of bounds>, len = 0}, dst = 0x1, rcv = 0x389c422190}
> keng = 0x0
> __FUNCTION__ = "receive_msg"
> #10 0x00000000006456a3 in udp_rcv_loop () at core/udp_server.c:515
> len = 781
> buf = "KDMQ sip:usrloc@10.112.103.64:5080 SIP/2.0\r\nVia:
> SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0' <repeats
24
> times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080>\r\nFrom: <
> sip:usrloc@10.112.101.128:5060>;tag=8a3"...
> tmp = 0x100000000 <Address 0x100000000 out of bounds>
> from = 0x7ff028b2c728
> fromlen = 16
> ri = {src_ip = {af = 2, len = 4, u = {addrl = {140722462617610,
> 140722621864592}, addr32 = {2154131466, 32764, 2313378448, 32764}, addr16 =
> {
> 28682, 32869, 32764, 0, 23184, 35299, 32764, 0}, addr =
> "\npe\200\374\177\000\000\220Z\343\211\374\177\000"}}, dst_ip = {af = 2,
> len = 4, u = {addrl = {1080520714, 0}, addr32 = {1080520714,
> 0, 0, 0}, addr16 = {28682, 16487, 0, 0, 0, 0, 0, 0},
> addr = "\npg@", '\000' <repeats 11 times>}},
src_port =
> 5060, dst_port = 5080, proto_reserved1 = 0, proto_reserved2 = 0, src_su =
> {s = {
> sa_family = 2, sa_data =
> "\023\304\npe\200\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
> sin_port = 50195, sin_addr = {
> s_addr = 2154131466}, sin_zero =
> "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
> ---Type <return> to continue, or q <return> to quit---
> sin6_flowinfo = 2154131466, sin6_addr = {__in6_u =
> {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0,
0,
> 0, 0},
> __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
> bind_address = 0x7ff02830fff0, proto = 1 '\001'}
> __FUNCTION__ = "udp_rcv_loop"
> #11 0x0000000000423c7d in main_loop () at main.c:1623
> i = 0
> pid = 0
> si = 0x7ff02830fff0
> si_desc = "udp receiver child=0 sock=10.112.103.64:5080\000\000\000\000
> _\343\211\374\177\000\000\062\000\000\000\000\000\000\000\000\000\000@
>
\000\000\000\000\000\000\000\002\000\000\000\000PrA\000\000\000\000\000PSn\234\070\000\000\000\060\000\000\000\060\000\000\000\020]\343\211\374\177\000\000@
> \\\343\211\374\177\000\000\000px\347\357\177\000"
> nrprocs = 4
> woneinit = 0
> __FUNCTION__ = "main_loop"
> #12 0x000000000042b0ce in main (argc=13, argv=0x7ffc89e36008) at
> main.c:2646
> cfg_stream = 0x1bb5010
> c = -1
> r = 0
> tmp = 0x7ffc89e36ef7 ""
> tmp_len = 32764
> port = -1981587768
> proto = 56
> options = 0x7434f8
> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
> ret = -1
> seed = 4012862799
> rfd = 4
> debug_save = 0
> debug_flag = 0
> dont_fork_cnt = 0
> n_lst = 0x7ffc89e35f08
> p = 0x100000000 <Address 0x100000000 out of bounds>
> st = {st_dev = 64769, st_ino = 148592, st_nlink = 2, st_mode =
> 16877, st_uid = 498, st_gid = 498, __pad0 = 0, st_rdev = 0, st_size = 4096,
> st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec =
> 1576522467, tv_nsec = 972000117}, st_mtim = {tv_sec = 1519927576,
> tv_nsec = 668342155}, st_ctim = {tv_sec = 1584026268, tv_nsec
> = 165523404}, __unused = {0, 0, 0}}
> __FUNCTION__ = "main"
> (gdb)
>
> Em ter., 17 de mar. de 2020 às 09:52, Daniel-Constantin Mierla <
> miconda(a)gmail.com> escreveu:
>
>> Hello,
>>
>> does it happen only when doing restart of kamailio? Can you send the
>> full backtrace from gdb?
>>
>> Also, 5.0.x is no longer a supported release for quite some time, try to
>> upgrade to 5.2.x or even better to the latest one 5.3.x and see if you can
>> reproduce it with one of the supported releases, maybe it is something that
>> was already fixed.
>>
>> Cheers,
>> Daniel
>> On 17.03.20 10:32, Virgílio Cunha wrote:
>>
>> Hello,
>>
>> I have in production environment with 4 proxies using DMQ to sync
>> terminal registers and the DMQ configuration parameters is following:
>> loadmodule "dmq.so"
>> #modparam("dmq", "server_address",
"sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
>> modparam("dmq", "notification_address",
"DMQ_HOSTS")
>> modparam("dmq", "multi_notify", 1)
>> modparam("dmq", "num_workers", 4)
>>
>>
>> In the lab environment with the same configuration that in prod, I can
>> simulate this problem with 12000 terminal REGISTERS and, when doing several
>> restarts on a proxy, one of them is generated a coredump.
>> Analyzing the core, I can see that the problem is concurrency in the dmq
>> synchronization.
>>
>> Job_queue_size (...) is called with a valid *queue *and, but inside
>> this function, the *queue *is null.
>>
>>
>> (gdb) up
>> #1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at worker.c:254
>> 254 return atomic_get(&queue->count);
>> (gdb) print queue
>> $1 = (job_queue_t *) 0x0
>> (gdb) up
>> #2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
>> peer=0x7fefebcc4df8) at worker.c:184
>> 184 if(job_queue_size(workers[i].queue) == 0) {
>> (gdb) print i
>> $2 = 1
>> (gdb) print workers[i].queue
>> $3 = (job_queue_t *) 0x7fefebce49b0
>>
>> If I configure DMQ with num_workers=1 I can't reproduce this problem.
>> I'm using the kamailio 5.0.8 release.
>>
>> Is this problem known to you? What is the right way to solve it?
>>
>> Best Regards,
>> Virgílio Cunha
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing
Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
>>
>> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda --
www.linkedin.com/in/miconda
>
>
27 Mar
27 Mar
8:39 a.m.
Thanks for testing and the feedback!
Check also if this one can affect you:
* https://github.com/kamailio/kamailio/issues/2224#issuecomment-602730307
I just backported all of them to 5.3, so if you want to test, you can
use directly the latest branch 5.3.
Cheers,
Daniel
On 26.03.20 13:20, Virgílio Cunha wrote:
Hello Daniel,
I followed your recommendation and upgraded to kamailio-5.3.3 but the
problem remains with this version.
So I applied this
patch: https://github.com/kamailio/kamailio/commit/21bdbff049571ca22f1205a041c8b11… and
I run the tests again.
After applying the patch, I was no longer able to replicate the problem :)
Thank you for your great support.
Best Regards,
Virgílio Cunha
Em qua., 18 de mar. de 2020 às 18:22, Virgílio Cunha
<virgilio18vsc(a)gmail.com <mailto:virgilio18vsc@gmail.com>> escreveu:
Hi,
Sorry for using this topic for this question, but where can I find
Kamailio's release policy?
I needed to know how long the kamailio 5.3.x branch will maintain
its support.
Best Regards,
Virgílio Cunha
Em ter., 17 de mar. de 2020 às 11:03, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> escreveu:
Hello,
if it is a runtime issue, then it is better to upgrade to a
supported release series and try to reproduce it.
Looking at startup/shutdown code is easier than the rest used
during runtime by a module, but it is not the case here. So we
may waste a lot of time to dig into it and discover it was
fixed or the code in an older release was refactored not
matching any longer what is now in stable releases.
Cheers,
Daniel
On 17.03.20 11:35, Virgílio Cunha wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Hi,
The problem does not just occur when kamailio is restarted.
The restart was the way I found to reproduce the problem,
because when a proxy is restarted, there is more traffic from
DMQ messages, so the problem is more likely to occur.
The full backtrace is following:
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and
redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type
"show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy...Reading
symbols from /usr/lib/debug/usr/sbin/kamailio.debug...done.
done.
[New Thread 506]
Reading symbols from /lib64/libdl.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no
debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from
/usr/lib64/kamailio/modules/db_postgres.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/db_postgres.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/db_postgres.so
Reading symbols from /usr/lib64/libpq.so.5...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from
/usr/lib64/kamailio/libsrdb2.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/libsrdb2.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrdb2.so.1
Reading symbols from
/usr/lib64/kamailio/libsrdb1.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/libsrdb1.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
Reading symbols from /usr/lib64/libssl.so.10...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...(no
debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libldap_r-2.4.so.2...(no
debugging symbols found)...done.
Loaded symbols for /lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging
symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libkrb5.so.3...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no
debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libselinux.so.1...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/librt.so.1...(no debugging
symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from
/usr/lib64/kamailio/modules/ctl.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/ctl.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from
/usr/lib64/kamailio/modules/kex.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/kex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/kex.so
Reading symbols from
/usr/lib64/kamailio/libsrutils.so.1...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/libsrutils.so.1.0.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from
/usr/lib64/kamailio/modules/tm.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/tm.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/tm.so
Reading symbols from
/usr/lib64/kamailio/modules/tmx.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/tmx.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/tmx.so
Reading symbols from
/usr/lib64/kamailio/modules/corex.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/corex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/corex.so
Reading symbols from
/usr/lib64/kamailio/modules/cfg_rpc.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/cfg_rpc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
Reading symbols from
/usr/lib64/kamailio/modules/sl.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sl.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sl.so
Reading symbols from
/usr/lib64/kamailio/modules/rr.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/rr.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/rr.so
Reading symbols from
/usr/lib64/kamailio/modules/maxfwd.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/maxfwd.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/maxfwd.so
Reading symbols from
/usr/lib64/kamailio/modules/siputils.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/siputils.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
Reading symbols from
/usr/lib64/kamailio/modules/sanity.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sanity.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sanity.so
Reading symbols from
/usr/lib64/kamailio/modules/textops.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/textops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/textops.so
Reading symbols from
/usr/lib64/kamailio/modules/textopsx.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/textopsx.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/textopsx.so
Reading symbols from
/usr/lib64/kamailio/modules/dmq.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/dmq.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
Reading symbols from
/usr/lib64/kamailio/modules/htable.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/htable.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/htable.so
Reading symbols from
/usr/lib64/kamailio/modules/pv.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/pv.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/pv.so
Reading symbols from
/usr/lib64/kamailio/modules/xlog.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/xlog.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
Reading symbols from
/usr/lib64/kamailio/modules/jsonrpcs.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/jsonrpcs.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/jsonrpcs.so
Reading symbols from
/usr/lib64/kamailio/modules/uac.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/uac.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/uac.so
Reading symbols from
/usr/lib64/kamailio/modules/uac_redirect.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/uac_redirect.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/uac_redirect.so
Reading symbols from
/usr/lib64/kamailio/modules/db_text.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/db_text.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/db_text.so
Reading symbols from
/usr/lib64/kamailio/modules/dispatcher.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/dispatcher.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dispatcher.so
Reading symbols from
/usr/lib64/kamailio/modules/sqlops.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/sqlops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/sqlops.so
Reading symbols from
/usr/lib64/kamailio/modules/regex.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/regex.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/regex.so
Reading symbols from /lib64/libpcre.so.0...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from
/usr/lib64/kamailio/modules/permissions.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/permissions.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/permissions.so
Reading symbols from
/usr/lib64/kamailio/modules/dialplan.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/dialplan.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dialplan.so
Reading symbols from
/usr/lib64/kamailio/modules/mtree.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/mtree.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/mtree.so
Reading symbols from
/usr/lib64/kamailio/modules/cfgutils.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/cfgutils.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/cfgutils.so
Reading symbols from
/usr/lib64/kamailio/modules/statsd.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/statsd.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/statsd.so
Reading symbols from
/usr/lib64/kamailio/modules/app_lua.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/app_lua.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/app_lua.so
Reading symbols from /usr/lib64/liblua-5.1.so...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/liblua-5.1.so
<http://liblua-5.1.so>
Reading symbols from
/usr/lib64/kamailio/modules/rtimer.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/rtimer.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/rtimer.so
Reading symbols from
/usr/lib64/kamailio/modules/mqueue.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/mqueue.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/mqueue.so
Reading symbols from
/usr/lib64/kamailio/modules/auth.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/auth.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/auth.so
Reading symbols from
/usr/lib64/kamailio/modules/usrloc.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/usrloc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
Reading symbols from
/usr/lib64/kamailio/modules/registrar.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/registrar.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
Reading symbols from
/usr/lib64/kamailio/modules/dmq_usrloc.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/dmq_usrloc.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
Reading symbols from
/usr/lib64/kamailio/modules/xhttp.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/xhttp.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/xhttp.so
Reading symbols from
/usr/lib64/kamailio/modules/exec.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/exec.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/exec.so
Reading symbols from
/usr/lib64/kamailio/modules/http_client.so...Reading symbols
from
/usr/lib/debug/usr/lib64/kamailio/modules/http_client.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/http_client.so
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /usr/lib64/libcurl.so.4...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libcurl.so.4
Reading symbols from /lib64/libidn.so.11...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libidn.so.11
Reading symbols from /lib64/libldap-2.4.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /usr/lib64/libssh2.so.1...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib64/libssh2.so.1
Reading symbols from
/usr/lib64/kamailio/modules/json.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/json.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/json.so
Reading symbols from /lib64/libjson-c.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libjson-c.so.2
Reading symbols from
/usr/lib64/kamailio/modules/ipops.so...Reading symbols from
/usr/lib/debug/usr/lib64/kamailio/modules/ipops.so.debug...done.
done.
Loaded symbols for /usr/lib64/kamailio/modules/ipops.so
Reading symbols from /lib64/libnss_dns.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Core was generated by
`/opt/ptin/kamailio-ksipadp/bin/ksipadp-proxy -f
/opt/ptin/kamailio-ksipadp/prox'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
../../core/atomic/atomic_common.h:66
66 return atomic_get_int(&(v->val));
Missing separate debuginfos, use: debuginfo-install
kamailio-ksipadp-2.8.0-1.ptin.el6.x86_64
(gdb) bt full
#0 0x00007fefe58ea7c4 in atomic_get (v=0x0) at
../../core/atomic/atomic_common.h:66
No locals.
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at
worker.c:254
No locals.
#2 0x00007fefe58ed920 in add_dmq_job (msg=0x7ff028b453f8,
peer=0x7fefebcc4df8) at worker.c:184
i = 1
found_available = 0
new_job = {f = 0x7fefe2613797
<usrloc_dmq_handle_msg>, msg = 0x7fefebce67c8, orig_peer =
0x7fefebcc4df8, next = 0x0, prev = 0x0}
worker = 0x7fefe7a1d890
cloned_msg = 0x7fefebce67c8
cloned_msg_len = 3888
__FUNCTION__ = "add_dmq_job"
#3 0x00007fefe58dfec1 in dmq_handle_message
(msg=0x7ff028b453f8, str1=0x0, str2=0x0) at message.c:66
peer = 0x7fefebcc4df8
__FUNCTION__ = "dmq_handle_message"
#4 0x0000000000433729 in do_action (h=0x7ffc89e35670,
a=0x7ff0285a9340, msg=0x7ff028b453f8) at core/action.c:1054
ret = -5
v = 1
dst = {send_sock = 0x7ff0285a8810, to = {s =
{sa_family = 21376, sa_data =
"\343\211\374\177\000\000\370S\264(\360\177\000"}, sin = {
sin_family = 21376, sin_port = 35299, sin_addr
= {s_addr = 32764}, sin_zero = "\370S\264(\360\177\000"},
sin6 = {sin6_family = 21376,
sin6_port = 35299, sin6_flowinfo = 32764,
sin6_addr = {__in6_u = {__u6_addr8 =
"\370S\264(\360\177\000\000pV\343\211\374\177\000",
__u6_addr16 = {21496, 10420, 32752, 0,
22128, 35299, 32764, 0}, __u6_addr32 = {682906616, 32752,
2313377392, 32764}}},
sin6_scope_id = 1}}, id = 32764, proto = 0
'\000', send_flags = {f = 0, blst_imask = 0}}
tmp = 0x389c422190 ""
new_uri = 0x7c00000000 <Address 0x7c00000000 out of
bounds>
end = 0x6e00000077 <Address 0x6e00000077 out of bounds>
crt = 0x389c98e120 ""
cmd = 0x7ff0283219b0
len = 0
user = 0
uri = {user = {s = 0x7ffc89e34a10
"@K\343\211\374\177", len = 4459394}, passwd = {s =
0x7ffc89e34e80 "\356Q\343\211\374\177",
len = 682906616}, host = {s = 0x7ff0285a9020
"\177\001", len = -1981589904}, port = {s = 0x0, len = 1},
params = {
s = 0x7ff028b453f8 "\017", len = 295169284},
sip_params = {s = 0x32 <Address 0x32 out of bounds>, len =
1073741824}, headers = {
s = 0x2000000 <Address 0x2000000 out of bounds>,
len = 4289104}, port_no = 60676, proto = 5047, type =
2312043322, flags = 2168712452,
transport = {s = 0x0, len = -1675583518}, ttl = {s
= 0x89e34e90 <Address 0x89e34e90 out of bounds>, len =
35715777}, user_param = {
s = 0xffffffff0000003c <Address
0xffffffff0000003c out of bounds>, len = -1671419584}, maddr
= {s = 0x0, len = -1981592704}, method = {
s = 0x389c603c48 "", len = -1671391960}, lr = {s
= 0x0, len = -1981592544}, r2 = {s = 0x1dde9a0 " 9\336\001",
len = 0}, gr = {
s = 0x4 <Address 0x4 out of bounds>, len = 2479},
transport_val = {s = 0x389c422190 "", len = 1}, ttl_val = {
s = 0x7ffc89e34b40 "pM\343\211\374\177", len =
5683491}, user_param_val = {s = 0x40e7e8
"\370\377\377\377\377\377\377\377@ݘ\234\070",
len = -1671362728}, maddr_val = {s = 0x404130 "",
len = 0}, method_val = {s = 0x1000003b4 <Address 0x1000003b4
out of bounds>, len = 0},
lr_val = {s = 0x7fefe7374b9a "%x%c%x%c", len =
-1673386776}, r2_val = {s = 0x7ffc89e34bd0 "", len =
-1981592584}, gr_val = {
---Type <return> to continue, or q <return> to quit---
s = 0x389c422190 "", len = 0}}
next_hop = {user = {s = 0x1 <Address 0x1 out of
bounds>, len = 40}, passwd = {s = 0x0, len = -1407107317},
host = {
s = 0x2000000 <Address 0x2000000 out of bounds>,
len = 31398688}, port = {s = 0x1dde9a0 " 9\336\001", len =
31320480}, params = {
s = 0x7ffc89e34c20 "", len = -1981593152},
sip_params = {s = 0x7ffc89e34910 "\200N\343\211\374\177", len
= 682906616}, headers = {
s = 0x32 <Address 0x32 out of bounds>, len =
1073741824}, port_no = 0, proto = 512, type = ERROR_URI_T,
flags = 4289104, transport = {
s = 0x7ffc89e34900 "\020J\343\211\374\177", len =
4457737}, ttl = {s = 0x7ffc89e348f0 "", len = -1981593520},
user_param = {
s = 0x7ffc89e34858 "\r", len = -1981593264},
maddr = {s = 0x7ffc89e34870 "", len = -1981593456}, method =
{s = 0x7ffc89e348b0 "\r",
len = -1981593392}, lr = {s = 0x8cd2d0 <Address
0x8cd2d0 out of bounds>, len = 13}, r2 = {
s = 0x10100000002 <Address 0x10100000002 out of
bounds>, len = -1}, gr = {s = 0x0, len = 0}, transport_val =
{s = 0x0, len = 682906616},
ttl_val = {s = 0x7ff0285a9020 "\177\001", len =
-1981593296}, user_param_val = {s = 0x7ffc89e34fa0
"\001\200\255\373\374\177", len = 0},
maddr_val = {s = 0x10000000d <Address 0x10000000d
out of bounds>, len = -1981591296}, method_val = {
s = 0x7fef00000000 <Address 0x7fef00000000 out of
bounds>, len = -1671151657}, lr_val = {s = 0x7fefe7374ba2 "",
len = 682906616},
r2_val = {s = 0x32 <Address 0x32 out of bounds>,
len = 1073741824}, gr_val = {s = 0x2000000 <Address 0x2000000
out of bounds>,
len = 4289104}}
u = 0x389c4224e8
port = 0
dst_host = 0x89e34bd0
i = 32764
flags = 52
avp = 0x189e35040
st = {flags = 2313374752, id = 32764, name = {n =
-1981593512, s = {s = 0x7ffc89e34858 "\r", len =
-1981593152}, re = 0x7ffc89e34858},
avp = 0x7ffc89e347a0}
sct = 0x0
sjt = 0x7ff028b453f8
rve = 0x32
mct = 0x40000000
rv = 0x7ffc89e34c20
rv1 = 0x2000000
c1 = {cache_type = 2313374144, val_type = 32764, c =
{avp_val = {n = 1, s = {s = 0x1 <Address 0x1 out of bounds>,
len = 1}, re = 0x1}, pval = {
rs = {s = 0x1 <Address 0x1 out of bounds>, len
= 1}, ri = -1407115050, flags = 52}},
i2s = "\000\000\000\000\000\000\000\000\274\224
\254\064\000\000\000\240G\343\211\374\177"}
s = {s = 0x1 <Address 0x1 out of bounds>, len =
-1981593512}
srevp = {0x7ffc89e34960, 0x34ac208a97}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len =
0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u =
{number = 0, string = 0x0, str = {
s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
= 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data
= 0x0, attr = 0x0,
select = 0x0}}}
---Type <return> to continue, or q <return> to quit---
__FUNCTION__ = "do_action"
#5 0x00000000004404b8 in run_actions (h=0x7ffc89e35670,
a=0x7ff0285a9340, msg=0x7ff028b453f8) at core/action.c:1552
t = 0x7ff0285a9340
ret = -1
mod = 0x1dde9a0
ms = 176570657
__FUNCTION__ = "run_actions"
#6 0x0000000000433692 in do_action (h=0x7ffc89e35670,
a=0x7ff0285a9668, msg=0x7ff028b453f8) at core/action.c:1045
ret = 1
v = 1
dst = {send_sock = 0x7ffc89e35380, to = {s =
{sa_family = 61042, sa_data = "I", '\000' <repeats 12
times>}, sin = {sin_family = 61042,
sin_port = 73, sin_addr = {s_addr = 0},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 61042, sin6_port = 73,
sin6_flowinfo = 0, sin6_addr = {__in6_u =
{__u6_addr8 =
"\000\000\000\000\000\000\000\000\020\000\000\000\005\000\000",
__u6_addr16 = {
0, 0, 0, 0, 16, 0, 5, 0}, __u6_addr32 =
{0, 0, 16, 5}}}, sin6_scope_id = 673678288}}, id = 32752,
proto = 0 '\000', send_flags = {
f = 0, blst_imask = 32752}}
tmp = 0x66307b "H\213E\270H\203\300\070\351O\003"
new_uri = 0x7ff028a93048 ""
end = 0x7ff028a93038 "\303l\256"
crt = 0x2000000 <Address 0x2000000 out of bounds>
cmd = 0x1773508
len = 6791248
user = 0
uri = {user = {s = 0xae6ecd "", len = 11430897},
passwd = {s = 0x7c95f8 "core: core/parser/msg_parser.c", len
= 682507016}, host = {
s = 0xae6c43 "To: <sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
<sip:usrloc@10.112.101.128:5060
<http://sip:usrloc@10.112.101.128:5060>>;tag=8a3534eb92149bbe647f599d354f2511-090e\r\nCSeq:
10 KDMQ\r\nCall-ID: 409f4ac5788a8cc4-24333(a)10.112.101.128
<mailto:409f4ac5788a8cc4-24333@10.112.101.128>\r\nContent-Length:
39"..., len = 11430924}, port = {s = 0x0, len = 682507072},
params = {
s = 0x7ff028ae3c90 <incomplete sequence \350>,
len = 0}, sip_params = {s = 0x7ffc89e35330 "\246", len =
6697083}, headers = {
s = 0x7c9170 "core", len = -1981590832}, port_no
= 8, proto = 0, type = ERROR_URI_T, flags = 30415600,
transport = {
s = 0xa8428197 <Address 0xa8428197 out of
bounds>, len = -1675583518}, ttl = {s = 0x3000000030 <Address
0x3000000030 out of bounds>,
len = 44108294}, user_param = {s = 0x400000017
<Address 0x400000017 out of bounds>, len = 8176032}, maddr =
{s = 0x0, len = -1981590448},
method = {s = 0x389c603c48 "", len = 11431629}, lr
= {s = 0x7ffc89e35450 "\220T\343\211\374\177", len =
6869803}, r2 = {
s = 0x7cc190 "get_hdr_field", len = 6730416}, gr
= {s = 0x7ffc89e35300 "", len = 673677328}, transport_val =
{s = 0x389c422190 "",
len = 673677328}, ttl_val = {s = 0x7ffc89e35350
"\200S\343\211\374\177", len = -413652166}, user_param_val =
{s = 0x40e7f9 "", len = 0},
maddr_val = {s = 0x2000000 <Address 0x2000000 out
of bounds>, len = 24588552}, method_val = {
s = 0x67a450
"\030A\211\321A)\301D\211ȉ\302H\213\205(\377\377\377\211P
\276\004", len = 8965880}, lr_val = {
s = 0x8cd2d0 <Address 0x8cd2d0 out of bounds>,
len = 8}, r2_val = {s = 0xa6 <Address 0xa6 out of bounds>,
len = 8136007}, gr_val = {
s = 0x7c95f8 "core: core/parser/msg_parser.c",
len = 682176488}}
next_hop = {user = {s = 0x1e7374b9a <Address
0x1e7374b9a out of bounds>, len = 674472240}, passwd = {
s = 0xffffffff <Address 0xffffffff out of
bounds>, len = 4289104}, host = {s = 0x4e2cb1825 <Address
0x4e2cb1825 out of bounds>,
len = -1667705792}, port = {s = 0x0, len =
1073741824}, params = {s = 0x2000000 <Address 0x2000000 out
of bounds>, len = 4289104},
sip_params = {s = 0x7ffc89e35180
"\360Q\343\211\374\177", len = 6031673}, headers = {s =
0x7fefe2cb183f "", len = -1981591296},
port_no = 23013, proto = 58022, type = 32751, flags
= 2313376240, transport = {s = 0x7fefe7374b9a "%x%c%x%c", len
= -1981590944}, ttl = {
---Type <return> to continue, or q <return> to quit---
s = 0x1a <Address 0x1a out of bounds>, len =
4289104}, user_param = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-",
len = -1670974014}, maddr = {
s = 0x30fbad8001 <Address 0x30fbad8001 out of
bounds>, len = -492414491}, method = {s = 0x7fefe2a659e5
"5e6a5a14-1fa-", len = -492414491},
lr = {s = 0x7fefe2a659e5 "5e6a5a14-1fa-", len =
-492414478}, r2 = {s = 0x7fefe2a659ff "", len = -492414491},
gr = {s = 0x7fefe2a659ff "",
len = 0}, transport_val = {s = 0x0, len =
6730416}, ttl_val = {s = 0x7ffc89e351a0 "", len = 673677328},
user_param_val = {
s = 0x2000000 <Address 0x2000000 out of bounds>,
len = 673677328}, maddr_val = {s = 0x7ffc89e351f0 "Cl\256",
len = -413652166},
method_val = {s = 0x7ff028278010 "\001", len = 0},
lr_val = {s = 0x2000000 <Address 0x2000000 out of bounds>,
len = 24589344}, r2_val = {
s = 0x67a138 "", len = 8965088}, gr_val = {s =
0x8cd2d0 <Address 0x8cd2d0 out of bounds>, len = 682507072}}
u = 0x66273d
port = 0
dst_host = 0x40000000
i = 28
flags = 0
avp = 0xae6cc7
st = {flags = 31129600, id = 0, name = {n = 0, s = {s
= 0x0, len = 93}, re = 0x0}, avp = 0x7ffc89e35080}
sct = 0x7ffc89e35490
sjt = 0x6943d3
rve = 0x7ff0285a8808
mct = 0x6a283b
rv = 0x0
rv1 = 0x7c9170
c1 = {cache_type = 24, val_type = 48, c = {avp_val =
{n = -1981590720, s = {s = 0x7ffc89e35340 "\370\225|", len =
-1981590912},
re = 0x7ffc89e35340}, pval = {rs = {s =
0x7ffc89e35340 "\370\225|", len = -1981590912}, ri = 0, flags
= 0}},
i2s = '\000' <repeats 21 times>}
s = {s = 0x7fefe2cb183f "", len = -490006491}
srevp = {0x7fefe2cb1825, 0x7fefe2cb1832}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
0x0}}, {type = NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len =
0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u =
{number = 0, string = 0x0, str = {
s = 0x0, len = 0}, data = 0x0, attr = 0x0,
select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0},
data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
= 0}, data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE,
u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data
= 0x0, attr = 0x0,
select = 0x0}}}
__FUNCTION__ = "do_action"
#7 0x00000000004404b8 in run_actions (h=0x7ffc89e35670,
a=0x7ff0285a85d0, msg=0x7ff028b453f8) at core/action.c:1552
t = 0x7ff0285a9668
ret = 1
mod = 0x417250
ms = 176570657
__FUNCTION__ = "run_actions"
#8 0x0000000000440c4a in run_top_route (a=0x7ff0285a85d0,
msg=0x7ff028b453f8, c=0x0) at core/action.c:1641
---Type <return> to continue, or q <return> to quit---
ctx = {rec_lev = 2, run_flags = 0, last_retcode = 1,
jmp_env = {{__jmpbuf = {140669451785720,
-8516593617994257148, 50, 1073741824, 33554432,
4289104, -8516593618092823292,
8518532807277145348}, __mask_was_saved = 0, __saved_mask =
{__val = {140668355198191, 4302791168, 48,
48, 18446744073709551615, 140722621863712,
140668355198919, 50, 0, 4611686022755909632, 140669451785208,
140722621863776, 5814046,
6608344928, 140669451785208, 0}}}}}
p = 0x7ffc89e35670
ret = 682907128
sfbk = 0
#9 0x000000000052e6b9 in receive_msg (
buf=0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia:
SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0'
<repeats 24 times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
<sip:usrloc@10.112.101.128
<mailto:sip%3Ausrloc@10.112.101.128>"..., len=781,
rcv_info=0x7ffc89e35a70) at core/receive.c:264
msg = 0x7ff028b453f8
ctx = {rec_lev = -1981589208, run_flags = 32764,
last_retcode = 8, jmp_env = {{__jmpbuf = {163, 4253893,
8100093, 140669451683568,
140722621864176, 4845170, 0, 21474836480},
__mask_was_saved = 673677840, __saved_mask = {__val =
{140668768878592, 140722621864448,
6697083, 8094080, 5501958, 243141751568,
140669476122624, 140722621864896, 4294967295, 88212724672, 1,
140668357584160, 0, 33554432,
1024, 6608345520}}}}}
bctx = 0x7ffc89e359b0
ret = 0
stats_on = 0
tvb = {tv_sec = 140722621864208, tv_usec = 8961768}
tve = {tv_sec = 0, tv_usec = 243139749096}
tz = {tz_minuteswest = 50, tz_dsttime = 0}
diff = 0
inb = {
s = 0xae6bc0 "KDMQ sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia:
SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0'
<repeats 24 times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
<sip:usrloc@10.112.101.128
<mailto:sip%3Ausrloc@10.112.101.128>"..., len = 781}
netinfo = {data = {s = 0x7ffc00000001 <Address
0x7ffc00000001 out of bounds>, len = 0}, dst = 0x1, rcv =
0x389c422190}
keng = 0x0
__FUNCTION__ = "receive_msg"
#10 0x00000000006456a3 in udp_rcv_loop () at
core/udp_server.c:515
len = 781
buf = "KDMQ sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080> SIP/2.0\r\nVia:
SIP/2.0/UDP 10.112.101.128;branch=z9hG4bK8791.0c350dd3", '0'
<repeats 24 times>, ".0\r\nTo: <sip:usrloc@10.112.103.64:5080
<http://sip:usrloc@10.112.103.64:5080>>\r\nFrom:
<sip:usrloc@10.112.101.128:5060
<http://sip:usrloc@10.112.101.128:5060>>;tag=8a3"...
tmp = 0x100000000 <Address 0x100000000 out of bounds>
from = 0x7ff028b2c728
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl =
{140722462617610, 140722621864592}, addr32 = {2154131466,
32764, 2313378448, 32764}, addr16 = {
28682, 32869, 32764, 0, 23184, 35299, 32764,
0}, addr =
"\npe\200\374\177\000\000\220Z\343\211\374\177\000"}}, dst_ip
= {af = 2,
len = 4, u = {addrl = {1080520714, 0}, addr32 =
{1080520714, 0, 0, 0}, addr16 = {28682, 16487, 0, 0, 0, 0, 0, 0},
addr = "\npg@", '\000' <repeats 11
times>}},
src_port = 5060, dst_port = 5080, proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023\304\npe\200\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {
s_addr = 2154131466}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2,
sin6_port = 50195,
---Type <return> to continue, or q <return> to quit---
sin6_flowinfo = 2154131466, sin6_addr =
{__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id
= 0}}, bind_address = 0x7ff02830fff0, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#11 0x0000000000423c7d in main_loop () at main.c:1623
i = 0
pid = 0
si = 0x7ff02830fff0
si_desc = "udp receiver child=0
sock=10.112.103.64:5080
<http://10.112.103.64:5080>\000\000\000\000
_\343\211\374\177\000\000\062\000\000\000\000\000\000\000\000\000\000@\000\000\000\000\000\000\000\002\000\000\000\000PrA\000\000\000\000\000PSn\234\070\000\000\000\060\000\000\000\060\000\000\000\020]\343\211\374\177\000\000@\\\343\211\374\177\000\000\000px\347\357\177\000"
nrprocs = 4
woneinit = 0
__FUNCTION__ = "main_loop"
#12 0x000000000042b0ce in main (argc=13, argv=0x7ffc89e36008)
at main.c:2646
cfg_stream = 0x1bb5010
c = -1
r = 0
tmp = 0x7ffc89e36ef7 ""
tmp_len = 32764
port = -1981587768
proto = 56
options = 0x7434f8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 4012862799
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x7ffc89e35f08
p = 0x100000000 <Address 0x100000000 out of bounds>
st = {st_dev = 64769, st_ino = 148592, st_nlink = 2,
st_mode = 16877, st_uid = 498, st_gid = 498, __pad0 = 0,
st_rdev = 0, st_size = 4096,
st_blksize = 4096, st_blocks = 8, st_atim = {tv_sec
= 1576522467, tv_nsec = 972000117}, st_mtim = {tv_sec =
1519927576,
tv_nsec = 668342155}, st_ctim = {tv_sec =
1584026268, tv_nsec = 165523404}, __unused = {0, 0, 0}}
__FUNCTION__ = "main"
(gdb)
Em ter., 17 de mar. de 2020 às 09:52, Daniel-Constantin
Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> escreveu:
Hello,
does it happen only when doing restart of kamailio? Can
you send the full backtrace from gdb?
Also, 5.0.x is no longer a supported release for quite
some time, try to upgrade to 5.2.x or even better to the
latest one 5.3.x and see if you can reproduce it with one
of the supported releases, maybe it is something that was
already fixed.
Cheers,
Daniel
On 17.03.20 10:32, Virgílio Cunha wrote:
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Hello,
I have in production environment with 4 proxies using
DMQ to sync terminal registers and the DMQ configuration
parameters is following:
loadmodule "dmq.so"
#modparam("dmq", "server_address",
"sip:MY_IP_ADDRESS:MY_PORT_ADDRESS")
modparam("dmq", "notification_address",
"DMQ_HOSTS")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
In the lab environment with the same configuration that
in prod, I can simulate this problem with 12000 terminal
REGISTERS and, when doing several restarts on a proxy,
one of them is generated a coredump.
Analyzing the core, I can see that the problem is
concurrency in the dmq synchronization.
Job_queue_size (...) is called with a valid /queue /and,
but inside this function, the /queue /is null.
(gdb) up
#1 0x00007fefe58ee246 in job_queue_size (queue=0x0) at
worker.c:254
254 return atomic_get(&queue->count);
(gdb) print queue
$1 = (job_queue_t *) 0x0
(gdb) up
#2 0x00007fefe58ed920 in add_dmq_job
(msg=0x7ff028b453f8, peer=0x7fefebcc4df8) at worker.c:184
184
if(job_queue_size(workers[i].queue) == 0) {
(gdb) print i
$2 = 1
(gdb) print workers[i].queue
$3 = (job_queue_t *) 0x7fefebce49b0
If I configure DMQ with num_workers=1 I can't reproduce
this problem.
I'm using the kamailio 5.0.8 release.
Is this problem known to you? What is the right way to
solve it?
Best Regards,
Virgílio Cunha
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
1488
days inactive
1498
days old
7 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Henning Westerholt -
Virgílio Cunha