<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello,</p>
<p>I pushed two patches to prevent the crash, even the modules is
not used as expected in the config.</p>
<p>Charles: can you check and see if both makes sense? The one in
worker_loop() function is to prevent the crash:</p>
<p> *
<a class="moz-txt-link-freetext" href="https://github.com/kamailio/kamailio/commit/a675ab88fefac75145a7d563fee0431458630529">https://github.com/kamailio/kamailio/commit/a675ab88fefac75145a7d563fee0431458630529</a></p>
<p>This should be backported if all goes fine with it.</p>
<p>The second one in empty_peer_callback() is to generated a
202-Accepted response, otherwise in such cases the sender will do
retransmissions:<br>
</p>
<p> *
<a class="moz-txt-link-freetext" href="https://github.com/kamailio/kamailio/commit/7f618c2d855ac268df905eb3d6e18733c8773047">https://github.com/kamailio/kamailio/commit/7f618c2d855ac268df905eb3d6e18733c8773047</a></p>
<p>But maybe it was on purpose not to send a response (i.e., to
allow sending the response from config), in such case it can be
reverted.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 24.04.20 20:57, Charles Chance
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAOvxgzBqvkUSFDey27zv+MpbbcFWxbFgPMvA69oETsKgdvD2MA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div>
<div>
<div dir="auto">Hi,</div>
<div dir="auto"><br>
</div>
<div dir="auto">Did you try the config snippet I provided?</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Basically dmq_handle_message() must be called if
the message is not your own, otherwise the node
discovery/health check will not work and you will see nodes
disappearing as you described.</div>
</div>
<div>
<div dir="auto"><br>
</div>
<div dir="auto">Here it is again:</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div dir="auto" style="color:rgb(49,49,49);word-spacing:1px"><font
style="font-size:1rem" face="monospace">
if(is_method("KDMQ")){
<div class="quoted-text"
style="color:rgb(117,117,117);font-size:1rem"><br>
if($rU =~ "userOnline"){<br>
//user came online in cluster, resume
transactions if-any suspended<br>
$avp(remoteUser) = $rb; <br>
</div>
} else {<br>
</font><span
style="font-family:monospace;font-size:1rem">dmq_handle_message();</span></div>
<div dir="auto" style="color:rgb(49,49,49);word-spacing:1px"><font
style="font-size:1rem" face="monospace"> }<br>
}<br>
</font></div>
<br class="Apple-interchange-newline"
style="-webkit-tap-highlight-color: rgba(26, 26, 26,
0.301961); -webkit-text-size-adjust: auto;">
</div>
<div dir="auto">Notice that we check for your own/custom message
first, then call handle message if not matched.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Let me know if it works.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Cheers,</div>
<div dir="auto"><br>
</div>
<div dir="auto">Charles</div>
<div dir="auto"><br>
</div>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 24 Apr 2020 at
19:52, SamyGo <<a href="mailto:govoiper@gmail.com"
target="_blank" moz-do-not-send="true">govoiper@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Yes,<br>
I did read all(past 3+ years) his replies specific to
DMQ and DMQ USRLOC and only one matched exact
description and there has no resolution to it.
<div>Github open+closed issues for DMQ didn't have
anything similar either. Could it be something I'm
doing wrong !?
<div><br>
Additional info:
One of the server is direct on Public IP and Other
one is behind NAT. Another test setup where it
consistently reproducible is two server behind
NAT(AWS)
</div>
<div>Here are the mod params. Only usrloc sync is
done via DMQ and no other module is using DMQ. <br>
<font face="monospace"><br>
</font></div>
<div><font face="monospace">listen=<a class="moz-txt-link-freetext" href="udp:LocalIP:5060">udp:LocalIP:5060</a>
advertise PublicIP:5060</font><br>
</div>
<div><span style="font-family:monospace"><br>
</span></div>
<div><span style="font-family:monospace">modparam("dmq","server_</span><span
style="font-family:monospace">address",
DMQ_LOCAL_SERVER)</span><br>
</div>
<div><font face="monospace">modparam("dmq",
"notification_address", DMQ_REMOTE_SERVER)<br>
modparam("dmq", "multi_notify", 0) //1 for DNS SRV<br>
modparam("dmq", "num_workers", 10)<br>
modparam("dmq", "ping_interval", 60)<br>
<br>
modparam("dmq_usrloc", "enable", 1)<br>
modparam("dmq_usrloc", "sync", 1)<br>
modparam("dmq_usrloc", "batch_size", 4000)<br>
modparam("dmq_usrloc", "batch_usleep", 1000)<br>
modparam("dmq_usrloc", "usrloc_domain",
"location")</font><br>
</div>
<div><br>
</div>
<div>Where:
<font face="monospace">DMQ_REMOTE_SERVER =
<a class="moz-txt-link-freetext" href="sip:PublicIP2:5060">sip:PublicIP2:5060</a>
</font><br>
<br>
</div>
<div>GDB info as requested:</div>
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">Core was generated by
`/usr/local/sbin/kamailio -w /tmp/kamailio -P
/var/run/kamailio/kamailio.pid -f'.<br>
Program terminated with signal SIGSEGV,
Segmentation fault.<br>
#0 0x00007f248c4cef15 in send_reply
(msg=0x7f2469f88d40, code=0,
reason=0x7ffd775e3ab8) at sl.c:276<br>
276
if(reason->s[reason->len-1]=='\0') {<br>
(gdb)<br>
(gdb)<br>
(gdb) frame 0<br>
#0 0x00007f248c4cef15 in send_reply
(msg=0x7f2469f88d40, code=0,
reason=0x7ffd775e3ab8) at sl.c:276<br>
276
if(reason->s[reason->len-1]=='\0') {<br>
(gdb) p *reason<br>
$1 = {s = 0x0, len = 0}<br>
(gdb)<br>
(gdb) frame 1<br>
#1 0x00007f24656c6549 in worker_loop (id=2) at
worker.c:129<br>
129
if(slb.freply(current_job->msg,
peer_response.resp_code,<br>
(gdb) p *worker<br>
$3 = {queue = 0x7f2469f240a8, jobs_processed = 5,
lock = {val = 2}, pid = 935}<br>
(gdb)<br>
(gdb)<br>
(gdb) p *current_job<br>
$6 = {f = 0x7f24656d6d8d
<empty_peer_callback>, msg = 0x7f2469f88d40,
orig_peer = 0x7f2469f6ed50, next = 0x0, prev =
0x0}<br>
(gdb)<br>
</font><br>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Apr 24, 2020
at 1:30 PM Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com" target="_blank"
moz-do-not-send="true">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>have you tried the suggestion from Charles in the
other response? It can help figuring out where the
problem resides.</p>
<p>Now, from C point of view, I would need the
following output from gdb of the core file:</p>
<p>frame 0<br>
p *reason</p>
<p>frame 1<br>
p *worker<br>
p *current_job</p>
<p>I would also need to know the modparams for dmq
and other dmq_* module, plus the list if modules
for which you enabled dmq (eg, htable, dialog,
presence, ...).</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 24.04.20 18:10, SamyGo wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Oops,apologize, missed that:
<div><font face="monospace"><br>
</font></div>
<div><font face="monospace">version: kamailio
5.3.3 (x86_64/linux) 44ccb9-dirty<br>
flags: USE_TCP, USE_TLS, USE_SCTP,
TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP,
PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS 1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE
1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB<br>
poll method support: poll, epoll_lt,
epoll_et, sigio_rt, select.<br>
id: 44ccb9 -dirty<br>
compiled on 17:04:55 Apr 17 2020 with gcc
4.9.2</font><br>
</div>
<div><font face="monospace"><br>
</font></div>
<div><font face="arial, sans-serif">Tried this
with version 5.0, 5.2, and now 5.3 same
situation.. </font></div>
<div><font face="arial, sans-serif"><br>
</font></div>
<div><font face="arial, sans-serif">Thankyou for
looking into this,</font></div>
<div><font face="arial, sans-serif">Sammy</font></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Apr
24, 2020 at 2:33 AM Daniel-Constantin Mierla
<<a href="mailto:miconda@gmail.com"
target="_blank" moz-do-not-send="true">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>you have to provide the version of
kamailio for each reported kamailio issue,
otherwise is hard to match with the source
code. Use 'kamailio -v' to get version
details.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 23.04.20 23:36, SamyGo wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,<br>
<br>
<div>Is there a way to broadcast KDMQ to
the cluster but not expect a reply
back !?as far as I've read the source
code dmq_bcast_message is exactly like
dmq_send_message in a way that it
expects a callback to be executed on
response i.e expects a reply.</div>
<div><br>
</div>
<div>So, the situation I'm facing is I'm
broadcasting message to cluster and I
do not want a reply back. The
following two options result in crash
& core dump.</div>
<div><br>
</div>
<div>1 - If my script doesn't respond
back, by use of dmq_handle_message, it
marks the destined servers as
"inactive" and stops usrloc sync
process which isn't desirable.</div>
<div>2 - If I respond back with the
dmq_handle_message it crashes the
Kamailio which just received this
broadcasted message.</div>
<div><br>
</div>
<div>Here is how its done in script:<br>
</div>
<div><br>
</div>
<div><b>broadcasting message to cluster:</b></div>
<div><font face="monospace">
dmq_bcast_message("userOnline",
"$fu", "text/plain");</font><br>
</div>
<div><font face="arial, sans-serif"><br>
</font></div>
<div><font face="arial, sans-serif"><b>Receiving
and handling a broadcast message:</b></font></div>
<div><font face="monospace">route[DMQ_HANDLE]
{</font></div>
<font face="monospace">
if(!(is_method("KDMQ") || $rm ==
"KDMQ")) return;<br>
<br>
if(is_method("KDMQ") || $rm ==
"KDMQ"){<br>
if($rU =~ "userOnline"){<br>
//user came online
in cluster, resume transactions if-any
suspended<br>
$avp(remoteUser) =
$rb; <br>
}<br>
dmq_handle_message();<br>
exit;<br>
}<br>
}</font>
<div><font face="arial, sans-serif"><br>
</font></div>
<div><font face="arial, sans-serif"><b>Related
log lines:</b></font></div>
<div><font face="monospace">Apr 23
21:15:48 kamailio[916]: ALERT:
<script>: [da2c1-2f499] ------
DMQ_HANDLE: UserOnline Event
Received ------<br>
Apr 23 21:15:48 kamailio[916]:
DEBUG: dmq [message.c:53]:
ki_dmq_handle_message_rc():
dmq_handle_message [KDMQ <a
href="http://sip:userOnline@9.8.7.123:5060"
target="_blank"
moz-do-not-send="true">sip:userOnline@9.8.7.123:5060</a>]<br>
Apr 23 21:15:48 kamailio[916]:
DEBUG: dmq [message.c:66]:
ki_dmq_handle_message_rc():
dmq_handle_message peer found:
userOnline<br>
Apr 23 21:15:48 kamailio[916]:
DEBUG: <core>
[core/receive.c:437]: receive_msg():
request-route executed in: 401461
usec<br>
Apr 23 21:15:48 kamailio[935]:
DEBUG: dmq [worker.c:87]:
worker_loop(): dmq_worker [2 935]
lock acquired<br>
</font></div>
<div><font face="monospace">and
crash/segfault..<br>
<br>
</font></div>
<div><font face="monospace">Core dump: </font><a
href="https://pastebin.com/S7ekCPfF"
target="_blank"
moz-do-not-send="true">https://pastebin.com/S7ekCPfF</a><br>
<br>
Any help or pointers to solve this
would be really appreciated.</div>
<div><br>
</div>
<div>Best Regards,</div>
<div>Sammy</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.kamailio.org" target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org"
target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a><br>
<a
href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote>
</div>
</div>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr"><font face="arial, helvetica, sans-serif"><b><font
size="2">Charles Chance</font></b><br>
<font size="2">Managing Director</font></font><br>
<div><font face="arial, helvetica, sans-serif"><font
size="2"><br>
</font></font></div>
<div><font face="arial, helvetica, sans-serif"><font
size="2">t. 0330 120 1200 m. 07932 063 891</font></font></div>
</div>
</div>
</div>
</div>
<br>
<div><font
style="font-size:10pt;font-family:Helvetica,Arial,sans-serif"
color="gray">Sipcentric Ltd. Company registered in England
& Wales no. 7365592.</font><span
style="font-size:10pt;font-family:Helvetica,Arial,sans-serif"> </span><font
style="font-size:10pt;font-family:Helvetica,Arial,sans-serif"
color="gray">Registered office: Faraday Wharf, Innovation
Birmingham Campus, Holt Street, Birmingham Science Park,
Birmingham B7 4BB.</font></div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a></pre>
</body>
</html>