<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>can you share with me the full config along with sipp scenario
files and commands you used for testing? I would like to reproduce
on my test environment to be able to troubleshoot.</p>
<p>Also, can you try with latest version from 5.2 branch? I pushed
some fixes recently to rtpengine as well as a rework for reply
handling inside the tm module -- these because there were some
similar reports before, but none of them had a way to reproduce.
Since you can reproduce it, if I can test it here I can be sure
the proper fix was done or the issue is somewhere else.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 21.01.19 18:48, Yufei Tao wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJwP0iSUo9z-sdZPGZ6EYk6Snjrc4kPtFWY9zM6Cd=Dqk31XgQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Hi,<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">I
have been testing one Kamailio v5.1.6 instance with
one rtpengine
instance, using sipp playing media files at 40 cps (-r
40) with up to 1600
concurrent calls. During the load tests if rtpengine
is pkill'ed/restarted a few times Kamailio would
crash. It is quite repeatable and every time the
backtrace
from gdb points to the same place as shown below.<br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>
However the same tests on Kamailio v5.0.7 with the
same cfg files and the same rtpengine instance did
not cause any crash. <br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Here’s
what I got from gdb backtrace for v5.1.6 using a dbg
build: 2 core dump files:</p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">1. <span>
UDP receiver processes</span>
<span>14483</span>
<br>
<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>[New
LWP 14483]<br>
Core was generated by `/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGSEGV, Segmentation
fault.<br>
#0 0x00007fadfa824d8e in t_should_relay_response
(Trans=0x7fadf4207730, new_code=200, branch=0,
should_store=0x7ffd5038fce4,
should_relay=0x7ffd5038fce0,
cancel_data=0x7ffd5038fed0, <br>
reply=0x7fadfb545210) at t_reply.c:1282<br>
1282 t_reply.c: No such file or directory.<br>
(gdb) bt<br>
#0 0x00007fadfa824d8e in t_should_relay_response
(Trans=0x7fadf4207730, new_code=200, branch=0,
should_store=0x7ffd5038fce4,
should_relay=0x7ffd5038fce0,
cancel_data=0x7ffd5038fed0, <br>
reply=0x7fadfb545210) at t_reply.c:1282<br>
#1 0x00007fadfa829577 in relay_reply
(t=0x7fadf4207730, p_msg=0x7fadfb545210, branch=0,
msg_status=200, cancel_data=0x7ffd5038fed0,
do_put_on_wait=1) at t_reply.c:1786<br>
#2 0x00007fadfa82f54c in reply_received
(p_msg=0x7fadfb545210) at t_reply.c:2537<br>
#3 0x000000000054624b in do_forward_reply
(msg=0x7fadfb545210, mode=0) at core/forward.c:747<br>
#4 0x0000000000547e4c in forward_reply
(msg=0x7fadfb545210) at core/forward.c:852<br>
#5 0x000000000058e186 in receive_msg (<br>
buf=0xa595a0 <buf> "SIP/2.0 200 OK\r\nVia:
SIP/2.0/UDP
192.168.70.102;branch=z9hG4bKa042.afac8eb973f1dfad7a549af0ab1a8ccc.0,
SIP/2.0/UDP
192.168.60.80:5060;branch=z9hG4bK-3750-978-0\r\nFrom:
sipp <<a
href="mailto:sip%3ACustomer68@192.168.60.8"
moz-do-not-send="true">sip:Customer68@192.168.60.8</a>"...,
len=888, rcv_info=0x7ffd50390480) at
core/receive.c:364<br>
#6 0x00000000004af6b1 in udp_rcv_loop () at
core/udp_server.c:554<br>
#7 0x00000000004246ac in main_loop () at
main.c:1619<br>
#8 0x000000000042bd5c in main (argc=13,
argv=0x7ffd50390b38) at main.c:2638<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
</div>
<div dir="ltr"><br>
</div>
<div>2. Main process 14468<br>
</div>
<div>{{{</div>
<div>[New LWP 14468]<br>
Core was generated by `/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGABRT, Aborted.<br>
#0 0x00007fadfbc77428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54<br>
54 ../sysdeps/unix/sysv/linux/raise.c: No such file
or directory.<br>
(gdb) bt<br>
#0 0x00007fadfbc77428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54<br>
#1 0x00007fadfbc7902a in __GI_abort () at abort.c:89<br>
#2 0x000000000041a029 in sig_alarm_abort (signo=14) at
main.c:646<br>
#3 <signal handler called><br>
#4 syscall () at
../sysdeps/unix/sysv/linux/x86_64/syscall.S:37<br>
#5 0x00007fadf354e67d in futex_get
(lock=0x7fadf3e94e50) at
../../core/parser/../mem/../futexlock.h:121<br>
#6 0x00007fadf3561113 in mod_destroy () at
rtpengine.c:1810<br>
#7 0x000000000055132b in destroy_modules () at
core/sr_module.c:832<br>
#8 0x0000000000418c9f in cleanup (show_status=1) at
main.c:521<br>
#9 0x000000000041a313 in shutdown_children (sig=15,
show_status=1) at main.c:663<br>
#10 0x000000000041cfa5 in handle_sigs () at main.c:768<br>
#11 0x0000000000425fb5 in main_loop () at main.c:1752<br>
#12 0x000000000042bd5c in main (argc=13,
argv=0x7ffd50390b38) at main.c:2638<br>
}}}<br>
</div>
<div><br>
</div>
<div dir="ltr">
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>The
parameters for rtpengine:</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>loadmodule
"rtpengine.so"<br>
modparam("rtpengine", "db_url",
"text:///usr/share/kamailio/dbtext/kamailio")<br>
modparam("rtpengine", "hash_table_size", 4)<br>
modparam("rtpengine", "setid_default", 1)<br>
modparam("rtpengine", "rtpengine_disable_tout", 20)<br>
modparam("rtpengine", "rtpengine_retr", 1)<br>
modparam("rtpengine", "setid_avp", "$avp(setid)")<br>
modparam("rtpengine", "rtp_inst_pvar",
"$avp(rtpInstance)")<br>
modparam("rtpengine", "rtpengine_tout_ms", 1000)<br>
modparam("rtpengine", "read_sdp_pv",
"$var(sdpToRtpengine)")<br>
modparam("rtpengine", "write_sdp_pv",
"$var(sdpFromRtpengine)")<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>I'm
using a simplified kamailio.cfg from installation,
and here are calls to rtpengine:<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>...
<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>route[INVITE]<br>
{<br>
$var(sdpToRtpengine) = $rb;<br>
$var(ret) =
rtpengine_manage("direction=dirty direction=clean
ICE=remove");<br>
xlog("L_INFO", "$ci INVITE: rtpengine
chosen: $avp(rtpInstance)");<br>
remove_body();<br>
replace_body(".*", $var(sdpFromRtpengine));<br>
t_on_reply("RESPONSE");<br>
<br>
route(RELAY);<br>
}<br>
<br>
onreply_route[RESPONSE]<br>
{<br>
$var(sdpToRtpengine) = $rb;<br>
$var(ret) =
rtpengine_manage("direction=clean direction=dirty
ICE=remove");<br>
remove_body();<br>
replace_body(".*", $var(sdpFromRtpengine));<br>
xlog("L_INFO", "$ci RESPONSE: $rm - $rs
$rr, cseq=$cs, by [$hdr(Server)], from $si:$sp");<br>
}<br>
...</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">When
rtpengine is down for a
couple of seconds, there were a lot of SIP
retransmissions and timeouts. Doing
a netstat and I can see Kamailio’s receive buffer is
quite filled up.<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Please
let me know if more information is needed.<span></span>
Thank you!<br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Cheers,<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Yufei<span></span></p>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a>
Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in Washington, DC, USA -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a></pre>
</body>
</html>