<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi Daniel,</div><div><br></div><div>I tested latest v5.2.1 Debian package and created the crash as well. <br></div><div><br></div><div>Two core dump files again similar to 5.1.6:<br></div><div>1. 1687 - udp receiver process</div><div>{{{</div><div>[New LWP 1687]<br>Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.<br>Program terminated with signal SIGSEGV, Segmentation fault.<br>#0 0x00007f015b3795cc in t_should_relay_response (Trans=0x7f015cce9e98, new_code=200, branch=0, should_store=0x7ffd5b4c5e24, should_relay=0x7ffd5b4c5e20, cancel_data=0x7ffd5b4c6010, <br> reply=0x7f0161407ab8) at t_reply.c:1279<br>1279 t_reply.c: No such file or directory.<br>(gdb) bt<br>#0 0x00007f015b3795cc in t_should_relay_response (Trans=0x7f015cce9e98, new_code=200, branch=0, should_store=0x7ffd5b4c5e24, should_relay=0x7ffd5b4c5e20, cancel_data=0x7ffd5b4c6010, <br> reply=0x7f0161407ab8) at t_reply.c:1279<br>#1 0x00007f015b37dec7 in relay_reply (t=0x7f015cce9e98, p_msg=0x7f0161407ab8, branch=0, msg_status=200, cancel_data=0x7ffd5b4c6010, do_put_on_wait=1) at t_reply.c:1804<br>#2 0x00007f015b383eaa in reply_received (p_msg=0x7f0161407ab8) at t_reply.c:2539<br>#3 0x000000000054e7f0 in do_forward_reply (msg=0x7f0161407ab8, mode=0) at core/forward.c:747<br>#4 0x0000000000550415 in forward_reply (msg=0x7f0161407ab8) at core/forward.c:852<br>#5 0x0000000000599159 in receive_msg (<br> buf=0xa6ec80 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 192.168.70.101;branch=z9hG4bK155f.f4284a7086985c9b088dc7c0dd32c63e.0, SIP/2.0/UDP 192.168.60.80:5060;branch=z9hG4bK-5164-4615-0\r\nFrom: sipp <<a href="mailto:sip%3ACustomer69@192.168.60.">sip:Customer69@192.168.60.</a>"..., len=886, rcv_info=0x7ffd5b4c65d0) at core/receive.c:433<br>#6 0x00000000004b22e8 in udp_rcv_loop () at core/udp_server.c:541<br>#7 0x0000000000425205 in main_loop () at main.c:1645<br>#8 0x000000000042c9a5 in main (argc=13, argv=0x7ffd5b4c6c98) at main.c:2675</div><div>}}}</div><div><br></div><div>2. 1673 - main process</div><div>{{{</div><div>[New LWP 1673]<br>Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.<br>Program terminated with signal SIGABRT, Aborted.<br>#0 0x00007f0162344428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54<br>54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.<br>(gdb) bt<br>#0 0x00007f0162344428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54<br>#1 0x00007f016234602a in __GI_abort () at abort.c:89<br>#2 0x000000000041a836 in sig_alarm_abort (signo=14) at main.c:663<br>#3 <signal handler called><br>#4 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:37<br>#5 0x00007f0160a259ed in futex_get (lock=0x7f015c2739c8) at ../../core/parser/../mem/../futexlock.h:121<br>#6 0x00007f0160a395bc in mod_destroy () at rtpengine.c:1941<br>#7 0x00000000005589e2 in destroy_modules () at core/sr_module.c:732<br>#8 0x000000000041940b in cleanup (show_status=1) at main.c:537<br>#9 0x000000000041ab21 in shutdown_children (sig=15, show_status=1) at main.c:680<br>#10 0x000000000041d7c3 in handle_sigs () at main.c:785<br>#11 0x0000000000426b23 in main_loop () at main.c:1780<br>#12 0x000000000042c9a5 in main (argc=13, argv=0x7ffd5b4c6c98) at main.c:2675<br>}}}<br></div><div><br></div><div>I used the example kamailio-minimal-proxy.cfg from 5.2.1 source /misc/examples/mixed/ directory and added rtpengine parameters and calls to rtpengine functions, as the cfg file for 5.1.6 didn't work for 5.2.1. Attached is the kamailio.cfg for v5.2.1 that I used in the tests.<br></div><div><br></div><div>Cheers,</div><div>Yufei<br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 22 Jan 2019 at 07:33, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hello,</p>
<p>can you share with me the full config along with sipp scenario
files and commands you used for testing? I would like to reproduce
on my test environment to be able to troubleshoot.</p>
<p>Also, can you try with latest version from 5.2 branch? I pushed
some fixes recently to rtpengine as well as a rework for reply
handling inside the tm module -- these because there were some
similar reports before, but none of them had a way to reproduce.
Since you can reproduce it, if I can test it here I can be sure
the proper fix was done or the issue is somewhere else.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="gmail-m_-6412462747132725978moz-cite-prefix">On 21.01.19 18:48, Yufei Tao wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Hi,<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">I
have been testing one Kamailio v5.1.6 instance with
one rtpengine
instance, using sipp playing media files at 40 cps (-r
40) with up to 1600
concurrent calls. During the load tests if rtpengine
is pkill'ed/restarted a few times Kamailio would
crash. It is quite repeatable and every time the
backtrace
from gdb points to the same place as shown below.<br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>
However the same tests on Kamailio v5.0.7 with the
same cfg files and the same rtpengine instance did
not cause any crash. <br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Here’s
what I got from gdb backtrace for v5.1.6 using a dbg
build: 2 core dump files:</p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">1. <span>
UDP receiver processes</span>
<span>14483</span>
<br>
<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>[New
LWP 14483]<br>
Core was generated by `/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGSEGV, Segmentation
fault.<br>
#0 0x00007fadfa824d8e in t_should_relay_response
(Trans=0x7fadf4207730, new_code=200, branch=0,
should_store=0x7ffd5038fce4,
should_relay=0x7ffd5038fce0,
cancel_data=0x7ffd5038fed0, <br>
reply=0x7fadfb545210) at t_reply.c:1282<br>
1282 t_reply.c: No such file or directory.<br>
(gdb) bt<br>
#0 0x00007fadfa824d8e in t_should_relay_response
(Trans=0x7fadf4207730, new_code=200, branch=0,
should_store=0x7ffd5038fce4,
should_relay=0x7ffd5038fce0,
cancel_data=0x7ffd5038fed0, <br>
reply=0x7fadfb545210) at t_reply.c:1282<br>
#1 0x00007fadfa829577 in relay_reply
(t=0x7fadf4207730, p_msg=0x7fadfb545210, branch=0,
msg_status=200, cancel_data=0x7ffd5038fed0,
do_put_on_wait=1) at t_reply.c:1786<br>
#2 0x00007fadfa82f54c in reply_received
(p_msg=0x7fadfb545210) at t_reply.c:2537<br>
#3 0x000000000054624b in do_forward_reply
(msg=0x7fadfb545210, mode=0) at core/forward.c:747<br>
#4 0x0000000000547e4c in forward_reply
(msg=0x7fadfb545210) at core/forward.c:852<br>
#5 0x000000000058e186 in receive_msg (<br>
buf=0xa595a0 <buf> "SIP/2.0 200 OK\r\nVia:
SIP/2.0/UDP
192.168.70.102;branch=z9hG4bKa042.afac8eb973f1dfad7a549af0ab1a8ccc.0,
SIP/2.0/UDP
192.168.60.80:5060;branch=z9hG4bK-3750-978-0\r\nFrom:
sipp <<a href="mailto:sip%3ACustomer68@192.168.60.8" target="_blank">sip:Customer68@192.168.60.8</a>"...,
len=888, rcv_info=0x7ffd50390480) at
core/receive.c:364<br>
#6 0x00000000004af6b1 in udp_rcv_loop () at
core/udp_server.c:554<br>
#7 0x00000000004246ac in main_loop () at
main.c:1619<br>
#8 0x000000000042bd5c in main (argc=13,
argv=0x7ffd50390b38) at main.c:2638<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
</div>
<div dir="ltr"><br>
</div>
<div>2. Main process 14468<br>
</div>
<div>{{{</div>
<div>[New LWP 14468]<br>
Core was generated by `/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGABRT, Aborted.<br>
#0 0x00007fadfbc77428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54<br>
54 ../sysdeps/unix/sysv/linux/raise.c: No such file
or directory.<br>
(gdb) bt<br>
#0 0x00007fadfbc77428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54<br>
#1 0x00007fadfbc7902a in __GI_abort () at abort.c:89<br>
#2 0x000000000041a029 in sig_alarm_abort (signo=14) at
main.c:646<br>
#3 <signal handler called><br>
#4 syscall () at
../sysdeps/unix/sysv/linux/x86_64/syscall.S:37<br>
#5 0x00007fadf354e67d in futex_get
(lock=0x7fadf3e94e50) at
../../core/parser/../mem/../futexlock.h:121<br>
#6 0x00007fadf3561113 in mod_destroy () at
rtpengine.c:1810<br>
#7 0x000000000055132b in destroy_modules () at
core/sr_module.c:832<br>
#8 0x0000000000418c9f in cleanup (show_status=1) at
main.c:521<br>
#9 0x000000000041a313 in shutdown_children (sig=15,
show_status=1) at main.c:663<br>
#10 0x000000000041cfa5 in handle_sigs () at main.c:768<br>
#11 0x0000000000425fb5 in main_loop () at main.c:1752<br>
#12 0x000000000042bd5c in main (argc=13,
argv=0x7ffd50390b38) at main.c:2638<br>
}}}<br>
</div>
<div><br>
</div>
<div dir="ltr">
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>The
parameters for rtpengine:</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>loadmodule
"rtpengine.so"<br>
modparam("rtpengine", "db_url",
"text:///usr/share/kamailio/dbtext/kamailio")<br>
modparam("rtpengine", "hash_table_size", 4)<br>
modparam("rtpengine", "setid_default", 1)<br>
modparam("rtpengine", "rtpengine_disable_tout", 20)<br>
modparam("rtpengine", "rtpengine_retr", 1)<br>
modparam("rtpengine", "setid_avp", "$avp(setid)")<br>
modparam("rtpengine", "rtp_inst_pvar",
"$avp(rtpInstance)")<br>
modparam("rtpengine", "rtpengine_tout_ms", 1000)<br>
modparam("rtpengine", "read_sdp_pv",
"$var(sdpToRtpengine)")<br>
modparam("rtpengine", "write_sdp_pv",
"$var(sdpFromRtpengine)")<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>I'm
using a simplified kamailio.cfg from installation,
and here are calls to rtpengine:<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>...
<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>route[INVITE]<br>
{<br>
$var(sdpToRtpengine) = $rb;<br>
$var(ret) =
rtpengine_manage("direction=dirty direction=clean
ICE=remove");<br>
xlog("L_INFO", "$ci INVITE: rtpengine
chosen: $avp(rtpInstance)");<br>
remove_body();<br>
replace_body(".*", $var(sdpFromRtpengine));<br>
t_on_reply("RESPONSE");<br>
<br>
route(RELAY);<br>
}<br>
<br>
onreply_route[RESPONSE]<br>
{<br>
$var(sdpToRtpengine) = $rb;<br>
$var(ret) =
rtpengine_manage("direction=clean direction=dirty
ICE=remove");<br>
remove_body();<br>
replace_body(".*", $var(sdpFromRtpengine));<br>
xlog("L_INFO", "$ci RESPONSE: $rm - $rs
$rr, cseq=$cs, by [$hdr(Server)], from $si:$sp");<br>
}<br>
...</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">When
rtpengine is down for a
couple of seconds, there were a lot of SIP
retransmissions and timeouts. Doing
a netstat and I can see Kamailio’s receive buffer is
quite filled up.<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Please
let me know if more information is needed.<span></span>
Thank you!<br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Cheers,<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Yufei<span></span></p>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_-6412462747132725978mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-6412462747132725978moz-quote-pre">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a class="gmail-m_-6412462747132725978moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="gmail-m_-6412462747132725978moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in Washington, DC, USA -- <a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
</div>
</blockquote></div></div></div>