<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>thanks for spending more time on it! I will try to reproduce in
my side during the next days, currently being out of the office,
and then see what I can find.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 22.01.19 15:36, Yufei Tao wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJwP0iRK5UkSOfuXnOOFqTjNW_OeApXr1vPjDh0MkUxy4iRTCw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Daniel,</div>
<div><br>
</div>
<div>I tested latest v5.2.1 Debian package and created
the crash as well. <br>
</div>
<div><br>
</div>
<div>Two core dump files again similar to 5.1.6:<br>
</div>
<div>1. 1687 - udp receiver process</div>
<div>{{{</div>
<div>[New LWP 1687]<br>
Core was generated by `/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGSEGV, Segmentation
fault.<br>
#0 0x00007f015b3795cc in t_should_relay_response
(Trans=0x7f015cce9e98, new_code=200, branch=0,
should_store=0x7ffd5b4c5e24,
should_relay=0x7ffd5b4c5e20,
cancel_data=0x7ffd5b4c6010, <br>
reply=0x7f0161407ab8) at t_reply.c:1279<br>
1279 t_reply.c: No such file or directory.<br>
(gdb) bt<br>
#0 0x00007f015b3795cc in t_should_relay_response
(Trans=0x7f015cce9e98, new_code=200, branch=0,
should_store=0x7ffd5b4c5e24,
should_relay=0x7ffd5b4c5e20,
cancel_data=0x7ffd5b4c6010, <br>
reply=0x7f0161407ab8) at t_reply.c:1279<br>
#1 0x00007f015b37dec7 in relay_reply
(t=0x7f015cce9e98, p_msg=0x7f0161407ab8, branch=0,
msg_status=200, cancel_data=0x7ffd5b4c6010,
do_put_on_wait=1) at t_reply.c:1804<br>
#2 0x00007f015b383eaa in reply_received
(p_msg=0x7f0161407ab8) at t_reply.c:2539<br>
#3 0x000000000054e7f0 in do_forward_reply
(msg=0x7f0161407ab8, mode=0) at core/forward.c:747<br>
#4 0x0000000000550415 in forward_reply
(msg=0x7f0161407ab8) at core/forward.c:852<br>
#5 0x0000000000599159 in receive_msg (<br>
buf=0xa6ec80 <buf> "SIP/2.0 200 OK\r\nVia:
SIP/2.0/UDP
192.168.70.101;branch=z9hG4bK155f.f4284a7086985c9b088dc7c0dd32c63e.0,
SIP/2.0/UDP
192.168.60.80:5060;branch=z9hG4bK-5164-4615-0\r\nFrom:
sipp <<a href="mailto:sip%3ACustomer69@192.168.60."
moz-do-not-send="true">sip:Customer69@192.168.60.</a>"...,
len=886, rcv_info=0x7ffd5b4c65d0) at
core/receive.c:433<br>
#6 0x00000000004b22e8 in udp_rcv_loop () at
core/udp_server.c:541<br>
#7 0x0000000000425205 in main_loop () at main.c:1645<br>
#8 0x000000000042c9a5 in main (argc=13,
argv=0x7ffd5b4c6c98) at main.c:2675</div>
<div>}}}</div>
<div><br>
</div>
<div>2. 1673 - main process</div>
<div>{{{</div>
<div>[New LWP 1673]<br>
Core was generated by `/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGABRT, Aborted.<br>
#0 0x00007f0162344428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54<br>
54 ../sysdeps/unix/sysv/linux/raise.c: No such file
or directory.<br>
(gdb) bt<br>
#0 0x00007f0162344428 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/unix/sysv/linux/raise.c:54<br>
#1 0x00007f016234602a in __GI_abort () at abort.c:89<br>
#2 0x000000000041a836 in sig_alarm_abort (signo=14)
at main.c:663<br>
#3 <signal handler called><br>
#4 syscall () at
../sysdeps/unix/sysv/linux/x86_64/syscall.S:37<br>
#5 0x00007f0160a259ed in futex_get
(lock=0x7f015c2739c8) at
../../core/parser/../mem/../futexlock.h:121<br>
#6 0x00007f0160a395bc in mod_destroy () at
rtpengine.c:1941<br>
#7 0x00000000005589e2 in destroy_modules () at
core/sr_module.c:732<br>
#8 0x000000000041940b in cleanup (show_status=1) at
main.c:537<br>
#9 0x000000000041ab21 in shutdown_children (sig=15,
show_status=1) at main.c:680<br>
#10 0x000000000041d7c3 in handle_sigs () at main.c:785<br>
#11 0x0000000000426b23 in main_loop () at main.c:1780<br>
#12 0x000000000042c9a5 in main (argc=13,
argv=0x7ffd5b4c6c98) at main.c:2675<br>
}}}<br>
</div>
<div><br>
</div>
<div>I used the example kamailio-minimal-proxy.cfg from
5.2.1 source /misc/examples/mixed/ directory and added
rtpengine parameters and calls to rtpengine functions,
as the cfg file for 5.1.6 didn't work for 5.2.1.
Attached is the kamailio.cfg for v5.2.1 that I used in
the tests.<br>
</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Yufei<br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Tue, 22 Jan 2019 at
07:33, Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com" moz-do-not-send="true">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hello,</p>
<p>can you share with me the full config along with sipp
scenario files and commands you used for testing? I
would like to reproduce on my test environment to be
able to troubleshoot.</p>
<p>Also, can you try with latest version from 5.2
branch? I pushed some fixes recently to rtpengine as
well as a rework for reply handling inside the tm
module -- these because there were some similar
reports before, but none of them had a way to
reproduce. Since you can reproduce it, if I can test
it here I can be sure the proper fix was done or the
issue is somewhere else.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="gmail-m_-6412462747132725978moz-cite-prefix">On
21.01.19 18:48, Yufei Tao wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Hi,<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">I
have been testing one Kamailio v5.1.6
instance with one rtpengine instance,
using sipp playing media files at 40 cps
(-r 40) with up to 1600 concurrent calls.
During the load tests if rtpengine is
pkill'ed/restarted a few times Kamailio
would crash. It is quite repeatable and
every time the backtrace from gdb points
to the same place as shown below.<br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>
However the same tests on Kamailio
v5.0.7 with the same cfg files and the
same rtpengine instance did not cause
any crash. <br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Here’s
what I got from gdb backtrace for v5.1.6
using a dbg build: 2 core dump files:</p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">1. <span>
UDP receiver processes</span> <span>14483</span>
<br>
<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>[New
LWP 14483]<br>
Core was generated by
`/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGSEGV,
Segmentation fault.<br>
#0 0x00007fadfa824d8e in
t_should_relay_response
(Trans=0x7fadf4207730, new_code=200,
branch=0, should_store=0x7ffd5038fce4,
should_relay=0x7ffd5038fce0,
cancel_data=0x7ffd5038fed0, <br>
reply=0x7fadfb545210) at
t_reply.c:1282<br>
1282 t_reply.c: No such file or
directory.<br>
(gdb) bt<br>
#0 0x00007fadfa824d8e in
t_should_relay_response
(Trans=0x7fadf4207730, new_code=200,
branch=0, should_store=0x7ffd5038fce4,
should_relay=0x7ffd5038fce0,
cancel_data=0x7ffd5038fed0, <br>
reply=0x7fadfb545210) at
t_reply.c:1282<br>
#1 0x00007fadfa829577 in relay_reply
(t=0x7fadf4207730, p_msg=0x7fadfb545210,
branch=0, msg_status=200,
cancel_data=0x7ffd5038fed0,
do_put_on_wait=1) at t_reply.c:1786<br>
#2 0x00007fadfa82f54c in reply_received
(p_msg=0x7fadfb545210) at t_reply.c:2537<br>
#3 0x000000000054624b in
do_forward_reply (msg=0x7fadfb545210,
mode=0) at core/forward.c:747<br>
#4 0x0000000000547e4c in forward_reply
(msg=0x7fadfb545210) at
core/forward.c:852<br>
#5 0x000000000058e186 in receive_msg (<br>
buf=0xa595a0 <buf> "SIP/2.0
200 OK\r\nVia: SIP/2.0/UDP
192.168.70.102;branch=z9hG4bKa042.afac8eb973f1dfad7a549af0ab1a8ccc.0,
SIP/2.0/UDP
192.168.60.80:5060;branch=z9hG4bK-3750-978-0\r\nFrom:
sipp <<a
href="mailto:sip%3ACustomer68@192.168.60.8"
target="_blank" moz-do-not-send="true">sip:Customer68@192.168.60.8</a>"...,
len=888, rcv_info=0x7ffd50390480) at
core/receive.c:364<br>
#6 0x00000000004af6b1 in udp_rcv_loop
() at core/udp_server.c:554<br>
#7 0x00000000004246ac in main_loop ()
at main.c:1619<br>
#8 0x000000000042bd5c in main (argc=13,
argv=0x7ffd50390b38) at main.c:2638<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
</div>
<div dir="ltr"><br>
</div>
<div>2. Main process 14468<br>
</div>
<div>{{{</div>
<div>[New LWP 14468]<br>
Core was generated by `/usr/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGABRT,
Aborted.<br>
#0 0x00007fadfbc77428 in __GI_raise
(sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54<br>
54 ../sysdeps/unix/sysv/linux/raise.c: No
such file or directory.<br>
(gdb) bt<br>
#0 0x00007fadfbc77428 in __GI_raise
(sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54<br>
#1 0x00007fadfbc7902a in __GI_abort () at
abort.c:89<br>
#2 0x000000000041a029 in sig_alarm_abort
(signo=14) at main.c:646<br>
#3 <signal handler called><br>
#4 syscall () at
../sysdeps/unix/sysv/linux/x86_64/syscall.S:37<br>
#5 0x00007fadf354e67d in futex_get
(lock=0x7fadf3e94e50) at
../../core/parser/../mem/../futexlock.h:121<br>
#6 0x00007fadf3561113 in mod_destroy () at
rtpengine.c:1810<br>
#7 0x000000000055132b in destroy_modules ()
at core/sr_module.c:832<br>
#8 0x0000000000418c9f in cleanup
(show_status=1) at main.c:521<br>
#9 0x000000000041a313 in shutdown_children
(sig=15, show_status=1) at main.c:663<br>
#10 0x000000000041cfa5 in handle_sigs () at
main.c:768<br>
#11 0x0000000000425fb5 in main_loop () at
main.c:1752<br>
#12 0x000000000042bd5c in main (argc=13,
argv=0x7ffd50390b38) at main.c:2638<br>
}}}<br>
</div>
<div><br>
</div>
<div dir="ltr">
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>The
parameters for rtpengine:</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>loadmodule
"rtpengine.so"<br>
modparam("rtpengine", "db_url",
"text:///usr/share/kamailio/dbtext/kamailio")<br>
modparam("rtpengine", "hash_table_size",
4)<br>
modparam("rtpengine", "setid_default",
1)<br>
modparam("rtpengine",
"rtpengine_disable_tout", 20)<br>
modparam("rtpengine", "rtpengine_retr",
1)<br>
modparam("rtpengine", "setid_avp",
"$avp(setid)")<br>
modparam("rtpengine", "rtp_inst_pvar",
"$avp(rtpInstance)")<br>
modparam("rtpengine",
"rtpengine_tout_ms", 1000)<br>
modparam("rtpengine", "read_sdp_pv",
"$var(sdpToRtpengine)")<br>
modparam("rtpengine", "write_sdp_pv",
"$var(sdpFromRtpengine)")<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>I'm
using a simplified kamailio.cfg from
installation, and here are calls to
rtpengine:<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>{{{</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>...
<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>route[INVITE]<br>
{<br>
$var(sdpToRtpengine) = $rb;<br>
$var(ret) =
rtpengine_manage("direction=dirty
direction=clean ICE=remove");<br>
xlog("L_INFO", "$ci INVITE:
rtpengine chosen: $avp(rtpInstance)");<br>
remove_body();<br>
replace_body(".*",
$var(sdpFromRtpengine));<br>
t_on_reply("RESPONSE");<br>
<br>
route(RELAY);<br>
}<br>
<br>
onreply_route[RESPONSE]<br>
{<br>
$var(sdpToRtpengine) = $rb;<br>
$var(ret) =
rtpengine_manage("direction=clean
direction=dirty ICE=remove");<br>
remove_body();<br>
replace_body(".*",
$var(sdpFromRtpengine));<br>
xlog("L_INFO", "$ci RESPONSE:
$rm - $rs $rr, cseq=$cs, by
[$hdr(Server)], from $si:$sp");<br>
}<br>
...</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span>}}}<br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span><br>
</span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">When
rtpengine is down for a couple of seconds,
there were a lot of SIP retransmissions
and timeouts. Doing a netstat and I can
see Kamailio’s receive buffer is quite
filled up.<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Please
let me know if more information is needed.<span></span>
Thank you!<br>
</p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span> </span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Cheers,<span></span></p>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif">Yufei<span></span></p>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset
class="gmail-m_-6412462747132725978mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-6412462747132725978moz-quote-pre">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="gmail-m_-6412462747132725978moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="gmail-m_-6412462747132725978moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank" moz-do-not-send="true">www.kamailioworld.com</a>
Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in Washington, DC, USA -- <a class="gmail-m_-6412462747132725978moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a></pre>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a>
Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in Washington, DC, USA -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a></pre>
</body>
</html>