[SR-Users] Kamailio v5.1.6 crashes with rtpengine restarts
Daniel-Constantin Mierla
miconda at gmail.com
Wed Jan 23 09:01:01 CET 2019
Hello,
thanks for spending more time on it! I will try to reproduce in my side
during the next days, currently being out of the office, and then see
what I can find.
Cheers,
Daniel
On 22.01.19 15:36, Yufei Tao wrote:
> Hi Daniel,
>
> I tested latest v5.2.1 Debian package and created the crash as well.
>
> Two core dump files again similar to 5.1.6:
> 1. 1687 - udp receiver process
> {{{
> [New LWP 1687]
> Core was generated by `/usr/sbin/kamailio -P
> /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007f015b3795cc in t_should_relay_response
> (Trans=0x7f015cce9e98, new_code=200, branch=0,
> should_store=0x7ffd5b4c5e24, should_relay=0x7ffd5b4c5e20,
> cancel_data=0x7ffd5b4c6010,
> reply=0x7f0161407ab8) at t_reply.c:1279
> 1279 t_reply.c: No such file or directory.
> (gdb) bt
> #0 0x00007f015b3795cc in t_should_relay_response
> (Trans=0x7f015cce9e98, new_code=200, branch=0,
> should_store=0x7ffd5b4c5e24, should_relay=0x7ffd5b4c5e20,
> cancel_data=0x7ffd5b4c6010,
> reply=0x7f0161407ab8) at t_reply.c:1279
> #1 0x00007f015b37dec7 in relay_reply (t=0x7f015cce9e98,
> p_msg=0x7f0161407ab8, branch=0, msg_status=200,
> cancel_data=0x7ffd5b4c6010, do_put_on_wait=1) at t_reply.c:1804
> #2 0x00007f015b383eaa in reply_received (p_msg=0x7f0161407ab8) at
> t_reply.c:2539
> #3 0x000000000054e7f0 in do_forward_reply (msg=0x7f0161407ab8,
> mode=0) at core/forward.c:747
> #4 0x0000000000550415 in forward_reply (msg=0x7f0161407ab8) at
> core/forward.c:852
> #5 0x0000000000599159 in receive_msg (
> buf=0xa6ec80 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
> 192.168.70.101;branch=z9hG4bK155f.f4284a7086985c9b088dc7c0dd32c63e.0,
> SIP/2.0/UDP 192.168.60.80:5060;branch=z9hG4bK-5164-4615-0\r\nFrom:
> sipp <sip:Customer69 at 192.168.60.
> <mailto:sip%3ACustomer69 at 192.168.60.>"..., len=886,
> rcv_info=0x7ffd5b4c65d0) at core/receive.c:433
> #6 0x00000000004b22e8 in udp_rcv_loop () at core/udp_server.c:541
> #7 0x0000000000425205 in main_loop () at main.c:1645
> #8 0x000000000042c9a5 in main (argc=13, argv=0x7ffd5b4c6c98) at
> main.c:2675
> }}}
>
> 2. 1673 - main process
> {{{
> [New LWP 1673]
> Core was generated by `/usr/sbin/kamailio -P
> /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007f0162344428 in __GI_raise (sig=sig at entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0 0x00007f0162344428 in __GI_raise (sig=sig at entry=6) at
> ../sysdeps/unix/sysv/linux/raise.c:54
> #1 0x00007f016234602a in __GI_abort () at abort.c:89
> #2 0x000000000041a836 in sig_alarm_abort (signo=14) at main.c:663
> #3 <signal handler called>
> #4 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:37
> #5 0x00007f0160a259ed in futex_get (lock=0x7f015c2739c8) at
> ../../core/parser/../mem/../futexlock.h:121
> #6 0x00007f0160a395bc in mod_destroy () at rtpengine.c:1941
> #7 0x00000000005589e2 in destroy_modules () at core/sr_module.c:732
> #8 0x000000000041940b in cleanup (show_status=1) at main.c:537
> #9 0x000000000041ab21 in shutdown_children (sig=15, show_status=1) at
> main.c:680
> #10 0x000000000041d7c3 in handle_sigs () at main.c:785
> #11 0x0000000000426b23 in main_loop () at main.c:1780
> #12 0x000000000042c9a5 in main (argc=13, argv=0x7ffd5b4c6c98) at
> main.c:2675
> }}}
>
> I used the example kamailio-minimal-proxy.cfg from 5.2.1 source
> /misc/examples/mixed/ directory and added rtpengine parameters and
> calls to rtpengine functions, as the cfg file for 5.1.6 didn't work
> for 5.2.1. Attached is the kamailio.cfg for v5.2.1 that I used in the
> tests.
>
> Cheers,
> Yufei
>
> On Tue, 22 Jan 2019 at 07:33, Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
> can you share with me the full config along with sipp scenario
> files and commands you used for testing? I would like to reproduce
> on my test environment to be able to troubleshoot.
>
> Also, can you try with latest version from 5.2 branch? I pushed
> some fixes recently to rtpengine as well as a rework for reply
> handling inside the tm module -- these because there were some
> similar reports before, but none of them had a way to reproduce.
> Since you can reproduce it, if I can test it here I can be sure
> the proper fix was done or the issue is somewhere else.
>
> Cheers,
> Daniel
>
> On 21.01.19 18:48, Yufei Tao wrote:
>>
>> Hi,
>>
>>
>>
>> I have been testing one Kamailio v5.1.6 instance with one
>> rtpengine instance, using sipp playing media files at 40 cps (-r
>> 40) with up to 1600 concurrent calls. During the load tests if
>> rtpengine is pkill'ed/restarted a few times Kamailio would crash.
>> It is quite repeatable and every time the backtrace from gdb
>> points to the same place as shown below.
>>
>>
>> However the same tests on Kamailio v5.0.7 with the same cfg files
>> and the same rtpengine instance did not cause any crash.
>>
>>
>> Here’s what I got from gdb backtrace for v5.1.6 using a dbg
>> build: 2 core dump files:
>>
>>
>> 1. UDP receiver processes 14483
>>
>> {{{
>>
>> [New LWP 14483]
>> Core was generated by `/usr/sbin/kamailio -P
>> /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x00007fadfa824d8e in t_should_relay_response
>> (Trans=0x7fadf4207730, new_code=200, branch=0,
>> should_store=0x7ffd5038fce4, should_relay=0x7ffd5038fce0,
>> cancel_data=0x7ffd5038fed0,
>> reply=0x7fadfb545210) at t_reply.c:1282
>> 1282 t_reply.c: No such file or directory.
>> (gdb) bt
>> #0 0x00007fadfa824d8e in t_should_relay_response
>> (Trans=0x7fadf4207730, new_code=200, branch=0,
>> should_store=0x7ffd5038fce4, should_relay=0x7ffd5038fce0,
>> cancel_data=0x7ffd5038fed0,
>> reply=0x7fadfb545210) at t_reply.c:1282
>> #1 0x00007fadfa829577 in relay_reply (t=0x7fadf4207730,
>> p_msg=0x7fadfb545210, branch=0, msg_status=200,
>> cancel_data=0x7ffd5038fed0, do_put_on_wait=1) at t_reply.c:1786
>> #2 0x00007fadfa82f54c in reply_received (p_msg=0x7fadfb545210)
>> at t_reply.c:2537
>> #3 0x000000000054624b in do_forward_reply (msg=0x7fadfb545210,
>> mode=0) at core/forward.c:747
>> #4 0x0000000000547e4c in forward_reply (msg=0x7fadfb545210) at
>> core/forward.c:852
>> #5 0x000000000058e186 in receive_msg (
>> buf=0xa595a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
>> 192.168.70.102;branch=z9hG4bKa042.afac8eb973f1dfad7a549af0ab1a8ccc.0,
>> SIP/2.0/UDP 192.168.60.80:5060;branch=z9hG4bK-3750-978-0\r\nFrom:
>> sipp <sip:Customer68 at 192.168.60.8
>> <mailto:sip%3ACustomer68 at 192.168.60.8>"..., len=888,
>> rcv_info=0x7ffd50390480) at core/receive.c:364
>> #6 0x00000000004af6b1 in udp_rcv_loop () at core/udp_server.c:554
>> #7 0x00000000004246ac in main_loop () at main.c:1619
>> #8 0x000000000042bd5c in main (argc=13, argv=0x7ffd50390b38) at
>> main.c:2638
>>
>> }}}
>>
>>
>>
>> 2. Main process 14468
>> {{{
>> [New LWP 14468]
>> Core was generated by `/usr/sbin/kamailio -P
>> /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
>> Program terminated with signal SIGABRT, Aborted.
>> #0 0x00007fadfbc77428 in __GI_raise (sig=sig at entry=6) at
>> ../sysdeps/unix/sysv/linux/raise.c:54
>> 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
>> (gdb) bt
>> #0 0x00007fadfbc77428 in __GI_raise (sig=sig at entry=6) at
>> ../sysdeps/unix/sysv/linux/raise.c:54
>> #1 0x00007fadfbc7902a in __GI_abort () at abort.c:89
>> #2 0x000000000041a029 in sig_alarm_abort (signo=14) at main.c:646
>> #3 <signal handler called>
>> #4 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:37
>> #5 0x00007fadf354e67d in futex_get (lock=0x7fadf3e94e50) at
>> ../../core/parser/../mem/../futexlock.h:121
>> #6 0x00007fadf3561113 in mod_destroy () at rtpengine.c:1810
>> #7 0x000000000055132b in destroy_modules () at core/sr_module.c:832
>> #8 0x0000000000418c9f in cleanup (show_status=1) at main.c:521
>> #9 0x000000000041a313 in shutdown_children (sig=15,
>> show_status=1) at main.c:663
>> #10 0x000000000041cfa5 in handle_sigs () at main.c:768
>> #11 0x0000000000425fb5 in main_loop () at main.c:1752
>> #12 0x000000000042bd5c in main (argc=13, argv=0x7ffd50390b38) at
>> main.c:2638
>> }}}
>>
>> The parameters for rtpengine:
>>
>> {{{
>>
>> loadmodule "rtpengine.so"
>> modparam("rtpengine", "db_url",
>> "text:///usr/share/kamailio/dbtext/kamailio")
>> modparam("rtpengine", "hash_table_size", 4)
>> modparam("rtpengine", "setid_default", 1)
>> modparam("rtpengine", "rtpengine_disable_tout", 20)
>> modparam("rtpengine", "rtpengine_retr", 1)
>> modparam("rtpengine", "setid_avp", "$avp(setid)")
>> modparam("rtpengine", "rtp_inst_pvar", "$avp(rtpInstance)")
>> modparam("rtpengine", "rtpengine_tout_ms", 1000)
>> modparam("rtpengine", "read_sdp_pv", "$var(sdpToRtpengine)")
>> modparam("rtpengine", "write_sdp_pv", "$var(sdpFromRtpengine)")
>>
>> }}}
>>
>>
>> I'm using a simplified kamailio.cfg from installation, and here
>> are calls to rtpengine:
>>
>> {{{
>>
>> ...
>>
>> route[INVITE]
>> {
>> $var(sdpToRtpengine) = $rb;
>> $var(ret) = rtpengine_manage("direction=dirty
>> direction=clean ICE=remove");
>> xlog("L_INFO", "$ci INVITE: rtpengine chosen:
>> $avp(rtpInstance)");
>> remove_body();
>> replace_body(".*", $var(sdpFromRtpengine));
>> t_on_reply("RESPONSE");
>>
>> route(RELAY);
>> }
>>
>> onreply_route[RESPONSE]
>> {
>> $var(sdpToRtpengine) = $rb;
>> $var(ret) = rtpengine_manage("direction=clean
>> direction=dirty ICE=remove");
>> remove_body();
>> replace_body(".*", $var(sdpFromRtpengine));
>> xlog("L_INFO", "$ci RESPONSE: $rm - $rs $rr, cseq=$cs,
>> by [$hdr(Server)], from $si:$sp");
>> }
>> ...
>>
>> }}}
>>
>>
>> When rtpengine is down for a couple of seconds, there were a lot
>> of SIP retransmissions and timeouts. Doing a netstat and I can
>> see Kamailio’s receive buffer is quite filled up.
>>
>>
>>
>> Please let me know if more information is needed.Thank you!
>>
>>
>>
>> Cheers,
>>
>> Yufei
>>
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com <http://www.kamailioworld.com>
> Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in Washington, DC, USA -- www.asipto.com <http://www.asipto.com>
>
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
Kamailio Advanced Training - Mar 4-6, 2019 in Berlin; Mar 25-27, 2019, in Washington, DC, USA -- www.asipto.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190123/6bde02a5/attachment.html>
More information about the sr-users
mailing list