Recently, I've had to be on a lot of long conference calls and have noticed that after about 18-22 minutes, the audio on the call drops out. There are no errors logged from Asterisk, Kamailio, or RTPengine. The signalling portion of the call is unaffected and continues as normal.
My setup is as follows asterisk-14.3.0-7.gita047b25.fc25.x86_64 (14 branch) kamailio-5.0.0-5.git2608015.fc25.x86_64 (5.0 branch) kernel-4.10.8-200.fc25.x86_64 (stable Fedora kernel) rtpengine-5.1.1.1-9.git27af349.fc25.x86_64 (master branch) Yealink or Zoiper softphones
Calls are originated as follows using TLS & SRTP Yealink/Zoiper -> Kamailio/RTPengine+kernel -> Asterisk/DAHDI/PSTN
Related IPtables output below (generated by firewalld) ~]# iptables -S |grep INPUT -P INPUT ACCEPT -N INPUT_ZONES -N INPUT_ZONES_SOURCE -N INPUT_direct -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -j INPUT_direct -A INPUT -j INPUT_ZONES_SOURCE -A INPUT -j INPUT_ZONES -A INPUT -m conntrack --ctstate INVALID -j DROP -A INPUT -j REJECT --reject-with icmp-host-prohibited -A INPUT_direct -p udp -m udp --dport 30000:40000 -j RTPENGINE --id 0 -A INPUT_ZONES_SOURCE -m set --match-set mss4 src -g IN_mss -A INPUT_ZONES -g IN_public -A IN_public -j IN_public_allow -A IN_public_allow -p udp -m udp --dport 30000:40000 -m conntrack \ --ctstate NEW -j ACCEPT
I saw https://github.com/sipwise/rtpengine/issues/102 and https://github.com/sipwise/rtpengine/issues/108, but do not seem to have any issues (that I know of) with the SRTP, though the latter ticket references "call that would last longer than 20 minutes".
Can you share suggestions for looking into the cause of the audio drops, especially as nothing appears in the logs?
Thanks. -A
On Tuesday, April 4, 2017 5:44:47 PM CDT Anthony Joseph Messina wrote:
Recently, I've had to be on a lot of long conference calls and have noticed that after about 18-22 minutes, the audio on the call drops out. There are no errors logged from Asterisk, Kamailio, or RTPengine. The signalling portion of the call is unaffected and continues as normal.
My setup is as follows asterisk-14.3.0-7.gita047b25.fc25.x86_64 (14 branch) kamailio-5.0.0-5.git2608015.fc25.x86_64 (5.0 branch) kernel-4.10.8-200.fc25.x86_64 (stable Fedora kernel) rtpengine-5.1.1.1-9.git27af349.fc25.x86_64 (master branch) Yealink or Zoiper softphones
Calls are originated as follows using TLS & SRTP Yealink/Zoiper -> Kamailio/RTPengine+kernel -> Asterisk/DAHDI/PSTN
Related IPtables output below (generated by firewalld) ~]# iptables -S |grep INPUT -P INPUT ACCEPT -N INPUT_ZONES -N INPUT_ZONES_SOURCE -N INPUT_direct -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -j INPUT_direct -A INPUT -j INPUT_ZONES_SOURCE -A INPUT -j INPUT_ZONES -A INPUT -m conntrack --ctstate INVALID -j DROP -A INPUT -j REJECT --reject-with icmp-host-prohibited -A INPUT_direct -p udp -m udp --dport 30000:40000 -j RTPENGINE --id 0 -A INPUT_ZONES_SOURCE -m set --match-set mss4 src -g IN_mss -A INPUT_ZONES -g IN_public -A IN_public -j IN_public_allow -A IN_public_allow -p udp -m udp --dport 30000:40000 -m conntrack \ --ctstate NEW -j ACCEPT
I saw https://github.com/sipwise/rtpengine/issues/102 and https://github.com/sipwise/rtpengine/issues/108, but do not seem to have any issues (that I know of) with the SRTP, though the latter ticket references "call that would last longer than 20 minutes".
Can you share suggestions for looking into the cause of the audio drops, especially as nothing appears in the logs?
After more digging, I see (from the Asterisk perspective) that after a certain amount of time, the "RTCP report" size gets smaller and this is the point at which the audio from Asterisk back to the softphone is dropped. Again, this audio drop occurred around 19 minutes into the call.
I'm not sure this means anything, but perhaps it can point someone more knowledgeable in the right direction.
The following is debug output from Asterisk
[Apr 4 20:05:33] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 88 bytes [Apr 4 20:05:38] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 72 bytes [Apr 4 20:05:43] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 72 bytes [Apr 4 20:05:48] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 92 bytes [Apr 4 20:05:53] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 72 bytes [Apr 4 20:05:58] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 72 bytes [Apr 4 20:06:03] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 88 bytes [Apr 4 20:06:08] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 72 bytes
###### This is where the audio stops ######
[Apr 4 20:06:13] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 48 bytes [Apr 4 20:06:18] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 68 bytes [Apr 4 20:06:23] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 48 bytes [Apr 4 20:06:28] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 48 bytes [Apr 4 20:06:33] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 64 bytes [Apr 4 20:06:38] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 48 bytes [Apr 4 20:06:43] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 48 bytes [Apr 4 20:06:48] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 68 bytes [Apr 4 20:06:53] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report of 48 bytes
On 04/04/2017 09:33 PM, Anthony Joseph Messina wrote:
After more digging, I see (from the Asterisk perspective) that after a certain amount of time, the "RTCP report" size gets smaller and this is the point at which the audio from Asterisk back to the softphone is dropped. Again, this audio drop occurred around 19 minutes into the call.
I'm not sure this means anything, but perhaps it can point someone more knowledgeable in the right direction.
A good place to start is to inspect /proc/rtpengine/0/list and check the packet and byte counters for the respective local ports. This way you can check if incoming packets are actually arriving at rtpengine.
Cheers
On Wednesday, April 5, 2017 8:55:36 AM CDT Richard Fuchs wrote:
On 04/04/2017 09:33 PM, Anthony Joseph Messina wrote:
After more digging, I see (from the Asterisk perspective) that after a certain amount of time, the "RTCP report" size gets smaller and this is the point at which the audio from Asterisk back to the softphone is dropped. Again, this audio drop occurred around 19 minutes into the call.
I'm not sure this means anything, but perhaps it can point someone more knowledgeable in the right direction.
A good place to start is to inspect /proc/rtpengine/0/list and check the packet and byte counters for the respective local ports. This way you can check if incoming packets are actually arriving at rtpengine.
Thanks, Richard. I am amidst a call right now which shows it's kernelized. The output from cat /proc/rtpengine/0/list shows nothing changing throughout the call (after repeating the command).
rtpengine[29508]: INFO: [0_4088656235@10.77.79.165 port 30520]: Confirmed peer address as 10.77.79.3:10646 rtpengine[29508]: INFO: [0_4088656235@10.77.79.165 port 30534]: Confirmed peer address as 10.77.79.165:12668 rtpengine[29508]: INFO: [0_4088656235@10.77.79.165 port 30534]: Kernelizing media stream: 10.77.79.165:12668 rtpengine[29508]: DEBUG: [0_4088656235@10.77.79.165 port 30534]: enabling kernel intercept with stream idx 2 rtpengine[29508]: INFO: [0_4088656235@10.77.79.165 port 30520]: Kernelizing media stream: 10.77.79.3:10646 rtpengine[29508]: DEBUG: [0_4088656235@10.77.79.165 port 30520]: enabling kernel intercept with stream idx 0 rtpengine[29508]: INFO: [0_4088656235@10.77.79.165 port 30521]: Confirmed peer address as 10.77.79.3:10647 rtpengine[29508]: INFO: [0_4088656235@10.77.79.165 port 30535]: Confirmed peer address as 10.77.79.165:12669
~]# cat /proc/rtpengine/0/list local inet4 10.77.79.3:30520 src inet4 10.77.79.3:30534 dst inet4 10.77.79.165:12668 stats: 0 bytes, 0 packets, 0 errors RTP payload type 0: 0 bytes, 0 packets RTP payload type 9: 0 bytes, 0 packets RTP payload type 101: 0 bytes, 0 packets RTP payload type 106: 0 bytes, 0 packets SRTP encryption (outgoing) parameters: cipher: AES-CM-128 HMAC: HMAC-SHA1 auth tag length: 10 local inet4 10.77.79.3:30534 src inet4 10.77.79.3:30520 dst inet4 10.77.79.3:10646 stats: 0 bytes, 0 packets, 0 errors RTP payload type 0: 0 bytes, 0 packets RTP payload type 9: 0 bytes, 0 packets RTP payload type 101: 0 bytes, 0 packets RTP payload type 106: 0 bytes, 0 packets SRTP decryption (incoming) parameters: cipher: AES-CM-128 HMAC: HMAC-SHA1 auth tag length: 10
On 05/04/17 02:53 PM, Anthony Joseph Messina wrote:
On Wednesday, April 5, 2017 8:55:36 AM CDT Richard Fuchs wrote:
On 04/04/2017 09:33 PM, Anthony Joseph Messina wrote:
After more digging, I see (from the Asterisk perspective) that after a certain amount of time, the "RTCP report" size gets smaller and this is the point at which the audio from Asterisk back to the softphone is dropped. Again, this audio drop occurred around 19 minutes into the call.
I'm not sure this means anything, but perhaps it can point someone more knowledgeable in the right direction.
A good place to start is to inspect /proc/rtpengine/0/list and check the packet and byte counters for the respective local ports. This way you can check if incoming packets are actually arriving at rtpengine.
Thanks, Richard. I am amidst a call right now which shows it's kernelized. The output from cat /proc/rtpengine/0/list shows nothing changing throughout the call (after repeating the command).
That means your iptables rule isn't effective. Packets don't get delivered to the RTPENGINE iptables target. That doesn't explain audio stopping but that's the first thing you should fix.
Cheers
On Wednesday, April 5, 2017 2:24:12 PM CDT Richard Fuchs wrote:
On 05/04/17 02:53 PM, Anthony Joseph Messina wrote:
On Wednesday, April 5, 2017 8:55:36 AM CDT Richard Fuchs wrote:
On 04/04/2017 09:33 PM, Anthony Joseph Messina wrote:
After more digging, I see (from the Asterisk perspective) that after a certain amount of time, the "RTCP report" size gets smaller and this is the point at which the audio from Asterisk back to the softphone is dropped. Again, this audio drop occurred around 19 minutes into the call.
I'm not sure this means anything, but perhaps it can point someone more knowledgeable in the right direction.
A good place to start is to inspect /proc/rtpengine/0/list and check the packet and byte counters for the respective local ports. This way you can check if incoming packets are actually arriving at rtpengine.
Thanks, Richard. I am amidst a call right now which shows it's kernelized. The output from cat /proc/rtpengine/0/list shows nothing changing throughout the call (after repeating the command).
That means your iptables rule isn't effective. Packets don't get delivered to the RTPENGINE iptables target. That doesn't explain audio stopping but that's the first thing you should fix.
You are correct about that. firewalld inserts
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
as the first rule in the INPUT chain, so it never got to my
-A INPUT_direct -p udp -m udp -j RTPENGINE --id 0
The following works at least for the kernel module part... firewall-cmd --permanent --direct --passthrough ipv4 -I INPUT -p udp -m udp -j RTPENGINE --id 0
firewall-cmd --permanent --direct --passthrough ipv6 -I INPUT -p udp -m udp -j RTPENGINE --id 0
On Wednesday, April 5, 2017 2:54:08 PM CDT Anthony Joseph Messina wrote:
Signed by amessina@messinet.com,webmaster@messinet.com,admin@messinet.com. Show Details On Wednesday, April 5, 2017 2:24:12 PM CDT Richard Fuchs wrote:
On 05/04/17 02:53 PM, Anthony Joseph Messina wrote:
On Wednesday, April 5, 2017 8:55:36 AM CDT Richard Fuchs wrote:
On 04/04/2017 09:33 PM, Anthony Joseph Messina wrote:
After more digging, I see (from the Asterisk perspective) that after a certain amount of time, the "RTCP report" size gets smaller and this is the point at which the audio from Asterisk back to the softphone is dropped. Again, this audio drop occurred around 19 minutes into the call.
I'm not sure this means anything, but perhaps it can point someone more knowledgeable in the right direction.
A good place to start is to inspect /proc/rtpengine/0/list and check the packet and byte counters for the respective local ports. This way you can check if incoming packets are actually arriving at rtpengine.
Thanks, Richard. I am amidst a call right now which shows it's kernelized. The output from cat /proc/rtpengine/0/list shows nothing changing throughout the call (after repeating the command).
That means your iptables rule isn't effective. Packets don't get delivered to the RTPENGINE iptables target. That doesn't explain audio stopping but that's the first thing you should fix.
Richard, thank you for pointing me in the right direction. firewalld's direct rule was in fact the problem in my case. As the direct rule
INPUT_direct -p udp -m udp -j RTPENGINE --id 0
was after
INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
it was not allowing the calls to actually kernelize, and at some point the audio would just drop (still not sure why this would happen).
For the rest of you using firewalld, having the following in /etc/firewalld/ direct.xml works for me.
<?xml version="1.0" encoding="utf-8"?> <direct> <!--unrelated direct rules here--> <passthrough ipv="ipv4">-I INPUT -p udp -m udp --dport 30000:40000 -j RTPENGINE --id 0</passthrough> <passthrough ipv="ipv6">-I INPUT -p udp -m udp --dport 30000:40000 -j RTPENGINE --id 0</passthrough> </direct>
I've now had several calls last nearly an hour (a first in a long time). I don't know how I didn't put this together with firewalld upstream reports:
https://github.com/t-woerner/firewalld/issues/44 https://github.com/t-woerner/firewalld/issues/191
Thanks again. -A
You are correct about that. firewalld inserts
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
as the first rule in the INPUT chain, so it never got to my
-A INPUT_direct -p udp -m udp -j RTPENGINE --id 0
The following works at least for the kernel module part... firewall-cmd --permanent --direct --passthrough ipv4 -I INPUT -p udp -m udp -j RTPENGINE --id 0
firewall-cmd --permanent --direct --passthrough ipv6 -I INPUT -p udp -m udp -j RTPENGINE --id 0
-
Anthony Joseph Messina -
Richard Fuchs