5 Apr
2017
5 Apr
'17
1:44 a.m.
Recently, I've had to be on a lot of long conference calls and have noticed
that after about 18-22 minutes, the audio on the call drops out. There are no
errors logged from Asterisk, Kamailio, or RTPengine. The signalling portion
of the call is unaffected and continues as normal.
My setup is as follows
asterisk-14.3.0-7.gita047b25.fc25.x86_64 (14 branch)
kamailio-5.0.0-5.git2608015.fc25.x86_64 (5.0 branch)
kernel-4.10.8-200.fc25.x86_64 (stable Fedora kernel)
rtpengine-5.1.1.1-9.git27af349.fc25.x86_64 (master branch)
Yealink or Zoiper softphones
Calls are originated as follows using TLS & SRTP
Yealink/Zoiper -> Kamailio/RTPengine+kernel -> Asterisk/DAHDI/PSTN
Related IPtables output below (generated by firewalld)
~]# iptables -S |grep INPUT
-P INPUT ACCEPT
-N INPUT_ZONES
-N INPUT_ZONES_SOURCE
-N INPUT_direct
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT_direct -p udp -m udp --dport 30000:40000 -j RTPENGINE --id 0
-A INPUT_ZONES_SOURCE -m set --match-set mss4 src -g IN_mss
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_allow
-A IN_public_allow -p udp -m udp --dport 30000:40000 -m conntrack \
--ctstate NEW -j ACCEPT
I saw https://github.com/sipwise/rtpengine/issues/102 and
https://github.com/sipwise/rtpengine/issues/108, but do not seem to have any
issues (that I know of) with the SRTP, though the latter ticket references
"call that would last longer than 20 minutes".
Can you share suggestions for looking into the cause of the audio drops,
especially as nothing appears in the logs?
Thanks. -A
--
Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery
F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6
5 Apr
5 Apr
4:33 a.m.
On Tuesday, April 4, 2017 5:44:47 PM CDT Anthony Joseph Messina wrote:
Recently, I've had to be on a lot of long
conference calls and have noticed
that after about 18-22 minutes, the audio on the call drops out. There are
no errors logged from Asterisk, Kamailio, or RTPengine. The signalling
portion of the call is unaffected and continues as normal.
My setup is as follows
asterisk-14.3.0-7.gita047b25.fc25.x86_64 (14 branch)
kamailio-5.0.0-5.git2608015.fc25.x86_64 (5.0 branch)
kernel-4.10.8-200.fc25.x86_64 (stable Fedora kernel)
rtpengine-5.1.1.1-9.git27af349.fc25.x86_64 (master branch)
Yealink or Zoiper softphones
Calls are originated as follows using TLS & SRTP
Yealink/Zoiper -> Kamailio/RTPengine+kernel -> Asterisk/DAHDI/PSTN
Related IPtables output below (generated by firewalld)
~]# iptables -S |grep INPUT
-P INPUT ACCEPT
-N INPUT_ZONES
-N INPUT_ZONES_SOURCE
-N INPUT_direct
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT_direct -p udp -m udp --dport 30000:40000 -j RTPENGINE --id 0
-A INPUT_ZONES_SOURCE -m set --match-set mss4 src -g IN_mss
-A INPUT_ZONES -g IN_public
-A IN_public -j IN_public_allow
-A IN_public_allow -p udp -m udp --dport 30000:40000 -m conntrack \
--ctstate NEW -j ACCEPT
I saw https://github.com/sipwise/rtpengine/issues/102 and
https://github.com/sipwise/rtpengine/issues/108, but do not seem to have any
issues (that I know of) with the SRTP, though the latter ticket references
"call that would last longer than 20 minutes".
Can you share suggestions for looking into the cause of the audio drops,
especially as nothing appears in the logs?
After more digging, I see (from the Asterisk perspective) that after a certain
amount of time, the "RTCP report" size gets smaller and this is the point at
which the audio from Asterisk back to the softphone is dropped. Again, this
audio drop occurred around 19 minutes into the call.
I'm not sure this means anything, but perhaps it can point someone more
knowledgeable in the right direction.
The following is debug output from Asterisk
[Apr 4 20:05:33] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 88 bytes
[Apr 4 20:05:38] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 72 bytes
[Apr 4 20:05:43] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 72 bytes
[Apr 4 20:05:48] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 92 bytes
[Apr 4 20:05:53] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 72 bytes
[Apr 4 20:05:58] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 72 bytes
[Apr 4 20:06:03] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 88 bytes
[Apr 4 20:06:08] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 72 bytes
###### This is where the audio stops ######
[Apr 4 20:06:13] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 48 bytes
[Apr 4 20:06:18] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 68 bytes
[Apr 4 20:06:23] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 48 bytes
[Apr 4 20:06:28] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 48 bytes
[Apr 4 20:06:33] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 64 bytes
[Apr 4 20:06:38] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 48 bytes
[Apr 4 20:06:43] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 48 bytes
[Apr 4 20:06:48] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 68 bytes
[Apr 4 20:06:53] res_rtp_asterisk.c:4214 ast_rtcp_interpret: Got RTCP report
of 48 bytes
--
Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery
F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6
4:55 p.m.
On 04/04/2017 09:33 PM, Anthony Joseph Messina wrote:
After more digging, I see (from the Asterisk perspective) that after a certain
amount of time, the "RTCP report" size gets smaller and this is the point at
which the audio from Asterisk back to the softphone is dropped. Again, this
audio drop occurred around 19 minutes into the call.
I'm not sure this means anything, but perhaps it can point someone more
knowledgeable in the right direction.
A good place to start is to inspect /proc/rtpengine/0/list and check the
packet and byte counters for the respective local ports. This way you
can check if incoming packets are actually arriving at rtpengine.
Cheers
9:53 p.m.
On Wednesday, April 5, 2017 8:55:36 AM CDT Richard Fuchs wrote:
On 04/04/2017 09:33 PM, Anthony Joseph Messina wrote:
Thanks, Richard. I am amidst a call right now which shows it's kernelized.
The output from cat /proc/rtpengine/0/list shows nothing changing throughout
the call (after repeating the command).
rtpengine[29508]: INFO: [0_4088656235(a)10.77.79.165 port 30520]: Confirmed peer
address as 10.77.79.3:10646
rtpengine[29508]: INFO: [0_4088656235(a)10.77.79.165 port 30534]: Confirmed peer
address as 10.77.79.165:12668
rtpengine[29508]: INFO: [0_4088656235(a)10.77.79.165 port 30534]: Kernelizing
media stream: 10.77.79.165:12668
rtpengine[29508]: DEBUG: [0_4088656235(a)10.77.79.165 port 30534]: enabling
kernel intercept with stream idx 2
rtpengine[29508]: INFO: [0_4088656235(a)10.77.79.165 port 30520]: Kernelizing
media stream: 10.77.79.3:10646
rtpengine[29508]: DEBUG: [0_4088656235(a)10.77.79.165 port 30520]: enabling
kernel intercept with stream idx 0
rtpengine[29508]: INFO: [0_4088656235(a)10.77.79.165 port 30521]: Confirmed peer
address as 10.77.79.3:10647
rtpengine[29508]: INFO: [0_4088656235(a)10.77.79.165 port 30535]: Confirmed peer
address as 10.77.79.165:12669
~]# cat /proc/rtpengine/0/list
local inet4 10.77.79.3:30520
src inet4 10.77.79.3:30534
dst inet4 10.77.79.165:12668
stats: 0 bytes, 0 packets,
0 errors
RTP payload type 0: 0 bytes, 0
packets
RTP payload type 9: 0 bytes, 0
packets
RTP payload type 101: 0 bytes, 0
packets
RTP payload type 106: 0 bytes, 0
packets
SRTP encryption (outgoing) parameters:
cipher: AES-CM-128
HMAC: HMAC-SHA1
auth tag length: 10
local inet4 10.77.79.3:30534
src inet4 10.77.79.3:30520
dst inet4 10.77.79.3:10646
stats: 0 bytes, 0 packets,
0 errors
RTP payload type 0: 0 bytes, 0
packets
RTP payload type 9: 0 bytes, 0
packets
RTP payload type 101: 0 bytes, 0
packets
RTP payload type 106: 0 bytes, 0
packets
SRTP decryption (incoming) parameters:
cipher: AES-CM-128
HMAC: HMAC-SHA1
auth tag length: 10
--
Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery
F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6
After more digging, I see (from the Asterisk
perspective) that after a
certain amount of time, the "RTCP report" size gets smaller and this is
the point at which the audio from Asterisk back to the softphone is
dropped. Again, this audio drop occurred around 19 minutes into the
call.
I'm not sure this means anything, but perhaps it can point someone more
knowledgeable in the right direction.
A good place to start is to inspect /proc/rtpengine/0/list and check the
packet and byte counters for the respective local ports. This way you
can check if incoming packets are actually arriving at rtpengine.
10:24 p.m.
On 05/04/17 02:53 PM, Anthony Joseph Messina wrote:
On Wednesday, April 5, 2017 8:55:36 AM CDT Richard
Fuchs wrote:
That means your iptables rule isn't
effective. Packets don't get
delivered to the RTPENGINE iptables target. That doesn't explain audio
stopping but that's the first thing you should fix.
Cheers
On 04/04/2017 09:33 PM, Anthony Joseph Messina
wrote:
Thanks, Richard.
I am amidst a call right now which shows it's kernelized.
The output from cat /proc/rtpengine/0/list shows nothing changing throughout
the call (after repeating the command). After more digging, I see (from the Asterisk
perspective) that after a
certain amount of time, the "RTCP report" size gets smaller and this is
the point at which the audio from Asterisk back to the softphone is
dropped. Again, this audio drop occurred around 19 minutes into the
call.
I'm not sure this means anything, but perhaps it can point someone more
knowledgeable in the right direction.
A good place to start is to inspect
/proc/rtpengine/0/list and check the
packet and byte counters for the respective local ports. This way you
can check if incoming packets are actually arriving at rtpengine.
10:54 p.m.
On Wednesday, April 5, 2017 2:24:12 PM CDT Richard Fuchs wrote:
On 05/04/17 02:53 PM, Anthony Joseph Messina wrote:
You are correct about that. firewalld inserts
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
as the first rule in the INPUT chain, so it never got to my
-A INPUT_direct -p udp -m udp -j RTPENGINE --id 0
The following works at least for the kernel module part...
firewall-cmd --permanent --direct --passthrough ipv4 -I INPUT -p udp -m udp -j
RTPENGINE --id 0
firewall-cmd --permanent --direct --passthrough ipv6 -I INPUT -p udp -m udp -j
RTPENGINE --id 0
--
Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery
F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6
On Wednesday, April 5, 2017 8:55:36 AM CDT
Richard Fuchs wrote:
That means your iptables rule isn't effective. Packets don't get
delivered to the RTPENGINE iptables target. That doesn't explain audio
stopping but that's the first thing you should fix. On 04/04/2017 09:33 PM, Anthony Joseph Messina
wrote:
Thanks, Richard. I am amidst a call right now which shows it's kernelized.
The output from cat /proc/rtpengine/0/list shows nothing changing
throughout the call (after repeating the command). After more digging, I see (from the Asterisk
perspective) that after a
certain amount of time, the "RTCP report" size gets smaller and this is
the point at which the audio from Asterisk back to the softphone is
dropped. Again, this audio drop occurred around 19 minutes into the
call.
I'm not sure this means anything, but perhaps it can point someone more
knowledgeable in the right direction.
A good place to start is to inspect /proc/rtpengine/0/list and check the
packet and byte counters for the respective local ports. This way you
can check if incoming packets are actually arriving at rtpengine.
11:56 p.m.
New subject: RTPengine + kernel module long call RTP drops [SOLVED]
On Wednesday, April 5, 2017 2:54:08 PM CDT Anthony Joseph Messina wrote:
Signed by
amessina@messinet.com,webmaster@messinet.com,admin@messinet.com. Show
Details
On Wednesday, April 5, 2017 2:24:12 PM CDT Richard Fuchs wrote:
> On 05/04/17 02:53 PM, Anthony Joseph Messina wrote:
> > On Wednesday, April 5, 2017 8:55:36 AM CDT Richard Fuchs wrote:
> >> On 04/04/2017 09:33 PM, Anthony Joseph Messina wrote:
> >>> After more digging, I see (from the Asterisk perspective) that after a
> >>> certain amount of time, the "RTCP report" size gets smaller
and this
> >>> is
> >>> the point at which the audio from Asterisk back to the softphone is
> >>> dropped. Again, this audio drop occurred around 19 minutes into the
> >>> call.
> >>>
> >>> I'm not sure this means anything, but perhaps it can point someone
> >>> more
> >>> knowledgeable in the right direction.
> >>
> >> A good place to start is to inspect /proc/rtpengine/0/list and check
> >> the
> >> packet and byte counters for the respective local ports. This way you
> >> can check if incoming packets are actually arriving at rtpengine.
> >
> > Thanks, Richard. I am amidst a call right now which shows it's
> > kernelized.
> > The output from cat /proc/rtpengine/0/list shows nothing changing
> > throughout the call (after repeating the command).
>
> That means your iptables rule isn't effective. Packets don't get
> delivered to the RTPENGINE iptables target. That doesn't explain audio
> stopping but that's the first thing you should fix.
Richard, thank you for pointing me in the right direction. firewalld's direct
rule was in fact the problem in my case. As the direct rule
INPUT_direct -p udp -m udp -j RTPENGINE --id 0
was after
INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
it was not allowing the calls to actually kernelize, and at some point the
audio would just drop (still not sure why this would happen).
For the rest of you using firewalld, having the following in /etc/firewalld/
direct.xml works for me.
<?xml version="1.0" encoding="utf-8"?>
<direct>
<!--unrelated direct rules here-->
<passthrough ipv="ipv4">-I INPUT -p udp -m udp --dport 30000:40000 -j
RTPENGINE --id 0</passthrough>
<passthrough ipv="ipv6">-I INPUT -p udp -m udp --dport 30000:40000 -j
RTPENGINE --id 0</passthrough>
</direct>
I've now had several calls last nearly an hour (a first in a long time). I
don't know how I didn't put this together with firewalld upstream reports:
https://github.com/t-woerner/firewalld/issues/44
https://github.com/t-woerner/firewalld/issues/191
Thanks again. -A
You are correct about that. firewalld inserts
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
as the first rule in the INPUT chain, so it never got to my
-A INPUT_direct -p udp -m udp -j RTPENGINE --id 0
The following works at least for the kernel module part...
firewall-cmd --permanent --direct --passthrough ipv4 -I INPUT -p udp -m udp
-j RTPENGINE --id 0
firewall-cmd --permanent --direct --passthrough ipv6 -I INPUT -p udp -m udp
-j RTPENGINE --id 0
--
Anthony - https://messinet.com/ - https://messinet.com/~amessina/gallery
F9B6 560E 68EA 037D 8C3D D1C9 FF31 3BDB D9D8 99B6
2794
days inactive
2795
days old
6 comments
2 participants
participants (2)
-
Anthony Joseph Messina -
Richard Fuchs