16 Sep
2015
16 Sep
'15
10 a.m.
Hi,
we have a load balancer which is handling a lot of SIP traffic all day.
There's always 20-40 Mbit SIP traffic going through. From time to time we
see in our logs messages like these:
Sep 16 09:46:28 ecker /usr/sbin/kamailio[25505]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f2d9d6b3ce0,1321,0,46.237.225.126:5060,16): Operation not
permitted(1)
Sep 16 09:46:38 ecker /usr/sbin/kamailio[25194]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7efc982b8fc8,420,0,82.113.121.183:35794,16): Operation not
permitted(1)
Sep 16 09:46:40 ecker /usr/sbin/kamailio[25505]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f2d9d6b3ce0,1338,0,5.158.137.9:55067,16): Operation not
permitted(1)
Sep 16 09:46:44 ecker /usr/sbin/kamailio[25183]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7efc982d9f48,450,0,178.165.131.197:37515,16): Operation not
permitted(1)
Sep 16 09:46:49 ecker /usr/sbin/kamailio[25643]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f93fb624530,496,0,172.56.7.69:25643,16): Operation not
permitted(1)
Sep 16 09:46:55 ecker /usr/sbin/kamailio[25335]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f41632cda98,598,0,80.215.234.139:3396,16): Operation not
permitted(1)
Sep 16 09:46:56 ecker /usr/sbin/kamailio[25345]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f41632f4840,459,0,94.197.120.191:8225,16): Operation not
permitted(1)
I know that these messages can be produced by iptables blocking the
outbound traffic. But our outbound chain looks basically like this:
root@ecker:~# iptables-save | grep OUTPUT
:OUTPUT DROP [0:0]
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-j ACCEPT
-A OUTPUT -m state --state INVALID -j DROP
-A OUTPUT -o lo -m state --state NEW -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m state --state NEW -j ACCEPT
We don't have the nf_ct_sip module loaded, syslog doesn't say anything, and
even clearing all iptables rules doesn't eliminate those errors.
Has anyone ever seen this? It looks like a load thing, because at weekends
there are significantly less errors.
Thanks,
Sebastian
16 Sep
16 Sep
10:21 a.m.
Am 16.09.2015 um 10:00 schrieb Sebastian Damm:
Has anyone ever seen this? It looks like a load thing,
because at
weekends there are significantly less errors.
Could it be that you are trying to send from an ip address that is down
at the time?
We see log messages like this on the standby machine that doesn't own
the network interfaces. It'd be nice if the Kamailio dispatcher module
could be told not do health checks temporarily.
-Sven
10:29 a.m.
On Wed, Sep 16, 2015 at 10:21 AM, Sven Neuhaus <neuhaus(a)tyntec.com> wrote:
Could it be that you are trying to send from an ip
address that is down
at the time?
Interesting thought, but all interfaces on the machine are up at this time.
And we are not manipulating the sending socket, so the address where the
packet comes into the system is the same where it is sent out from.
Since I forgot to mention it: The system is running Debian Wheezy with it's
standard kernel.
Best Regards
Sebastian
10:44 a.m.
On Wednesday 16 September 2015 10:00:53 Sebastian Damm wrote:
Has anyone ever seen this? It looks like a load thing,
because at weekends
there are significantly less errors.
You should look at the OS level, the error is from the kernel.
Are you runing out of sockets/files? It the connection tracker full?
BTW you accept related and new state, this makes no sense, you could just as
well have no rules for the OUTPUT chain (which is much better for perfomance).
12:15 p.m.
On Wed, Sep 16, 2015 at 10:44 AM, Daniel Tryba <d.tryba(a)pocos.nl> wrote:
You should look at the OS level, the error is from the
kernel.
I know, but dmesg, syslog or kernel log don't say anything.
Are you runing out of sockets/files? It the connection
tracker full?
The connection tracking table is monitored and never close to full. How
could I check the sockets/files?
BTW you accept related and new state, this makes no
sense, you could just
as
well have no rules for the OUTPUT chain (which is much better for
perfomance).
I know. My old hand-written firewall was much smaller and almost stateless.
But according to our administrators policy all firewalls should be
generated by FWbuilder, which generates pretty ugly rules, and also
implicitly injects the related rule. (I'm not really happy with that.)
Sebastian
6:02 p.m.
On 9/16/15 4:00 AM, Sebastian Damm wrote:
Hi,
we have a load balancer which is handling a lot of SIP traffic all
day. There's always 20-40 Mbit SIP traffic going through. From time to
time we see in our logs messages like these:
Sep 16 09:46:28 ecker /usr/sbin/kamailio[25505]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f2d9d6b3ce0,1321,0,46.237.225.126:5060
<http://46.237.225.126:5060>,16): Operation not permitted(1)
Sep 16 09:46:38 ecker /usr/sbin/kamailio[25194]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7efc982b8fc8,420,0,82.113.121.183:35794
<http://82.113.121.183:35794>,16): Operation not permitted(1)
Sep 16 09:46:40 ecker /usr/sbin/kamailio[25505]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f2d9d6b3ce0,1338,0,5.158.137.9:55067
<http://5.158.137.9:55067>,16): Operation not permitted(1)
Sep 16 09:46:44 ecker /usr/sbin/kamailio[25183]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7efc982d9f48,450,0,178.165.131.197:37515
<http://178.165.131.197:37515>,16): Operation not permitted(1)
Sep 16 09:46:49 ecker /usr/sbin/kamailio[25643]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f93fb624530,496,0,172.56.7.69:25643
<http://172.56.7.69:25643>,16): Operation not permitted(1)
Sep 16 09:46:55 ecker /usr/sbin/kamailio[25335]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f41632cda98,598,0,80.215.234.139:3396
<http://80.215.234.139:3396>,16): Operation not permitted(1)
Sep 16 09:46:56 ecker /usr/sbin/kamailio[25345]: ERROR: <core>
[udp_server.c:591]: udp_send(): ERROR: udp_send:
sendto(sock,0x7f41632f4840,459,0,94.197.120.191:8225
<http://94.197.120.191:8225>,16): Operation not permitted(1)
I know that these messages can be produced by iptables blocking the
outbound traffic. But our outbound chain looks basically like this:
root@ecker:~# iptables-save | grep OUTPUT
:OUTPUT DROP [0:0]
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-j ACCEPT
-A OUTPUT -m state --state INVALID -j DROP
-A OUTPUT -o lo -m state --state NEW -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m state --state NEW -j ACCEPT
We don't have the nf_ct_sip module loaded, syslog doesn't say
anything, and even clearing all iptables rules doesn't eliminate those
errors.
Has anyone ever seen this? It looks like a load thing, because at
weekends there are significantly less errors.
Try to see if the transmit queue is
high (third column) using netstat.
You always want that to drop down to zero fast when it goes up. If it
starts to grow, you definitely have a problem between the application
and operating system.
# netstat -aupn | grep kama
udp 0 0 23.239.16.141:50963 23.239.16.141:22555
ESTABLISHED 26101/kamailio
udp 0 0 23.239.16.141:50195 198.58.110.117:22555
ESTABLISHED 26081/kamailio
udp 0 0 23.239.16.141:42772 198.58.110.117:22555
ESTABLISHED 26122/kamailio
udp 0 0 23.239.16.141:49046 23.239.16.141:22555
ESTABLISHED 26112/kamailio
udp 0 0 23.239.16.141:49181 198.58.110.117:22555
ESTABLISHED 26118/kamailio
udp 0 0 23.239.16.141:36898 23.239.16.141:22555
ESTABLISHED 26122/kamailio
udp 0 0 23.239.16.141:59813 23.239.16.141:22555
ESTABLISHED 26120/kamailio
udp 0 0 23.239.16.141:36016 23.239.16.141:22555
ESTABLISHED 26110/kamailio
udp 0 0 23.239.16.141:47541 198.58.110.117:22555
ESTABLISHED 26103/kamailio
udp 0 0 23.239.16.141:60727 23.239.16.141:22555
ESTABLISHED 26096/kamailio
udp 0 0 23.239.16.141:52282 198.58.110.117:22555
ESTABLISHED 26120/kamailio
udp 0 0 23.239.16.141:57275 198.58.110.117:22555
ESTABLISHED 26114/kamailio
udp 0 0 23.239.16.141:51137 23.239.16.141:22555
ESTABLISHED 26114/kamailio
udp 0 0 23.239.16.141:53574 23.239.16.141:22555
ESTABLISHED 26095/kamailio
udp 0 0 23.239.16.141:5065
0.0.0.0:* 26081/kamailio
Thanks,
Sebastian
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Technical Support
http://www.cellroute.net
11 Mar
11 Mar
2:49 p.m.
Hi,
just to resolve this thread, we found the reason for the problem. It
occurs, when we try sending out packets to a customer, which look
identical to netfilter, at roughly the same time. Those could be for
example forked calls to two extensions registered on the same device
(a FRITZ Box for example). Then netfilter tries to insert the same
packet into its conntrack table twice, causing a collision, leading to
a rejection of one of the packets.
We played around with different kernels, without success. The errors
kept on coming as long as the nf_conntrack module was loaded, even if
there was no iptables rule using it.
The only solution right now seems to be a stateless firewall and
unloading the module.
Best Regards,
Sebastian
6:07 p.m.
Hello,
this proves again my theory that best options one could get for contrack
and selinux is to disable them completely ...
Anyhow, great that you reported back, I am sure it will help others over
the time.
Cheers,
Daniel
On 11/03/16 14:49, Sebastian Damm wrote:
Hi,
just to resolve this thread, we found the reason for the problem. It
occurs, when we try sending out packets to a customer, which look
identical to netfilter, at roughly the same time. Those could be for
example forked calls to two extensions registered on the same device
(a FRITZ Box for example). Then netfilter tries to insert the same
packet into its conntrack table twice, causing a collision, leading to
a rejection of one of the packets.
We played around with different kernels, without success. The errors
kept on coming as long as the nf_conntrack module was loaded, even if
there was no iptables rule using it.
The only solution right now seems to be a stateless firewall and
unloading the module.
Best Regards,
Sebastian
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, Berlin, May 18-20, 2016 - http://www.kamailioworld.com
2966
days inactive
3143
days old
7 comments
5 participants
participants (5)
-
Andres -
Daniel Tryba -
Daniel-Constantin Mierla -
Sebastian Damm -
Sven Neuhaus