On 9/16/15 4:00 AM, Sebastian Damm wrote:
Hi,

we have a load balancer which is handling a lot of SIP traffic all day. There's always 20-40 Mbit SIP traffic going through. From time to time we see in our logs messages like these:

Sep 16 09:46:28 ecker /usr/sbin/kamailio[25505]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f2d9d6b3ce0,1321,0,46.237.225.126:5060,16): Operation not permitted(1)
Sep 16 09:46:38 ecker /usr/sbin/kamailio[25194]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7efc982b8fc8,420,0,82.113.121.183:35794,16): Operation not permitted(1)
Sep 16 09:46:40 ecker /usr/sbin/kamailio[25505]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f2d9d6b3ce0,1338,0,5.158.137.9:55067,16): Operation not permitted(1)
Sep 16 09:46:44 ecker /usr/sbin/kamailio[25183]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7efc982d9f48,450,0,178.165.131.197:37515,16): Operation not permitted(1)
Sep 16 09:46:49 ecker /usr/sbin/kamailio[25643]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f93fb624530,496,0,172.56.7.69:25643,16): Operation not permitted(1)
Sep 16 09:46:55 ecker /usr/sbin/kamailio[25335]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f41632cda98,598,0,80.215.234.139:3396,16): Operation not permitted(1)
Sep 16 09:46:56 ecker /usr/sbin/kamailio[25345]: ERROR: <core> [udp_server.c:591]: udp_send(): ERROR: udp_send: sendto(sock,0x7f41632f4840,459,0,94.197.120.191:8225,16): Operation not permitted(1)

I know that these messages can be produced by iptables blocking the outbound traffic. But our outbound chain looks basically like this:

root@ecker:~# iptables-save | grep OUTPUT
:OUTPUT DROP [0:0]
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-j ACCEPT
-A OUTPUT -m state --state INVALID -j DROP
-A OUTPUT -o lo -m state --state NEW -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -m state --state NEW -j ACCEPT

We don't have the nf_ct_sip module loaded, syslog doesn't say anything, and even clearing all iptables rules doesn't eliminate those errors.

Has anyone ever seen this? It looks like a load thing, because at weekends there are significantly less errors.
Try to see if the transmit queue is high (third column) using netstat.  You always want that to drop down to zero fast when it goes up.  If it starts to grow, you definitely have a problem between the application and operating system.

# netstat -aupn | grep kama
udp        0      0 23.239.16.141:50963         23.239.16.141:22555         ESTABLISHED 26101/kamailio     
udp        0      0 23.239.16.141:50195         198.58.110.117:22555        ESTABLISHED 26081/kamailio     
udp        0      0 23.239.16.141:42772         198.58.110.117:22555        ESTABLISHED 26122/kamailio     
udp        0      0 23.239.16.141:49046         23.239.16.141:22555         ESTABLISHED 26112/kamailio     
udp        0      0 23.239.16.141:49181         198.58.110.117:22555        ESTABLISHED 26118/kamailio     
udp        0      0 23.239.16.141:36898         23.239.16.141:22555         ESTABLISHED 26122/kamailio     
udp        0      0 23.239.16.141:59813         23.239.16.141:22555         ESTABLISHED 26120/kamailio     
udp        0      0 23.239.16.141:36016         23.239.16.141:22555         ESTABLISHED 26110/kamailio     
udp        0      0 23.239.16.141:47541         198.58.110.117:22555        ESTABLISHED 26103/kamailio     
udp        0      0 23.239.16.141:60727         23.239.16.141:22555         ESTABLISHED 26096/kamailio     
udp        0      0 23.239.16.141:52282         198.58.110.117:22555        ESTABLISHED 26120/kamailio     
udp        0      0 23.239.16.141:57275         198.58.110.117:22555        ESTABLISHED 26114/kamailio     
udp        0      0 23.239.16.141:51137         23.239.16.141:22555         ESTABLISHED 26114/kamailio     
udp        0      0 23.239.16.141:53574         23.239.16.141:22555         ESTABLISHED 26095/kamailio     
udp        0      0 23.239.16.141:5065          0.0.0.0:*                               26081/kamailio 



Thanks,
Sebastian


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


-- 
Technical Support
http://www.cellroute.net