sr-dev December 2023

sr-dev@lists.kamailio.org

14 participants
232 discussions

[kamailio/kamailio] Kamailio Fails to Release Connected WebSocket TCP Connections (Issue #3589)

by pklinus

### Description  Kamailio does not properly release connected WebSocket TCP connections, leading to a situation where the maximum number of allowable connections is reached and new connection requests are rejected. ### Troubleshooting When examining the logs, we identified an issue where Kamailio is failing to release connected WebSocket TCP connections, causing a backlog of TIME_WAIT connections. The logs show a warning message indicating that the maximum number of allowable sockets per IP has been reached, leading to the rejection of further WebSocket requests. #### Reproduction Establish WebSocket connections with Kamailio. Monitor the number of TIME_WAIT connections using netstat. Observe the warning message in Kamailio logs when the maximum allowable sockets per IP are reached.  #### Debugging Data  ``` (paste your debugging data here) ``` #### Log Messages  ``` 2023-10-03T13:19:33.699043+00:00 apps001 kamailio[23781]: WARNING: |<null>|websockets-role.cfg:224 (websocket_request) 223.233.86.93 is at the maximum 50 allowable sockets per IP, rejecting request for another websocket netstat -punta | grep 223.233.86.93 | grep 5065 tcp 0 0 45.79.166.214:5065 223.233.86.93:52158 TIME_WAIT - tcp 0 0 45.79.166.214:5065 223.233.86.93:52195 TIME_WAIT - tcp 0 0 45.79.166.214:5065 223.233.86.93:52292 TIME_WAIT - tcp 0 0 45.79.166.214:5065 223.233.86.93:52248 TIME_WAIT - tcp 0 0 45.79.166.214:5065 223.233.86.93:52108 TIME_WAIT - tcp 0 0 45.79.166.214:5065 223.233.86.93:52071 TIME_WAIT - ``` #### SIP Traffic  ``` (paste your sip traffic here) ``` ### Possible Solutions  restarting kamailio solves the problem. ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.7.2 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 14:39:54 Sep 27 2023 with gcc 7.3.1 ``` * **Operating System**:  ``` CentOS Linux release 7.9.2009 (Core) ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3589 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3589(a)github.com>

9 months, 1 week

[kamailio/kamailio] no corresponding socket found error (Issue #3583)

by descartin

Hello all we are seeing these kind of logs in a Debian GNU/Linux 11 using kamailio 5.5.6 ```Sep 21 17:41:13 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155702]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[164.152.22.248:5060](http://164.152.22.248:5060/)) Sep 21 17:41:13 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155698]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[208.74.138.184:5060](http://208.74.138.184:5060/)) Sep 21 17:41:13 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155695]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[208.74.138.181:5060](http://208.74.138.181:5060/)) Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155693]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[38.102.250.60:5060](http://38.102.250.60:5060/)) Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155707]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[87.1.1.27:5060](http://87.1.1.27:5060/)) Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155707]: ERROR: tm [ut.h:302]: uri2dst2(): no corresponding socket for "87.1.1.27" af 2 Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155707]: ERROR: tm [t_fwd.c:470]: prepare_new_uac(): can't fwd to af 2, proto 1 (no corresponding listening socket) Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155697]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[208.74.138.184:5060](http://208.74.138.184:5060/)) Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155694]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[192.40.216.97:5060](http://192.40.216.97:5060/)) Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155694]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[208.74.138.180:5060](http://208.74.138.180:5060/)) Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155714]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[87.1.1.27:5060](http://87.1.1.27:5060/)) ``` The kamailio instance we are using is receiving around 1500 calls per second in average when we do start seeing these errors more frequently In this instance we are using multihomed the listen address list is ``` children=14 socket_workers=1 listen=udp:[192.168.99.70:5081](http://192.168.99.70:5081/) listen=udp:[192.168.99.81:5060](http://192.168.99.81:5060/) listen=udp:[87.1.1.27:5060](http://87.1.1.27:5060/) listen=tcp:[192.168.96.105:8095](http://192.168.96.105:8095/) listen=tcp:[87.1.1.27:5060](http://87.1.1.27:5060/) tcp_children=6 port=5060 ``` We use the first worker only to perform the OPTIONS requests from the dispatcher module. The sockets listen=udp:[192.168.99.81:5060](http://192.168.99.81:5060/) listen=udp:[87.1.1.27:5060](http://87.1.1.27:5060/) are used to communicate both A and B legs so when a message is received at [192.168.99.81:5060](http://192.168.99.81:5060/) we send it to [87.1.1.27:5060](http://87.1.1.27:5060/) and viceversa We were using $fs variable before doing the t_relay() function. This way is working, but when load increases, seems sometimes kamailio doesn't get the proper socket to forward the reply. I think the errors mainly are related to responses which are being forwarded, like if the function get_sock_info_list was not able to retrieve the listen interfaces we have tried to set in the onreply routes the commands set_recv_socket("udp:[192.168.99.81:5060](http://192.168.99.81:5060/)"); set_send_socket("udp:[87.1.1.27:5060](http://87.1.1.27:5060/)"); (when reply goes from private to public domain) and it seems it reduces the number of ``` Sep 21 17:41:13 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155702]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[164.152.22.248:5060](http://164.152.22.248:5060/)) Sep 21 17:41:13 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155698]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[208.74.138.184:5060](http://208.74.138.184:5060/)) Sep 21 17:41:13 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155695]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[208.74.138.181:5060](http://208.74.138.181:5060/)) Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155693]: ERROR: <core> [core/forward.c:183]: get_out_socket(): no corresponding socket found for(udp:[38.102.250.60:5060](http://38.102.250.60:5060/)) errors ``` But i honestly don't know why the ```Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155707]: ERROR: tm [ut.h:302]: uri2dst2(): no corresponding socket for "87.1.1.27" af 2 Sep 21 17:41:14 lax-dedge-1 /usr/local/kamailio/sbin/kamailio[3155707]: ERROR: tm [t_fwd.c:470]: prepare_new_uac(): can't fwd to af 2, proto 1 (no corresponding listening socket) appear ``` We also tried to increase the kernel buffers rmem and wmem from 208KB (as we had by default) to 4MB Do you know a reason which could cause these logs to appear? Any setup that could mitigate them? thanks a lot and regards david escartin -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3583 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3583(a)github.com>

9 months, 1 week

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

sr-dev December 2023