we are getting the following errors in Kamailio logs
**kamailio.log**
Jun 29 08:08:14 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7f524f7862d0, fd 30406 Jun 29 08:08:14 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7f525a6c5f90, fd 30407 Jun 29 08:08:14 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7f52449e0978, fd 30415
Jun 29 08:06:27 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 30355, not in [0, 4607) Jun 29 08:06:27 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 30356, not in [0, 4607) Jun 29 08:06:27 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 30358, not in [0, 4607)
**Description**
On checking this issue we found that Kamailio was not accepting TLS connections. We were getting an error which is mentioned above.
We double-checked TLS and TCP connections and they were normal. We have 5000+ TLS connections and the max connection limit is 30,000. So we were not exceeding the TLS connection limit.
We also verified open_files_limits. It is around 1 million which is more than enough.
**Kamailio Version** - output of kamailio -v
version: kamailio 5.4.2 (x86_64/linux) f8885c flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: f8885c
**Operating System:**
NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"
What is the output of:
``` kamctl rpc core.tcp_info ```
actually, we don't have jsonrpcs module loaded so command throwing following error
/usr/local/fep-kamailio/sbin/kamctl rpc core.tcp_info ERROR: Error opening Kamailio's FIFO /run/kamailio/kamailio_rpc.fifo ERROR: Make sure you have loaded the jsonrpcs module and set FIFO transport parameters
Is your config making other tcp connections from modules? What is the list of loaded modules?
``` grep loadmodule kamailio.cfg ```
list of modules
loadmodule "cfgutils.so" loadmodule "db_mysql.so" loadmodule "xhttp.so" loadmodule "jsonrpcs.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "maxfwd.so" loadmodule "textops.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "ctl.so" loadmodule "counters.so" loadmodule "sqlops.so" loadmodule "http_client.so" loadmodule "permissions.so" #loadmodule "alias_db.so" loadmodule "nathelper.so" loadmodule "rtpproxy.so" loadmodule "htable.so" loadmodule "pike.so" loadmodule "sdpops.so" #loadmodule "debugger.so" loadmodule "ipops.so" loadmodule "tcpops.so" loadmodule "avpops.so" loadmodule "tls.so" loadmodule "websocket.so" loadmodule "textopsx.so"
Only http_client does own connections, is it configure to reuse connections or creates one each time?
no, we don't do that if you are talking about the following param
modparam("http_client", "keep_connections", 1)
It is strange that the file descriptor value in the error messages is so high (e.g., 30406), given the open connections is not close to the limit:
``` Jun 29 08:08:14 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7f524f7862d0, fd 30406 ```
The POSIX requires to use always the lowest available file descriptor value - `File Descriptor Allocation`:
* https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#ta...
Maybe the implementation in your OS has bugs or doesn't follow the POSIX. You can try to investigate the list of file descriptors listed in the proc for Kamailio processes, should be in paths like:
``` /proc/_PID_/fd/ ```
And you have to replace `_PID_` with the values corresponding to Kamailio processes, interesting would be for `tcp main process` (usually the last process in `kamctl ps`) and for the `tcp receiver ...`.
Were you able to gather more details for file descriptors based on my remarks above?
Closing it -- more than two months without providing requested data. Issue can be reopened if troubleshooting data is posted here as requested above.
Closed #2796.
-
aatif218 -
Daniel-Constantin Mierla