11 Oct
2023
11 Oct
'23
11:59 a.m.
Hello!
It often happens to me that the service begins to refuse service (503) and so far I have
not been able to understand the exact reason. Typically, before clients start receiving
503, many before them receive a 408 code. As soon as the service starts responding with
503, it does not recover on its own, you have to restart. There is no way to understand
from the logs what led to this (nothing unusual). My server is powerful enough for the
number of users of my service (what I mean is that there is no problem with resources).
For example, 64 cores, 125GB of DDR4 RAM, 2TB of disk, of which 1TB SSD for the
database,
10Gigabit channel, this is only a server for Kamailio for Rtpengines separate servers.
Please help me figure out what causes a service failure!?
OS: Debian 11, 64 cores, 125GB of DDR4 RAM; DB: MariaDB 10.5.15;
kamailio -v
version: kamailio 5.5.3 (x86_64/linux) 473cef
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE
65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
udp_children: 30; tcp_children: 34; TLS: YES; async_workers: 64.
Incoming calls are sent via push notifications( Federico Cabiddu method:
https://www.voztovoice.org/sites/default/files/KamilioWorld2015%20-Federico…).
NetBridging(for SIP and RTPEngine).
ims_charging for billing (integration with our billing system using the Diameter
protocol).
In the logs exactly at the moment when the service began to refuse, the recording
disappears completely for 7
minutes, absolutely no entries, after which the recording of the TCP queue full begins.
Oct 3 18:38:07 sip1-life3 kamailio[3380553]: CRITICAL: {2 3691 INVITE
672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer():
tcp_send failed
Oct 3 18:38:09 sip1-life3 kamailio[3380539]: CRITICAL: {2 3691 INVITE
672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer():
tcp_send failed
Oct 3 18:38:12 sip1-life3 kamailio[3380534]: CRITICAL: {2 30106 INVITE
032f8650-1d90-49ed-82f1-c50ccbef191d} tm [../../core/forward.h:292]: msg_send_buffer():
tcp_send failed
Oct 3 18:45:14 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 285 requests
queued (total handled 2899418)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 286 requests
queued (total handled 2964063)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 286 requests
queued (total handled 2923484)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 286 requests
queued (total handled 2911633)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 286 requests
queued (total handled 2907132)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 286 requests
queued (total handled 2903281)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 286 requests
queued (total handled 2903438)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 286 requests
queued (total handled 2899419)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 286 requests
queued (total handled 2897842)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 286 requests
queued (total handled 2895429)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 286 requests
queued (total handled 2895303)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 22, socket 246: queue full, 286 requests
queued (total handled 2891873)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 26, socket 254: queue full, 286 requests
queued (total handled 2892619)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 27, socket 256: queue full, 286 requests
queued (total handled 2891769)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry
0,
retries 98, connection 0x7f0b6d974348, tcp so
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 30, socket 262: queue full, 286 requests
queued (total handled 2891614)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 31, socket 264: queue full, 286 requests
queued (total handled 2890778)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 33, socket 268: queue full, 286 requests
queued (total handled 2890852)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 0, socket 202: queue full, 287 requests
queued (total handled 3071438)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 287 requests
queued (total handled 2964064)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 2, socket 206: queue full, 287 requests
queued (total handled 2944928)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 287 requests
queued (total handled 2923485)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 5, socket 212: queue full, 287 requests
queued (total handled 2915778)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 287 requests
queued (total handled 2911634)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 287 requests
queued (total handled 2907133)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 287 requests
queued (total handled 2903282)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 287 requests
queued (total handled 2903439)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 10, socket 222: queue full, 287 requests
queued (total handled 2900227)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 287 requests
queued (total handled 2899420)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 287 requests
queued (total handled 2897843)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 204 , queue entry
0,
retries 88, connection 0x7f0b742a1968, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 13, socket 228: queue full, 287 requests
queued (total handled 2896210)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 210 , queue entry
0,
retries 86, connection 0x7f0b74ffaa08, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 287 requests
queued (total handled 2895430)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 214 , queue entry
0,
retries 88, connection 0x7f0b75003218, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 216 , queue entry
1,
retries 86, connection 0x7f0b75007620, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 218 , queue entry
2,
retries 84, connection 0x7f0b74772f70, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 287 requests
queued (total handled 2895304)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 220 , queue entry
0,
retries 87, connection 0x7f0b74777378, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry
1,
retries 83, connection 0x7f0b7477fb88, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 21, socket 244: queue full, 287 requests
queued (total handled 2893042)
11 Oct
11 Oct
1:18 p.m.
Hello,
Kamailio 5.5.x series are no longer maintained, you should upgrade to a
supported version, for example 5.7.x series.
Then, it seems there is a problem with transmission, either the targets
of tcp/tls connections do not accept them, or the corresponding
connections get stuck.
If you look at kamailio core cookbook, there are some parameters to tune
the size of outgoing tcp buffers, but probably getting the queue filled
up is the effect of another cause and trying to handle better the effect
is not going to help much in long term, just prolong a bit more. You
should figure out the cause, maybe you have end points behind the nat
and try to open connections towards them?
You can also look at the output generated by "kamctl trap" (it requires
gdb) to see what each kamailio process is doing. You can also increase
the debug level when such log messages appear to get more details in the
syslog (there is a rpc command to change debug level at runtime without
restart).
Cheers,
Daniel
On 11.10.23 11:59, Masud via sr-users wrote:
Hello!
It often happens to me that the service begins to refuse service (503) and so far I
have
not been able to understand the exact reason. Typically, before clients start receiving
503, many before them receive a 408 code. As soon as the service starts responding with
503, it does not recover on its own, you have to restart. There is no way to understand
from the logs what led to this (nothing unusual). My server is powerful enough for the
number of users of my service (what I mean is that there is no problem with resources).
For example, 64 cores, 125GB of DDR4 RAM, 2TB of disk, of which 1TB SSD for the
database,
10Gigabit channel, this is only a server for Kamailio for Rtpengines separate servers.
Please help me figure out what causes a service failure!?
OS: Debian 11, 64 cores, 125GB of DDR4 RAM; DB: MariaDB 10.5.15;
kamailio -v
version: kamailio 5.5.3 (x86_64/linux) 473cef
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE
65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
udp_children: 30; tcp_children: 34; TLS: YES; async_workers: 64.
Incoming calls are sent via push notifications( Federico Cabiddu method:
https://www.voztovoice.org/sites/default/files/KamilioWorld2015%20-Federico…).
NetBridging(for SIP and RTPEngine).
ims_charging for billing (integration with our billing system using the Diameter
protocol).
In the logs exactly at the moment when the service began to refuse, the recording
disappears completely for 7
minutes, absolutely no entries, after which the recording of the TCP queue full begins.
Oct 3 18:38:07 sip1-life3 kamailio[3380553]: CRITICAL: {2 3691 INVITE
672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer():
tcp_send failed
Oct 3 18:38:09 sip1-life3 kamailio[3380539]: CRITICAL: {2 3691 INVITE
672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer():
tcp_send failed
Oct 3 18:38:12 sip1-life3 kamailio[3380534]: CRITICAL: {2 30106 INVITE
032f8650-1d90-49ed-82f1-c50ccbef191d} tm [../../core/forward.h:292]: msg_send_buffer():
tcp_send failed
Oct 3 18:45:14 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 285
requests
queued (total handled 2899418)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 286 requests
queued (total handled 2964063)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 286 requests
queued (total handled 2923484)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 286 requests
queued (total handled 2911633)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 286 requests
queued (total handled 2907132)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 286 requests
queued (total handled 2903281)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 286 requests
queued (total handled 2903438)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 286
requests
queued (total handled 2899419)
Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 286
requests
queued (total handled 2897842)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 286
requests
queued (total handled 2895429)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 286
requests
queued (total handled 2895303)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 22, socket 246: queue full, 286
requests
queued (total handled 2891873)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 26, socket 254: queue full, 286
requests
queued (total handled 2892619)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 27, socket 256: queue full, 286
requests
queued (total handled 2891769)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry
0,
retries 98, connection 0x7f0b6d974348, tcp so
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 30, socket 262: queue full, 286
requests
queued (total handled 2891614)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 31, socket 264: queue full, 286
requests
queued (total handled 2890778)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 33, socket 268: queue full, 286
requests
queued (total handled 2890852)
Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 0, socket 202: queue full, 287 requests
queued (total handled 3071438)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 287 requests
queued (total handled 2964064)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 2, socket 206: queue full, 287 requests
queued (total handled 2944928)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 287 requests
queued (total handled 2923485)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 5, socket 212: queue full, 287 requests
queued (total handled 2915778)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 287 requests
queued (total handled 2911634)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 287 requests
queued (total handled 2907133)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 287 requests
queued (total handled 2903282)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 287 requests
queued (total handled 2903439)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 10, socket 222: queue full, 287
requests
queued (total handled 2900227)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 287
requests
queued (total handled 2899420)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 287
requests
queued (total handled 2897843)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 204 , queue entry
0,
retries 88, connection 0x7f0b742a1968, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 13, socket 228: queue full, 287
requests
queued (total handled 2896210)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 210 , queue entry
0,
retries 86, connection 0x7f0b74ffaa08, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 287
requests
queued (total handled 2895430)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 214 , queue entry
0,
retries 88, connection 0x7f0b75003218, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 216 , queue entry
1,
retries 86, connection 0x7f0b75007620, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 218 , queue entry
2,
retries 84, connection 0x7f0b74772f70, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 287
requests
queued (total handled 2895304)
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 220 , queue entry
0,
retries 87, connection 0x7f0b74777378, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR:
[core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry
1,
retries 83, connection 0x7f0b7477fb88, tcp so
Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL:
[core/tcp_main.c:4170]: send2child(): tcp child 21, socket 244: queue full, 287
requests
queued (total handled 2893042)
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy and Development Services
Kamailio Advanced Training - Online - Nov 14-16, 2023 -- asipto.com
2:27 p.m.
Thank you very much for your quick response!
I am considering the option of switching to 5.7, but I would like to understand this error
now and fix it.
Most of the clients are behind NAT, but Kamailio and RTPEngiene have a white IP, and
billing to whom Kamailio contacts to receive a quota and further billing are located in
the same subnet. The database is on the same server as Kamailio.
There is one more point that I did not mention above, before the 2nd accident occurs:
cdp[api_process.c:110] api_callback(): Recived diameter response outside of threshold
(500) - [520-800].
CDP config:
Vendor_Id="10415"
Product_Name="CDiameterPeer"
AcceptUnknownPeers="0"
DropUnknownOnDisconnect="0"
Tc="30"
Workers="10"
QueueLength="2"
ConnectTimeout="5"
TransactionTimeout="5"
SessionsHashSize="128"
DefaultAuthSessionTimeout="60"
MaxAuthSessionTimeout="300"
Are these parameters enough for cps 70-100?
I tried "kamctl trap" several times but couldn't figure it out. The main
question that worries me is why, when the workers are full, the service freezes and does
not recover on its own? Only restarting the service helps.
14 Oct
14 Oct
10:11 p.m.
Hi,
I would go with the advise from Daniel to upgrade your Kamailio and if the
error stays then come back or fill a issue on Github.
And your server is really, really good sized for your use. I use VMs with
much less specs and a higher CPS.
Kind regards
Masud via sr-users <sr-users(a)lists.kamailio.org> schrieb am Fr., 13. Okt.
2023, 09:10:
Hello.
Daniel can you help with these questions?)
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
16 Oct
16 Oct
6:59 a.m.
Hi Karsten,
I will definitely switch to the current version later, but I am wondering if I have
configured the Kamailio and CDP module configs correctly enough for my load?
15 Oct
15 Oct
6 a.m.
Hello,
Try to check the limits to kamailio processes in OS
cat /proc/pid/limits
On Fri, 13 Oct 2023 at 13:10 Masud via sr-users <sr-users(a)lists.kamailio.org>
wrote:
Hello.
Daniel can you help with these questions?)
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
16 Oct
16 Oct
6:51 a.m.
Hello,
cat /proc/3461653/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 514004 514004 processes
Max open files 65000 65000 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 514004 514004 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
217
days inactive
222
days old
7 comments
4 participants
participants (4)
-
Daniel-Constantin Mierla -
Karsten Horsmann -
masoud96＠mail.ru -
Nick F