Hello,
We have a production Kamailio 5.2.7 server which has suddenly started reporting spikes in the shared memory use. When looking into this, we noticed that at the same time as the memory spikes Kamailio also logs a huge amount of errors like the following.
Can anyone help with an explanation for what's going on? The xx.xx.52.202 address is the Kamailio server itself. We record historical numbers of network connections for the server overall, and that doesn't show any increase in the number of TCP connections, either at the time of the spikes or as a general trend.
Thanks in advance!
Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: <core> [core/tcp_main.c:2703]: tcpconn_1st_send(): connect xx.xx.52.202:42726 failed (RST) Connection refused Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: <core> [core/tcp_main.c:2711]: tcpconn_1st_send(): xx.xx.52.202:42726: connect & send for 0x14bbf44c9f40 failed: Connection refused (111) Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: WARNING: <core> [core/tcp_main.c:1149]: tcp_do_connect(): xx.xx.52.202:42726: could not find corresponding listening socket for xx.xx.52.202, using default...
Hello,
if Kamailio can’t send out messages due to network issues I would expect some spikes in private and/or shared memory usage.
Have you investigated e.g., with some traces to get more information about the possible causes of this issues?
Besides that, with the given information I can only recommend the usual update to the latest stable version (e.g. 5.8.1) as the 5.2.x release series is long end of life.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: David Cunningham via sr-users sr-users@lists.kamailio.org Sent: Freitag, 26. April 2024 02:27 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Cc: David Cunningham dcunningham@voisonics.com Subject: [SR-Users] Sudden TCP errors and shared memory spikes
Hello,
We have a production Kamailio 5.2.7 server which has suddenly started reporting spikes in the shared memory use. When looking into this, we noticed that at the same time as the memory spikes Kamailio also logs a huge amount of errors like the following.
Can anyone help with an explanation for what's going on? The xx.xx.52.202 address is the Kamailio server itself. We record historical numbers of network connections for the server overall, and that doesn't show any increase in the number of TCP connections, either at the time of the spikes or as a general trend.
Thanks in advance!
Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: <core> [core/tcp_main.c:2703]: tcpconn_1st_send(): connect xx.xx.52.202:42726 failed (RST) Connection refused Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: <core> [core/tcp_main.c:2711]: tcpconn_1st_send(): xx.xx.52.202:42726: connect & send for 0x14bbf44c9f40 failed: Connection refused (111) Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: WARNING: <core> [core/tcp_main.c:1149]: tcp_do_connect(): xx.xx.52.202:42726: could not find corresponding listening socket for xx.xx.52.202, using default...
-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782
Hi Henning,
Thank you for that. I was not sure what direction to take traces in, so possible network issues with clients using TCP is a good indicator. We'll look into that.
On Mon, 29 Apr 2024 at 22:20, Henning Westerholt hw@gilawa.com wrote:
Hello,
if Kamailio can’t send out messages due to network issues I would expect some spikes in private and/or shared memory usage.
Have you investigated e.g., with some traces to get more information about the possible causes of this issues?
Besides that, with the given information I can only recommend the usual update to the latest stable version (e.g. 5.8.1) as the 5.2.x release series is long end of life.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com
*From:* David Cunningham via sr-users sr-users@lists.kamailio.org *Sent:* Freitag, 26. April 2024 02:27 *To:* Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Cc:* David Cunningham dcunningham@voisonics.com *Subject:* [SR-Users] Sudden TCP errors and shared memory spikes
Hello,
We have a production Kamailio 5.2.7 server which has suddenly started reporting spikes in the shared memory use. When looking into this, we noticed that at the same time as the memory spikes Kamailio also logs a huge amount of errors like the following.
Can anyone help with an explanation for what's going on? The xx.xx.52.202 address is the Kamailio server itself. We record historical numbers of network connections for the server overall, and that doesn't show any increase in the number of TCP connections, either at the time of the spikes or as a general trend.
Thanks in advance!
Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: <core> [core/tcp_main.c:2703]: tcpconn_1st_send(): connect xx.xx.52.202:42726 failed (RST) Connection refused Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: <core> [core/tcp_main.c:2711]: tcpconn_1st_send(): xx.xx.52.202:42726: connect & send for 0x14bbf44c9f40 failed: Connection refused (111) Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed Apr 26 09:23:16 vpbx11 /sbin/kamailio[44272]: WARNING: <core> [core/tcp_main.c:1149]: tcp_do_connect(): xx.xx.52.202:42726: could not find corresponding listening socket for xx.xx.52.202, using default...
--
David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782
-
David Cunningham -
Henning Westerholt