Hello,
I´ve googled around about tcp timeout in kamailio tcp connections when the tcp connection is broken.
Found this old question and to me it feels like I’m having that same situation as original question https://www.mail-archive.com/sr-users@lists.kamailio.org/msg15020.html Also found this https://kamailio.org/mailman3/hyperkitty/list/sr-dev@lists.kamailio.org/thre... and it seems to be the same description pretty much that I´m seeing. When the tcp connection (or tls) is interrupted in the network it takes around 15min for kamailio to reset the outgoing tcp connection.
I see in the logs that when restarting kamailio process it logs 2025-03-27 12:25:18.697 { "level": "INFO", "module": "core", "file": "core/tcp_main.c", "line": 3282, "function": "tcp_init", "logprefix": "", "message": "Set TCP_USER_TIMEOUT=10000 ms" } So I think the fix from this https://github.com/kamailio/kamailio/commit/d893f3af1444c8c4c5db6cd53fb57770... is applied.
This is tested with 5.8.5 and I have tested this by setting up with dispatcher tcp connection to external host Then with iptables drop traffic to that host, waiting kamailio to notice that destination is down, removing the iptables input and it takes around 15min to recover. (also restarting the kamailio helps and resolves the connection) And this is what kamailio prints during the test: Apr 4 08:29:15 kamailio[156947]: { "level": "ERROR", "module": "xlog", "file": "xlog.c", "line": 278, "function": "", "logprefix": "", "message": "Destination down: OPTIONS sip:ext-host;transport=tcp (<null>)" } Apr 4 08:44:43 kamailio[156957]: { "level": "ERROR", "module": "core", "file": "core/tcp_read.c", "line": 267, "function": "tcp_read_data", "logprefix": "", "message": "error reading: Connection timed out (110) ([]:5060 -> []:47492)" } Apr 4 08:44:43 kamailio[156957]: { "level": "ERROR", "module": "core", "file": "core/tcp_read.c", "line": 1524, "function": "tcp_read_req", "logprefix": "", "message": "error reading - c: 0x7f8625dea9b0 r: 0x7f8625deaad8 (-1)" } Apr 4 08:45:05 kamailio[156957]: { "level": "ERROR", "module": "xlog", "file": "xlog.c", "line": 278, "function": "", "logprefix": "Source:[ext-host]:5060, Call-id:4d0e1ccd315c817f-156947@int-host, CSeq:10", "message": "Destination up: OPTIONS sip:ext.host;transport=tcp (<null>)" }
Any advice how to lower the timeout to be quicker in such event?
-Pyry
On 4 Apr 2025, at 10:55, Pyry Aaltonen via sr-users sr-users@lists.kamailio.org wrote:
Hello,
I´ve googled around about tcp timeout in kamailio tcp connections when the tcp connection is broken.
Found this old question and to me it feels like I’m having that same situation as original question https://www.mail-archive.com/sr-users@lists.kamailio.org/msg15020.html Also found this https://kamailio.org/mailman3/hyperkitty/list/sr-dev@lists.kamailio.org/thre... and it seems to be the same description pretty much that I´m seeing. When the tcp connection (or tls) is interrupted in the network it takes around 15min for kamailio to reset the outgoing tcp connection.
I see in the logs that when restarting kamailio process it logs 2025-03-27 12:25:18.697 { "level": "INFO", "module": "core", "file": "core/tcp_main.c", "line": 3282, "function": "tcp_init", "logprefix": "", "message": "Set TCP_USER_TIMEOUT=10000 ms" } So I think the fix from this https://github.com/kamailio/kamailio/commit/d893f3af1444c8c4c5db6cd53fb57770... is applied.
This is tested with 5.8.5 and I have tested this by setting up with dispatcher tcp connection to external host Then with iptables drop traffic to that host, waiting kamailio to notice that destination is down, removing the iptables input and it takes around 15min to recover. (also restarting the kamailio helps and resolves the connection) And this is what kamailio prints during the test: Apr 4 08:29:15 kamailio[156947]: { "level": "ERROR", "module": "xlog", "file": "xlog.c", "line": 278, "function": "", "logprefix": "", "message": "Destination down: OPTIONS sip:ext-host;transport=tcp (<null>)" } Apr 4 08:44:43 kamailio[156957]: { "level": "ERROR", "module": "core", "file": "core/tcp_read.c", "line": 267, "function": "tcp_read_data", "logprefix": "", "message": "error reading: Connection timed out (110) ([]:5060 -> []:47492)" } Apr 4 08:44:43 kamailio[156957]: { "level": "ERROR", "module": "core", "file": "core/tcp_read.c", "line": 1524, "function": "tcp_read_req", "logprefix": "", "message": "error reading - c: 0x7f8625dea9b0 r: 0x7f8625deaad8 (-1)" } Apr 4 08:45:05 kamailio[156957]: { "level": "ERROR", "module": "xlog", "file": "xlog.c", "line": 278, "function": "", "logprefix": "Source:[ext-host]:5060, Call-id:4d0e1ccd315c817f-156947@int-host, CSeq:10", "message": "Destination up: OPTIONS sip:ext.host;transport=tcp (<null>)" }
Any advice how to lower the timeout to be quicker in such event?
That’s dependent on the operating system. This is one of the reasons why SIP outbound required two active TCP connections in order to have a fast failover.
If you google for “unix tcp timeout” you will find many documents with hints. I think Geoff Houston wrote a good summary once upon a time, but I can’t find it any more.
/O
https://datatracker.ietf.org/doc/html/rfc5626
" For a UA to receive incoming requests, the UA has to connect to a server. Since the server can't connect to the UA, the UA has to make sure that a flow is always active. This requires the UA to detect when a flow fails. Since such detection takes time and leaves a window of opportunity for missed incoming requests, this mechanism allows the UA to register over multiple flows at the same time.”
I understand, yes, that it comes from the os default settings how the tcp connection is handled, but as far as I understood that the patch done in the commit is adding the TCP_USER_TIMEOUT which should somehow at least try to fix this behaviour?
Kamailio core cookbook says: ”tcp_send_timeout Time in seconds after a TCP connection will be closed if it is not available for writing in this interval (and Kamailio wants to send something on it). Lower this value for faster detection of broken TCP connections. The default value is 10s. Example of usage: tcp_send_timeout=3”
And I understood that tcp_send_timeout is the one that sets the TCP_USER_TIMEOUT, which by default is set to 10seconds in kamailio. I was expecting this would change the os default behaviour?
And I know that yes, I can modify the system defaults also but rather tweak it in kamailio if its possible.
-Pyry
From: Olle E. Johansson oej@edvina.net Date: Friday, 4. April 2025 at 13.11 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Cc: Pyry Aaltonen pyry.aaltonen@cuuma.com Subject: Re: [SR-Users] TCP timeout
On 4 Apr 2025, at 10:55, Pyry Aaltonen via sr-users sr-users@lists.kamailio.org wrote:
Hello,
I´ve googled around about tcp timeout in kamailio tcp connections when the tcp connection is broken.
Found this old question and to me it feels like I’m having that same situation as original question https://www.mail-archive.com/sr-users@lists.kamailio.org/msg15020.html Also found this https://kamailio.org/mailman3/hyperkitty/list/sr-dev@lists.kamailio.org/thre... and it seems to be the same description pretty much that I´m seeing. When the tcp connection (or tls) is interrupted in the network it takes around 15min for kamailio to reset the outgoing tcp connection.
I see in the logs that when restarting kamailio process it logs 2025-03-27 12:25:18.697 { "level": "INFO", "module": "core", "file": "core/tcp_main.c", "line": 3282, "function": "tcp_init", "logprefix": "", "message": "Set TCP_USER_TIMEOUT=10000 ms" } So I think the fix from this https://github.com/kamailio/kamailio/commit/d893f3af1444c8c4c5db6cd53fb57770... is applied.
This is tested with 5.8.5 and I have tested this by setting up with dispatcher tcp connection to external host Then with iptables drop traffic to that host, waiting kamailio to notice that destination is down, removing the iptables input and it takes around 15min to recover. (also restarting the kamailio helps and resolves the connection) And this is what kamailio prints during the test: Apr 4 08:29:15 kamailio[156947]: { "level": "ERROR", "module": "xlog", "file": "xlog.c", "line": 278, "function": "", "logprefix": "", "message": "Destination down: OPTIONS sip:ext-host;transport=tcp (<null>)" } Apr 4 08:44:43 kamailio[156957]: { "level": "ERROR", "module": "core", "file": "core/tcp_read.c", "line": 267, "function": "tcp_read_data", "logprefix": "", "message": "error reading: Connection timed out (110) ([]:5060 -> []:47492)" } Apr 4 08:44:43 kamailio[156957]: { "level": "ERROR", "module": "core", "file": "core/tcp_read.c", "line": 1524, "function": "tcp_read_req", "logprefix": "", "message": "error reading - c: 0x7f8625dea9b0 r: 0x7f8625deaad8 (-1)" } Apr 4 08:45:05 kamailio[156957]: { "level": "ERROR", "module": "xlog", "file": "xlog.c", "line": 278, "function": "", "logprefix": "Source:[ext-host]:5060, Call-id:4d0e1ccd315c817f-156947@int-host, CSeq:10", "message": "Destination up: OPTIONS sip:ext.host;transport=tcp (<null>)" }
Any advice how to lower the timeout to be quicker in such event?
That’s dependent on the operating system. This is one of the reasons why SIP outbound required two active TCP connections in order to have a fast failover.
If you google for “unix tcp timeout” you will find many documents with hints. I think Geoff Houston wrote a good summary once upon a time, but I can’t find it any more.
/O
https://datatracker.ietf.org/doc/html/rfc5626
" For a UA to receive incoming requests, the UA has to connect to a
server. Since the server can't connect to the UA, the UA has to make
sure that a flow is always active. This requires the UA to detect
when a flow fails. Since such detection takes time and leaves a
window of opportunity for missed incoming requests, this mechanism
allows the UA to register over multiple flows at the same time.”
On Fri, 4 Apr 2025 at 14:05, Pyry Aaltonen via sr-users sr-users@lists.kamailio.org wrote:
I understand, yes, that it comes from the os default settings how the tcp connection is handled, but as far as I understood that the patch done in the commit is adding the TCP_USER_TIMEOUT which should somehow at least try to fix this behaviour?
I think you are misinterpreting what TCP User Timeout does.
It detects a broken TCP connection when *actively* writing to this socket. It cannot detect a broken connection in absence of any data written to the socket.
I think you are probably looking for TCP keepalives instead and Kamailio provides tcp_keepalive / tcp_keepcnt / tcp_keepidle / tcp_keepintvl options for this purpose.
In both cases (TCP User timeout vs TCP keepalives) Kamalio sets socket options advising the kernel about the configuration. I suggest you read the kernel documentation to understand more about those OS features.
If you are doubting that Kamailio sets the correct socket options I'd suggest using strace to verify.
Lukas
We have the same issue with outbound TCP connections Issue described here https://github.com/kamailio/kamailio/issues/4187
On Fri, 2025-04-04 at 14:54 +0200, Lukas Tribus via sr-users wrote:
On Fri, 4 Apr 2025 at 14:05, Pyry Aaltonen via sr-users sr-users@lists.kamailio.org wrote:
I understand, yes, that it comes from the os default settings how the tcp connection is handled, but as far as I understood that the patch done in the commit is adding the TCP_USER_TIMEOUT which should somehow at least try to fix this behaviour?
I think you are misinterpreting what TCP User Timeout does.
It detects a broken TCP connection when *actively* writing to this socket. It cannot detect a broken connection in absence of any data written to the socket.
I think you are probably looking for TCP keepalives instead and Kamailio provides tcp_keepalive / tcp_keepcnt / tcp_keepidle / tcp_keepintvl options for this purpose.
In both cases (TCP User timeout vs TCP keepalives) Kamalio sets socket options advising the kernel about the configuration. I suggest you read the kernel documentation to understand more about those OS features.
If you are doubting that Kamailio sets the correct socket options I'd suggest using strace to verify.
Lukas __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
Thanks for the response,
I think the dispatcher module should be actively trying to probe with OPTIONS ping the connection, which I suppose is meaning the socket should be actively used?
Im using dispatcher ds_probing_mode with value 1 and ping interval is also seconds instead of minutes.
I checked with ss -tin dst <trunk-ip>, and with that the ESTAB connections don’t have any “timeout” attribute, but I understood that it might depend on kernel version or socket config to show it there or not.
Need to think a bit how to setup strace to see what is set and signaled in the kamailio startup when it logs that 2025-03-27 12:25:18.697 { "level": "INFO", "module": "core", "file": "core/tcp_main.c", "line": 3282, "function": "tcp_init", "logprefix": "", "message": "Set TCP_USER_TIMEOUT=10000 ms" }, but will look forward to check that soon.
-Pyry
From: Lukas Tribus lukas@ltri.eu Date: Friday, 4. April 2025 at 15.54 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org, Pyry Aaltonen pyry.aaltonen@cuuma.com Subject: Re: [SR-Users] Re: TCP timeout On Fri, 4 Apr 2025 at 14:05, Pyry Aaltonen via sr-users sr-users@lists.kamailio.org wrote:
I understand, yes, that it comes from the os default settings how the tcp connection is handled, but as far as I understood that the patch done in the commit is adding the TCP_USER_TIMEOUT which should somehow at least try to fix this behaviour?
I think you are misinterpreting what TCP User Timeout does.
It detects a broken TCP connection when *actively* writing to this socket. It cannot detect a broken connection in absence of any data written to the socket.
I think you are probably looking for TCP keepalives instead and Kamailio provides tcp_keepalive / tcp_keepcnt / tcp_keepidle / tcp_keepintvl options for this purpose.
In both cases (TCP User timeout vs TCP keepalives) Kamalio sets socket options advising the kernel about the configuration. I suggest you read the kernel documentation to understand more about those OS features.
If you are doubting that Kamailio sets the correct socket options I'd suggest using strace to verify.
Lukas
Hello,
yes, the dispatcher module should send an OPTIONS ping out, also over TCP. Could you maybe test what actually happens, and also look to the log output, and socket stats?
Cheers,
Henning
From: Pyry Aaltonen via sr-users sr-users@lists.kamailio.org Sent: Freitag, 11. April 2025 07:09 To: Lukas Tribus lukas@ltri.eu; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Cc: Pyry Aaltonen pyry.aaltonen@cuuma.com Subject: [SR-Users] Re: TCP timeout
Thanks for the response,
I think the dispatcher module should be actively trying to probe with OPTIONS ping the connection, which I suppose is meaning the socket should be actively used?
Im using dispatcher ds_probing_mode with value 1 and ping interval is also seconds instead of minutes.
I checked with ss -tin dst <trunk-ip>, and with that the ESTAB connections don't have any "timeout" attribute, but I understood that it might depend on kernel version or socket config to show it there or not.
Need to think a bit how to setup strace to see what is set and signaled in the kamailio startup when it logs that 2025-03-27 12:25:18.697 { "level": "INFO", "module": "core", "file": "core/tcp_main.c", "line": 3282, "function": "tcp_init", "logprefix": "", "message": "Set TCP_USER_TIMEOUT=10000 ms" }, but will look forward to check that soon.
-Pyry
From: Lukas Tribus <lukas@ltri.eumailto:lukas@ltri.eu> Date: Friday, 4. April 2025 at 15.54 To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org>, Pyry Aaltonen <pyry.aaltonen@cuuma.commailto:pyry.aaltonen@cuuma.com> Subject: Re: [SR-Users] Re: TCP timeout On Fri, 4 Apr 2025 at 14:05, Pyry Aaltonen via sr-users <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> wrote:
I understand, yes, that it comes from the os default settings how the tcp connection is handled, but as far as I understood that the patch done in the commit is adding the TCP_USER_TIMEOUT which should somehow at least try to fix this behaviour?
I think you are misinterpreting what TCP User Timeout does.
It detects a broken TCP connection when *actively* writing to this socket. It cannot detect a broken connection in absence of any data written to the socket.
I think you are probably looking for TCP keepalives instead and Kamailio provides tcp_keepalive / tcp_keepcnt / tcp_keepidle / tcp_keepintvl options for this purpose.
In both cases (TCP User timeout vs TCP keepalives) Kamalio sets socket options advising the kernel about the configuration. I suggest you read the kernel documentation to understand more about those OS features.
If you are doubting that Kamailio sets the correct socket options I'd suggest using strace to verify.
Lukas
-
Henning Westerholt -
Lukas Tribus -
Olle E. Johansson -
Pyry Aaltonen -
Sergei Safarov