[Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
Hi Using Kamailio 1.5 we periodically get the following errors: Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110) Connection timed out Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_send: connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:tm:msg_send: tcp_send failed They are several emails in this mailing list explaining we should play with tcp parameters. I will try to change my configuration and see if we can solve this issue. But when this kind of error happen we observe that the kamailio processes are using 100% of CPU (top). Our client is using SIP over TCP only (no UDP). Do you have any hint on how to configure Kamailio to improve TCP support ? Is it normal to kamailio processes to overload CPU ? Regards, Pascal
2009/5/27 Pascal Maugeri <pascal.maugeri@gmail.com>:
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110) Connection timed out Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_send: connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:tm:msg_send: tcp_send failed
They are several emails in this mailing list explaining we should play with tcp parameters. I will try to change my configuration and see if we can solve this issue.
But when this kind of error happen we observe that the kamailio processes are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ? Is it normal to kamailio processes to overload CPU ?
Unfortunatelly a kamailio TCP listeners blocks when initiates a outgoing TCP connection. If it fails, you get a kamailio process "stopped" waiting for that connection to end. If you have 8 kamailio TCP listeners and have this issue in all of them, Kamailio cannot handle TCP connection for a while. This is of course something that should change radically (there have been already threads about this issue). However I've ever seen CPU at 100% due to this issues. How many concurrent TCP connection does your kamailio handle when it occurs? -- Iñaki Baz Castillo <ibc@aliax.net>
On Wed, May 27, 2009 at 10:57 AM, Pascal Maugeri <pascal.maugeri@gmail.com> wrote:
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110) Connection timed out Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_send: connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:tm:msg_send: tcp_send failed
They are several emails in this mailing list explaining we should play with tcp parameters. I will try to change my configuration and see if we can solve this issue.
But when this kind of error happen we observe that the kamailio processes are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ? Is it normal to kamailio processes to overload CPU ?
To my best knowledge, TCP support in Kamailio is blocking. This has been addressed for sip-router (tcp_async=yes). Anyway, there are still some issues for tls. How many simultaneous TCP connections are you handling? You could try to limit the max number of connections and check the impact on the CPU consumption. Cheers, -- Victor Pascual Ávila
To my best knowledge, TCP support in Kamailio is blocking. This has been addressed for sip-router (tcp_async=yes). Anyway, there are still some issues for tls.
This is a good argument to switch to sip-router :-)
How many simultaneous TCP connections are you handling? You could try to limit the max number of connections and check the impact on the CPU consumption.
Theoretically we're using few simultaneous client connections (<10). My first intention is to increase tcp_children and to decrease tcp_send_timeout and tcp_connect_timeout. Do you believe it is a good idea ? Cheers Pascal
Cheers, -- Victor Pascual Ávila
FYI, I've changed the configuration to: tcp_children=8 tcp_send_timeout=3 tcp_connect_timeout=5 and it looks we get less often the problem described here ( tcp_connect/send time out error). I confirm that the problem was triggered by the fact that our client is using TCP only and that the tcp connections were unavailable after some presence test (many NOTIFY messages sent to TCP). I'm curious to know how many children do you use in a typical configuration ? 4, 8, more? Of course it depends on the traffic expected but let me know that this is parameter you always increase in a production environment. Regards, Pascal 2009/5/27 Pascal Maugeri <pascal.maugeri@gmail.com>
To my best knowledge, TCP support in Kamailio is blocking. This has been addressed for sip-router (tcp_async=yes). Anyway, there are still some issues for tls.
This is a good argument to switch to sip-router :-)
How many simultaneous TCP connections are you handling? You could try to limit the max number of connections and check the impact on the CPU consumption.
Theoretically we're using few simultaneous client connections (<10). My first intention is to increase tcp_children and to decrease tcp_send_timeout and tcp_connect_timeout. Do you believe it is a good idea ?
Cheers Pascal
Cheers, -- Victor Pascual Ávila
2009/5/28 Pascal Maugeri <pascal.maugeri@gmail.com>:
FYI, I've changed the configuration to:
tcp_children=8 tcp_send_timeout=3 tcp_connect_timeout=5
and it looks we get less often the problem described here ( tcp_connect/send time out error).
This is a good workaround, but the problem is that it replies on the clients behaviour and it shouldn't occur (a server cannot get "blocked" due to clients problems). I hope TCP handling will be improved in SIP-Router.
I confirm that the problem was triggered by the fact that our client is using TCP only and that the tcp connections were unavailable after some presence test (many NOTIFY messages sent to TCP).
I'm curious to know how many children do you use in a typical configuration ? 4, 8, more? Of course it depends on the traffic expected but let me know that this is parameter you always increase in a production environment.
Theorically you can add so many TCP listeners as you desire (depending on your CPU's and memory). But if your environment makes high usage of TCP then increase it as much as possible. -- Iñaki Baz Castillo <ibc@aliax.net>
On Thu, May 28, 2009 at 12:59 PM, Iñaki Baz Castillo <ibc@aliax.net> wrote:
2009/5/28 Pascal Maugeri <pascal.maugeri@gmail.com>:
FYI, I've changed the configuration to:
tcp_children=8 tcp_send_timeout=3 tcp_connect_timeout=5
and it looks we get less often the problem described here ( tcp_connect/send time out error).
This is a good workaround, but the problem is that it replies on the clients behaviour and it shouldn't occur (a server cannot get "blocked" due to clients problems). I hope TCP handling will be improved in SIP-Router.
Indeed the very same problem occured after a while :-( It looks my new configuration only "delayed" it. The symptoms are: - errors in log file: May 27 14:20:11 server2-operations /usr/local/sbin/kamailio[26227]: WARNING:core:send2child: no free tcp receiver, connection passed to the leastbusy one (1) May 27 14:20:13 server2-operations /usr/local/sbin/kamailio[26221]: ERROR:core:tcp_blocking_connect: timeout 10 s elapsed from 10 s May 27 14:20:13 server2-operations /usr/local/sbin/kamailio[26221]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed May 27 14:20:13 server2-operations /usr/local/sbin/kamailio[26221]: ERROR:core:tcp_send: connect failed May 27 14:20:13 server2-operations /usr/local/sbin/kamailio[26221]: ERROR:sl:msg_send: tcp_send failed - in top <http://unixhelp.ed.ac.uk/CGI/man-cgi?top> kamailio process have high values in the %CPU column such as 50% or 99% !? - when all kamailio processes have reached this top values, it is impossible to connect a new client untill we reset kamailio :-/ I will try the parameter modparam("registrar", "tcp_persistent_flag", 7) from the registrar module. -pascal
I confirm that the problem was triggered by the fact that our client is using TCP only and that the tcp connections were unavailable after some presence test (many NOTIFY messages sent to TCP).
I'm curious to know how many children do you use in a typical configuration ? 4, 8, more? Of course it depends on the traffic expected but let me know that this is parameter you always increase in a production environment.
Theorically you can add so many TCP listeners as you desire (depending on your CPU's and memory). But if your environment makes high usage of TCP then increase it as much as possible.
-- Iñaki Baz Castillo <ibc@aliax.net>
2009/5/29 Pascal Maugeri <pascal.maugeri@gmail.com>:
May 27 14:20:11 server2-operations /usr/local/sbin/kamailio[26227]: WARNING:core:send2child: no free tcp receiver, connection passed to the leastbusy one (1)
Yes, this is the warning I told about in other mail.
- in top kamailio process have high values in the %CPU column such as 50% or 99% !?
The same occurs (kamailio takes 50-99% CPU) in case there is a problem with MySQL and Kamailio cannot write into it. It seems that when kamailio process must wait for something (TCP connection, MysQL access) it consumes CPU resources (waiting).
- when all kamailio processes have reached this top values, it is impossible to connect a new client untill we reset kamailio :-/
Makes sense since no TCP listener is available :(
I will try the parameter
modparam("registrar", "tcp_persistent_flag", 7)
It will not solve the issue. The only it does it setting the TCP connection duration for the registration expires time. It has nothing to do with failed TCP connections. -- Iñaki Baz Castillo <ibc@aliax.net>
I have also found this problem with 1.5.3. Modified Kamailio's source code to fix that problem. Now I don't have this problem. Pascal Maugeri wrote:
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110) Connection timed out Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_send: connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:tm:msg_send: tcp_send failed
They are several emails in this mailing list explaining we should play with tcp parameters. I will try to change my configuration and see if we can solve this issue.
But when this kind of error happen we observe that the kamailio processes are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ? Is it normal to kamailio processes to overload CPU ?
Regards, Pascal
_______________________________________________ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
-- View this message in context: http://old.nabble.com/tcp_blocking_connect-related-errors-and-Kamailio-using... Sent from the OpenSER Users Mailing List mailing list archive at Nabble.com.
On 1/29/10 8:40 PM, Zunnun wrote:
I have also found this problem with 1.5.3. Modified Kamailio's source code to fix that problem. Now I don't have this problem.
can you send a patch? I will check it and commit it to svn if all ok. Thanks, Daniel
Pascal Maugeri wrote:
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: poll error: flags 18 Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110) Connection timed out Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcpconn_connect: tcp_blocking_connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:core:tcp_send: connect failed Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:tm:msg_send: tcp_send failed
They are several emails in this mailing list explaining we should play with tcp parameters. I will try to change my configuration and see if we can solve this issue.
But when this kind of error happen we observe that the kamailio processes are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ? Is it normal to kamailio processes to overload CPU ?
Regards, Pascal
_______________________________________________ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
-- Daniel-Constantin Mierla * http://www.asipto.com/
participants (5)
-
Daniel-Constantin Mierla -
Iñaki Baz Castillo -
Pascal Maugeri -
Victor Pascual Ávila -
Zunnun