Problem with max tcp connections
Hi I have a kamailio 5.1.2 as load balancer and registration offloading, but I have a problem with the max tcp connections that it can handle. I suspect that is a linux limit, but I don't find the reason or config. When that limit arrives, I can't connect to kamailio and I receive "Connection reset by peer", but I can't view any error message in the logs. If I check the connections in kamailio, I view that it have "free" connections: # kamctl kamcmd core.tcp_info { readers: 8 max_connections: 4096 max_tls_connections: 2048 opened_connections: 2655 opened_tls_connections: 0 write_queued_bytes: 0 } I have this configs in kamailio.conf (related to tcp) disable_tcp=no tcp_connection_lifetime=3610 tcp_connect_timeout=5 tcp_crlf_ping=yes tcp_accept_aliases=no tcp_keepalive=yes tcp_keepidle=5 tcp_rd_buf_size=65536 tcp_conn_wq_max=131072 mlock_pages=yes shm_force_alloc=yes tcp_max_connections=4096 The shm memory to 256 and the pkg memory to 32. And, following this doc: https://github.com/kamailio/kamailio/blob/master/doc/tutorials/tcp_tunning.t... I have setted this values: net.ipv4.ip_local_port_range = 1024 65535 net.core.somaxconn = 65535 net.core.netdev_max_backlog = 182757 Also, I had checked the limits for the main process pid: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 386297 386297 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us The service is running inside a lxc container, without any resource limit, connected to the outside word throught macvlan interface. Where can I find problem source? Best regards
I thought that the 5.1.2 is the latest release in this branch, sorry. First I will upgrade to the latest version (5.1.9) and if the problem persists, I'll resend the email. Best regards El jue., 21 nov. 2019 a las 14:46, Jose Fco. Irles Durá (<josefu@gmail.com>) escribió:
Hi
I have a kamailio 5.1.2 as load balancer and registration offloading, but I have a problem with the max tcp connections that it can handle. I suspect that is a linux limit, but I don't find the reason or config. When that limit arrives, I can't connect to kamailio and I receive "Connection reset by peer", but I can't view any error message in the logs.
If I check the connections in kamailio, I view that it have "free" connections: # kamctl kamcmd core.tcp_info { readers: 8 max_connections: 4096 max_tls_connections: 2048 opened_connections: 2655 opened_tls_connections: 0 write_queued_bytes: 0 }
I have this configs in kamailio.conf (related to tcp) disable_tcp=no tcp_connection_lifetime=3610 tcp_connect_timeout=5 tcp_crlf_ping=yes tcp_accept_aliases=no tcp_keepalive=yes tcp_keepidle=5 tcp_rd_buf_size=65536 tcp_conn_wq_max=131072 mlock_pages=yes shm_force_alloc=yes tcp_max_connections=4096
The shm memory to 256 and the pkg memory to 32.
And, following this doc: https://github.com/kamailio/kamailio/blob/master/doc/tutorials/tcp_tunning.t... I have setted this values: net.ipv4.ip_local_port_range = 1024 65535 net.core.somaxconn = 65535 net.core.netdev_max_backlog = 182757
Also, I had checked the limits for the main process pid: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 1048576 1048576 files Max locked memory 16777216 16777216 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 386297 386297 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
The service is running inside a lxc container, without any resource limit, connected to the outside word throught macvlan interface.
Where can I find problem source?
Best regards
-- Jose Fco. Irles Durá
On Thu, Nov 21, 2019 at 02:46:39PM +0100, Jose Fco. Irles Dur?? wrote:
Hi
I have a kamailio 5.1.2 as load balancer and registration offloading, but I have a problem with the max tcp connections that it can handle. I suspect that is a linux limit, but I don't find the reason or config. When that limit arrives, I can't connect to kamailio and I receive "Connection reset by peer", but I can't view any error message in the logs.
I experienced the same with versions pre 5.1.9. It hasn't happened to me with 5.1.9 yet, so I cannot say it was fixed. Additional changes I applied where some sysctl tweaks: net.core.somaxconn = 128 net.ipv4.tcp_max_syn_backlog = 1024 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_tw_buckets = 4096 net.ipv4.ip_local_port_range = 4096 65534 net.ipv4.tcp_syncookies = 0 Again, not really sure if these are related.
Thanks for the info! Tomorrow I'll test it with the 5.1.9 version. Best regards El jue., 21 nov. 2019 a las 17:35, Daniel Tryba (<d.tryba@pocos.nl>) escribió:
On Thu, Nov 21, 2019 at 02:46:39PM +0100, Jose Fco. Irles Dur?? wrote:
Hi
I have a kamailio 5.1.2 as load balancer and registration offloading, but I have a problem with the max tcp connections that it can handle. I suspect that is a linux limit, but I don't find the reason or config. When that limit arrives, I can't connect to kamailio and I receive "Connection reset by peer", but I can't view any error message in the logs.
I experienced the same with versions pre 5.1.9. It hasn't happened to me with 5.1.9 yet, so I cannot say it was fixed. Additional changes I applied where some sysctl tweaks:
net.core.somaxconn = 128 net.ipv4.tcp_max_syn_backlog = 1024 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_tw_buckets = 4096 net.ipv4.ip_local_port_range = 4096 65534 net.ipv4.tcp_syncookies = 0
Again, not really sure if these are related.
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Jose Fco. Irles Durá
On Thu, Nov 21, 2019 at 07:49:28PM +0100, Jose Fco. Irles Dur?? wrote:
Thanks for the info!
Tomorrow I'll test it with the 5.1.9 version.
Well, the problem happened to me on 2 different loadbalancers (withing 24 hours where the loadbalancers had a near identical uptime) For about 35m no new connections can be established. Already established connections work fine. I'm not seeing any queueing in to OS (netstat/ss) After some time all works well again without doing anything to the machine/kamailio. I could try to make a core dump if this happens again and the timing is more appropriate for that. kamcmd> core.tcp_info { readers: 4 max_connections: 4096 max_tls_connections: 2048 opened_connections: 617 opened_tls_connections: 0 write_queued_bytes: 0 } tcp options in .cg: tcp_max_connections=4096 tcp_connection_lifetime=3605 Only funny thing I see is that there are some TCP connections with a very high timeout: { id: 15453 type: TCP state: CONN_OK timeout: 268435455 lifetime: 3605 ref_count: 2 src_ip: a.b.c.d src_port: 51196 dst_ip: e.f.g.h dst_port: 5060 } The timeout is either [0,3605] or 268435455. The high timeout ones dissear quickly: while sleep 1 ; do date; kamcmd core.tcp_list | grep 268435455; done Tue Nov 26 10:29:21 CET 2019 Tue Nov 26 10:29:22 CET 2019 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 Tue Nov 26 10:29:23 CET 2019 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 Tue Nov 26 10:29:24 CET 2019 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 Tue Nov 26 10:29:25 CET 2019 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 Tue Nov 26 10:29:26 CET 2019 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 Tue Nov 26 10:29:27 CET 2019 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 timeout: 268435455 Tue Nov 26 10:29:28 CET 2019 timeout: 268435455 timeout: 268435455 Tue Nov 26 10:29:29 CET 2019 timeout: 268435455 timeout: 268435455
On Tue, Nov 26, 2019 at 10:32:59AM +0100, Daniel Tryba wrote:
Well, the problem happened to me on 2 different loadbalancers (withing 24 hours where the loadbalancers had a near identical uptime) For about 35m no new connections can be established. Already established connections work fine. I'm not seeing any queueing in to OS (netstat/ss) After some time all works well again without doing anything to the machine/kamailio.
I could try to make a core dump if this happens again and the timing is more appropriate for that.
And then it happened again. Coredumps weren't reacted sadly enough. But after restarting I still couldn't reconnect. I'm starting to think this is an OS issue.
In my case, from the upgrade I haven't issues (Kamailio has ~2700 tcp connections) But when I restarted the service (for the upgrade), tcp connections grow until ~2850 and in that moment I can't create new connections. I will continue investigating El mar., 26 nov. 2019 a las 10:48, Daniel Tryba (<d.tryba@pocos.nl>) escribió:
On Tue, Nov 26, 2019 at 10:32:59AM +0100, Daniel Tryba wrote:
Well, the problem happened to me on 2 different loadbalancers (withing 24 hours where the loadbalancers had a near identical uptime) For about 35m no new connections can be established. Already established connections work fine. I'm not seeing any queueing in to OS (netstat/ss) After some time all works well again without doing anything to the machine/kamailio.
I could try to make a core dump if this happens again and the timing is more appropriate for that.
And then it happened again. Coredumps weren't reacted sadly enough. But after restarting I still couldn't reconnect. I'm starting to think this is an OS issue.
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Jose Fco. Irles Durá
That’s strange, I have 5000+ tcp connections and I can’t remember I did anything special at the os level. I’m using 5.2 now, but it was the same with 5.1 On Tue, 26 Nov 2019 at 11:37, Jose Fco. Irles Durá <josefu@gmail.com> wrote:
In my case, from the upgrade I haven't issues (Kamailio has ~2700 tcp connections)
But when I restarted the service (for the upgrade), tcp connections grow until ~2850 and in that moment I can't create new connections.
I will continue investigating
El mar., 26 nov. 2019 a las 10:48, Daniel Tryba (<d.tryba@pocos.nl>) escribió:
On Tue, Nov 26, 2019 at 10:32:59AM +0100, Daniel Tryba wrote:
Well, the problem happened to me on 2 different loadbalancers (withing 24 hours where the loadbalancers had a near identical uptime) For about 35m no new connections can be established. Already established connections work fine. I'm not seeing any queueing in to OS
(netstat/ss)
After some time all works well again without doing anything to the machine/kamailio.
I could try to make a core dump if this happens again and the timing is more appropriate for that.
And then it happened again. Coredumps weren't reacted sadly enough. But after restarting I still couldn't reconnect. I'm starting to think this is an OS issue.
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Jose Fco. Irles Durá
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Regards, David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337
I will try to upgrade to the latest version and re test. Best regards El mar., 26 nov. 2019 a las 13:15, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
That’s strange, I have 5000+ tcp connections and I can’t remember I did anything special at the os level. I’m using 5.2 now, but it was the same with 5.1
I checked that the problem persist. I have installed the version is 5.1.9, and the limit to 4096, but I can't connect more than 2758. I think that the problem is in the kernel, but i don't have any idea how debug it. Any idea? Best regards El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá (<josefu@gmail.com>) escribió:
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
That’s strange, I have 5000+ tcp connections and I can’t remember I did anything special at the os level. I’m using 5.2 now, but it was the same with 5.1
-- Jose Fco. Irles Durá
I know this is not recommended, but have you tried starting as root? If so and you don’t hit a limit, then there’s something wrong in your limits for the kamailio user. I have the same issue and it only works with root, I haven’t been able to run it with kamailio for so many tcp connections. On Tue, 10 Dec 2019 at 11:27, Jose Fco. Irles Durá <josefu@gmail.com> wrote:
I checked that the problem persist.
I have installed the version is 5.1.9, and the limit to 4096, but I can't connect more than 2758.
I think that the problem is in the kernel, but i don't have any idea how debug it. Any idea?
Best regards
El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá (<josefu@gmail.com>) escribió:
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
That’s strange, I have 5000+ tcp connections and I can’t remember I
did anything special at the os level. I’m using 5.2 now, but it was the same with 5.1
-- Jose Fco. Irles Durá
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Regards, David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337
I will test it. Thanks for the reply. Best regards El mar., 10 dic. 2019 a las 13:35, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
I know this is not recommended, but have you tried starting as root? If so and you don’t hit a limit, then there’s something wrong in your limits for the kamailio user. I have the same issue and it only works with root, I haven’t been able to run it with kamailio for so many tcp connections.
On Tue, 10 Dec 2019 at 11:27, Jose Fco. Irles Durá <josefu@gmail.com> wrote:
I checked that the problem persist.
I have installed the version is 5.1.9, and the limit to 4096, but I can't connect more than 2758.
I think that the problem is in the kernel, but i don't have any idea how debug it. Any idea?
Best regards
El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá (<josefu@gmail.com>) escribió:
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
That’s strange, I have 5000+ tcp connections and I can’t remember I did anything special at the os level. I’m using 5.2 now, but it was the same with 5.1
-- Jose Fco. Irles Durá
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Regards,
David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337 _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Jose Fco. Irles Durá
Today, running as root, kamailio has more connections. It seems that the problem is some user limit, I will recheck the limits. I will wait some days checking it. Best regards El mar., 10 dic. 2019 a las 13:39, Jose Fco. Irles Durá (<josefu@gmail.com>) escribió:
I will test it.
Thanks for the reply.
Best regards
El mar., 10 dic. 2019 a las 13:35, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
I know this is not recommended, but have you tried starting as root? If so and you don’t hit a limit, then there’s something wrong in your limits for the kamailio user. I have the same issue and it only works with root, I haven’t been able to run it with kamailio for so many tcp connections.
On Tue, 10 Dec 2019 at 11:27, Jose Fco. Irles Durá <josefu@gmail.com> wrote:
I checked that the problem persist.
I have installed the version is 5.1.9, and the limit to 4096, but I can't connect more than 2758.
I think that the problem is in the kernel, but i don't have any idea how debug it. Any idea?
Best regards
El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá (<josefu@gmail.com>) escribió:
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
That’s strange, I have 5000+ tcp connections and I can’t remember I did anything special at the os level. I’m using 5.2 now, but it was the same with 5.1
-- Jose Fco. Irles Durá
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Regards,
David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337 _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Jose Fco. Irles Durá
-- Jose Fco. Irles Durá
Where did you guys get to with this? Can you share details of the OS you were using? I'm very curious if you are still seeing this behaviour, and if not, what did you do to change it? Thanks Jay On Wed, 11 Dec 2019 at 22:26, Jose Fco. Irles Durá <josefu@gmail.com> wrote:
Today, running as root, kamailio has more connections. It seems that the problem is some user limit, I will recheck the limits.
I will wait some days checking it.
Best regards
El mar., 10 dic. 2019 a las 13:39, Jose Fco. Irles Durá (<josefu@gmail.com>) escribió:
I will test it.
Thanks for the reply.
Best regards
El mar., 10 dic. 2019 a las 13:35, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
I know this is not recommended, but have you tried starting as root?
If so and you don’t hit a limit, then there’s something wrong in your limits for the kamailio user. I have the same issue and it only works with root, I haven’t been able to run it with kamailio for so many tcp connections.
On Tue, 10 Dec 2019 at 11:27, Jose Fco. Irles Durá <josefu@gmail.com>
wrote:
I checked that the problem persist.
I have installed the version is 5.1.9, and the limit to 4096, but I can't connect more than 2758.
I think that the problem is in the kernel, but i don't have any idea how debug it. Any idea?
Best regards
El mar., 26 nov. 2019 a las 16:14, Jose Fco. Irles Durá (<josefu@gmail.com>) escribió:
I will try to upgrade to the latest version and re test.
Best regards
El mar., 26 nov. 2019 a las 13:15, David Villasmil (<david.villasmil.work@gmail.com>) escribió:
That’s strange, I have 5000+ tcp connections and I can’t remember
I did anything special at the os level. I’m using 5.2 now, but it was the same with 5.1
-- Jose Fco. Irles Durá
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Regards,
David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337 _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Jose Fco. Irles Durá
-- Jose Fco. Irles Durá
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Sincerely Jay
participants (4)
-
Daniel Tryba -
David Villasmil -
jay binks -
Jose Fco. Irles Durá