Hi Again,
Here is an issue with TCP connection being kept for more:
Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E (SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER over new established TCP socket *every 2 seconds*.
All the REGISTER was rejected with 401. (may be the device was misconfigured? or not receiving any of my answer? I can't tell)
NOTE: You can see the expires header was very large: 86400, ie: 24 hours...
I was checking the TCP/TLS connections on my server and discovered more than 1000 TCP established connection to that user/ip, and thus, I have tried to understand what happened.
Checking the logs, I received 4855 REGISTER from this device from "Mar 25 03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP connection every 2.5 seconds...
Today, I decided to check it again around 11am.
jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1857", "tcp:con_timeout = 35927", "tcp:connect_failed = 25", "tcp:connect_success = 2", "tcp:current_opened_connections = 2291", "tcp:current_write_queue_size = 0", "tcp:established = 80778", "tcp:local_reject = 0", "tcp:passive_open = 80776", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 7305 }
There was still A LOT of established connections. And the connections have been established more than 24 hours ago.
At 11H16: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1161 At 11H22: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1018 At 11H35: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 655 At 13H $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 0
So the established connections are all gone now.
Between 11h16 and 11H35, I was seeing the server regularly sending [FIN, ACK] over each TCP established connection, with retransmissions for all of them. (no incoming trafic)
I do not have numbers/capture/stats, but I think that kamailio was already closing some connection yesterday. I don't know when kamailio started to try closing those connections.
I'm now back with this status:
At 13pm: jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1896", "tcp:con_timeout = 38042", "tcp:connect_failed = 26", "tcp:connect_success = 2", "tcp:current_opened_connections = 939", "tcp:current_write_queue_size = 0", "tcp:established = 81950", "tcp:local_reject = 0", "tcp:passive_open = 81948", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 12734 }
With around 155 registration entries using TCP and TLS in my location database.
As you can see, tcp:current_opened_connections = 939 is still pretty high compared to my currently registred users.
I have "modparam("registrar", "max_expires", 86400)", because I'm keeping contact entries (even with TCP connection down) for push notifications.
I have "tcp_connection_lifetime=3600" configured.
Question 1
With "tcp_connection_lifetime=3600", I would expect kamailio to close the established connection after 3600 seconds without traffic. It is pretty obvious that no data has been exchanged over the 4855 established connection during a day.
Despite the issue with the Avaya phones is solved automatically after a day, I guess similar stuff or happening, at a different rate, for other users as well. (because current_opened_connections is way higher than registred TCP/TLS users)
Question 2
I can list TLS connection with "kamctl rpc tls.list" Can I get a similar list for TCP? (lsof returns a lot of duplicates...)
Tks Aymeric
Hello,
On 26.03.19 17:16, Aymeric Moizard wrote:
Hi Again,
Here is an issue with TCP connection being kept for more:
Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E (SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER over new established TCP socket *every 2 seconds*.
All the REGISTER was rejected with 401. (may be the device was misconfigured? or not receiving any of my answer? I can't tell)
NOTE: You can see the expires header was very large: 86400, ie: 24 hours...
I was checking the TCP/TLS connections on my server and discovered more than 1000 TCP established connection to that user/ip, and thus, I have tried to understand what happened.
Checking the logs, I received 4855 REGISTER from this device from "Mar 25 03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP connection every 2.5 seconds...
Today, I decided to check it again around 11am.
jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1857", "tcp:con_timeout = 35927", "tcp:connect_failed = 25", "tcp:connect_success = 2", "tcp:current_opened_connections = 2291", "tcp:current_write_queue_size = 0", "tcp:established = 80778", "tcp:local_reject = 0", "tcp:passive_open = 80776", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 7305 }
There was still A LOT of established connections. And the connections have been established more than 24 hours ago.
At 11H16: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1161 At 11H22: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1018 At 11H35: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 655 At 13H $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 0
So the established connections are all gone now.
Between 11h16 and 11H35, I was seeing the server regularly sending [FIN, ACK] over each TCP established connection, with retransmissions for all of them. (no incoming trafic)
I do not have numbers/capture/stats, but I think that kamailio was already closing some connection yesterday. I don't know when kamailio started to try closing those connections.
I'm now back with this status:
At 13pm: jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1896", "tcp:con_timeout = 38042", "tcp:connect_failed = 26", "tcp:connect_success = 2", "tcp:current_opened_connections = 939", "tcp:current_write_queue_size = 0", "tcp:established = 81950", "tcp:local_reject = 0", "tcp:passive_open = 81948", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 12734 }
With around 155 registration entries using TCP and TLS in my location database.
As you can see, tcp:current_opened_connections = 939 is still pretty high compared to my currently registred users.
I have "modparam("registrar", "max_expires", 86400)", because I'm keeping contact entries (even with TCP connection down) for push notifications.
I have "tcp_connection_lifetime=3600" configured.
Question 1
With "tcp_connection_lifetime=3600", I would expect kamailio to close the established connection after 3600 seconds without traffic. It is pretty obvious that no data has been exchanged over the 4855 established connection during a day.
Despite the issue with the Avaya phones is solved automatically after a day, I guess similar stuff or happening, at a different rate, for other users as well. (because current_opened_connections is way higher than registred TCP/TLS users)
Yes, tcp connections should be closed if no traffic on them for the lifetime duration.
Question 2
I can list TLS connection with "kamctl rpc tls.list" Can I get a similar list for TCP? (lsof returns a lot of duplicates...)
Yes, see:
http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_...
Maybe you can compare what is listed by the rpc command to see what kamailio actually sees as active connections.
Cheers, Daniel
Hi Daniel,
Tks for answering! Unfortunatly, core.tcp_list is only returning one connection. But core.tcp_info reports 184 opened connections (same for "kamctl stats tcp")
sudo kamctl rpc core.tcp_list { "jsonrpc": "2.0", "result": { "id": 439290, "type": "TCP", "state": "CONN_ACCEPT", "timeout": 2567, "lifetime": 3600, "ref_count": 1, "src_ip": "41.46.4.235", "src_port": 4957, "dst_ip": "91.121.30.149", "dst_port": 5060 }, "id": 9158 }
sudo kamctl rpc core.tcp_info { "jsonrpc": "2.0", "result": { "readers": 16, "max_connections": 50000, "max_tls_connections": 50000, "opened_connections": 184, "opened_tls_connections": 64, "write_queued_bytes": 0 }, "id": 9523 }
Did I missed something?
jack@sip:~$ /usr/sbin/kamailio -v version: kamailio 5.2.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 6.3.0
Regards Aymeric
Le lun. 15 avr. 2019 à 09:10, Daniel-Constantin Mierla miconda@gmail.com a écrit :
Hello, On 26.03.19 17:16, Aymeric Moizard wrote:
Hi Again,
Here is an issue with TCP connection being kept for more:
Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E (SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER over new established TCP socket *every 2 seconds*.
All the REGISTER was rejected with 401. (may be the device was misconfigured? or not receiving any of my answer? I can't tell)
NOTE: You can see the expires header was very large: 86400, ie: 24 hours...
I was checking the TCP/TLS connections on my server and discovered more than 1000 TCP established connection to that user/ip, and thus, I have tried to understand what happened.
Checking the logs, I received 4855 REGISTER from this device from "Mar 25 03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP connection every 2.5 seconds...
Today, I decided to check it again around 11am.
jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1857", "tcp:con_timeout = 35927", "tcp:connect_failed = 25", "tcp:connect_success = 2", "tcp:current_opened_connections = 2291", "tcp:current_write_queue_size = 0", "tcp:established = 80778", "tcp:local_reject = 0", "tcp:passive_open = 80776", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 7305 }
There was still A LOT of established connections. And the connections have been established more than 24 hours ago.
At 11H16: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1161 At 11H22: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1018 At 11H35: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 655 At 13H $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 0
So the established connections are all gone now.
Between 11h16 and 11H35, I was seeing the server regularly sending [FIN, ACK] over each TCP established connection, with retransmissions for all of them. (no incoming trafic)
I do not have numbers/capture/stats, but I think that kamailio was already closing some connection yesterday. I don't know when kamailio started to try closing those connections.
I'm now back with this status:
At 13pm: jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1896", "tcp:con_timeout = 38042", "tcp:connect_failed = 26", "tcp:connect_success = 2", "tcp:current_opened_connections = 939", "tcp:current_write_queue_size = 0", "tcp:established = 81950", "tcp:local_reject = 0", "tcp:passive_open = 81948", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 12734 }
With around 155 registration entries using TCP and TLS in my location database.
As you can see, tcp:current_opened_connections = 939 is still pretty high compared to my currently registred users.
I have "modparam("registrar", "max_expires", 86400)", because I'm keeping contact entries (even with TCP connection down) for push notifications.
I have "tcp_connection_lifetime=3600" configured.
Question 1
With "tcp_connection_lifetime=3600", I would expect kamailio to close the established connection after 3600 seconds without traffic. It is pretty obvious that no data has been exchanged over the 4855 established connection during a day.
Despite the issue with the Avaya phones is solved automatically after a day, I guess similar stuff or happening, at a different rate, for other users as well. (because current_opened_connections is way higher than registred TCP/TLS users)
Yes, tcp connections should be closed if no traffic on them for the lifetime duration.
Question 2
I can list TLS connection with "kamctl rpc tls.list" Can I get a similar list for TCP? (lsof returns a lot of duplicates...)
Yes, see:
http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_...
Maybe you can compare what is listed by the rpc command to see what kamailio actually sees as active connections.
Cheers, Daniel
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
Hello,
as you said it shows a single connections, I went to the code and I discovered a bug in exporting rpc command core.tcp_info, because it was missing the option that it returns an array. I fixed it in master branch with next commit:
- https://github.com/kamailio/kamailio/commit/24ca2e7760a8dada433b188348c768e7...
and I will backport to stable branches.
Meanwhile, you can use:
kamcmd core.tcp_list
which is not strict in validating the binrcp/jsonrpc response and eventually it will print all the tcp connections. Can you test that?
Cheers, Daniel
On 17.04.19 15:42, Aymeric Moizard wrote:
Hi Daniel,
Tks for answering! Unfortunatly, core.tcp_list is only returning one connection. But core.tcp_info reports 184 opened connections (same for "kamctl stats tcp")
sudo kamctl rpc core.tcp_list { "jsonrpc": "2.0", "result": { "id": 439290, "type": "TCP", "state": "CONN_ACCEPT", "timeout": 2567, "lifetime": 3600, "ref_count": 1, "src_ip": "41.46.4.235", "src_port": 4957, "dst_ip": "91.121.30.149", "dst_port": 5060 }, "id": 9158 }
sudo kamctl rpc core.tcp_info { "jsonrpc": "2.0", "result": { "readers": 16, "max_connections": 50000, "max_tls_connections": 50000, "opened_connections": 184, "opened_tls_connections": 64, "write_queued_bytes": 0 }, "id": 9523 }
Did I missed something?
jack@sip:~$ /usr/sbin/kamailio -v version: kamailio 5.2.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 6.3.0
Regards Aymeric
Le lun. 15 avr. 2019 à 09:10, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> a écrit :
Hello, On 26.03.19 17:16, Aymeric Moizard wrote:Hi Again, Here is an issue with TCP connection being kept for more: Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E (SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER over new established TCP socket *every 2 seconds*. All the REGISTER was rejected with 401. (may be the device was misconfigured? or not receiving any of my answer? I can't tell) NOTE: You can see the expires header was very large: 86400, ie: 24 hours... I was checking the TCP/TLS connections on my server and discovered more than 1000 TCP established connection to that user/ip, and thus, I have tried to understand what happened. Checking the logs, I received 4855 REGISTER from this device from "Mar 25 03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP connection every 2.5 seconds... Today, I decided to check it again around 11am. jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1857", "tcp:con_timeout = 35927", "tcp:connect_failed = 25", "tcp:connect_success = 2", "tcp:current_opened_connections = 2291", "tcp:current_write_queue_size = 0", "tcp:established = 80778", "tcp:local_reject = 0", "tcp:passive_open = 80776", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 7305 } There was still A LOT of established connections. And the connections have been established more than 24 hours ago. At 11H16: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1161 At 11H22: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1018 At 11H35: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 655 At 13H $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 0 So the established connections are all gone now. Between 11h16 and 11H35, I was seeing the server regularly sending [FIN, ACK] over each TCP established connection, with retransmissions for all of them. (no incoming trafic) I do not have numbers/capture/stats, but I think that kamailio was already closing some connection yesterday. I don't know when kamailio started to try closing those connections. I'm now back with this status: At 13pm: jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1896", "tcp:con_timeout = 38042", "tcp:connect_failed = 26", "tcp:connect_success = 2", "tcp:current_opened_connections = 939", "tcp:current_write_queue_size = 0", "tcp:established = 81950", "tcp:local_reject = 0", "tcp:passive_open = 81948", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 12734 } With around 155 registration entries using TCP and TLS in my location database. As you can see, tcp:current_opened_connections = 939 is still pretty high compared to my currently registred users. I have "modparam("registrar", "max_expires", 86400)", because I'm keeping contact entries (even with TCP connection down) for push notifications. I have "tcp_connection_lifetime=3600" configured. Question 1 With "tcp_connection_lifetime=3600", I would expect kamailio to close the established connection after 3600 seconds without traffic. It is pretty obvious that no data has been exchanged over the 4855 established connection during a day. Despite the issue with the Avaya phones is solved automatically after a day, I guess similar stuff or happening, at a different rate, for other users as well. (because current_opened_connections is way higher than registred TCP/TLS users)Yes, tcp connections should be closed if no traffic on them for the lifetime duration.Question 2 I can list TLS connection with "kamctl rpc tls.list" Can I get a similar list for TCP? (lsof returns a lot of duplicates...)Yes, see: http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list Maybe you can compare what is listed by the rpc command to see what kamailio actually sees as active connections. Cheers, Daniel -- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com <http://www.kamailioworld.com>-- Antisip - http://www.antisip.com
Hi Daniel!
Tks a lot:
$> kamcmd core.tcp_list
Worked!
Next time I meet the issue, I will have more data to analyse... I guess I will be able to compare "timeout" vs "lifetime"
Tks a lot!
Aymeric
Le mer. 17 avr. 2019 à 15:51, Daniel-Constantin Mierla miconda@gmail.com a écrit :
Hello,
as you said it shows a single connections, I went to the code and I discovered a bug in exporting rpc command core.tcp_info, because it was missing the option that it returns an array. I fixed it in master branch with next commit:
https://github.com/kamailio/kamailio/commit/24ca2e7760a8dada433b188348c768e7...
and I will backport to stable branches.
Meanwhile, you can use:
kamcmd core.tcp_list
which is not strict in validating the binrcp/jsonrpc response and eventually it will print all the tcp connections. Can you test that?
Cheers, Daniel On 17.04.19 15:42, Aymeric Moizard wrote:
Hi Daniel,
Tks for answering! Unfortunatly, core.tcp_list is only returning one connection. But core.tcp_info reports 184 opened connections (same for "kamctl stats tcp")
sudo kamctl rpc core.tcp_list { "jsonrpc": "2.0", "result": { "id": 439290, "type": "TCP", "state": "CONN_ACCEPT", "timeout": 2567, "lifetime": 3600, "ref_count": 1, "src_ip": "41.46.4.235", "src_port": 4957, "dst_ip": "91.121.30.149", "dst_port": 5060 }, "id": 9158 }
sudo kamctl rpc core.tcp_info { "jsonrpc": "2.0", "result": { "readers": 16, "max_connections": 50000, "max_tls_connections": 50000, "opened_connections": 184, "opened_tls_connections": 64, "write_queued_bytes": 0 }, "id": 9523 }
Did I missed something?
jack@sip:~$ /usr/sbin/kamailio -v version: kamailio 5.2.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 6.3.0
Regards Aymeric
Le lun. 15 avr. 2019 à 09:10, Daniel-Constantin Mierla miconda@gmail.com a écrit :
Hello, On 26.03.19 17:16, Aymeric Moizard wrote:
Hi Again,
Here is an issue with TCP connection being kept for more:
Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E (SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER over new established TCP socket *every 2 seconds*.
All the REGISTER was rejected with 401. (may be the device was misconfigured? or not receiving any of my answer? I can't tell)
NOTE: You can see the expires header was very large: 86400, ie: 24 hours...
I was checking the TCP/TLS connections on my server and discovered more than 1000 TCP established connection to that user/ip, and thus, I have tried to understand what happened.
Checking the logs, I received 4855 REGISTER from this device from "Mar 25 03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP connection every 2.5 seconds...
Today, I decided to check it again around 11am.
jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1857", "tcp:con_timeout = 35927", "tcp:connect_failed = 25", "tcp:connect_success = 2", "tcp:current_opened_connections = 2291", "tcp:current_write_queue_size = 0", "tcp:established = 80778", "tcp:local_reject = 0", "tcp:passive_open = 80776", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 7305 }
There was still A LOT of established connections. And the connections have been established more than 24 hours ago.
At 11H16: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1161 At 11H22: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 1018 At 11H35: $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 655 At 13H $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l 0
So the established connections are all gone now.
Between 11h16 and 11H35, I was seeing the server regularly sending [FIN, ACK] over each TCP established connection, with retransmissions for all of them. (no incoming trafic)
I do not have numbers/capture/stats, but I think that kamailio was already closing some connection yesterday. I don't know when kamailio started to try closing those connections.
I'm now back with this status:
At 13pm: jack@sip:~$ sudo kamctl stats tcp { "jsonrpc": "2.0", "result": [ "tcp:con_reset = 1896", "tcp:con_timeout = 38042", "tcp:connect_failed = 26", "tcp:connect_success = 2", "tcp:current_opened_connections = 939", "tcp:current_write_queue_size = 0", "tcp:established = 81950", "tcp:local_reject = 0", "tcp:passive_open = 81948", "tcp:send_timeout = 2", "tcp:sendq_full = 0" ], "id": 12734 }
With around 155 registration entries using TCP and TLS in my location database.
As you can see, tcp:current_opened_connections = 939 is still pretty high compared to my currently registred users.
I have "modparam("registrar", "max_expires", 86400)", because I'm keeping contact entries (even with TCP connection down) for push notifications.
I have "tcp_connection_lifetime=3600" configured.
Question 1
With "tcp_connection_lifetime=3600", I would expect kamailio to close the established connection after 3600 seconds without traffic. It is pretty obvious that no data has been exchanged over the 4855 established connection during a day.
Despite the issue with the Avaya phones is solved automatically after a day, I guess similar stuff or happening, at a different rate, for other users as well. (because current_opened_connections is way higher than registred TCP/TLS users)
Yes, tcp connections should be closed if no traffic on them for the lifetime duration.
Question 2
I can list TLS connection with "kamctl rpc tls.list" Can I get a similar list for TCP? (lsof returns a lot of duplicates...)
Yes, see:
http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_...
Maybe you can compare what is listed by the rpc command to see what kamailio actually sees as active connections.
Cheers, Daniel
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
-- Antisip - http://www.antisip.com
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
-
Aymeric Moizard -
Daniel-Constantin Mierla