TCP connection lifetime - sr-users

26 Mar 2019

Hi Again,

Here is an issue with TCP connection being kept for more:

Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E
(SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER
over new established TCP socket *every 2 seconds*.

All the REGISTER was rejected with 401. (may be the device was
misconfigured? or not receiving any of my answer? I can't tell)

NOTE: You can see the expires header was very large: 86400, ie: 24 hours...

I was checking the TCP/TLS connections on my server and discovered more
than 1000 TCP established connection to that user/ip, and thus, I have
tried to understand what happened.

Checking the logs, I received 4855 REGISTER from this device from "Mar 25
03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP
connection every 2.5 seconds...

Today, I decided to check it again around 11am.

jack@sip:~$ sudo kamctl stats tcp
{
  "jsonrpc":  "2.0",
  "result": [
    "tcp:con_reset = 1857",
    "tcp:con_timeout = 35927",
    "tcp:connect_failed = 25",
    "tcp:connect_success = 2",
    "tcp:current_opened_connections = 2291",
    "tcp:current_write_queue_size = 0",
    "tcp:established = 80778",
    "tcp:local_reject = 0",
    "tcp:passive_open = 80776",
    "tcp:send_timeout = 2",
    "tcp:sendq_full = 0"
  ],
  "id": 7305
}

There was still A LOT of established connections. And the connections have
been established more than 24 hours ago.

At 11H16:
$> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA |
wc -l
1161
At 11H22:
$> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA |
wc -l
1018
At 11H35:
$> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA |
wc -l
655
At 13H
$> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA |
wc -l
0

So the established connections are all gone now.

Between 11h16 and 11H35, I was seeing the server regularly sending [FIN,
ACK] over each TCP established connection, with retransmissions for all of
them. (no incoming trafic)

I do not have numbers/capture/stats, but I think that kamailio was already
closing some
connection yesterday. I don't know when kamailio started to try closing
those connections.

I'm now back with this status:

At 13pm:
jack@sip:~$ sudo kamctl stats tcp
{
  "jsonrpc":  "2.0",
  "result": [
    "tcp:con_reset = 1896",
    "tcp:con_timeout = 38042",
    "tcp:connect_failed = 26",
    "tcp:connect_success = 2",
    "tcp:current_opened_connections = 939",
    "tcp:current_write_queue_size = 0",
    "tcp:established = 81950",
    "tcp:local_reject = 0",
    "tcp:passive_open = 81948",
    "tcp:send_timeout = 2",
    "tcp:sendq_full = 0"
  ],
  "id": 12734
}

With around 155 registration entries using TCP and TLS in my location
database.

As you can see, tcp:current_opened_connections = 939 is still pretty high
compared to
my currently registred users.

I have "modparam("registrar", "max_expires", 86400)",
because I'm keeping
contact entries (even with TCP connection down) for push notifications.

I have "tcp_connection_lifetime=3600" configured.

Question 1

With "tcp_connection_lifetime=3600", I would expect kamailio to close the
established connection after 3600 seconds without traffic. It is pretty
obvious that no data has been exchanged over the 4855 established
connection during a day.

Despite the issue with the Avaya phones is solved automatically after a
day, I guess similar stuff or happening, at a different rate, for other
users as well. (because  current_opened_connections is way higher than
registred TCP/TLS users)

Question 2

I can list TLS connection with "kamctl rpc tls.list"
Can I get a similar list for TCP? (lsof returns a lot of duplicates...)

Tks
Aymeric

-- 
Antisip - http://www.antisip.com