Hello,

On 26.03.19 17:16, Aymeric Moizard wrote:

Hi Again,

Here is an issue with TCP connection being kept for more:

Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E (SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER over new established TCP socket *every 2 seconds*.

All the REGISTER was rejected with 401. (may be the device was misconfigured? or not receiving any of my answer? I can't tell)

NOTE: You can see the expires header was very large: 86400, ie: 24 hours...

I was checking the TCP/TLS connections on my server and discovered more than 1000 TCP established connection to that user/ip, and thus, I have tried to understand what happened.

Checking the logs, I received 4855 REGISTER from this device from "Mar 25 03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP connection every 2.5 seconds...

Today, I decided to check it again around 11am.

jack@sip:~$ sudo kamctl stats tcp

{

  "jsonrpc":  "2.0",

  "result": [

    "tcp:con_reset = 1857",

    "tcp:con_timeout = 35927",

    "tcp:connect_failed = 25",

    "tcp:connect_success = 2",

    "tcp:current_opened_connections = 2291",

    "tcp:current_write_queue_size = 0",

    "tcp:established = 80778",

    "tcp:local_reject = 0",

    "tcp:passive_open = 80776",

    "tcp:send_timeout = 2",

    "tcp:sendq_full = 0"

  ],

  "id": 7305

}

There was still A LOT of established connections. And the connections have been established more than 24 hours ago.

At 11H16:

$> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l

1161

At 11H22:

$> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l

1018

At 11H35:

$> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l

655

At 13H

$> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA | wc -l

0

So the established connections are all gone now.

Between 11h16 and 11H35, I was seeing the server regularly sending [FIN, ACK] over each TCP established connection, with retransmissions for all of them. (no incoming trafic)

I do not have numbers/capture/stats, but I think that kamailio was already closing some

connection yesterday. I don't know when kamailio started to try closing those connections.

I'm now back with this status:

At 13pm:

jack@sip:~$ sudo kamctl stats tcp

{

  "jsonrpc":  "2.0",

  "result": [

    "tcp:con_reset = 1896",

    "tcp:con_timeout = 38042",

    "tcp:connect_failed = 26",

    "tcp:connect_success = 2",

    "tcp:current_opened_connections = 939",

    "tcp:current_write_queue_size = 0",

    "tcp:established = 81950",

    "tcp:local_reject = 0",

    "tcp:passive_open = 81948",

    "tcp:send_timeout = 2",

    "tcp:sendq_full = 0"

  ],

  "id": 12734

}

With around 155 registration entries using TCP and TLS in my location database.

As you can see, tcp:current_opened_connections = 939 is still pretty high compared to

my currently registred users.

I have "modparam("registrar", "max_expires", 86400)", because I'm keeping contact entries (even with TCP connection down) for push notifications.

I have "tcp_connection_lifetime=3600" configured.

Question 1

With "tcp_connection_lifetime=3600", I would expect kamailio to close the established connection after 3600 seconds without traffic. It is pretty obvious that no data has been exchanged over the 4855 established connection during a day.

Despite the issue with the Avaya phones is solved automatically after a day, I guess similar stuff or happening, at a different rate, for other users as well. (because  current_opened_connections is way higher than registred TCP/TLS users)

Yes, tcp connections should be closed if no traffic on them for the lifetime duration.

Question 2

I can list TLS connection with "kamctl rpc tls.list"

Can I get a similar list for TCP? (lsof returns a lot of duplicates...)

Yes, see:

http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list

Maybe you can compare what is listed by the rpc command to see what kamailio actually sees as active connections.

Cheers,
Daniel

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com