<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,<br>
</p>
<div class="moz-cite-prefix">On 26.03.19 17:16, Aymeric Moizard
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALM7LKNMoQTqgMy5MzvEcdd8hWZ0uR2uRSV5=eLHBNUYTz+ogw@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div>Hi Again,</div>
<div><br>
</div>
<div>Here is an issue with TCP connection being kept for more:</div>
<div><br>
</div>
<div>
<div>Yesterday, I have discovered that a User-Agent
(<Avaya IP Phone 1120E (SIP1120e.04.04.30.00)> tried
to register a lot. It was sending REGISTER over new
established TCP socket *every 2 seconds*.</div>
<br class="gmail-Apple-interchange-newline">
</div>
<div>All the REGISTER was rejected with 401. (may be the
device was misconfigured? or not receiving any of my answer?
I can't tell)<br>
</div>
<div><br>
</div>
<div>NOTE: You can see the expires header was very large:
86400, ie: 24 hours...</div>
<div><br>
</div>
<div>I was checking the TCP/TLS connections on my server and
discovered more than 1000 TCP established connection to that
user/ip, and thus, I have tried to understand what happened.</div>
<div><br>
</div>
<div>Checking the logs, I received 4855 REGISTER from this
device from "Mar 25 03:47:09" to "Mar 25 07:56:13" which is
a rate of approx one new TCP connection every 2.5 seconds...</div>
<div><br>
</div>
<div>Today, I decided to check it again around 11am.</div>
<div><br>
</div>
<div>jack@<a class="moz-txt-link-freetext" href="sip:~$">sip:~$</a> sudo kamctl stats tcp</div>
<div>{</div>
<div> "jsonrpc": "2.0",</div>
<div> "result": [</div>
<div> "tcp:con_reset = 1857",</div>
<div> "tcp:con_timeout = 35927",</div>
<div> "tcp:connect_failed = 25",</div>
<div> "tcp:connect_success = 2",</div>
<div> "tcp:current_opened_connections = 2291",</div>
<div> "tcp:current_write_queue_size = 0",</div>
<div> "tcp:established = 80778",</div>
<div> "tcp:local_reject = 0",</div>
<div> "tcp:passive_open = 80776",</div>
<div> "tcp:send_timeout = 2",</div>
<div> "tcp:sendq_full = 0"</div>
<div> ],</div>
<div> "id": 7305</div>
<div>}</div>
<div><br>
</div>
<div>There was still A LOT of established connections. And the
connections have been established more than 24 hours ago.</div>
<div><br>
</div>
<div>At 11H16:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP | grep
41.234.242.69 | grep ESTA | wc -l</div>
<div>1161</div>
<div>At 11H22:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP | grep
41.234.242.69 | grep ESTA | wc -l</div>
<div>1018</div>
<div>At 11H35:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP | grep
41.234.242.69 | grep ESTA | wc -l</div>
<div>655</div>
<div>At 13H</div>
<div>$> lsof -n -l | grep kamailio | grep TCP | grep
41.234.242.69 | grep ESTA | wc -l</div>
<div>0</div>
<div><br>
</div>
<div>So the established connections are all gone now.</div>
<div><br>
</div>
<div>Between 11h16 and 11H35, I was seeing the server
regularly sending [FIN, ACK] over each TCP established
connection, with retransmissions for all of them. (no
incoming trafic)</div>
<div><br>
</div>
<div>I do not have numbers/capture/stats, but I think that
kamailio was already closing some</div>
<div>connection yesterday. I don't know when kamailio started
to try closing those connections.</div>
<div><br>
</div>
<div>I'm now back with this status:</div>
<div><br>
</div>
<div>At 13pm:</div>
<div>jack@<a class="moz-txt-link-freetext" href="sip:~$">sip:~$</a> sudo kamctl stats tcp</div>
<div>{</div>
<div> "jsonrpc": "2.0",</div>
<div> "result": [</div>
<div> "tcp:con_reset = 1896",</div>
<div> "tcp:con_timeout = 38042",</div>
<div> "tcp:connect_failed = 26",</div>
<div> "tcp:connect_success = 2",</div>
<div> "tcp:current_opened_connections = 939",</div>
<div> "tcp:current_write_queue_size = 0",</div>
<div> "tcp:established = 81950",</div>
<div> "tcp:local_reject = 0",</div>
<div> "tcp:passive_open = 81948",</div>
<div> "tcp:send_timeout = 2",</div>
<div> "tcp:sendq_full = 0"</div>
<div> ],</div>
<div> "id": 12734</div>
<div>}</div>
<div><br>
</div>
<div>With around 155 registration entries using TCP and TLS in
my location database.</div>
<div><br>
</div>
<div>As you can see, tcp:current_opened_connections = 939 is
still pretty high compared to</div>
<div>my currently registred users.</div>
<div><br>
</div>
<div>I have "modparam("registrar", "max_expires", 86400)",
because I'm keeping contact entries (even with TCP
connection down) for push notifications.</div>
<div><br>
</div>
<div>I have "tcp_connection_lifetime=3600" configured.</div>
<div><br>
</div>
<div>Question 1</div>
<div><br>
</div>
<div>With "tcp_connection_lifetime=3600", I would expect
kamailio to close the established connection after 3600
seconds without traffic. It is pretty obvious that no data
has been exchanged over the 4855 established connection
during a day.</div>
<div><br>
</div>
<div>Despite the issue with the Avaya phones is solved
automatically after a day, I guess similar stuff or
happening, at a different rate, for other users as well.
(because current_opened_connections is way higher than
registred TCP/TLS users)</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Yes, tcp connections should be closed if no traffic on them for
the lifetime duration.</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CALM7LKNMoQTqgMy5MzvEcdd8hWZ0uR2uRSV5=eLHBNUYTz+ogw@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>Question 2</div>
<div><br>
</div>
<div>I can list TLS connection with "kamctl rpc tls.list"</div>
<div>Can I get a similar list for TCP? (lsof returns a lot of
duplicates...)</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Yes, see:</p>
<p><a
href="http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list">http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list</a></p>
<p>Maybe you can compare what is listed by the rpc command to see
what kamailio actually sees as active connections.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a></pre>
</body>
</html>