<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Daniel,<div><br></div><div>Tks for answering! Unfortunatly, core.tcp_list is only returning one connection.</div><div>But core.tcp_info reports 184 opened connections (same for "kamctl stats tcp")</div><div><br></div><div><div>sudo kamctl rpc core.tcp_list</div><div>{</div><div> "jsonrpc": "2.0",</div><div> "result": {</div><div> "id": 439290,</div><div> "type": "TCP",</div><div> "state": "CONN_ACCEPT",</div><div> "timeout": 2567,</div><div> "lifetime": 3600,</div><div> "ref_count": 1,</div><div> "src_ip": "41.46.4.235",</div><div> "src_port": 4957,</div><div> "dst_ip": "91.121.30.149",</div><div> "dst_port": 5060</div><div> },</div><div> "id": 9158</div><div>}</div></div><div><br></div><div><div>sudo kamctl rpc core.tcp_info</div><div>{</div><div> "jsonrpc": "2.0",</div><div> "result": {</div><div> "readers": 16,</div><div> "max_connections": 50000,</div><div> "max_tls_connections": 50000,</div><div> "opened_connections": 184,</div><div> "opened_tls_connections": 64,</div><div> "write_queued_bytes": 0</div><div> },</div><div> "id": 9523</div><div>}</div></div><div><br></div><div>Did I missed something?</div><div><br></div><div><br></div><div><div>jack@sip:~$ /usr/sbin/kamailio -v</div><div>version: kamailio 5.2.2 (x86_64/linux)</div><div>flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES</div><div>ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB</div><div>poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.</div><div>id: unknown</div><div>compiled with gcc 6.3.0</div></div><div><br></div><div>Regards</div><div>Aymeric</div><div><br></div></div></div></div></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le lun. 15 avr. 2019 à 09:10, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hello,<br>
</p>
<div class="gmail-m_427341605902023125moz-cite-prefix">On 26.03.19 17:16, Aymeric Moizard
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Hi Again,</div>
<div><br>
</div>
<div>Here is an issue with TCP connection being kept for more:</div>
<div><br>
</div>
<div>
<div>Yesterday, I have discovered that a User-Agent
(<Avaya IP Phone 1120E (SIP1120e.04.04.30.00)> tried
to register a lot. It was sending REGISTER over new
established TCP socket *every 2 seconds*.</div>
<br class="gmail-m_427341605902023125gmail-Apple-interchange-newline">
</div>
<div>All the REGISTER was rejected with 401. (may be the
device was misconfigured? or not receiving any of my answer?
I can't tell)<br>
</div>
<div><br>
</div>
<div>NOTE: You can see the expires header was very large:
86400, ie: 24 hours...</div>
<div><br>
</div>
<div>I was checking the TCP/TLS connections on my server and
discovered more than 1000 TCP established connection to that
user/ip, and thus, I have tried to understand what happened.</div>
<div><br>
</div>
<div>Checking the logs, I received 4855 REGISTER from this
device from "Mar 25 03:47:09" to "Mar 25 07:56:13" which is
a rate of approx one new TCP connection every 2.5 seconds...</div>
<div><br>
</div>
<div>Today, I decided to check it again around 11am.</div>
<div><br>
</div>
<div>jack@<a class="gmail-m_427341605902023125moz-txt-link-freetext">sip:~$</a> sudo kamctl stats tcp</div>
<div>{</div>
<div> "jsonrpc": "2.0",</div>
<div> "result": [</div>
<div> "tcp:con_reset = 1857",</div>
<div> "tcp:con_timeout = 35927",</div>
<div> "tcp:connect_failed = 25",</div>
<div> "tcp:connect_success = 2",</div>
<div> "tcp:current_opened_connections = 2291",</div>
<div> "tcp:current_write_queue_size = 0",</div>
<div> "tcp:established = 80778",</div>
<div> "tcp:local_reject = 0",</div>
<div> "tcp:passive_open = 80776",</div>
<div> "tcp:send_timeout = 2",</div>
<div> "tcp:sendq_full = 0"</div>
<div> ],</div>
<div> "id": 7305</div>
<div>}</div>
<div><br>
</div>
<div>There was still A LOT of established connections. And the
connections have been established more than 24 hours ago.</div>
<div><br>
</div>
<div>At 11H16:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP | grep
41.234.242.69 | grep ESTA | wc -l</div>
<div>1161</div>
<div>At 11H22:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP | grep
41.234.242.69 | grep ESTA | wc -l</div>
<div>1018</div>
<div>At 11H35:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP | grep
41.234.242.69 | grep ESTA | wc -l</div>
<div>655</div>
<div>At 13H</div>
<div>$> lsof -n -l | grep kamailio | grep TCP | grep
41.234.242.69 | grep ESTA | wc -l</div>
<div>0</div>
<div><br>
</div>
<div>So the established connections are all gone now.</div>
<div><br>
</div>
<div>Between 11h16 and 11H35, I was seeing the server
regularly sending [FIN, ACK] over each TCP established
connection, with retransmissions for all of them. (no
incoming trafic)</div>
<div><br>
</div>
<div>I do not have numbers/capture/stats, but I think that
kamailio was already closing some</div>
<div>connection yesterday. I don't know when kamailio started
to try closing those connections.</div>
<div><br>
</div>
<div>I'm now back with this status:</div>
<div><br>
</div>
<div>At 13pm:</div>
<div>jack@<a class="gmail-m_427341605902023125moz-txt-link-freetext">sip:~$</a> sudo kamctl stats tcp</div>
<div>{</div>
<div> "jsonrpc": "2.0",</div>
<div> "result": [</div>
<div> "tcp:con_reset = 1896",</div>
<div> "tcp:con_timeout = 38042",</div>
<div> "tcp:connect_failed = 26",</div>
<div> "tcp:connect_success = 2",</div>
<div> "tcp:current_opened_connections = 939",</div>
<div> "tcp:current_write_queue_size = 0",</div>
<div> "tcp:established = 81950",</div>
<div> "tcp:local_reject = 0",</div>
<div> "tcp:passive_open = 81948",</div>
<div> "tcp:send_timeout = 2",</div>
<div> "tcp:sendq_full = 0"</div>
<div> ],</div>
<div> "id": 12734</div>
<div>}</div>
<div><br>
</div>
<div>With around 155 registration entries using TCP and TLS in
my location database.</div>
<div><br>
</div>
<div>As you can see, tcp:current_opened_connections = 939 is
still pretty high compared to</div>
<div>my currently registred users.</div>
<div><br>
</div>
<div>I have "modparam("registrar", "max_expires", 86400)",
because I'm keeping contact entries (even with TCP
connection down) for push notifications.</div>
<div><br>
</div>
<div>I have "tcp_connection_lifetime=3600" configured.</div>
<div><br>
</div>
<div>Question 1</div>
<div><br>
</div>
<div>With "tcp_connection_lifetime=3600", I would expect
kamailio to close the established connection after 3600
seconds without traffic. It is pretty obvious that no data
has been exchanged over the 4855 established connection
during a day.</div>
<div><br>
</div>
<div>Despite the issue with the Avaya phones is solved
automatically after a day, I guess similar stuff or
happening, at a different rate, for other users as well.
(because current_opened_connections is way higher than
registred TCP/TLS users)</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Yes, tcp connections should be closed if no traffic on them for
the lifetime duration.</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>Question 2</div>
<div><br>
</div>
<div>I can list TLS connection with "kamctl rpc tls.list"</div>
<div>Can I get a similar list for TCP? (lsof returns a lot of
duplicates...)</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Yes, see:</p>
<p><a href="http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list" target="_blank">http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list</a></p>
<p>Maybe you can compare what is listed by the rpc command to see
what kamailio actually sees as active connections.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<pre class="gmail-m_427341605902023125moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_427341605902023125moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="gmail-m_427341605902023125moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="gmail-m_427341605902023125moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="gmail-m_427341605902023125moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><img src="http://sip.antisip.com/am48.png">Antisip - <a href="http://www.antisip.com" target="_blank">http://www.antisip.com</a><br></div></div></div></div>