<div dir="ltr"><div dir="ltr"><div><br></div>Hi Daniel!<div><br></div><div>Tks a lot:</div><div><br><div>$> kamcmd core.tcp_list <br></div></div><div><br></div><div>Worked!</div><div><br></div><div>Next time I meet the issue, I will have more data to analyse...</div><div>I guess I will be able to compare "timeout" vs "lifetime"</div><div><br></div><div>Tks a lot!</div><div><br></div><div>Aymeric</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mer. 17 avr. 2019 à 15:51, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hello,</p>
<p>as you said it shows a single connections, I went to the code and
I discovered a bug in exporting rpc command core.tcp_info, because
it was missing the option that it returns an array. I fixed it in
master branch with next commit:</p>
<p> - <a href="https://github.com/kamailio/kamailio/commit/24ca2e7760a8dada433b188348c768e7e224f10d" target="_blank">https://github.com/kamailio/kamailio/commit/24ca2e7760a8dada433b188348c768e7e224f10d</a></p>
<p>and I will backport to stable branches.</p>
<p>Meanwhile, you can use:</p>
<p>kamcmd core.tcp_list</p>
<p>which is not strict in validating the binrcp/jsonrpc response and
eventually it will print all the tcp connections. Can you test
that?<br>
</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="gmail-m_7447434029031418155moz-cite-prefix">On 17.04.19 15:42, Aymeric Moizard
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Daniel,
<div><br>
</div>
<div>Tks for answering! Unfortunatly,
core.tcp_list is only returning one connection.</div>
<div>But core.tcp_info reports 184 opened
connections (same for "kamctl stats tcp")</div>
<div><br>
</div>
<div>
<div>sudo kamctl rpc core.tcp_list</div>
<div>{</div>
<div> "jsonrpc": "2.0",</div>
<div> "result": {</div>
<div> "id": 439290,</div>
<div> "type": "TCP",</div>
<div> "state": "CONN_ACCEPT",</div>
<div> "timeout": 2567,</div>
<div> "lifetime": 3600,</div>
<div> "ref_count": 1,</div>
<div> "src_ip": "41.46.4.235",</div>
<div> "src_port": 4957,</div>
<div> "dst_ip": "91.121.30.149",</div>
<div> "dst_port": 5060</div>
<div> },</div>
<div> "id": 9158</div>
<div>}</div>
</div>
<div><br>
</div>
<div>
<div>sudo kamctl rpc core.tcp_info</div>
<div>{</div>
<div> "jsonrpc": "2.0",</div>
<div> "result": {</div>
<div> "readers": 16,</div>
<div> "max_connections": 50000,</div>
<div> "max_tls_connections": 50000,</div>
<div> "opened_connections": 184,</div>
<div> "opened_tls_connections": 64,</div>
<div> "write_queued_bytes": 0</div>
<div> },</div>
<div> "id": 9523</div>
<div>}</div>
</div>
<div><br>
</div>
<div>Did I missed something?</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>jack@<a class="gmail-m_7447434029031418155moz-txt-link-freetext">sip:~$</a> /usr/sbin/kamailio -v</div>
<div>version: kamailio 5.2.2 (x86_64/linux)</div>
<div>flags: STATS: Off, USE_TCP, USE_TLS,
USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES</div>
<div>ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB</div>
<div>poll method support: poll, epoll_lt,
epoll_et, sigio_rt, select.</div>
<div>id: unknown</div>
<div>compiled with gcc 6.3.0</div>
</div>
<div><br>
</div>
<div>Regards</div>
<div>Aymeric</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Le lun. 15 avr. 2019
à 09:10, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hello,<br>
</p>
<div class="gmail-m_7447434029031418155gmail-m_427341605902023125moz-cite-prefix">On
26.03.19 17:16, Aymeric Moizard wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>Hi Again,</div>
<div><br>
</div>
<div>Here is an issue with TCP connection being
kept for more:</div>
<div><br>
</div>
<div>
<div>Yesterday, I have discovered that a
User-Agent (<Avaya IP Phone 1120E
(SIP1120e.04.04.30.00)> tried to register
a lot. It was sending REGISTER over new
established TCP socket *every 2 seconds*.</div>
<br class="gmail-m_7447434029031418155gmail-m_427341605902023125gmail-Apple-interchange-newline">
</div>
<div>All the REGISTER was rejected with 401.
(may be the device was misconfigured? or not
receiving any of my answer? I can't tell)<br>
</div>
<div><br>
</div>
<div>NOTE: You can see the expires header was
very large: 86400, ie: 24 hours...</div>
<div><br>
</div>
<div>I was checking the TCP/TLS connections on
my server and discovered more than 1000 TCP
established connection to that user/ip, and
thus, I have tried to understand what
happened.</div>
<div><br>
</div>
<div>Checking the logs, I received 4855 REGISTER
from this device from "Mar 25 03:47:09" to
"Mar 25 07:56:13" which is a rate of approx
one new TCP connection every 2.5 seconds...</div>
<div><br>
</div>
<div>Today, I decided to check it again around
11am.</div>
<div><br>
</div>
<div>jack@<a class="gmail-m_7447434029031418155gmail-m_427341605902023125moz-txt-link-freetext">sip:~$</a> sudo
kamctl stats tcp</div>
<div>{</div>
<div> "jsonrpc": "2.0",</div>
<div> "result": [</div>
<div> "tcp:con_reset = 1857",</div>
<div> "tcp:con_timeout = 35927",</div>
<div> "tcp:connect_failed = 25",</div>
<div> "tcp:connect_success = 2",</div>
<div> "tcp:current_opened_connections =
2291",</div>
<div> "tcp:current_write_queue_size = 0",</div>
<div> "tcp:established = 80778",</div>
<div> "tcp:local_reject = 0",</div>
<div> "tcp:passive_open = 80776",</div>
<div> "tcp:send_timeout = 2",</div>
<div> "tcp:sendq_full = 0"</div>
<div> ],</div>
<div> "id": 7305</div>
<div>}</div>
<div><br>
</div>
<div>There was still A LOT of established
connections. And the connections have been
established more than 24 hours ago.</div>
<div><br>
</div>
<div>At 11H16:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP
| grep 41.234.242.69 | grep ESTA | wc -l</div>
<div>1161</div>
<div>At 11H22:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP
| grep 41.234.242.69 | grep ESTA | wc -l</div>
<div>1018</div>
<div>At 11H35:</div>
<div>$> lsof -n -l | grep kamailio | grep TCP
| grep 41.234.242.69 | grep ESTA | wc -l</div>
<div>655</div>
<div>At 13H</div>
<div>$> lsof -n -l | grep kamailio | grep TCP
| grep 41.234.242.69 | grep ESTA | wc -l</div>
<div>0</div>
<div><br>
</div>
<div>So the established connections are all gone
now.</div>
<div><br>
</div>
<div>Between 11h16 and 11H35, I was seeing the
server regularly sending [FIN, ACK] over each
TCP established connection, with
retransmissions for all of them. (no incoming
trafic)</div>
<div><br>
</div>
<div>I do not have numbers/capture/stats, but I
think that kamailio was already closing some</div>
<div>connection yesterday. I don't know when
kamailio started to try closing those
connections.</div>
<div><br>
</div>
<div>I'm now back with this status:</div>
<div><br>
</div>
<div>At 13pm:</div>
<div>jack@<a class="gmail-m_7447434029031418155gmail-m_427341605902023125moz-txt-link-freetext">sip:~$</a> sudo
kamctl stats tcp</div>
<div>{</div>
<div> "jsonrpc": "2.0",</div>
<div> "result": [</div>
<div> "tcp:con_reset = 1896",</div>
<div> "tcp:con_timeout = 38042",</div>
<div> "tcp:connect_failed = 26",</div>
<div> "tcp:connect_success = 2",</div>
<div> "tcp:current_opened_connections = 939",</div>
<div> "tcp:current_write_queue_size = 0",</div>
<div> "tcp:established = 81950",</div>
<div> "tcp:local_reject = 0",</div>
<div> "tcp:passive_open = 81948",</div>
<div> "tcp:send_timeout = 2",</div>
<div> "tcp:sendq_full = 0"</div>
<div> ],</div>
<div> "id": 12734</div>
<div>}</div>
<div><br>
</div>
<div>With around 155 registration entries using
TCP and TLS in my location database.</div>
<div><br>
</div>
<div>As you can see,
tcp:current_opened_connections = 939 is still
pretty high compared to</div>
<div>my currently registred users.</div>
<div><br>
</div>
<div>I have "modparam("registrar",
"max_expires", 86400)", because I'm keeping
contact entries (even with TCP connection
down) for push notifications.</div>
<div><br>
</div>
<div>I have "tcp_connection_lifetime=3600"
configured.</div>
<div><br>
</div>
<div>Question 1</div>
<div><br>
</div>
<div>With "tcp_connection_lifetime=3600", I
would expect kamailio to close the established
connection after 3600 seconds without traffic.
It is pretty obvious that no data has been
exchanged over the 4855 established connection
during a day.</div>
<div><br>
</div>
<div>Despite the issue with the Avaya phones is
solved automatically after a day, I guess
similar stuff or happening, at a different
rate, for other users as well. (because
current_opened_connections is way higher than
registred TCP/TLS users)</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Yes, tcp connections should be closed if no traffic
on them for the lifetime duration.</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>Question 2</div>
<div><br>
</div>
<div>I can list TLS connection with "kamctl rpc
tls.list"</div>
<div>Can I get a similar list for TCP? (lsof
returns a lot of duplicates...)</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Yes, see:</p>
<p><a href="http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list" target="_blank">http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list</a></p>
<p>Maybe you can compare what is listed by the rpc
command to see what kamailio actually sees as active
connections.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<pre class="gmail-m_7447434029031418155gmail-m_427341605902023125moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_7447434029031418155gmail-m_427341605902023125moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="gmail-m_7447434029031418155gmail-m_427341605902023125moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="gmail-m_7447434029031418155gmail-m_427341605902023125moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="gmail-m_7447434029031418155gmail-m_427341605902023125moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a></pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail-m_7447434029031418155gmail_signature"><img src="http://sip.antisip.com/am48.png">Antisip - <a href="http://www.antisip.com" target="_blank">http://www.antisip.com</a><br>
</div>
</div>
</div>
</div>
</blockquote>
<pre class="gmail-m_7447434029031418155moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_7447434029031418155moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="gmail-m_7447434029031418155moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="gmail-m_7447434029031418155moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="gmail-m_7447434029031418155moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><img src="http://sip.antisip.com/am48.png">Antisip - <a href="http://www.antisip.com" target="_blank">http://www.antisip.com</a><br></div>