[SR-Users] TCP connection lifetime

Aymeric Moizard amoizard at gmail.com
Wed Apr 17 15:42:49 CEST 2019


Hi Daniel,

Tks for answering! Unfortunatly, core.tcp_list is only returning one
connection.
But core.tcp_info reports 184 opened connections (same for "kamctl stats
tcp")

sudo kamctl rpc core.tcp_list
{
  "jsonrpc":  "2.0",
  "result": {
    "id": 439290,
    "type": "TCP",
    "state":  "CONN_ACCEPT",
    "timeout":  2567,
    "lifetime": 3600,
    "ref_count":  1,
    "src_ip": "41.46.4.235",
    "src_port": 4957,
    "dst_ip": "91.121.30.149",
    "dst_port": 5060
  },
  "id": 9158
}

sudo kamctl rpc core.tcp_info
{
  "jsonrpc":  "2.0",
  "result": {
    "readers":  16,
    "max_connections":  50000,
    "max_tls_connections":  50000,
    "opened_connections": 184,
    "opened_tls_connections": 64,
    "write_queued_bytes": 0
  },
  "id": 9523
}

Did I missed something?


jack at sip:~$ /usr/sbin/kamailio -v
version: kamailio 5.2.2 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 6.3.0

Regards
Aymeric

Le lun. 15 avr. 2019 à 09:10, Daniel-Constantin Mierla <miconda at gmail.com>
a écrit :

> Hello,
> On 26.03.19 17:16, Aymeric Moizard wrote:
>
> Hi Again,
>
> Here is an issue with TCP connection being kept for more:
>
> Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E
> (SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER
> over new established TCP socket *every 2 seconds*.
>
> All the REGISTER was rejected with 401. (may be the device was
> misconfigured? or not receiving any of my answer? I can't tell)
>
> NOTE: You can see the expires header was very large: 86400, ie: 24 hours...
>
> I was checking the TCP/TLS connections on my server and discovered more
> than 1000 TCP established connection to that user/ip, and thus, I have
> tried to understand what happened.
>
> Checking the logs, I received 4855 REGISTER from this device from "Mar 25
> 03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP
> connection every 2.5 seconds...
>
> Today, I decided to check it again around 11am.
>
> jack at sip:~$ sudo kamctl stats tcp
> {
>   "jsonrpc":  "2.0",
>   "result": [
>     "tcp:con_reset = 1857",
>     "tcp:con_timeout = 35927",
>     "tcp:connect_failed = 25",
>     "tcp:connect_success = 2",
>     "tcp:current_opened_connections = 2291",
>     "tcp:current_write_queue_size = 0",
>     "tcp:established = 80778",
>     "tcp:local_reject = 0",
>     "tcp:passive_open = 80776",
>     "tcp:send_timeout = 2",
>     "tcp:sendq_full = 0"
>   ],
>   "id": 7305
> }
>
> There was still A LOT of established connections. And the connections have
> been established more than 24 hours ago.
>
> At 11H16:
> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA
> | wc -l
> 1161
> At 11H22:
> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA
> | wc -l
> 1018
> At 11H35:
> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA
> | wc -l
> 655
> At 13H
> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA
> | wc -l
> 0
>
> So the established connections are all gone now.
>
> Between 11h16 and 11H35, I was seeing the server regularly sending [FIN,
> ACK] over each TCP established connection, with retransmissions for all of
> them. (no incoming trafic)
>
> I do not have numbers/capture/stats, but I think that kamailio was already
> closing some
> connection yesterday. I don't know when kamailio started to try closing
> those connections.
>
> I'm now back with this status:
>
> At 13pm:
> jack at sip:~$ sudo kamctl stats tcp
> {
>   "jsonrpc":  "2.0",
>   "result": [
>     "tcp:con_reset = 1896",
>     "tcp:con_timeout = 38042",
>     "tcp:connect_failed = 26",
>     "tcp:connect_success = 2",
>     "tcp:current_opened_connections = 939",
>     "tcp:current_write_queue_size = 0",
>     "tcp:established = 81950",
>     "tcp:local_reject = 0",
>     "tcp:passive_open = 81948",
>     "tcp:send_timeout = 2",
>     "tcp:sendq_full = 0"
>   ],
>   "id": 12734
> }
>
> With around 155 registration entries using TCP and TLS in my location
> database.
>
> As you can see, tcp:current_opened_connections = 939 is still pretty high
> compared to
> my currently registred users.
>
> I have "modparam("registrar", "max_expires", 86400)", because I'm keeping
> contact entries (even with TCP connection down) for push notifications.
>
> I have "tcp_connection_lifetime=3600" configured.
>
> Question 1
>
> With "tcp_connection_lifetime=3600", I would expect kamailio to close the
> established connection after 3600 seconds without traffic. It is pretty
> obvious that no data has been exchanged over the 4855 established
> connection during a day.
>
> Despite the issue with the Avaya phones is solved automatically after a
> day, I guess similar stuff or happening, at a different rate, for other
> users as well. (because  current_opened_connections is way higher than
> registred TCP/TLS users)
>
>
> Yes, tcp connections should be closed if no traffic on them for the
> lifetime duration.
>
>
>
> Question 2
>
> I can list TLS connection with "kamctl rpc tls.list"
> Can I get a similar list for TCP? (lsof returns a lot of duplicates...)
>
>
> Yes, see:
>
>
> http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list
>
> Maybe you can compare what is listed by the rpc command to see what
> kamailio actually sees as active connections.
>
> Cheers,
> Daniel
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
>
>

-- 
Antisip - http://www.antisip.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190417/4c79b531/attachment.html>


More information about the sr-users mailing list