[SR-Users] TCP connection lifetime
Daniel-Constantin Mierla
miconda at gmail.com
Wed Apr 17 15:51:42 CEST 2019
Hello,
as you said it shows a single connections, I went to the code and I
discovered a bug in exporting rpc command core.tcp_info, because it was
missing the option that it returns an array. I fixed it in master branch
with next commit:
-
https://github.com/kamailio/kamailio/commit/24ca2e7760a8dada433b188348c768e7e224f10d
and I will backport to stable branches.
Meanwhile, you can use:
kamcmd core.tcp_list
which is not strict in validating the binrcp/jsonrpc response and
eventually it will print all the tcp connections. Can you test that?
Cheers,
Daniel
On 17.04.19 15:42, Aymeric Moizard wrote:
> Hi Daniel,
>
> Tks for answering! Unfortunatly, core.tcp_list is only returning one
> connection.
> But core.tcp_info reports 184 opened connections (same for
> "kamctl stats tcp")
>
> sudo kamctl rpc core.tcp_list
> {
> "jsonrpc": "2.0",
> "result": {
> "id": 439290,
> "type": "TCP",
> "state": "CONN_ACCEPT",
> "timeout": 2567,
> "lifetime": 3600,
> "ref_count": 1,
> "src_ip": "41.46.4.235",
> "src_port": 4957,
> "dst_ip": "91.121.30.149",
> "dst_port": 5060
> },
> "id": 9158
> }
>
> sudo kamctl rpc core.tcp_info
> {
> "jsonrpc": "2.0",
> "result": {
> "readers": 16,
> "max_connections": 50000,
> "max_tls_connections": 50000,
> "opened_connections": 184,
> "opened_tls_connections": 64,
> "write_queued_bytes": 0
> },
> "id": 9523
> }
>
> Did I missed something?
>
>
> jack at sip:~$ /usr/sbin/kamailio -v
> version: kamailio 5.2.2 (x86_64/linux)
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
> SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
> USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
> USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE
> 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: unknown
> compiled with gcc 6.3.0
>
> Regards
> Aymeric
>
> Le lun. 15 avr. 2019 à 09:10, Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> a écrit :
>
> Hello,
>
> On 26.03.19 17:16, Aymeric Moizard wrote:
>> Hi Again,
>>
>> Here is an issue with TCP connection being kept for more:
>>
>> Yesterday, I have discovered that a User-Agent (<Avaya IP Phone
>> 1120E (SIP1120e.04.04.30.00)> tried to register a lot. It was
>> sending REGISTER over new established TCP socket *every 2 seconds*.
>>
>> All the REGISTER was rejected with 401. (may be the device was
>> misconfigured? or not receiving any of my answer? I can't tell)
>>
>> NOTE: You can see the expires header was very large: 86400, ie:
>> 24 hours...
>>
>> I was checking the TCP/TLS connections on my server and
>> discovered more than 1000 TCP established connection to that
>> user/ip, and thus, I have tried to understand what happened.
>>
>> Checking the logs, I received 4855 REGISTER from this device from
>> "Mar 25 03:47:09" to "Mar 25 07:56:13" which is a rate of approx
>> one new TCP connection every 2.5 seconds...
>>
>> Today, I decided to check it again around 11am.
>>
>> jack at sip:~$ sudo kamctl stats tcp
>> {
>> "jsonrpc": "2.0",
>> "result": [
>> "tcp:con_reset = 1857",
>> "tcp:con_timeout = 35927",
>> "tcp:connect_failed = 25",
>> "tcp:connect_success = 2",
>> "tcp:current_opened_connections = 2291",
>> "tcp:current_write_queue_size = 0",
>> "tcp:established = 80778",
>> "tcp:local_reject = 0",
>> "tcp:passive_open = 80776",
>> "tcp:send_timeout = 2",
>> "tcp:sendq_full = 0"
>> ],
>> "id": 7305
>> }
>>
>> There was still A LOT of established connections. And the
>> connections have been established more than 24 hours ago.
>>
>> At 11H16:
>> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 |
>> grep ESTA | wc -l
>> 1161
>> At 11H22:
>> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 |
>> grep ESTA | wc -l
>> 1018
>> At 11H35:
>> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 |
>> grep ESTA | wc -l
>> 655
>> At 13H
>> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 |
>> grep ESTA | wc -l
>> 0
>>
>> So the established connections are all gone now.
>>
>> Between 11h16 and 11H35, I was seeing the server regularly
>> sending [FIN, ACK] over each TCP established connection, with
>> retransmissions for all of them. (no incoming trafic)
>>
>> I do not have numbers/capture/stats, but I think that kamailio
>> was already closing some
>> connection yesterday. I don't know when kamailio started to try
>> closing those connections.
>>
>> I'm now back with this status:
>>
>> At 13pm:
>> jack at sip:~$ sudo kamctl stats tcp
>> {
>> "jsonrpc": "2.0",
>> "result": [
>> "tcp:con_reset = 1896",
>> "tcp:con_timeout = 38042",
>> "tcp:connect_failed = 26",
>> "tcp:connect_success = 2",
>> "tcp:current_opened_connections = 939",
>> "tcp:current_write_queue_size = 0",
>> "tcp:established = 81950",
>> "tcp:local_reject = 0",
>> "tcp:passive_open = 81948",
>> "tcp:send_timeout = 2",
>> "tcp:sendq_full = 0"
>> ],
>> "id": 12734
>> }
>>
>> With around 155 registration entries using TCP and TLS in my
>> location database.
>>
>> As you can see, tcp:current_opened_connections = 939 is still
>> pretty high compared to
>> my currently registred users.
>>
>> I have "modparam("registrar", "max_expires", 86400)", because I'm
>> keeping contact entries (even with TCP connection down) for push
>> notifications.
>>
>> I have "tcp_connection_lifetime=3600" configured.
>>
>> Question 1
>>
>> With "tcp_connection_lifetime=3600", I would expect kamailio to
>> close the established connection after 3600 seconds without
>> traffic. It is pretty obvious that no data has been exchanged
>> over the 4855 established connection during a day.
>>
>> Despite the issue with the Avaya phones is solved automatically
>> after a day, I guess similar stuff or happening, at a different
>> rate, for other users as well. (because
>> current_opened_connections is way higher than registred TCP/TLS
>> users)
>
>
> Yes, tcp connections should be closed if no traffic on them for
> the lifetime duration.
>
>
>>
>> Question 2
>>
>> I can list TLS connection with "kamctl rpc tls.list"
>> Can I get a similar list for TCP? (lsof returns a lot of
>> duplicates...)
>
>
> Yes, see:
>
> http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list
>
> Maybe you can compare what is listed by the rpc command to see
> what kamailio actually sees as active connections.
>
> Cheers,
> Daniel
>
> --
> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com <http://www.kamailioworld.com>
>
>
>
> --
> Antisip - http://www.antisip.com
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190417/70e3dd32/attachment.html>
More information about the sr-users
mailing list