[SR-Users] TCP connection lifetime
Aymeric Moizard
amoizard at gmail.com
Wed Apr 17 16:41:23 CEST 2019
Hi Daniel!
Tks a lot:
$> kamcmd core.tcp_list
Worked!
Next time I meet the issue, I will have more data to analyse...
I guess I will be able to compare "timeout" vs "lifetime"
Tks a lot!
Aymeric
Le mer. 17 avr. 2019 à 15:51, Daniel-Constantin Mierla <miconda at gmail.com>
a écrit :
> Hello,
>
> as you said it shows a single connections, I went to the code and I
> discovered a bug in exporting rpc command core.tcp_info, because it was
> missing the option that it returns an array. I fixed it in master branch
> with next commit:
>
> -
> https://github.com/kamailio/kamailio/commit/24ca2e7760a8dada433b188348c768e7e224f10d
>
> and I will backport to stable branches.
>
> Meanwhile, you can use:
>
> kamcmd core.tcp_list
>
> which is not strict in validating the binrcp/jsonrpc response and
> eventually it will print all the tcp connections. Can you test that?
>
> Cheers,
> Daniel
> On 17.04.19 15:42, Aymeric Moizard wrote:
>
> Hi Daniel,
>
> Tks for answering! Unfortunatly, core.tcp_list is only returning one
> connection.
> But core.tcp_info reports 184 opened connections (same for "kamctl stats
> tcp")
>
> sudo kamctl rpc core.tcp_list
> {
> "jsonrpc": "2.0",
> "result": {
> "id": 439290,
> "type": "TCP",
> "state": "CONN_ACCEPT",
> "timeout": 2567,
> "lifetime": 3600,
> "ref_count": 1,
> "src_ip": "41.46.4.235",
> "src_port": 4957,
> "dst_ip": "91.121.30.149",
> "dst_port": 5060
> },
> "id": 9158
> }
>
> sudo kamctl rpc core.tcp_info
> {
> "jsonrpc": "2.0",
> "result": {
> "readers": 16,
> "max_connections": 50000,
> "max_tls_connections": 50000,
> "opened_connections": 184,
> "opened_tls_connections": 64,
> "write_queued_bytes": 0
> },
> "id": 9523
> }
>
> Did I missed something?
>
>
> jack at sip:~$ /usr/sbin/kamailio -v
> version: kamailio 5.2.2 (x86_64/linux)
> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
> USE_DST_BLACKLIST, HAVE_RESOLV_RES
> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024,
> BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
> id: unknown
> compiled with gcc 6.3.0
>
> Regards
> Aymeric
>
> Le lun. 15 avr. 2019 à 09:10, Daniel-Constantin Mierla <miconda at gmail.com>
> a écrit :
>
>> Hello,
>> On 26.03.19 17:16, Aymeric Moizard wrote:
>>
>> Hi Again,
>>
>> Here is an issue with TCP connection being kept for more:
>>
>> Yesterday, I have discovered that a User-Agent (<Avaya IP Phone 1120E
>> (SIP1120e.04.04.30.00)> tried to register a lot. It was sending REGISTER
>> over new established TCP socket *every 2 seconds*.
>>
>> All the REGISTER was rejected with 401. (may be the device was
>> misconfigured? or not receiving any of my answer? I can't tell)
>>
>> NOTE: You can see the expires header was very large: 86400, ie: 24
>> hours...
>>
>> I was checking the TCP/TLS connections on my server and discovered more
>> than 1000 TCP established connection to that user/ip, and thus, I have
>> tried to understand what happened.
>>
>> Checking the logs, I received 4855 REGISTER from this device from "Mar 25
>> 03:47:09" to "Mar 25 07:56:13" which is a rate of approx one new TCP
>> connection every 2.5 seconds...
>>
>> Today, I decided to check it again around 11am.
>>
>> jack at sip:~$ sudo kamctl stats tcp
>> {
>> "jsonrpc": "2.0",
>> "result": [
>> "tcp:con_reset = 1857",
>> "tcp:con_timeout = 35927",
>> "tcp:connect_failed = 25",
>> "tcp:connect_success = 2",
>> "tcp:current_opened_connections = 2291",
>> "tcp:current_write_queue_size = 0",
>> "tcp:established = 80778",
>> "tcp:local_reject = 0",
>> "tcp:passive_open = 80776",
>> "tcp:send_timeout = 2",
>> "tcp:sendq_full = 0"
>> ],
>> "id": 7305
>> }
>>
>> There was still A LOT of established connections. And the connections
>> have been established more than 24 hours ago.
>>
>> At 11H16:
>> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA
>> | wc -l
>> 1161
>> At 11H22:
>> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA
>> | wc -l
>> 1018
>> At 11H35:
>> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA
>> | wc -l
>> 655
>> At 13H
>> $> lsof -n -l | grep kamailio | grep TCP | grep 41.234.242.69 | grep ESTA
>> | wc -l
>> 0
>>
>> So the established connections are all gone now.
>>
>> Between 11h16 and 11H35, I was seeing the server regularly sending [FIN,
>> ACK] over each TCP established connection, with retransmissions for all of
>> them. (no incoming trafic)
>>
>> I do not have numbers/capture/stats, but I think that kamailio was
>> already closing some
>> connection yesterday. I don't know when kamailio started to try closing
>> those connections.
>>
>> I'm now back with this status:
>>
>> At 13pm:
>> jack at sip:~$ sudo kamctl stats tcp
>> {
>> "jsonrpc": "2.0",
>> "result": [
>> "tcp:con_reset = 1896",
>> "tcp:con_timeout = 38042",
>> "tcp:connect_failed = 26",
>> "tcp:connect_success = 2",
>> "tcp:current_opened_connections = 939",
>> "tcp:current_write_queue_size = 0",
>> "tcp:established = 81950",
>> "tcp:local_reject = 0",
>> "tcp:passive_open = 81948",
>> "tcp:send_timeout = 2",
>> "tcp:sendq_full = 0"
>> ],
>> "id": 12734
>> }
>>
>> With around 155 registration entries using TCP and TLS in my location
>> database.
>>
>> As you can see, tcp:current_opened_connections = 939 is still pretty high
>> compared to
>> my currently registred users.
>>
>> I have "modparam("registrar", "max_expires", 86400)", because I'm keeping
>> contact entries (even with TCP connection down) for push notifications.
>>
>> I have "tcp_connection_lifetime=3600" configured.
>>
>> Question 1
>>
>> With "tcp_connection_lifetime=3600", I would expect kamailio to close the
>> established connection after 3600 seconds without traffic. It is pretty
>> obvious that no data has been exchanged over the 4855 established
>> connection during a day.
>>
>> Despite the issue with the Avaya phones is solved automatically after a
>> day, I guess similar stuff or happening, at a different rate, for other
>> users as well. (because current_opened_connections is way higher than
>> registred TCP/TLS users)
>>
>>
>> Yes, tcp connections should be closed if no traffic on them for the
>> lifetime duration.
>>
>>
>>
>> Question 2
>>
>> I can list TLS connection with "kamctl rpc tls.list"
>> Can I get a similar list for TCP? (lsof returns a lot of duplicates...)
>>
>>
>> Yes, see:
>>
>>
>> http://www.kamailio.org/docs/docbooks/devel/rpc_list/rpc_list.html#core.tcp_list
>>
>> Maybe you can compare what is listed by the rpc command to see what
>> kamailio actually sees as active connections.
>>
>> Cheers,
>> Daniel
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
>>
>>
>
> --
> Antisip - http://www.antisip.com
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio World Conference - May 6-8, 2019 -- www.kamailioworld.com
>
>
--
Antisip - http://www.antisip.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190417/c7d60486/attachment.html>
More information about the sr-users
mailing list