I can check with UDP next week. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-325139566

What happened was that the firewall closed the TCP connection after a while, Kamailio continued sending but got dropped. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-325330425

Not iptables -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-325331558

Note for the future: Maybe an event_route for failed replications would be a good thing. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-325332824

After fixing the firewall we still have issues. I see the TCP keepalives coming, but replication stopped, no SIP messages flowing and tm.stats has a big pile of open transactions, no more shm memory available. Current and waiting seems to follow each other. No calls during the night, so this is only registrations. 300 devices connected. I will have to go back to replication over UDP or stateless replication using forward to get stability. ``` # kamcmd tm.stats { current: 19965 waiting: 19965 total: 42187 total_local: 0 rpl_received: 42207 rpl_generated: 10 rpl_sent: 42211 6xx: 0 5xx: 1 4xx: 4 3xx: 0 2xx: 42191 created: 42187 freed: 22221 delayed_free: 0 } ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-326213777

I tried to reproduce using master, but all ok. Based on default kamailio.cfg, I have added t_replicate_tcp: ``` # Handle SIP registrations route[REGISTRAR] { if (!is_method("REGISTER")) return; if(isflagset(FLT_NATS)) { setbflag(FLB_NATB); #!ifdef WITH_NATSIPPING # do SIP NAT pinging setbflag(FLB_NATSIPPING); #!endif } if (!save("location")) { sl_reply_error(); } t_replicate_tcp("127.0.0.1", 9); exit; } ``` In logs I get: ``` 39(81014) ERROR: <core> [core/tcp_main.c:4135]: handle_tcpconn_ev(): connect 127.0.0.1:9 failed ``` and after a while (delete wait time) transaction is cleaned: ``` 25(81000) DEBUG: tm [t_reply.c:1263]: t_should_relay_response(): ->>>>>>>>> T_code=0, new_code=408 25(81000) DEBUG: tm [t_reply.c:2072]: local_reply(): branch=0, save=0, winner=0 25(81000) DEBUG: tm [t_reply.c:2111]: local_reply(): local transaction completed 408/0 (totag retr: 0/0) 26(81001) DEBUG: tm [h_table.c:129]: free_cell_helper(): freeing transaction 0x105712170 from timer.c:651 ``` `tm.stats` showing 0 current transactions. What version are you using? What are the parameters you set for tm module? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-326297453

Latest release as snapshot/tarball 5.0.2, or latest branch 5.0 at some point in time? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-326302491

No strange tm modparams: ``` # ----- tm params ----- # auto-discard branches from previous serial forking leg modparam("tm", "failure_reply_mode", 3) # default retransmission timeout: 30sec modparam("tm", "fr_timer", 30000) # default invite retransmission timeout after 1xx: 120sec modparam("tm", "fr_inv_timer", 120000) modparam("tm", "retr_timer1", 100) modparam("tm", "retr_timer2", 4000) ``` After the TCP problems started, we added som tcp params to the config, just to verify the settings. Mostly default. ``` tcp_connection_lifetime=3200 tcp_accept_no_cl=yes # Maximum write queue. If this is reached, the connection is killed. tcp_conn_wq_max=10000 # Enables keepalive for tcp (sets SO_KEEPALIVE socket option) tcp_keepalive = yes # Time before starting to send keepalives, if the connection is idle (TCP_KEEPIDLE socket option). Linux only. # Seconds tcp_keepidle = 10 # Number of keepalives sent before dropping the connection (TCP_KEEPCNT socket option). Linux only. tcp_keepcnt = 20 # Time interval between keepalive probes, when the previous probe failed (TCP_KEEPINTVL socket option). Linux only. # Seconds tcp_keepintvl = 20 # Time in seconds after a TCP connection will be closed if it is not available for writing in this # interval (and Kamailio wants to send something on it). Lower this value for faster detection of broken # TCP connections. The default value is 10s. tcp_send_timeout=10 # Number of SYN retransmissions before aborting a connect attempt (see linux tcp(7) TCP_SYNCNT). Linux only. # Default in LInux seems to be 5, which corresponds to 180 seconds tcp_syncnt = 2 ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-326885930

After three hours I have 14726 transactions waiting using around 300 accounts for REGISTER. I will restart now and see how fast I notice a new queue. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-326930453

Started test 12:42, all seems quiet and current goes up and down around 20. At 13:14 it begins to climb again, so around 30 minutes quiet. We have two-minute registration timeouts, so at the time it starts running astray we have processed 6762 transactions. All over UDP. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-326965067

replication stops. Can't find anything unusual in the syslogs. Can attach GDB, but have no clue on how to look at the transaction structures. Need some help :-) -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-326979140

There is something fishy going on here. I removed t_replicate and the server has been running for six days without getting into trouble. However when I run kamcmd tm.stats there are 767 waiting transactions, not freed. It's a very basic configuration using the dialog module, the dialog event routes. A few calls, but not much over the weekend. I will try to debug a big harder and find out where these come from. T_replicate really forced the issue and the server bogged down quickly, but it seems like ordinary calls also get transactions that hang around. That's not good. ``` { current: 780 waiting: 767 total: 792 total_local: 0 rpl_received: 1389 rpl_generated: 339 rpl_sent: 1514 6xx: 4 5xx: 285 4xx: 172 3xx: 0 2xx: 519 created: 792 freed: 11 delayed_free: 0 } ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-328437649

I couldn't find a reason after looking around, likely something is triggered at some point, in certain conditions. As a workaround, I am considering to add a hard cleanup on timer to the transactions that leave too long after lifetime, with a log message to gather some useful information for troubleshooting. If you can still get the output for gdb script on such transactions is still useful. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-328803937

I will try installing latest branch. This is the current stats, something is definitely keeping transactions open. ``` # kamcmd tm.stats { current: 5060 waiting: 5055 total: 5065 total_local: 0 rpl_received: 5415 rpl_generated: 266 rpl_sent: 5543 6xx: 1 5xx: 4614 4xx: 53 3xx: 0 2xx: 447 created: 5065 freed: 4 delayed_free: 0 } # kamcmd core.uptime { now: Mon Sep 18 09:39:51 2017 up_since: Tue Sep 12 12:12:25 2017 uptime: 509246 } ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-330148402

Have you tried with gdb script? That should work with any version. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-330154626

Finally have a day where it's possible to run som tests. Will try to fire up GDB :-) Building latest 5.0 from git. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-345624236

What version are you running now there? If the issue is reproducible with a lot of calls via sipp, then it sounds like not being related to t_replicate() of registrations... -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-357607076

While running SIPP and checking statistics, I see that inuse_transactions keep going up, but active_transactions stay low ``` tmx:active_transactions = 5 tmx:inuse_transactions = 1054 ``` What is the difference between these? -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-357656678

After a while, I can clearly see infuse_transactions growing ``` tmx:2xx_transactions = 3472 tmx:3xx_transactions = 0 tmx:4xx_transactions = 1078 tmx:5xx_transactions = 2913 tmx:6xx_transactions = 2 tmx:UAC_transactions = 5554 tmx:UAS_transactions = 7323 tmx:active_transactions = 2 tmx:inuse_transactions = 7322 tmx:rpl_absorbed = 463 tmx:rpl_generated = 1679 tmx:rpl_received = 8119 tmx:rpl_relayed = 7656 tmx:rpl_sent = 9335 ``` -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-357660042

This server was running on VM ware. We tested with another server and could not see the same behaviour. Still trying to find any differences while installing a third server to test with... Life. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-361590674

OK, good it was sorted out, tm is quite important and used pretty much everywhere. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-364099842

Closing it for now. If something new related appears, reopen or create a new item. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1220#issuecomment-366637925