[SR-Users] Seeing Large Waits

[Alex Balashov]

Hi Mack,

Happy to help!

There must be a hidden source of I/O wait somewhere. It may not be
database calls, but there's something. This shouldn't be happening in a
mostly static config. 

Can you share the numbers from your load testing? At what point did you
start seeing retransmissions, if any? What kind of delays are you
seeing?

-- Alex

On Thu, May 03, 2018 at 12:48:38AM -0400, Mack Hendricks wrote:

> Hey Alex,
> 
> Thanks for the insight. I remember reading your article before and it was super helpful. I think we disabled all of the heavy database operations, but I will double check.
> 
> We are building the RURI using the cfg_rpc module, so we aren’t hitting a database.   We have a failure route that causes another RURI to be created, which sends the call to a Freeswitch server to play a message.
> 
> Let me know if you see any issue with this, I will double check the database calls to be safe.
> 
> Sent from my iPhone
> 
> > On May 2, 2018, at 11:36 PM, Alex Balashov <abalashov at evaristesys.com> wrote:
> > 
> > Hi Mack,
> > 
> > What sort of end-to-end delays are you seeing, from receipt of request
> > until immediately before t_relay()?
> > 
> > You can profile this with the benchmark module:
> > 
> >   https://kamailio.org/docs/modules/5.1.x/modules/benchmark.html
> > 
> > Or my preferred method, the hard way:
> > 
> >        # At the practical start of request processing.
> > 
> >        $avp(proc_start) = $TV(Sn);
> > 
> >        # At the end, right before calling t_relay() or forward().
> > 
> >        $var(cur_time) = $TV(Sn);
> > 
> >        $var(proc_diff) = (
> >                (((
> >                  $(var(cur_time){s.select,0,.}{s.int}) -
> >                  $(avp(proc_start){s.select,0,.}{s.int})
> >                ) * 1000000)
> >                +
> >                (
> >                 $(var(cur_time){s.select,1,.}{s.int}) -
> >                 $(avp(proc_start){s.select,1,.}{s.int})
> >                ) / 1000) mod 1000
> >        );
> > 
> >        xlog("L_INFO", "[R-MAIN:$ci] -> Request processing delay: $var(proc_diff) ms\n"););
> > 
> > Also, for general background on this topic, you might consider my
> > article on the subject of Kamailio performance characteristics:
> > 
> >   http://blog.csrpswitch.com/tuning-kamailio-for-high-throughput-and-performance/
> > 
> > If I had to guess, shooting from the hip, you've got some really slow
> > database queries somewhere.
> > 
> > -- Alex
> > 
> >> On Wed, May 02, 2018 at 06:19:00PM -0400, Mack Hendricks wrote:
> >> 
> >> Hey All,
> >> 
> >> We are load testing Kamailio 4.2.3 and we are seeing the waiting within the transaction manager increasing.   I’ve increased the children from 8 to 16 to 24, but we are still seeing a large amount of waiting.  I don’t see an errors in the logs.   Can anyone give me a clue on where to look. 
> >> 
> >> 05-02-18_03:07:38 PM
> >> tm.stats
> >> {
> >>        current: 390
> >>        waiting: 226
> >>        total: 2361
> >>        total_local: 0
> >>        replied_locally: 1406
> >>        6xx: 0
> >>        5xx: 0
> >>        4xx: 88
> >>        3xx: 0
> >>        2xx: 3180
> >>        created: 2361
> >>        freed: 1971
> >>        delayed_free: 0
> >> }
> >> 
> >> Mack Hendricks / Head of Support / dOpenSource
> >> web: http://dopensource.com <http://dopensource.com/>
> >> support: +888-907-2085
> >> dSIPRouter <http://dsiprouter.org/> - GUI focused on implementing Kamailio to provide SIP Trunking and PBX Hosting Services
> >> 
> > 
> >> _______________________________________________
> >> Kamailio (SER) - Users Mailing List
> >> sr-users at lists.kamailio.org
> >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> > 
> > 
> > -- 
> > Alex Balashov | Principal | Evariste Systems LLC
> > 
> > Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
> > Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

-- 
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) 
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/