Does anyone know what causes this? */set_timer for 1 list called on a "detached" timer -- ignoring /* I also see */set_timer for 3 list called on a "detached" timer -- ignoring /* When this happens Openser seems to lock up for 10 seconds or so. what the race is or why this results in an unresponsive openser instance for multiple seconds. Transaction expiration racing reply? Desperately need to understand how this could be triggered so I can get customer to adjust system. Any way to adjust? tried tweaking fr_inv_timer but no joy. TR ------------------------------------------------------------------------ _______________________________________________ Devel mailing list Devel(a)openser.org http://openser.org/cgi-bin/mailman/listinfo/devel

Hi TR, it is race between expire even (from timer) and inserting again on a timer list. 1 is the final response timer list (fr_timer) 3 id the wait timer list (wt_timer) I would say there is no way this could leas to a any kind of lock. what version are you using? what makes you say it locks? regards, bogdan T.R. Missner wrote:

Is it possible the locked state I am seeing with openser leads to the "detached" timer? Since the "detached" timer is a race, it would make sense to see the race condition after openser locks up and messages buffer up in the stack. When a bunch of messages are processed all at once by multiple threads the race condition would occur. Does this make sense? Maybe I have been focusing on the wrong place. Ignoring the "detached" timer what could cause openser to hang for a couple seconds then clear every 5 - 10 minutes? Ideas? We are seeing this on 3 different productions servers. Thanks TR using openser1.1.1 T.R. Missner wrote: _______________________________________________ Users mailing list Users(a)openser.org http://openser.org/cgi-bin/mailman/listinfo/users

jiri, that is incorrect information - there was no plan / need / acceptance for the port you are mentioning. I thing you realize that the TM+timer code from SER and OpenSER is following different direction of development, which makes quite unrealistic your sayings.

wrong again :)

as I mentioned in my previous email, the "detached timer" was more an maker that something else was going wrong - there was no amplification.

and as TR clearly said, the problem was with DB connectivity and had nothing to do with TM timers.

regards, bogdan Jiri Kuthan wrote: > Actually more likely it has been both. The root problem lies in the timer subsystem > and may be amplified by other troubles (or amplify those). > > -jiri > > At 01:35 30/03/2007, T.R. Missner wrote: > > >> FYI All >> >> This turned out to be a database write ( acc ) that was blocking due to a raid card problem. >> >> >> >> T.R. Missner wrote: >> >> >>> Is it possible the locked state I am seeing with openser leads to the "detached" timer? >>> Since the "detached" timer is a race, it would make sense to see the race condition after openser locks up and messages buffer up in the stack. >>> When a bunch of messages are processed all at once by multiple threads the race condition would occur. >>> Does this make sense? >>> >>> Maybe I have been focusing on the wrong place. >>> >>> Ignoring the "detached" timer what could cause openser to hang for a couple seconds then clear every 5 - 10 minutes? >>> >>> Ideas? >>> >>> We are seeing this on 3 different productions servers. >>> >>> Thanks >>> >>> TR >>> >>> using openser1.1.1 >>> >>> >>> >>> T.R. Missner wrote: >>> >>> >>>> Bogdan, >>>> >>>> I have been chasing this for days and done lots of debugging. >>>> using 1.1.1 >>>> While looking at the network trace at the time of these messages ( I usually see at least 5 in a row with differing hex values ) I see many incoming packets coming into the box and no response from the proxy for somewhere between 5 - 10 seconds, then a flood a responses from the proxy. >>>> I can email you a sample pcap file if you like. >>>> As part of my debugging I forced a 100 reply at the very top of my cfg file. >>>> The forced 100 was not sent during the locked up time leading me to believe openser was not processing incoming packets. >>>> I have now seen this on multiple servers in different locations. Likely a particular customer call flow is causing this but I have not been able to pin it down to the exact customer. These proxies run pretty fast during the day so finding a pattern leading up the this issue is difficult. What could I add to the Log output to identify the offending sip-callid? Is sip-callid or branch tag or anything similar easily accessible in any of the data structs in timer.c? >>>> >>>> TR >>>> >>>> Bogdan-Andrei Iancu wrote: >>>> >>>> >>>>> Hi TR, >>>>> >>>>> it is race between expire even (from timer) and inserting again on a timer list. >>>>> 1 is the final response timer list (fr_timer) >>>>> 3 id the wait timer list (wt_timer) >>>>> >>>>> I would say there is no way this could leas to a any kind of lock. >>>>> >>>>> what version are you using? what makes you say it locks? >>>>> >>>>> regards, >>>>> bogdan >>>>> >>>>> T.R. Missner wrote: >>>>> >>>>> >>>>>> Does anyone know what causes this? >>>>>> >>>>>> */set_timer for 1 list called on a "detached" timer -- ignoring /* >>>>>> >>>>>> I also see >>>>>> >>>>>> */set_timer for 3 list called on a "detached" timer -- ignoring /* >>>>>> >>>>>> >>>>>> >>>>>> When this happens Openser seems to lock up for 10 seconds or so. >>>>>> >>>>>> >From searching it appears this is caused by a race but I am not sure what the race is or why this results in an unresponsive openser instance for multiple seconds. >>>>>> >>>>>> Transaction expiration racing reply? >>>>>> >>>>>> >>>>>> Desperately need to understand how this could be triggered so I can get customer to adjust system. >>>>>> >>>>>> Any way to adjust? >>>>>> >>>>>> tried tweaking fr_inv_timer but no joy. >>>>>> >>>>>> >>>>>> >>>>>> TR >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Jiri Kuthan http://iptel.org/~jiri/ >>>>>

Hi Jerome, right, sometime is good to have thinks as clear as possible :).

as update on the topic, openser 1.2 has a new, improved timer implementation (core and TM) - actually done by myself :) - and part of the performance boost roots from there.

For people interested in looking at what SER does, this message led me to take a look a SER 2.0 documentation and bits of code, and it is not immediatly evident that they are taking a radically different route ... they also improved timer granularity (down to a resolution of 62.5 ms).

They also are currently developping a very interesting module called "timer", which provides the ability to set timers on-the-fly, with callback implemented as routes called when the custom timers fire. This seems pretty simple in their model, the timer module being only 408 lines long (but I can't tell if this works already or not).

An other puzzling fact is that SER's implementation of timers in tm module is about half the size as OpenSER's .... I'm not sure we can infer anything from this fact, still it made me curious.

The key thing (in addition to minor) is elimination of race conditions.

Neither am I.

Not that there would not be good progress -- the 1.2.0 release list seems to have great deal of inspiration from ottendorf, it is just I don't understand why some folks are upset about fixing TM.