Re: [SR-Users] Strange application hang querying DB

Just thought I would post a quick update: After further digging around I stumbled upon what looked to a recursive loop in the freetds connection logs. Which led me to some posts on the freetds mailing list to do with handling closed connections by SQL Azure. The crux of the problem seems to be that the load balancer on SQL Azure will close “idle” connections before the SQL Server closes the connection, resulting in the connection being closed without any feedback to the odbc driver, which still maintains the connection reference in its connection pool, thus Kamailio will attempt to utilise a connection which is in effect orphaned. However if you shorten the TCP Keep alive time and interval from the default of 2 hours and 75 seconds respectively you can detect the closed connection and the ODBC driver handles it properly. I have currently set my keep alive time to 120 s (2 mins) with a 10 second interval, and so far I have maintained 4 hours of continuous service. I will continue to monitor the situation, but hopefully this has solved the intermittent issues I was experiencing. Tim. From: sr-users-bounces(a)lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Tim Chubb Sent: 26 June 2014 10:07 To: miconda(a)gmail.com; Kamailio (SER) - Users Mailing List Subject: Re: [SR-Users] Strange application hang querying DB Hi Daniel There is no load on the system as I’m developing the implementation at the moment, I have a softphone and hardphone registered and that is it. Im using Centos 6.5 32bit as the OS and have disabled SElinux whilst developing. There is no rhyme or reason as far as I can tell, sometime’s registration hangs (using auth_db against my Asterisk RT tables), sometimes it’s the dispatcher, and sometimes just calling ‘kamctl dispatcher reload’. One thing I have tried to help narrow the search focus down was run this when I experienced loss of connectivity ‘date && kamctl dispatcher reload && date’ which would give the start and end time of the query, where I seem to experience a 30min delay between start and finish (I have 2 entires in the dispatcher table, so its not a lack of index or anything like that). I have a nasty feeling this may be caused by SQL Azures idle connection handling which is quite different to the standard server implementation (which works frustratingly fine, i.e. if I work from home the kamailio process handles vpn disconnections and resumes operations as soon as I connect). Im going to try disabling connection pooling in unixodbc, see if that helps, if that doesn’t help then I fear I am going to have to brush off my c skills and dive into the db_unixodbc module and general DB logic to get a better understanding of how Kamailio is handling db connections. Tim. From: sr-users-bounces@lists.sip-router.org<mailto:sr-users-bounces@lists.sip-router.org> [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Daniel-Constantin Mierla Sent: 26 June 2014 09:08 To: Kamailio (SER) - Users Mailing List Subject: Re: [SR-Users] Strange application hang querying DB Hello, have you identified the config function that hangs? Is a sqlops function or something related to authentication, location, etc? If not, then you can use benchmark module to measure the duration of execution of various parts of config, in this way identifying the function that takes too long. You should eventually run directly with debug=3 and watch the logs in syslog file. Once you tell the function taking too long, I will try to dig further to see if I can spot any issue. Some other details would help: - is the system under load or it happens only for one request and always? - what is the operating system? I encountered many issues with selinux in the past, under high load or too many connections opened Cheers, Daniel On 25/06/14 20:06, Tim Chubb wrote: Hi, I’m currently wracking my brains trying to deduce where an application hang can be coming from. I have ported the PostGres schema to be compatible with MS Sql Server and have had no issues getting a working dev environment working. However when I try and use SQL Azure as the backend db as opposed to SQL Server 2012 (running locally on my dev machine) Kamailio will seem to hang after about 10 – 15 mins of flawless operation. I have recreated the SQL Azure schema from the working schema running on my dev machine and copied all data in a 1:1 manner so I can be certain it’s not a data quality issue. I am using the same FreeTDS/UnixODBC config that I am using for Asterisk realtime (which doesn’t exhibit this behaviour). The really strange thing is that I can see the queries being sent by Kmailio in the freetds trace log, and the response from SQL Azure, however its as if the db response handler is deadlocked/hung. The kamailio process continues to work fine, and as long as its not required to commiunicate with the DB all appears well i.e. no error messages in the log file. This behaviour only happens when im using SQL Azure as opposed to SQL Server, so there must be something causing the issue, that’s unique to SQL azure as opposed to SQL Server. As far as I can tell this is not something I can create a backtrace for as no core dumps are produced, as far as I can tell there is no easy way debug what the lock is. If anyone has any suggestions I would be very grateful to hear them. One thing that occurs to me is that the db_unixodbc module doesn’t have half as many parameters as its equilent in Asterisk, and that connection pooling may be in use, this could be a bit of a show stopper AFAIK you have to disable connection pooling with SQL server, certainly any of the guides I have come across configuring Asterisk to work with SQL server suggests that. Tim Chubb Developer [VS-logo-email] The Coach House, Heywood House, Park Lane, Heywood, Westbury, BA13 4NA Direct: +44 (0)333 0110164 Email: tim.chubb@voicesimplified.com<mailto:tim.chubb@voicesimplified.com> Web: www.VoiceSimplified.com<http://www.voicesimplified.com/> Voice Simplified Ltd (registered in England & Wales: 07171825) registered office: Curzon House, 2nd Floor, 24 High Street, Banstead, Surrey. SM7 2LJ. The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you are not the intended recipient, please return the message to the sender and delete it from your records. _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org<mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda