[SR-Users] Strange application hang querying DB

Thu Jun 26 11:06:38 CEST 2014

Hi Daniel

There is no load on the system as I’m developing the implementation at the moment, I have a softphone and hardphone registered and that is it.
Im using Centos 6.5 32bit as the OS and have disabled SElinux whilst developing.

There is no rhyme or reason as far as I can tell, sometime’s registration hangs (using auth_db against my Asterisk RT tables), sometimes it’s the dispatcher, and sometimes just calling ‘kamctl dispatcher reload’.  One thing I have tried to help narrow the search focus down was run this when I experienced loss of connectivity
‘date && kamctl dispatcher reload && date’ which would give the start and end time of the query, where I seem to experience a 30min delay between start and finish (I have 2 entires in the dispatcher table, so its not a lack of index or anything like that).

I have a nasty feeling this may be caused by SQL Azures idle connection handling which is quite different to the standard server implementation (which works frustratingly fine, i.e. if I work from home the kamailio process handles vpn disconnections and resumes operations as soon as I connect).

Im going to try disabling connection pooling in unixodbc, see if that helps, if that doesn’t help then I fear I am going to have to brush off my c skills and dive into the db_unixodbc module and general DB logic to get a better understanding of how Kamailio is handling db connections.

Tim.

From: sr-users-bounces at lists.sip-router.org [mailto:sr-users-bounces at lists.sip-router.org] On Behalf Of Daniel-Constantin Mierla
Sent: 26 June 2014 09:08
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Strange application hang querying DB

Hello,

have you identified the config function that hangs? Is a sqlops function or something related to authentication, location, etc?

If not, then you can use benchmark module to measure the duration of execution of various parts of config, in this way identifying the function that takes too long.

You should eventually run directly with debug=3 and watch the logs in syslog file.

Once you tell the function taking too long, I will try to dig further to see if I can spot any issue.

Some other details would help:
- is the system under load or it happens only for one request and always?
- what is the operating system? I encountered many issues with selinux in the past, under high load or too many connections opened

Cheers,
Daniel
On 25/06/14 20:06, Tim Chubb wrote:
Hi,

I’m currently wracking my brains trying to deduce where an application hang can be coming from.

I have ported the PostGres schema to be compatible with MS Sql Server and have had no issues getting a working dev environment working.  However when I try and use SQL Azure as the backend db as opposed to SQL Server 2012 (running locally on my dev machine) Kamailio will seem to hang after about 10 – 15 mins of flawless operation.  I have recreated the SQL Azure schema from the working schema running on my dev machine and copied all data in a 1:1 manner so I can be certain it’s not a data quality issue.  I am using the same FreeTDS/UnixODBC config that I am using for Asterisk realtime (which doesn’t exhibit this behaviour).  The really strange thing is that I can see the queries being sent by Kmailio in the freetds trace log, and the response from SQL Azure, however its as if the db response handler is deadlocked/hung.  The kamailio process continues to work fine, and as long as its not required to commiunicate with the DB all appears well i.e. no error messages in the log file.

This behaviour only happens when im using SQL Azure as opposed to SQL Server, so there must be something causing the issue, that’s unique to SQL azure as opposed to SQL Server.  As far as I can tell this is not something I can create a backtrace for as no core dumps are produced, as far as I can tell there is no easy way debug what the lock is.  If anyone has any suggestions I would be very grateful to hear them.

One thing that occurs to me is that the db_unixodbc module doesn’t have half as many parameters as its equilent in Asterisk, and that connection pooling may be in use, this could be a bit of a show stopper AFAIK you have to disable connection pooling with SQL server, certainly any of the guides I have come across configuring Asterisk to work with SQL server suggests that.

Tim Chubb
Developer

[VS-logo-email]

The Coach House, Heywood House, Park Lane, Heywood, Westbury, BA13 4NA

Direct: +44 (0)333 0110164
Email: tim.chubb at voicesimplified.com<mailto:tim.chubb at voicesimplified.com>
Web: www.VoiceSimplified.com<http://www.voicesimplified.com/>

Voice Simplified Ltd (registered in England & Wales: 07171825) registered office: Curzon House, 2nd Floor, 24 High Street, Banstead, Surrey. SM7 2LJ.
The information in this email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this email by anyone else
is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it,
is prohibited and may be unlawful. If you are not the intended recipient, please return the message to the sender and delete it from your records.

_______________________________________________

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list

sr-users at lists.sip-router.org<mailto:sr-users at lists.sip-router.org>

http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

--

Daniel-Constantin Mierla - http://www.asipto.com

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140626/6f06b9b3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 12700 bytes
Desc: image001.png
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20140626/6f06b9b3/attachment.png>