<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi all!</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Having an issue with MySQL db connections being dropped in a system running 4.4.7.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We're using db_mysql and db_cluster modules setup a cluster connecting two different DB servers. We have two cluster connections, one for acc and one for "other queries". One DB (A) is on the same network, another DB (B) is on another network. The default DB
connection is for the remote server B. Auto reconnect is enabled.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The specific issue seen is that the "timer" child process looses/drops both connections to DB A and B. Looking at the output from lsof when this happens, the connections usually does not both drop connections for A and B at the same time. Sometimes the connections
keep up for ~24h, sometimes for 10 minutes, but normally the problem re-occurs every 6 hours or so. We're seeing this problem on two Kamailio servers, both handling fairly high amount of calls.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
None of the other Kamailio child processes seems to get their connections dropped, only the "timer" process. To solve this we need to restart Kamailio.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Lately I've added the timer.so module to make a simple query on each cluster connection each 10 seconds.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This is an example output from when the problem appears and connections are dropped:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003)<br>
</span>
<div>Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server during query (2013)<br>
</div>
<div>Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003)<br>
</div>
<div>Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003)<br>
</div>
<div>Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003)<br>
</div>
<div>Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server during query (2013)<br>
</div>
<div>Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:15 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:15 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:16 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:16 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:17 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:17 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:18 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:18 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:27 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:27 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:28 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:28 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:37 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:37 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:38 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:38 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:47 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:47 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:48 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:48 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query<br>
</div>
<div>Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<div>Jun 15 09:39:57 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)<br>
</div>
<div>Jun 15 09:39:57 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query<br>
</div>
<span></span><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Again, it's always and only the timer child process where these error messages are seen. Have been trying to debug this for some time now but I'm running out of ideas. Do you have any ideas on what I could look for? Any idea on why it's just the timer-process?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Kind regards,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
/Tobias</div>
</body>
</html>