Hi all!
Having an issue with MySQL db connections being dropped in a system running 4.4.7.
We're using db_mysql and db_cluster modules setup a cluster connecting two different DB servers. We have two cluster connections, one for acc and one for "other queries". One DB (A) is on the same network, another DB (B) is on another network. The default DB connection is for the remote server B. Auto reconnect is enabled.
The specific issue seen is that the "timer" child process looses/drops both connections to DB A and B. Looking at the output from lsof when this happens, the connections usually does not both drop connections for A and B at the same time. Sometimes the connections keep up for ~24h, sometimes for 10 minutes, but normally the problem re-occurs every 6 hours or so. We're seeing this problem on two Kamailio servers, both handling fairly high amount of calls.
None of the other Kamailio child processes seems to get their connections dropped, only the "timer" process. To solve this we need to restart Kamailio.
Lately I've added the timer.so module to make a simple query on each cluster connection each 10 seconds.
This is an example output from when the problem appears and connections are dropped: Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003) Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server during query (2013) Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003) Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003) Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003) Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server during query (2013) Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:15 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:15 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:16 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:16 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:17 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:17 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:18 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:18 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:27 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:27 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:28 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:28 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:37 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:37 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:38 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:38 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:47 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:47 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:48 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:48 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query Jun 15 09:39:57 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013) Jun 15 09:39:57 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Again, it's always and only the timer child process where these error messages are seen. Have been trying to debug this for some time now but I'm running out of ideas. Do you have any ideas on what I could look for? Any idea on why it's just the timer-process?
Kind regards, /Tobias