[sr-dev] [kamailio/kamailio] kamailio crashes when attempting to query offline database (#1821)

Wed Jan 23 20:47:53 CET 2019

### Description

I am testing how kamailio reacts to various database conditions.   One such condition is if the database engine is simply shut down (that is, database server process no longer running, tcp listening socket closed, etc...)

I am utilizing the db_unixodbc module to connect to an Informix database engine.   

I am currently running on Kamailio version 5.0.1.

I have a test query that executes against the database engine every 10 seconds.

Here is what i have noticed if i shut down the database engine at some point after i run Kamailio.

The first test query that attempts to run against the db engine fails;  it tries to reconnect and fails.

The second test query (10 seconds after the 1st) results in a SIGCHILD and shuts down the entire Kamailio process.  

I communicated this info to the mailing list and was asked to open an issue regarding this and to also test the more recent version 5 releases.

Here is the update on said tests:

I have tested the master branch and 5.3.0-dev2 and neither branch resolves the issue.  

However I did notice in the master branch that there is new code that is related to this issue.  

https://github.com/kamailio/kamailio/issues/1681

In issue 1681 there is code that allows Kamailio to start even if a database connection can not be established.  Queries attempting to run against the offline database fail gracefully.  And once the database is back online, a connection is established and queries against it are successful.

However, if at some later point I shut down the database, we're back to the original issue that i reported.  Kamailio crashes with the same output as listed before except the first query that is attempted against the offline db causes the crash in this master branch unlike previously (branch 5.series) the first attempt fails, tries again and fails, and the second attempt causes the crash.  Regardless, the output is more or less the same and Kamailio is down.

I suspect this might be the same behavior even if one is not using an odbc driver; but maybe not. 

### Troubleshooting

#### Reproduction

Start Kamailio;  kill the database engine;  run test query from kamailo to said database engine;  Kamailio crashes.  This can be replicated using the db_unixodbc module;  not sure if it's the same for other types of database drivers.

#### Log Messages
Jan 17 20:07:25 [29297]: INFO: (s)  SQL query: FIRST TEST QUERY 
Jan 17 20:07:25 [29297]: ERROR: db_unixodbc [dbase.c:135]: db_unixodbc_submit_query(): rv=-1. Query= FIRST TEST QUERY
Jan 17 20:07:25 [29297]: ERROR: db_unixodbc [connection.c:220]: db_unixodbc_extract_error(): unixodbc:SQLExecDirect=08S01:1:-11020:[Informix][Informix ODBC Driver]Communication link failure.
Jan 17 20:07:25 [29297]: ERROR: db_unixodbc [dbase.c:59]: reconnect(): Attempting DB reconnect
Jan 17 20:07:25 [29297]: ERROR: db_unixodbc [dbase.c:74]: reconnect(): failed to connect
Jan 17 20:07:25 [29297]: ERROR: db_unixodbc [connection.c:220]: db_unixodbc_extract_error(): unixodbc:SQLDriverConnect=08002:1:0:[unixODBC][Driver Manager]Connection name in use
Jan 17 20:07:25 [29297]: ERROR: db_unixodbc [connection.c:220]: db_unixodbc_extract_error(): unixodbc:SQLDriverConnect=HY010:2:-11067:[Informix][Informix ODBC Driver]Function sequence error.
Jan 17 20:07:25 [29297]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jan 17 20:07:25 [29297]: ERROR: sqlops [sql_api.c:265]: sql_do_query(): cannot do the query FIRST TEST QUERY
Jan 17 20:07:25 [29297]: INFO: (s) [123] SQL ret: fail (-1)
Jan 17 20:07:25 [29297]: INFO: (s) [123] SQL res: no rows
Jan 17 20:07:35 [29297]: INFO: (s) [123] SQL query: 10 seconds later the SECOND TEST QUERY (it's the same query as the first one)
Jan 17 20:07:35 [29301]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 28
Jan 17 20:07:35 [29283]: ALERT: <core> [main.c:744]: handle_sigs(): child process 29297 exited by a signal 11
Jan 17 20:07:35 [29283]: ALERT: <core> [main.c:747]: handle_sigs(): core was not generated
Jan 17 20:07:35 [29283]: INFO: <core> [main.c:759]: handle_sigs(): terminating due to SIGCHLD
Jan 17 20:07:35 [29301]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29295]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29291]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29288]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29300]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29284]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29286]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29293]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29289]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29287]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29292]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29296]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29298]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29299]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29285]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29294]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29290]: INFO: <core> [main.c:814]: sig_usr(): signal 15 received
Jan 17 20:07:35 [29283]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized

### Additional Information

version: kamailio 5.0.1 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 11:25:09 May  2 2018 with gcc 4.1.2

version: kamailio 5.3.0-dev2 (x86_64/linux) 30857d-dirty
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 30857d -dirty
compiled on 14:30:54 Jan 22 2019 with gcc 4.1.2

* **Operating System**:
Red Hat Enterprise Linux Server release 5.11 (Tikanga)
Linux 2.6.18-404.el5 #1 SMP Sat Mar 7 04:14:13 EST 2015 x86_64 x86_64 x86_64 GNU/Linux

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1821
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20190123/a3e6602b/attachment.html>