### Description
Started up Kamailio with USRLOC modules db_mode=2 and expected Kamailio to start successfully. Upon running the startup command, Kamailio timed out, saying the service failed. Syslog shows a segfault. Kamailio processes had to be manually killed afterwards.
### Troubleshooting
-This issue is only occurring when the DB_MODE parameter of the USRLOC module is 1 or 2. When it is 0 Kamailio runs and operates without fault.
-Ran a SQL profiler trace and there were no errors triggered on the SQL side of things. The final query Kamailio ran before the crash was "select username from location where username='dummy_user' and domain = 'dummy_domain'"
#### Reproduction
I can reproduce this every time I try to run Kamailio with DB_MODE = 1 or 2. FYI I am using unixodbc tied to a Microsoft SQL server.
#### Debugging Data
Even after uninstalling and reinstalling kamailio-dbg I still get "No symbol table info available." Assistance to fix this would also be helpful.
``` [New LWP 56741] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:689 689 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory. (gdb) bt full #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:689 No locals. #1 0x00007fe0f5e16439 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #2 0x00007fe0f5e16a4a in lh_retrieve () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #3 0x00007fe0f5e18f71 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #4 0x00007fe0f5e1997e in ERR_get_state () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #5 0x00007fe0f5e19bb5 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #6 0x00007fe0f6151630 in SSL_get_error () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 No symbol table info available. #7 0x00007fe0f5870bed in Handshake (verifier=..., cbWriteBuffer=16384, pbWriteBuffer=0x19b5710 "\026\003\001\001\034\001", cbReadBuffer=1203, pbReadBuffer=<optimized out>, this=0x19199e0) at Sql/Common/DK/sni/src/OpenSSL_SNI_SslProvider.cpp:768 eqs = {<No data fields>} #8 Do (cbWriteBuffer=16384, pbWriteBuffer=0x19b5710 "\026\003\001\001\034\001", cbReadBuffer=1203, pbReadBuffer=<optimized out>, this=0x7fffb87077e0) at Sql/Common/DK/sni/src/OpenSSL_SNI_SslProvider.cpp:1009 No locals. #9 Ssl::Handshake (this=0x1919880) at Sql/Common/DK/sni/src/OpenSSL_SNI_SslProvider.cpp:1333 szHostName = "vpsl_a_sql.cluster.data.local.tatnvps.com", '\000' <repeats 219 times> cchHostName = <optimized out> handshake = {m_sslEngine = @0x19199e0, m_verifier = {static X509_V_UNINITIALIZED = 1, static s_x509StoreExDataIdx = 0, static s_sslExDataIdx = 0, m_errorReason = 4, m_szHostName = 0x7fffb8707810 "vpsl_a_sql.cluster.data.local.tatnvps.com", m_cchHostName = 41, m_err = 1}} #10 0x00007fe0f5871e4c in Ssl::InitX (this=0x1919880) at Sql/Common/DK/sni/src/OpenSSL_SNI_SslProvider.cpp:1275 No locals. #11 0x00007fe0f5866bc8 in AddProvider (pConn=pConn@entry=0x196fe60, ProvNum=ProvNum@entry=SSL_PROV, pInfo=pInfo@entry=0x7fffb87079d0) at Sql/Common/DK/sni/src/sni.cpp:5500 _bidScp = {m_hScp = 0xffffffffffffffff} __func__ = "AddProvider" dwError = <optimized out> pProv = 0x1919880 #12 0x00007fe0f58672b0 in SNIAddProvider (pConn=0x196fe60, ProvNum=SSL_PROV, pInfo=0x7fffb8707b40) at Sql/Common/DK/sni/src/sni.cpp:5555 st_authParams = {m_bTDSPacketType = 0 '\000', m_cbInPrefix = 0, m_cbInSuffix = 0, m_cbOutPrefix = 0, m_cbOutSuffix = 0, m_cryptoMode = SNICRYPT_INVALIDMODE, m_pfnAuthTimer = 0x0, m_pfnAuthPrintBuf = 0x0, m_pfnAuthProcessInputBuf = 0x0, m_pfnAuthFormatOutputBuf = 0x0, m_pfnAuthEnqueueTask = 0x0, m_pfnAuthFKill = 0x0} st_authProvInfo = {pParams = 0x7fe0f5b04d40 SNIAddProvider::st_authParams, pbInitToken = 0x0, cbInitToken = 0, pwszSpn = 0x0, cwchSpn = 0, dwFlags = 266, errOp = SNIAUTH_ERROP_INVALID, errState = SNIAUTH_ERRST_INVALID, pCertContext = 0x0, pwszCertId = 0x0, fCertHash = 0, pCallbackContext = 0x0, pfnCertificateCallback = 0x0, fInitialTokenSent = 0} _bidScp = {m_hScp = 0xffffffffffffffff} __func__ = "SNIAddProvider" dwError = 0 #13 0x00007fe0f583cd0f in BATCHCTX::SNIAddProv (this=0x1916350, ProvNum=<optimized out>, pInfo=<optimized out>) at Sql/Ntdbms/sqlncli/tds/sql_ums.cpp:1145 hr = <optimized out> #14 0x00007fe0f583aba1 in CTdsParser::DoConnect (this=0x7fe0fa60ffe0, this@entry=0x19acef0, pLoginInfo=0x19ad0e0, pvUsrData=0xd6df6ca9, pvUsrData@entry=0x199f450, dwTimeout=4294967280, phConn=0x6000, phConn@entry=0x7fffb8709bf8) at Sql/Ntdbms/sqlncli/tds/netconn.cpp:1043 authProvInfo = {pParams = 0x7fe0f5b049c0 <st_authSSLParams>, pbInitToken = 0x0, cbInitToken = 0, pwszSpn = 0x0, cwchSpn = 0, dwFlags = 258, errOp = SNIAUTH_ERROP_INVALID, errState = SNIAUTH_ERRST_INVALID, pCertContext = 0x0, pwszCertId = 0x0, fCertHash = 0, pCallbackContext = 0x0, pfnCertificateCallback = 0x0, fInitialTokenSent = 0} fOverrideInst = 0 eNetFailoverPartner = INVALID_PREFIX pwszLKGServer = 0x19a213e L"\166\160\163\154\137\141\137\163\161\154\056\143\154\165\163\164\145\162\056\144\141\164\141\056\154\157\143\141\154\056\164\141\164\156\166\160\163\056\143\157", <incomplete sequence \155> wszFailoverPartner = L'\000\000' <repeats 2050 times>... hConn = 0x196daa0 pBatchDefault = 0x1916350 dwFailoverRetryTimeout = 0 ---Type <return> to continue, or q <return> to quit--- fATTNSent = 0 fFatalErr = 1 __func__ = "DoConnect" #15 0x00007fe0f583b522 in CTdsParser::OpenOrRedirectConnection (this=0x19acef0, pLoginInfo=0x19ad0e0, pvOwner=0x199f450, dwTimeoutMilliSeconds=<optimized out>, phConnection=0x199f490) at Sql/Ntdbms/sqlncli/tds/netconn.cpp:693 hr = 0 timer = {m_timeout = 15000, m_timeStart = 1168} __func__ = "OpenOrRedirectConnection" hConnection = 0x0 #16 0x00007fe0f57b0cc2 in ConnectIt (lpdbc=lpdbc@entry=0x199f450, rgAttr=..., dwFlags=dwFlags@entry=1) at Sql/Ntdbms/sqlncli/odbc/sqlcconn.cpp:3186 hr = <optimized out> fNoProtectPwd = 0 LoginInfo = @0x19ad0e0: { pwszConnStr = 0x19a213e L"\166\160\163\154\137\141\137\163\161\154\056\143\154\165\163\164\145\162\056\144\141\164\141\056\154\157\143\141\154\056\164\141\164\156\166\160\163\056\143\157", <incomplete sequence \155>, pwszFailoverPartner = 0x19a4140 L"", pwszDatabase = 0x199fd34 L"\153\141\155\141\151\154\151", <incomplete sequence \157>, pwszLanguage = 0x199fc32 L"", pwszUserName = 0x199f6ec L"\144\142\125\163\145\162\101\143\143\145\163", <incomplete sequence \163>, pwszPassword = 0x199f7ee L"\064\065\064\160\151\154\144\154\063\070\060", <incomplete sequence \172>, pwszNewPassword = 0x0, pwszWorkstation = 0x199fb30 L"\166\160\163\154\063\062\156\060\061\061\056\164\141\164\156\166\160\163\056\143\157", <incomplete sequence \155>, pwszServer = 0x199ff38 L"\166\160\163\154\137\141\137\163\161\154\056\143\154\165\163\164\145\162\056\144\141\164\141\056\154\157\143\141\154\056\164\141\164\156\166\160\163\056\143\157", <incomplete sequence \155>, pwszRPCServer = 0x0, pwszReplServer = 0x0, pwszAppName = 0x199fa2e L"", pwszClientInterface = 0x7fe0f5896942 L"\117\104\102", <incomplete sequence \103>, pwszFileName = 0x19a6142 L"", pwszServerSPN = 0x0, pwszFailoverPartnerSPN = 0x0, cbPacketSize = 0, eUseIntegratedSecurity = None, fMARSConnection = 0, hCompletionEvent = 0x0, fIsMDACCompatible = 0, phrAsyncResult = 0x0, plAsyncAbortFlag = 0x19ab704, fDumpLoad = 0, fEncrypt = 0, fTrustServerCert = 0, lcidClient = 1033, rgbClientID = "\377\377\377\377\377\377", dwConnFlags = 0, fNetProtocolApplyToFailoverCache = 1, eNetProtocol = UNKNOWN_PREFIX, eNetProtocolFailover = UNKNOWN_PREFIX, fMultiSubnetFailover = 0, fTnir = 1, fTnirRetry = 0, eApplicationIntent = ReadWrite, fIsRoutingInfoSet = 0, fIsSessionRecoveryEnabled = 0, fIsRecovery = 0, fIsColumnEncryptionEnabled = 0, wTdsVerMajor = 0, wTdsVerMinor = 0, pszClientCertificate = 0x19a8b80 L"", pfnClientCertificateFallback = 0x0, pFedAuthToken = 0x0} retcode = 0 fRealLogin = 1 #17 0x00007fe0f57e657e in DoDlgConnection (hwnd=hwnd@entry=0x0, lpDlgStruct=lpDlgStruct@entry=0x196cb90, dwFlags=dwFlags@entry=1) at Sql/Ntdbms/sqlncli/odbc/sqlcdlg.cpp:333 retcode = 0 lpdbc = 0x199f450 rgAttr = @0x196cc08: {m_cATTRs = 44, m_rgATTRs = 0x196d280, m_pextbufATTRs = 0x19699b0, m_SecuredAttrs = {idxPwd = 7, fPwdProtected = 0, idxOldPwd = 43, fOldPwdProtected = 0}} authMode = <optimized out> #18 0x00007fe0f57b3de5 in SQLDriverConnectW (hdbc=0x199f450, hwnd=<optimized out>, szConnStrIn=<optimized out>, cchConnStrIn=<optimized out>, szConnStrOut=<optimized out>, cchConnStrOutMax=<optimized out>, pcchConnStrOut=0x7fffb870edea, fDriverCompletion=1) at Sql/Ntdbms/sqlncli/odbc/sqlcconn.cpp:1638 lpdbc = 0x199f450 hr = <optimized out> wDlgRet = 0 fIssue01S00 = 0 __func__ = "SQLDriverConnectW" retcode = 0 cchConnStrLen = <optimized out> lpDlgStruct = 0x196cb90 _bidScp = {m_hScp = 0xffffffffffffffff} #19 0x00007fe0ff3ae978 in SQLDriverConnect () from /usr/lib/x86_64-linux-gnu/libodbc.so.2 No symbol table info available. #20 0x00007fe0ff607392 in db_unixodbc_new_connection (id=0x7fe0ff8ef2f0) at connection.c:155 outstr = "\017\000\000\000\000\000\000\000\340q\214\377\340\177", '\000' <repeats 82 times>, "\270\067\221\001", '\000' <repeats 68 times>, "\004\000\000\000\000\000\000\000G\026\000\000\000\000\000\000.Qx", '\000' <repeats 21 times>... outstrlen = 0 ret = 0 ptr = 0x7fe0ff8ef548 conn_str = "DSN=MSSQL13;UID=dbUserAccess;PWD=***********;", '\000' <repeats 14 times>, "\060\000\000\000\000\000\000\000\377\177\000\000\377\377\377\377", '\000' <repeats 32 times>, "\003\000\000\000\377\377\377\377\377\377\377\377c\371\006\000\341\177\000\000\006\000\000\000\000\000\000\000\232\371\006\000\341\177\000\000HQx\000\000\000\000\000`\240<\000\341\177\000\000\260\370p\270\377\177\000\000\060ʑ\376---Type <return> to continue, or q <return> to quit--- 340\177\000\000\207(\255\373\000\000\000\000~\243\006\000\341\177", '\000' <repeats 18 times>... __FUNCTION__ = "db_unixodbc_new_connection" #21 0x00007fe0ff1807dc in db_do_init2 (url=0x7fe0fb6ba190 <db_url>, new_connection=0x7fe0ff605ab1 <db_unixodbc_new_connection>, pooling=DB_POOLING_PERMITTED) at db.c:316 id = 0x7fe0ff8ef2f0 con = 0x0 res = 0x7fe0ff8d7850 con_size = 32 __FUNCTION__ = "db_do_init2" #22 0x00007fe0ff17f785 in db_do_init (url=0x7fe0fb6ba190 <db_url>, new_connection=0x7fe0ff605ab1 <db_unixodbc_new_connection>) at db.c:269 No locals. #23 0x00007fe0ff60c61d in db_unixodbc_init (_url=0x7fe0fb6ba190 <db_url>) at dbase.c:181 c = 0x53ec77 <init_mod_child+59> #24 0x00007fe0fb4b0b02 in child_init (rank=-2) at auth_db_mod.c:159 __FUNCTION__ = "child_init" #25 0x000000000053ef59 in init_mod_child (m=0x7fe0ff88b408, rank=-2) at core/sr_module.c:921 __FUNCTION__ = "init_mod_child" #26 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88b960, rank=-2) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #27 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88be10, rank=-2) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #28 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88c2f8, rank=-2) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #29 0x000000000053ec77 in init_mod_child (m=0x7fe0ff8920e0, rank=-2) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #30 0x000000000053f287 in init_child (rank=-2) at core/sr_module.c:947 No locals. #31 0x00000000004beaf7 in fork_process (child_id=-2, desc=0x7fe0fc693722 "ctl handler", make_sock=1) at core/pt.c:332 pid = 0 child_process_no = 6 ret = -1 new_seed1 = 715194968 new_seed2 = 2009098725 sockfd = {12, 13} __FUNCTION__ = "fork_process" #32 0x00007fe0fc684f8e in mod_child (rank=0) at ctl.c:327 pid = 32736 cs = 0x7fe0f639671d <futex_release+29> rpc_handler = 1 __FUNCTION__ = "mod_child" #33 0x000000000053ef59 in init_mod_child (m=0x7fe0ff886720, rank=0) at core/sr_module.c:921 __FUNCTION__ = "init_mod_child" #34 0x000000000053ec77 in init_mod_child (m=0x7fe0ff886d88, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #35 0x000000000053ec77 in init_mod_child (m=0x7fe0ff8887b8, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #36 0x000000000053ec77 in init_mod_child (m=0x7fe0ff888bf0, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #37 0x000000000053ec77 in init_mod_child (m=0x7fe0ff889a90, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #38 0x000000000053ec77 in init_mod_child (m=0x7fe0ff889e60, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #39 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88a6a8, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" ---Type <return> to continue, or q <return> to quit--- #40 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88ad98, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #41 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88b408, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #42 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88b960, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #43 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88be10, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #44 0x000000000053ec77 in init_mod_child (m=0x7fe0ff88c2f8, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #45 0x000000000053ec77 in init_mod_child (m=0x7fe0ff8920e0, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #46 0x000000000053f287 in init_child (rank=0) at core/sr_module.c:947 No locals. #47 0x0000000000423a13 in main_loop () at main.c:1705 i = 0 pid = 56737 si = 0x0 si_desc = "\000\000\000\000\000\000\000\000\001\000\000\000\002\000\000\000 ȏ\377\340\177\000\000\005\000\000\000\000\000\000\000HQx\000\000\000\000\000~\330s\000\000\000\000\000a\355u\000\000\000\000\000\000\222O\264\375\022\034\340O\021\000\020\000\000\000\000HQx\000\000\000\000\000\021\000\000\000\000\000\000\000\360\bq\270\377\177", '\000' <repeats 18 times>, " \006q\270\377\177\000\000K\363^\000\000\000\000" nrprocs = 0 woneinit = 0 __FUNCTION__ = "main_loop" #48 0x000000000042a408 in main (argc=13, argv=0x7fffb87108f8) at main.c:2643 cfg_stream = 0x18a1010 c = -1 r = 0 tmp = 0x7fffb8711f0b "" tmp_len = 213416 port = 32737 proto = -1200552016 options = 0x73b8f8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 610617656 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0x1 <error: Cannot access memory at address 0x1> st = {st_dev = 15, st_ino = 13541, st_nlink = 2, st_mode = 16832, st_uid = 112, st_gid = 119, __pad0 = 0, st_rdev = 0, st_size = 120, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1505141172, tv_nsec = 114977459}, st_mtim = {tv_sec = 1505142145, tv_nsec = 549518688}, st_ctim = {tv_sec = 1505142145, tv_nsec = 549518688}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" ```
info locals = "No locals." list = "684 in ../sysdeps/x86_64/multiarch/memcmp-sse4.S"
#### Log Messages
``` Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: tls [tls_domain.c:1168]: load_private_key(): TLSs<default>: Key '/etc/kamailio/tls/privatekey.pem' successfully loaded Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: tls [tls_domain.c:1168]: load_private_key(): TLSc<default>: Key '/etc/kamailio/tls/privatekey.pem' successfully loaded Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: debugger Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: debugger [debugger_mod.c:223]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: jsonrpcs Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: kex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: kex [kex_mod.c:156]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [srid-59b6b03a-dda0-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: corex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: tm Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: tm [callid.c:137]: child_init_callid(): callid: '6e9c02e64bdb8061-56736@74.208.157.229' Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: tmx Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: tmx [tmx_mod.c:260]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: sl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: ctl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: acc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: usrloc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [ulcx-59b6b03a-dda0-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: <core> [db.c:314]: db_do_init2(): connection 0x7fe0ff8ef2f0 not found in pool Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 1(56736) DEBUG: db_unixodbc [connection.c:153]: db_unixodbc_new_connection(): opening connection: unixodbc://xxxx:xxxx@192.168.32.21/MSSQL13 Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: debugger Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: debugger Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: debugger [debugger_mod.c:223]: child_init(): rank is (0) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: jsonrpcs Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: debugger [debugger_mod.c:223]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: debugger Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: debugger [debugger_mod.c:223]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: jsonrpcs Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: kex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: kex [kex_mod.c:156]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [srid-59b6b03a-dda2-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: corex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: tm Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: tm [callid.c:137]: child_init_callid(): callid: '6e9c02e64bdb8061-56738@74.208.157.229' Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: tmx Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: tmx [tmx_mod.c:260]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: sl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: ctl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: acc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: usrloc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [ulcx-59b6b03a-dda2-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: <core> [db.c:314]: db_do_init2(): connection 0x7fe0ff8ef2f0 not found in pool Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 3(56738) DEBUG: db_unixodbc [connection.c:153]: db_unixodbc_new_connection(): opening connection: unixodbc://xxxx:xxxx@192.168.32.21/MSSQL13 Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: jsonrpcs [jsonrpcs_sock.c:332]: jsonrpc_dgram_init_server(): we have a unix socket: /var/run/kamailio/kamailio_rpc.sock Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: jsonrpcs Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: kex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: kex [kex_mod.c:156]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [srid-59b6b03a-dda1-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: corex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: kex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: kex [kex_mod.c:156]: child_init(): rank is (0) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [srid-59b6b03a-dd9f-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: corex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: tm Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: tm [callid.c:137]: child_init_callid(): callid: '6e9c02e64bdb8061-56735@74.208.157.229' Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: tmx Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: tmx [tmx_mod.c:260]: child_init(): rank is (0) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: sl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: ctl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: ctl [ctl.c:321]: mod_child(): ctl: mod_child(0), ctrl_sock_lst=0x1906740 Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 5(56740) INFO: jsonrpcs [jsonrpcs_sock.c:429]: jsonrpc_dgram_process(): a new child 0/56740 Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: tm Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: tm [callid.c:137]: child_init_callid(): callid: '6e9c02e64bdb8061-56737@74.208.157.229' Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: ctl [ctl.c:329]: mod_child(): ctl: mod_child(0), fork_process=56741, csl=0x1906740 Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: acc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank 0: usrloc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [ulcx-59b6b03a-dd9f-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: <core> [db.c:314]: db_do_init2(): connection 0x7fe0ff8ef2f0 not found in pool Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 4(56739) DEBUG: jsonrpcs [jsonrpcs_fifo.c:484]: jsonrpc_fifo_process(): new process with pid = 56739 created Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 0(56735) DEBUG: db_unixodbc [connection.c:153]: db_unixodbc_new_connection(): opening connection: unixodbc://xxxx:xxxx@192.168.32.21/MSSQL13 Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 4(56739) DEBUG: jsonrpcs [jsonrpcs_fifo.c:148]: jsonrpc_init_fifo_server(): FIFO created @ /var/run/kamailio/kamailio_rpc.fifo Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 4(56739) DEBUG: jsonrpcs [jsonrpcs_fifo.c:163]: jsonrpc_init_fifo_server(): fifo /var/run/kamailio/kamailio_rpc.fifo opened, mode=660 Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: tmx Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: debugger Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: debugger [debugger_mod.c:223]: child_init(): rank is (-2) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: jsonrpcs Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: kex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: kex [kex_mod.c:156]: child_init(): rank is (-2) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [srid-59b6b03a-dda5-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: corex Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: tm Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: tm [callid.c:137]: child_init_callid(): callid: '6e9c02e64bdb8061-56741@74.208.157.229' Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: tmx Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: tmx [tmx_mod.c:260]: child_init(): rank is (-2) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: sl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: ctl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: acc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: usrloc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [ulcx-59b6b03a-dda5-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: registrar Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [uloc-59b6b03a-dda5-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: uac Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -2: auth_db Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: <core> [db.c:314]: db_do_init2(): connection 0x7fe0ff8ef2f0 not found in pool Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 6(56741) DEBUG: db_unixodbc [connection.c:153]: db_unixodbc_new_connection(): opening connection: unixodbc://xxxx:xxxx@192.168.32.21/MSSQL13 Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: tmx [tmx_mod.c:260]: child_init(): rank is (-1) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: sl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: ctl Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: acc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [core/sr_module.c:920]: init_mod_child(): rank -1: usrloc Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [sruid.c:106]: sruid_init(): root for sruid is [ulcx-59b6b03a-dda1-] (0 / 19) Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: <core> [db.c:314]: db_do_init2(): connection 0x7fe0ff8ef2f0 not found in pool Sep 11 10:48:10 vpsl32n011 kamailio[56733]: 2(56737) DEBUG: db_unixodbc [connection.c:153]: db_unixodbc_new_connection(): opening connection: unixodbc://xxxx:xxxx@192.168.32.21/MSSQL13 Sep 11 10:48:10 vpsl32n011 kernel: [753687.700674] kamailio[56741]: segfault at 7fe0fa610000 ip 00007fe10016be58 sp 00007fffb8707438 error 4 in libc-2.19.so[7fe100024000+1a1000] Sep 11 10:48:28 vpsl32n011 snmpd[805]: error on subcontainer 'ia_addr' insert (-1) Sep 11 10:48:58 vpsl32n011 snmpd[805]: error on subcontainer 'ia_addr' insert (-1) Sep 11 10:49:28 vpsl32n011 snmpd[805]: error on subcontainer 'ia_addr' insert (-1) Sep 11 10:49:40 vpsl32n011 systemd[1]: kamailio.service start operation timed out. Terminating. Sep 11 10:49:40 vpsl32n011 kamailio[56733]: 4(56739) INFO: <core> [main.c:823]: sig_usr(): signal 15 received Sep 11 10:49:40 vpsl32n011 kamailio[56733]: 1(56736) INFO: <core> [main.c:823]: sig_usr(): signal 15 received Sep 11 10:49:40 vpsl32n011 systemd[1]: Failed to start Kamailio (OpenSER) - the Open Source SIP Server. Sep 11 10:49:40 vpsl32n011 systemd[1]: Unit kamailio.service entered failed state. ```
### Additional Information
``` version: kamailio 5.0.3 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 4.9.2 ```
* **Operating System**:
``` Linux vpsl32n011.tatnvps.com 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u3 (2017-08-15) x86_64 GNU/Linux ```
db_mode=0 makes usrloc to run without connecting to database. Can you try with db_mode=3 to see if there is same crash?
So I tried db_mode 3 and it worked the first time. Phones registered and USRLOC wrote to the location table successfully.
Stopped the service and then attempted to start again, and the time-out crash occurred. Tried once more, and it still crashed (however, the segfaults are different).
Crash 1:
Sep 12 09:52:20 vpsl32n011 kernel: [836803.080856] kamailio[8116]: segfault at 0 ip (null) sp 00007fff1030aa68 error 14 in kamailio[400000+42e000]
Crash 2:
Sep 12 09:58:21 vpsl32n011 kernel: [837164.256390] kamailio[8220]: segfault at 7fdc518b1000 ip 00007fdc5740ce3e sp 00007fff40f0e118 error 4 in libc-2.19.so[7fdc572c5000+1a1000]
Here is syslog for both crashes and core dump for the last one (accidentally overwrote the corefile from first).
[Crash1_Syslog.txt](https://github.com/kamailio/kamailio/files/1296451/Crash1_Syslog.txt) [Crash2_Core.txt](https://github.com/kamailio/kamailio/files/1296452/Crash2_Core.txt) [Crash2_Syslog.txt](https://github.com/kamailio/kamailio/files/1296453/Crash2_Syslog.txt)
It may also be worth noting that the same configuration works without issue using MySQL, on db_mode = 2.
I will try to look at it in details when I have some spare time. I am not using odbc nor MSSQL.
Few questions now:
* do you have tls module loaded or another module that depends on libssl? * is odbc sql driver open source? If yes, what's the link to the project just in case it would be useful to look at code based on backtrace.
Thanks!
Yes, I am using the TLS module which has been working succesfully for registering.
Unfortunately the ODBC driver comes directly from Microsoft and is not open source--hopefully this doesn't come down to a fix that they need to take care of.
[MSSQL13] Driver = ODBC Driver 13 for SQL Server Description = PBX Connection Trace = yes Server = vpsl_a_sql.cluster.data.local.tatnvps.com UserName = dbUserAccess Password = ******* Port = 1433 Database = kamailio
Close this one, we don't have a way to reproduce. If you still see this problem, re-open with more information, would be also great if you could provide a test setup etc..
Closed #1231.