[kamailio/kamailio] TLS crashes Kamailio 5.4 branch on Ubuntu 20.04 and Debian Buster (#2560)
<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description Kamailio crashes on launch with TLS enabled on both Ubuntu 20.04 and Debian Buster.
### Troubleshooting
#### Reproduction
<!-- If the issue can be reproduced, describe how it can be done. -->
1. cloned `https://github.com/kamailio/kamailio%60 2. checked out `5.4` 3. make cfg 4. include_modules=app_python3 crypto db_mongodb db_postgres dialplan http_async_client http_client jansson json jsonrpcc log_systemd outbound regex tls utils xcap_client xhttp_pi uuid 5. make all 6. make install 7. make install-systemd-debian 8. turn on TLS (WITH_TLS), copy certs & keys. 9. sudo systemctl start kamailio
#### Log Messages
``` Nov 20 00:20:06 kamailio-stage-3 /usr/local/sbin/kamailio[12570]: INFO: tls [tls_mod.c:389]: mod_init(): With ECDH-Support! Nov 20 00:20:06 kamailio-stage-3 /usr/local/sbin/kamailio[12570]: INFO: tls [tls_mod.c:392]: mod_init(): With Diffie Hellman Nov 20 00:20:06 kamailio-stage-3 kamailio[12569]: DIGEST-MD5 common mech free Nov 20 00:20:06 kamailio-stage-3 /usr/local/sbin/kamailio[12570]: INFO: auth [auth_mod.c:345]: mod_init(): qop set, but nonce-count (nonce_count) support disabled Nov 20 00:20:06 kamailio-stage-3 /usr/local/sbin/kamailio[12570]: INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded Nov 20 00:20:06 kamailio-stage-3 /usr/local/sbin/kamailio[12570]: INFO: rr [rr_mod.c:185]: mod_init(): outbound module not available Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: ERROR: <core> [core/daemonize.c:303]: daemonize(): Main process exited before writing to pipe Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f09fdb43de8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f09fdb44098), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f09fdb94230), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f09fdb41890), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f09fdb34998), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f09fdb347b8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f09fdb34858), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring Nov 20 00:20:06 kamailio-stage-3 kamailio[12568]: CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f09fdb348f8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring Nov 20 00:20:06 kamailio-stage-3 systemd[1]: kamailio.service: Control process exited, code=killed, status=11/SEGV Nov 20 00:20:06 kamailio-stage-3 systemd[1]: kamailio.service: Failed with result 'signal'. Nov 20 00:20:06 kamailio-stage-3 systemd[1]: Failed to start kamailio (OpenSER) - the Open Source SIP Server. Nov 20 00:20:06 kamailio-stage-3 systemd[1]: kamailio.service: Service RestartSec=100ms expired, scheduling restart. Nov 20 00:20:06 kamailio-stage-3 systemd[1]: kamailio.service: Scheduled restart job, restart counter is at 57. Nov 20 00:20:06 kamailio-stage-3 systemd[1]: Stopped kamailio (OpenSER) - the Open Source SIP Server. Nov 20 00:20:06 kamailio-stage-3 systemd[1]: kamailio.service: Start request repeated too quickly. Nov 20 00:20:06 kamailio-stage-3 systemd[1]: kamailio.service: Failed with result 'signal'. Nov 20 00:20:06 kamailio-stage-3 systemd[1]: Failed to start kamailio (OpenSER) - the Open Source SIP Server. ```
### Possible Solutions As a hack, we manually downgraded to https://www.openssl.org/source/openssl-1.0.2r.tar.gz and that seemed to allow the system to launch, but this is obviously not a stable, long-term solution.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.4.2 (x86_64/linux) 2d733e flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 2d733e compiled on 23:01:51 Nov 19 2020 with gcc 8.3.0 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) -->
``` root@kamailio-stage-3:/home/admin# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 10 (buster) Release: 10 Codename: buster root@kamailio-stage-3:/home/admin# uname -a Linux kamailio-stage-3 4.19.0-12-cloud-amd64 #1 SMP Debian 4.19.152-1 (2020-10-18) x86_64 GNU/Linux
ubuntu@kamailio-stage-2:~$ lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.1 LTS Release: 20.04 Codename: focal ubuntu@kamailio-stage-2:~$ uname -a Linux kamailio-stage-2 5.4.0-1029-aws #30-Ubuntu SMP Tue Oct 20 10:06:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ```
The logs do not indicate where the problem is. You have to set `debug=3` in `kamailio.cfg`, then run again and attach here all the log messages printed by kamailio from start up, there should be a lot containing `DEBUG`.
Also, it would be good to get a core dump file, and then grab the `bt full` with `gdb`.
It should be something specific for your config, not strictly related to`tls` module, during the past week or so I deployed two times kamailio 5.4.x with tls on Debian Buster and starts fine.
I just tested with default `kamailio.cfg` and TLS enabled on Debian Stable (Buster) and Testing. In both cases started fine. Kamailio was installed from sources, built against the libssl coming with the distro.
From Ubuntu 20 server, truncated logs (hopefully enough) and then `bt full` at the bottom:
``` 0(46158) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(46158) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(46158) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(46158) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(46158) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(46158) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(46158) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(46158) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(46158) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(46158) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <xhttp_reply> in module xhttp [/usr/local/lib64/kamailio/modules/xhttp.so] 0(46158) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type2, i/count 0/2: type 3(3)/ data 0x1 0(46158) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type2, i/count 1/2: type 3(3)/ data 0x1 0(46158) DEBUG: <core> [core/route.c:129]: route_add(): mapping routing block (0x555555b65e60)[xhttp:request] to 3 0(46158) DEBUG: <core> [core/ppcfg.c:224]: pp_ifdef_level_check(): same number of pairing preprocessor directives #!IF[N]DEF - #!ENDIF 0(46158) DEBUG: <core> [core/route.c:129]: route_add(): mapping routing block (0x555555b65e60)[core:receive-parse-error] to 4 0(46158) INFO: <core> [core/sctp_core.c:74]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module 0(46158) DEBUG: <core> [core/socket_info.c:1400]: add_interfaces(): If: eth0 Fam: 2 Flg: 11043 Adr: 172.31.70.61 0(46158) DEBUG: <core> [core/socket_info.c:1400]: add_interfaces(): If: eth0 Fam: 2 Flg: 11043 Adr: 172.31.70.61 0(46158) DEBUG: <core> [core/socket_info.c:1400]: add_interfaces(): If: eth0 Fam: 2 Flg: 11043 Adr: 172.31.70.61 Listening on udp: 172.31.70.61 [172.31.70.61]:5060 advertise 3.91.14.101:5060 tcp: 172.31.70.61 [172.31.70.61]:5060 advertise 3.91.14.101:5060 tcp: 127.0.0.1 [127.0.0.1]:80 tls: 172.31.70.61 [172.31.70.61]:5061 advertise 3.91.14.101:5061 Aliases: *: sip.rraccess.com:*
0(46158) DEBUG: <core> [core/mem/pkg.c:108]: pkg_print_manager(): pkg - using memory manager: q_malloc 0(46158) DEBUG: <core> [core/mem/shm.c:300]: shm_print_manager(): shm - using memory manager: q_malloc 0(46158) DEBUG: <core> [core/cfg/cfg.c:173]: cfg_declare(): new config group has been registered: 'core' (num=56, size=224) 0(46158) DEBUG: <core> [core/cfg/cfg.c:173]: cfg_declare(): new config group has been registered: 'tcp' (num=27, size=108) 0(46158) DEBUG: <core> [core/timer.c:239]: init_timer(): starting with *ticks=2089369456 0(46158) DEBUG: <core> [core/timer.c:280]: init_timer(): timer_list between 0x7ffff30d07f0 and 0x7ffff31147f0 0(46158) DEBUG: <core> [core/pt.c:139]: register_procs(): registering new processes: 0 (old) + 1 (new) = 1 (total) 0(46158) INFO: <core> [core/tcp_main.c:4983]: init_tcp(): using epoll_lt as the io watch method (auto detected) [Detaching after fork from child process 46162] 0(46163) DEBUG: <core> [core/daemonize.c:207]: enable_dumpable(): trying enable core dumping... 0(46163) DEBUG: <core> [core/daemonize.c:224]: enable_dumpable(): core dumping is enabled now (1)... 0(46163) DEBUG: <core> [core/daemonize.c:583]: set_core_dump(): core dump limits set to 18446744073709551615 0(46163) DEBUG: tls [tls_init.c:629]: tls_h_mod_pre_init_f(): preparing tls env for modules initialization 0(46163) DEBUG: tls [tls_init.c:631]: tls_h_mod_pre_init_f(): preparing tls env for modules initialization (libssl >=1.1) 0(46163) DEBUG: <core> [core/async_task.c:123]: async_task_init(): start initializing asynk task framework 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): tls 0(46163) DEBUG: <core> [core/cfg/cfg.c:173]: cfg_declare(): new config group has been registered: 'tls' (num=31, size=272) 0(46163) INFO: tls [tls_mod.c:389]: mod_init(): With ECDH-Support! 0(46163) INFO: tls [tls_mod.c:392]: mod_init(): With Diffie Hellman 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): auth 0(46163) DEBUG: auth [auth_mod.c:274]: mod_init(): auth module - initializing 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <bind_sl> in module sl [/usr/local/lib64/kamailio/modules/sl.so] 0(46163) DEBUG: <core> [core/usr_avp.c:887]: parse_avp_ident(): Parsing 'digest_challenge' 0(46163) INFO: auth [auth_mod.c:345]: mod_init(): qop set, but nonce-count (nonce_count) support disabled 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): jsonrpcs 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <bind_xhttp> in module xhttp [/usr/local/lib64/kamailio/modules/xhttp.so] 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): kex 0(46163) DEBUG: <core> [core/utils/sruid.c:105]: sruid_init(): root for sruid is [srid-5fc6b661-b453-] (0 / 19) 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): corex 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): tm 0(46163) DEBUG: tm [tm.c:711]: mod_init(): TM - (sizeof cell=624, sip_msg=2008) initializing... 0(46163) DEBUG: tm [callid.c:105]: init_callid(): Call-ID initialization: '0ee1f7516248a6d2' 0(46163) DEBUG: tm [lock.c:66]: lock_initialize(): lock initialization started 0(46163) DEBUG: tm [timer.c:195]: tm_init_timers(): tm init timers - fr=48 fr_inv=1440 wait=80 t1=500 t2=4000 max_inv_lifetime=2880 max_noninv_lifetime=512 0(46163) DEBUG: <core> [core/cfg/cfg.c:173]: cfg_declare(): new config group has been registered: 'tm' (num=36, size=168) 0(46163) DEBUG: <core> [core/crypto/md5utils.c:67]: MD5StringArray(): MD5 calculated: 07d7259915666eca6ddb2a39fe2cf7e5 0(46163) DEBUG: <core> [core/crypto/md5utils.c:67]: MD5StringArray(): MD5 calculated: 97693c4a35fd2f31d46cf848a755e076 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): tmx 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <load_tm> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_newtran> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_relay_to_tcp> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_relay_to_udp> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_relay> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_forward_nonack> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_release> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): sl 0(46163) DEBUG: <core> [core/crypto/md5utils.c:67]: MD5StringArray(): MD5 calculated: 3f99dd4a61db77a8012dbbcb5df8674f 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <load_tm> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_newtran> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_relay_to_tcp> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_relay_to_udp> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_relay> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_forward_nonack> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <t_release> in module tm [/usr/local/lib64/kamailio/modules/tm.so] 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): rr 0(46163) DEBUG: <core> [core/sr_module.c:638]: find_mod_export_record(): export of <bind_ob> not found (flags 0) 0(46163) INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded 0(46163) INFO: rr [rr_mod.c:185]: mod_init(): outbound module not available 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): pv 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): maxfwd 0(46163) DEBUG: <core> [core/cfg/cfg.c:173]: cfg_declare(): new config group has been registered: 'maxfwd' (num=1, size=4) 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): textops 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): siputils 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <bind_sl> in module sl [/usr/local/lib64/kamailio/modules/sl.so] 0(46163) DEBUG: <core> [core/usr_avp.c:887]: parse_avp_ident(): Parsing 's:rpid' 0(46163) DEBUG: <core> [core/cfg/cfg.c:173]: cfg_declare(): new config group has been registered: 'siputils' (num=1, size=4) 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): xlog 0(46163) DEBUG: <core> [core/cfg/cfg.c:173]: cfg_declare(): new config group has been registered: 'xlog' (num=1, size=4) 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): sanity 0(46163) DEBUG: sanity [sanity_mod.c:103]: mod_init(): sanity initializing 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <bind_sl> in module sl [/usr/local/lib64/kamailio/modules/sl.so] 0(46163) DEBUG: sanity [sanity_mod.c:113]: mod_init(): parsing proxy requires string: 0(46163) DEBUG: sanity [sanity.c:178]: parse_str_list(): list is empty 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): ctl 0(46163) DEBUG: ctl [ctl.c:250]: mod_init(): listening on: 0(46163) DEBUG: ctl [ctl.c:259]: mod_init(): [binrpc:unix stream] /var/run/kamailio//kamailio_ctl 0(46163) DEBUG: <core> [core/pt.c:139]: register_procs(): registering new processes: 1 (old) + 1 (new) = 2 (total) 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): cfg_rpc 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): uuid 0(46163) DEBUG: uuid [uuid_mod.c:87]: mod_init(): uuid initialized - probing value [24d718c8-9bc8-477e-81e2-0ee6a6d99f91] 0(46163) DEBUG: <core> [core/sr_module.c:847]: init_mod(): dispatcher 0(46163) DEBUG: <core> [core/cfg/cfg.c:173]: cfg_declare(): new config group has been registered: 'dispatcher' (num=3, size=24) 0(46163) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <db_bind_api> in module db_mongodb [/usr/local/lib64/kamailio/modules/db_mongodb.so] 0(46163) DEBUG: <core> [db.c:210]: db_bind_mod(): using db bind api for db_mongodb 0(46163) DEBUG: <core> [db.c:315]: db_do_init2(): connection 0x7ffff75d4c90 not found in pool 0(46163) DEBUG: db_mongodb [mongodb_connection.c:55]: db_mongodb_new_connection(): connection open to: mongodb://metrics-user:c2b16f0b3b34e52fb32cafdca1008759fe60679aa6305a56842757b2809adb2357281d63e8b19db1ea735e1cba1743df@sandbox-shard-00-00.ru1pe.mongodb.net,sandbox-shard-00-01.ru1pe.mongodb.net,sandbox-shard-00-02.ru1pe.mongodb.net/kamailio?ssl=true&replicaSet=atlas-f1lj9f-shard-0&authSource=admin&retryWrites=true&w=majority 0(46163) DEBUG: db_mongodb [mongodb_dbase.c:973]: db_mongodb_query(): query to collection [version] 0(46163) DEBUG: db_mongodb [mongodb_dbase.c:1007]: db_mongodb_query(): query filter: { "table_name" : "dispatcher" } 0(46163) DEBUG: db_mongodb [mongodb_dbase.c:1043]: db_mongodb_query(): columns filter: { "projection" : { "table_version" : 1 } } 0(46163) DEBUG: <core> [db_res.c:119]: db_new_result(): allocate 56 bytes for result set at 0x7ffff75d5248 /build/mongo-c-driver-RRvvb_/mongo-c-driver-1.16.1/src/libmongoc/src/mongoc/mongoc-openssl.c:476 _mongoc_openssl_ctx_new(): precondition failed: ctx 0(46158) ERROR: <core> [core/daemonize.c:303]: daemonize(): Main process exited before writing to pipe 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30679f8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067ca8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30b7fc0), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30654a0), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30585a8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30577b8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3057858), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30578f8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067600), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_realloc(299) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067020), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067738), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30b7e18), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067900), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067970), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067140), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067698), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30cf3d8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30cf350), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30cf2c8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067860), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30609b8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_realloc(299) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3058648), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3058848), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3067d20), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_malloc(293) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff306b990), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3076460), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff307fb48), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff30839c8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7ffff3085c48), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free(323) - ignoring 0(46158) CRITICAL: <core> [core/mem/q_malloc.c:123]: qm_debug_check_frag(): BUG: qm: fragm. 0x7ffff308bff8 (address 0x7ffff308c030) beginning overwritten (0)! Memory allocator was called from tls: tls_init.c:323. Fragment marked by (null):0. Exec from core/mem/q_malloc.c:511.
Program received signal SIGABRT, Aborted. __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt full #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 set = {__val = {0, 140737350214984, 140737352245490, 1508362478, 23568163, 140737488345860, 0, 140737488346064, 140737352234024, 140737488346048, 0, 140737353886272, 7, 93823560581120, 4294967295, 0}} pid = <optimized out> tid = <optimized out> ret = <optimized out> #1 0x00007ffff7c60859 in __GI_abort () at abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x7ffff73fdb14, sa_sigaction = 0x7ffff73fdb14}, sa_mask = {__val = {12187122887200642304, 140737270609440, 140737270374400, 140737341545236, 12187122887200642304, 299, 93824995563840, 93824997234282, 93824997234096, 93824997234048, 93824997234456, 140737270824952, 12187122887200642304, 0, 93824997234282, 0}}, sa_flags = 0, sa_restorer = 0x0} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x0000555555881ad1 in qm_debug_check_frag (qm=0x7ffff301e000, f=0x7ffff308bff8, file=0x7ffff73fdb14 "tls: tls_init.c", line=323, efile=0x555555a19599 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:129 p = 0x7ffff73fdb14 __func__ = "qm_debug_check_frag" #3 0x00005555558862ba in qm_free (qmp=0x7ffff301e000, p=0x7ffff308c030, file=0x7ffff73fdb14 "tls: tls_init.c", func=0x7ffff73ff0a0 <__func__.23420> "ser_free", line=323, mname=0x7ffff73fdb10 "tls") at core/mem/q_malloc.c:511 qm = 0x7ffff301e000 f = 0x7ffff308bff8 size = 18446744064901936792 next = 0x7ffff301e000 prev = 0xfffffffdf30b7ec0 __func__ = "qm_free" #4 0x0000555555891d89 in qm_shm_free (qmp=0x7ffff301e000, p=0x7ffff308c030, file=0x7ffff73fdb14 "tls: tls_init.c", func=0x7ffff73ff0a0 <__func__.23420> "ser_free", line=323, mname=0x7ffff73fdb10 "tls") at core/mem/q_malloc.c:1275 No locals. #5 0x00007ffff73a9078 in ser_free (ptr=0x7ffff308c030, fname=0x7ffff725e24b "../crypto/lhash/lhash.c", fline=88) at tls_init.c:323 __func__ = "ser_free" #6 0x00007ffff71979c4 in OPENSSL_LH_free () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #7 0x00007ffff71750c5 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #8 0x00007ffff7195480 in OPENSSL_cleanup () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #9 0x00007ffff7c84a27 in __run_exit_handlers (status=-1, listp=0x7ffff7e26718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 atfct = <optimized out> onfct = <optimized out> cxafct = <optimized out> f = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- new_exitfn_called = 10 cur = 0x7ffff7e28980 <initial> #10 0x00007ffff7c84be0 in __GI_exit (status=<optimized out>) at exit.c:139 No locals. #11 0x00005555557e97ae in daemonize (name=0x7fffffffe897 "/usr/local/sbin/kamailio", status_wait=1) at core/daemonize.c:304 pid_stream = 0x0 pid = 46162 r = 1436369193 p = -1 pipe_status = 85 'U' pid_uid = 4294967295 pid_gid = 4294967295 __func__ = "daemonize" #12 0x000055555559652e in main (argc=1, argv=0x7fffffffe688) at main.c:2763 cfg_stream = 0x555555b672a0 c = -1 r = -134414416 tmp = 0x0 tmp_len = 32767 port = 8064 proto = 65535 ahost = 0x0 aport = 0 options = 0x5555559b3d48 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3286661736 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0x0 st = {st_dev = 25, st_ino = 461, st_nlink = 2, st_mode = 16893, st_uid = 115, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1606858266, tv_nsec = 984701676}, st_mtim = {tv_sec = 1606858266, tv_nsec = 984701676}, st_ctim = {tv_sec = 1606858315, tv_nsec = 714658382}, __glibc_reserved = {0, 0, 0}} tbuf = '\000' <repeats 32 times>, "Լ\252\252\252\252\377\377\000\000\000\000\000\000\000\000\002", '\000' <repeats 351 times>... option_index = 0 long_options = {{name = 0x5555559b5f76 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x5555559b1514 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x5555559b5f7b "alias", has_arg = 1, --Type <RET> for more, q to quit, c to continue without paging-- flag = 0x0, val = 1024}, {name = 0x5555559b5f81 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x5555559b5f87 "substdef", has_arg = 1, flag = 0x0, val = 1026}, { name = 0x5555559b5f90 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x5555559b5f9a "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x5555559b5fa4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x5555559b5faf "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x5555559b5fb8 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, { name = 0x5555559b5fc3 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" ```
Given the mongo issue, any way we can get a mongo driver that accepts a `mongodb+srv://` url
Hello,
i have the same creash on Centos 8.
# lsb_release -a LSB Version: :core-4.1-amd64:core-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 8.2.2004 (Core) Release: 8.2.2004 Codename: Core
# cat /etc/os-release NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" PRETTY_NAME="CentOS Linux 8 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:8" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8" CENTOS_MANTISBT_PROJECT_VERSION="8" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="8"
# uname -a Linux dev-nexios-fsrhel 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
I can attach the entire log from start till the crash if needed ... my logs are pretty much the same as knightcode :)
Here is a small snip: 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f3537194d68), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_mallo c(293) - ignoring 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f35371959e8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_mallo c(293) - ignoring 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f3537195be8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_mallo c(293) - ignoring 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f3537195668), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_free( 323) - ignoring 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f35371955e8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_mallo c(293) - ignoring 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f3537195868), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_mallo c(293) - ignoring 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f3537195ee8), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_mallo c(293) - ignoring 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f353716f450), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_reall oc(299) - ignoring 0(88367) CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f353712d160), called from tls: tls_init.c: ser_free(323), first free tls: tls_init.c: ser_mallo c(293) - ignoring
bt full:
warning: Loadable section ".note.gnu.property" outside of ELF segments Core was generated by `/usr/local/kamailio-devel/sbin/kamailio -DD -P /run/kamailio/kamailio.pid -f /u'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f3541c6318a in BN_clear_free () from /lib64/libcrypto.so.1.1 Missing separate debuginfos, use: yum debuginfo-install brotli-1.0.6-1.el8.x86_64 cyrus-sasl-lib-2.1.27-1.el8.x86_64 glibc-2.28-101.el8.x86_64 jansson-2.11-3.el8.x86_64 json-c-0.13.1-0.2.el8.x86_64 keyutils-libs-1.5.10-6.el8.x86_64 krb5-libs-1.17-18.el8.x86_64 libblkid-2.32.1-22.el8.x86_64 libbson-1.17.2-1.el8.x86_64 libcap-2.26-3.el8.x86_64 libcom_err-1.45.4-3.el8.x86_64 libcurl-7.61.1-12.el8.x86_64 libdb-5.3.28-37.el8.x86_64 libgcc-8.3.1-5.el8.0.2.x86_64 libicu-60.3-2.el8_1.x86_64 libidn2-2.2.0-1.el8.x86_64 libmount-2.32.1-22.el8.x86_64 libnghttp2-1.39.2-1.el8.x86_64 libpq-12.4-1.el8_2.x86_64 libpsl-0.20.2-5.el8.x86_64 libselinux-2.9-3.el8.x86_64 libssh-0.9.0-4.el8.x86_64 libstdc++-8.3.1-5.el8.0.2.x86_64 libunistring-0.9.9-3.el8.x86_64 libuuid-2.32.1-22.el8.x86_64 libxcrypt-4.1.1-4.el8.x86_64 mongo-c-driver-libs-1.17.2-1.el8.x86_64 openldap-2.4.46-11.el8_1.x86_64 openssl-libs-1.1.1c-15.el8.x86_64 pcre-8.42-4.el8.x86_64 snappy-1.1.7-5.el8.x86_64 sssd-client-2.2.3-20.el8.x86_64 systemd-libs-239-31.el8_2.2.x86_64 zlib-1.2.11-16.el8_2.x86_64 (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) (gdb) bt #0 0x00007f3541c6318a in BN_clear_free () from /lib64/libcrypto.so.1.1 #1 0x00007f353230d945 in ssh_dh_finalize.part () from /lib64/libssh.so.4 #2 0x00007f35322ff5fd in libssh_destructor () from /lib64/libssh.so.4 #3 0x00007f3543d8c2a6 in _dl_fini () from /lib64/ld-linux-x86-64.so.2 #4 0x00007f3543036e9c in __run_exit_handlers () from /lib64/libc.so.6 #5 0x00007f3543036fd0 in exit () from /lib64/libc.so.6 #6 0x0000000000421c21 in handle_sigs () at main.c:810 #7 0x000000000042c537 in main_loop () at main.c:1817 #8 0x0000000000434d46 in main (argc=15, argv=0x7ffc92a2e8e8) at main.c:2856 (gdb) bt full #0 0x00007f3541c6318a in BN_clear_free () from /lib64/libcrypto.so.1.1 No symbol table info available. #1 0x00007f353230d945 in ssh_dh_finalize.part () from /lib64/libssh.so.4 No symbol table info available. #2 0x00007f35322ff5fd in libssh_destructor () from /lib64/libssh.so.4 No symbol table info available. #3 0x00007f3543d8c2a6 in _dl_fini () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #4 0x00007f3543036e9c in __run_exit_handlers () from /lib64/libc.so.6 No symbol table info available. #5 0x00007f3543036fd0 in exit () from /lib64/libc.so.6 No symbol table info available. #6 0x0000000000421c21 in handle_sigs () at main.c:810 chld = 0 chld_status = 65280 any_chld_stopped = 1 memlog = 926740624 __func__ = "handle_sigs" #7 0x000000000042c537 in main_loop () at main.c:1817 i = 8 pid = 88396 si = 0x0 si_desc = "udp receiver child=7 sock=192.168.5.238:5060\000\177\000\000\060\000\000\000\060\000\000\000\310㢒\374\177\000\000\320⢒\374\177\000\000\000\312\004j\314\021Fz 㢒\374\177\000\000M\vo", '\000' <repeats 13 times>, "/Y\001\000\000\000\000\000\323\022\203\000\000\000\000\000\070x\326B5\177\000" nrprocs = 8 woneinit = 1 __func__ = "main_loop" #8 0x0000000000434d46 in main (argc=15, argv=0x7ffc92a2e8e8) at main.c:2856 cfg_stream = 0x22472a0 c = -1 r = 0 tmp = 0x7ffc92a3083f "" tmp_len = 32565 port = 1140483840 proto = 0 ahost = 0x0 aport = 0 options = 0x7ef1b8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2338164144 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0x7f3543fa6700 p = 0x7f35430a124d <handle_intel.constprop+269> "H\205\300u\036A\215U\001D9l$\f\017\207?\377\377\377\201", <incomplete sequence \355\277> st = {st_dev = 23, st_ino = 19256, st_nlink = 2, st_mode = 16832, st_uid = 991, st_gid = 989, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = { tv_sec = 1606640254, tv_nsec = 237000000}, st_mtim = {tv_sec = 1606876315, tv_nsec = 8047879}, st_ctim = {tv_sec = 1606876315, tv_nsec = 8047879}, __glibc_reserved = {0, 0, 0}} tbuf = "|c\001C5\177\000\000\330\361\262\222\374\177\000\000\000\000\000\000 ", '\000' <repeats 19 times>, " \317\327C5\177\000\000\300墒\374\177\000\000\a\000\000\000\000\000\000\000\330\304\327C5\177\000\000\a\000\000\000\t\000\000\000\220Y\372C5\177\000\000̀\330C5\177\000\000\000\000\000\000\000\000\000\000\370\216\330C5\177\000\000\020墒\374\177\000\000\240c\001C5\177\000\000\300c\371C5\177\000\000\000\000\000\000\000\000\000\000\220墒\374\177", '\000' <repeats 26 times>, "H\307\327C5\177\000\000\270d\372C5\177\000\000\220墒\374\177\000\000\000\300\327C5\177\000\000|"... option_index = 0 long_options = {{name = 0x7f1326 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7ec8f4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7f132b "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7f1331 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7f1337 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7f1340 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7f134a "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7f1354 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, { name = 0x7f135f "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7f1368 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7f1373 "debug", has_arg = 1, flag = 0x0, --Type <RET> for more, q to quit, c to continue without paging-- val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" (gdb) (gdb) q
The problem seems to be with the mongodb client library. Is it installed from packages or built from sources? Can you try to load db_mongodb module after tls module?
@knightcode - you can extend the c code of srdb1 internal lib or db_mongodb module to support extended schema like `mongodb+srv`.
hi,
@miconda i have this order from the start and it still crashes:
#!ifdef WITH_TLS loadmodule "tls.so" #!endif
#!ifdef WITH_ANTIFLOOD loadmodule "htable.so" loadmodule "pike.so" #!endif
#!ifdef WITH_DEBUG loadmodule "debugger.so" #!endif
#!ifdef WITH_MDB loadmodule "db_mongodb.so" #!endif
We also already load it first. And we're compiling from the HEAD of 5.4...
I don't think I'll be given the time to bring the module, or at least its interface, up to date for this project.
@miconda mongo mongodb client library is installed from packet manager:
``` dnf install mongo-c-driver-devel.x86_64
dnf list installed mongo-c-driver-devel.x86_64 Installed Packages mongo-c-driver-devel.x86_64 1.17.2-1.el8 @epel ```
let me know if i need to install a specific version of mongodb driver
@knightcode
do you ever get any response from mongo regarding version for dispatcher ? You should have an entry in version collection for dispatcher table.
``` db.getCollection("version").find({"table_name":"dispatcher"}).pretty() { "_id" : ObjectId("5fce0a29d09896665dd585b4"), "table_name" : "dispatcher", "table_version" : 4 } ```
I can see it starts crashing right after trying to get the table version. ``` 0(46163) DEBUG: db_mongodb [mongodb_dbase.c:973]: db_mongodb_query(): query to collection [version] 0(46163) DEBUG: db_mongodb [mongodb_dbase.c:1007]: db_mongodb_query(): query filter: { "table_name" : "dispatcher" } 0(46163) DEBUG: db_mongodb [mongodb_dbase.c:1043]: db_mongodb_query(): columns filter: { "projection" : { "table_version" : 1 } } 0(46163) DEBUG: <core> [db_res.c:119]: db_new_result(): allocate 56 bytes for result set at 0x7ffff75d5248 /build/mongo-c-driver-RRvvb_/mongo-c-driver-1.16.1/src/libmongoc/src/mongoc/mongoc-openssl.c:476 _mongoc_openssl_ctx_new(): precondition failed: ctx 0(46158) ERROR: <core> [core/daemonize.c:303]: daemonize(): Main process exited before writing to pipe ```
In my case it was crashing on userloc ... I had an entry in location collection and "path" key was missing (the entry was made by kamailio when it was run without path module). It crashed when i loaded path module because kamailio was expecting path ke/field in teh location table/collection.
Yeah, we have an entry for it:
``` db.getCollection("version").find({"table_name":"dispatcher"}).pretty() { "_id" : ObjectId("5f8898080cb4ca423800169c"), "table_name" : "dispatcher", "table_version" : 3 } ```
is version 3 what is kamailio 5.4.x looking for ?
Im not saying the table version mismatch is crashing it, but i got it crash on a missing key:val in a mongo document :) so it might.. just might help in your case as well.
check this on your kamailio machine: /usr/share/kamailio/postgres or /usr/share/kamailio/mysql
This is what is needed as table version for 5.4.x
``` kamailio2=# select * from version; id | table_name | table_version ----+---------------------+--------------- 1 | version | 1 2 | acc | 5 3 | acc_cdrs | 2 4 | missed_calls | 4 5 | lcr_gw | 3 6 | lcr_rule_target | 1 7 | lcr_rule | 3 8 | domain | 2 9 | domain_attrs | 1 10 | grp | 2 11 | re_grp | 1 12 | trusted | 6 13 | address | 6 14 | aliases | 8 15 | location | 9 16 | location_attrs | 1 17 | silo | 8 18 | dbaliases | 1 19 | uri | 1 20 | speed_dial | 2 21 | usr_preferences | 2 22 | subscriber | 7 23 | pdt | 1 24 | dialog | 7 25 | dialog_vars | 1 26 | dispatcher | 4 27 | dialplan | 2 28 | topos_d | 1 29 | topos_t | 1 30 | presentity | 5 31 | active_watchers | 12 32 | watchers | 3 33 | xcap | 4 34 | pua | 7 35 | rls_presentity | 1 36 | rls_watchers | 3 37 | imc_rooms | 1 38 | imc_members | 1 39 | cpl | 1 40 | sip_trace | 4 41 | domainpolicy | 2 42 | carrierroute | 3 43 | carrierfailureroute | 2 44 | carrier_name | 1 45 | domain_name | 1 46 | dr_gateways | 3 47 | dr_rules | 3 48 | dr_gw_lists | 1 49 | dr_groups | 2 50 | userblacklist | 1 51 | globalblacklist | 1 52 | htable | 2 53 | purplemap | 1 54 | uacreg | 4 55 | pl_pipes | 1 56 | mtree | 1 57 | mtrees | 2 58 | sca_subscriptions | 2 59 | mohqcalls | 1 60 | mohqueues | 1 61 | rtpproxy | 1 62 | rtpengine | 1 63 | secfilter | 1 ```
added version, location, location_attrs, rtpproxy, rtpengine, htable, acc, and acc_cdrs since those seemed to be related to the modules we have enabled. No change.
MongoDB atlas is configured to require TLS. If we could disable it, we would try that.... but I'm not sure that we can nor would we want to.
We reconfigured to remove our MongoDB dependency. But enabling TLS still causes a crash ...on Ubuntu 20 at least.
DEBUG output before crash: ``` 0(173591) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <xhttp_reply> in module xhttp [/usr/local/lib64/kamailio/modules/xhttp.so] 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type2, i/count 0/2: type 3(3)/ data 0x1 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type2, i/count 1/2: type 3(3)/ data 0x1 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type16, i/count 0/3: type 22(16)/ data 0x7f7155646b60 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type16, i/count 1/3: type 8(8)/ data 0x7f71556473b0 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type16, i/count 2/3: type 0(0)/ data (nil) 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type24, i/count 0/2: type 9(9)/ data (nil) 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type24, i/count 1/2: type 3(3)/ data (nil) 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/re.c:436]: subst_run(): running. r=1 0(173591) DEBUG: <core> [core/re.c:504]: subst_str(): no match 0(173591) DEBUG: <core> [core/sr_module.c:632]: find_mod_export_record(): found export of <xhttp_reply> in module xhttp [/usr/local/lib64/kamailio/modules/xhttp.so] 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type2, i/count 0/2: type 3(3)/ data 0x1 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type2, i/count 1/2: type 3(3)/ data 0x1 0(173591) DEBUG: <core> [core/route.c:129]: route_add(): mapping routing block (0x55d00859be60)[xhttp:request] to 3 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type5, i/count 0/1: type 1(1)/ data 0x7f7155648c00 0(173591) DEBUG: <core> [core/route_struct.c:128]: mk_action(): ACTION type5, i/count 0/1: type 1(1)/ data 0x7f7155648e38 0(173591) DEBUG: <core> [core/route.c:129]: route_add(): mapping routing block (0x55d00859be60)[htable:mod-init] to 4 0(173591) DEBUG: <core> [core/ppcfg.c:224]: pp_ifdef_level_check(): same number of pairing preprocessor directives #!IF[N]DEF - #!ENDIF 0(173591) DEBUG: <core> [core/route.c:129]: route_add(): mapping routing block (0x55d00859be60)[core:receive-parse-error] to 5 0(173591) INFO: <core> [core/sctp_core.c:74]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module 0(173591) ERROR: <core> [main.c:2625]: main(): failed to create runtime dir /var/run/kamailio/, check directory permissions failed to create runtime dir /var/run/kamailio/, check directory permissions 0(173591) DEBUG: tm [t_funcs.c:84]: tm_shutdown(): start 0(173591) DEBUG: tm [t_funcs.c:87]: tm_shutdown(): emptying hash table 0(173591) DEBUG: tm [t_funcs.c:89]: tm_shutdown(): removing semaphores 0(173591) DEBUG: tm [t_funcs.c:91]: tm_shutdown(): destroying tmcb lists 0(173591) DEBUG: tm [t_funcs.c:94]: tm_shutdown(): done 0(173591) DEBUG: tls [tls_init.c:859]: tls_h_mod_destroy_f(): tls module final tls destroy 0(173591) DEBUG: tls [tls_init.c:869]: tls_h_mod_destroy_f(): executing openssl v1.1+ cleanup 0(173591) INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized 0(173591) DEBUG: <core> [core/mem/shm.c:288]: shm_destroy_manager(): destroying memory manager: q_malloc 0(173591) DEBUG: <core> [core/mem/q_malloc.c:1185]: qm_shm_lock_destroy(): destroying the shared memory lock 0(173591) DEBUG: <core> [core/mem/pkg.c:97]: pkg_destroy_manager(): destroying memory manager: q_malloc Segmentation fault ```
backtrace: ``` 11(173938) DEBUG: <core> [core/sr_module.c:778]: init_mod_child(): idx 11 rank 4: siptrace [tcp receiver (generic) child=1] 11(173938) DEBUG: <core> [core/local_timer.c:60]: init_local_timer(): timer_list between 0x555555b163c8 and 0x555555b5a3c8 11(173938) DEBUG: <core> [core/io_wait.h:374]: io_watch_add(): DBG: io_watch_add(0x555555b16200, 25, 1, (nil)), fd_no=0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt full #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 set = {__val = {0, 93824997137864, 93824997137864, 1, 140737352204704, 0, 140737352016304, 140737488346976, 140737488346592, 140737350737331, 140737488347504, 93824996790201, 205470458, 1597694572, 0, 10346731307112276224}} pid = <optimized out> tid = <optimized out> ret = <optimized out> #1 0x00007ffff7c60859 in __GI_abort () at abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x7ffff7e275c0 <_IO_2_1_stderr_>, sa_sigaction = 0x7ffff7e275c0 <_IO_2_1_stderr_>}, sa_mask = {__val = {10346731307112276224, 140737270609440, 140737270374400, 140737352201664, 93824996942784, 140737488346864, 93824995565158, 93824997234282, 93824997234624, 93824997234048, 93824997234456, 140737270977464, 10346731307112276224, 0, 93824997234282, 0}}, sa_flags = 0, sa_restorer = 0x7ffff73fdb14} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x000055555588202c in qm_debug_check_frag (qm=0x7ffff301e000, f=0x7ffff30b13b8, file=0x7ffff73fdb14 "tls: tls_init.c", line=323, efile=0x555555a19599 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:140 p = 0x7ffff7ccead1 <_IO_new_do_write+177> __func__ = "qm_debug_check_frag" #3 0x00005555558862ba in qm_free (qmp=0x7ffff301e000, p=0x7ffff30b13f0, file=0x7ffff73fdb14 "tls: tls_init.c", func=0x7ffff73ff0a0 <__func__.23420> "ser_free", line=323, mname=0x7ffff73fdb10 "tls") at core/mem/q_malloc.c:511 qm = 0x7ffff301e000 f = 0x7ffff30b13b8 size = 281479271743489 next = 0x1000100010001 prev = 0x1000100010001 __func__ = "qm_free" #4 0x0000555555891d89 in qm_shm_free (qmp=0x7ffff301e000, p=0x7ffff30b13f0, file=0x7ffff73fdb14 "tls: tls_init.c", func=0x7ffff73ff0a0 <__func__.23420> "ser_free", line=323, mname=0x7ffff73fdb10 "tls") at core/mem/q_malloc.c:1275 No locals. #5 0x00007ffff73a9078 in ser_free (ptr=0x7ffff30b13f0, fname=0x7ffff723b960 "../crypto/bn/bn_lib.c", fline=194) at tls_init.c:323 __func__ = "ser_free" #6 0x00007ffff70d8b4d in BN_clear_free () from /lib/x86_64-linux-gnu/libcrypto.so.1.1 No symbol table info available. #7 0x00007ffff27fdf35 in ?? () from /lib/x86_64-linux-gnu/libssh.so.4 No symbol table info available. #8 0x00007ffff27ef4fd in ?? () from /lib/x86_64-linux-gnu/libssh.so.4 No symbol table info available. #9 0x00007ffff7fe0f5b in ?? () from /lib64/ld-linux-x86-64.so.2 No symbol table info available. #10 0x00007ffff7c84a27 in __run_exit_handlers (status=0, listp=0x7ffff7e26718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 atfct = <optimized out> onfct = <optimized out> cxafct = <optimized out> f = <optimized out> new_exitfn_called = 9 cur = 0x7ffff7e28980 <initial> --Type <RET> for more, q to quit, c to continue without paging-- #11 0x00007ffff7c84be0 in __GI_exit (status=<optimized out>) at exit.c:139 No locals. #12 0x00005555557e948e in daemonize (name=0x7fffffffe8a3 "/usr/local/sbin/kamailio", status_wait=1) at core/daemonize.c:301 pid_stream = 0x0 pid = 173923 r = 1436369193 p = -1 pipe_status = 0 '\000' pid_uid = 4294967295 pid_gid = 4294967295 __func__ = "daemonize" #13 0x000055555559652e in main (argc=1, argv=0x7fffffffe6a8) at main.c:2763 cfg_stream = 0x555555b672a0 c = -1 r = -134414416 tmp = 0x0 tmp_len = 0 port = -134422088 proto = 32767 ahost = 0x0 aport = 0 options = 0x5555559b3d48 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2333055721 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0x0 st = {st_dev = 0, st_ino = 0, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 0, st_blksize = 0, st_blocks = 0, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 0, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __glibc_reserved = {0, 0, 0}} tbuf = "p\345\377\377\377\177\000\000\253\067\375\367\377\177", '\000' <repeats 18 times>, "Լ\252\252\252\252\377\377\000\000\000\000\000\000\000\000\004\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\000\347\377\367\377\177\000\000\200\241\303\367\377\177\000\000\002", '\000' <repeats 319 times>... option_index = 0 long_options = {{name = 0x5555559b5f76 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x5555559b1514 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x5555559b5f7b "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x5555559b5f81 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x5555559b5f87 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x5555559b5f90 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x5555559b5f9a "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x5555559b5fa4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x5555559b5faf "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x5555559b5fb8 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x5555559b5fc3 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" ```
@miconda I sent an email to office@asipto.com (hopefully, you) regarding this issue
All these for Ubuntu 20.04 should be fixed by `--atexit` commits, to be backported to branch 5.4. Thanks @knightcode for supporting the troubleshooting and the work to fix them.
Closed #2560.
-
Daniel-Constantin Mierla -
Dylan
-
tculjaga