Re: [SR-Users] segmentation fault when mongodb & tls enabled. ( kamailio 4.3.2)

Hi, I tried with mongo-c-driver 1.2.1 and crash occurring with that library as well. So i took the kamailio basic script and able to reproduce it. So the issue is slightly diffrent now. Core occurs when i enable pike(ANTIFLOOD). Another thing which i noticed is, core doesn't occur If i disable TLS and enable pike. Please find the attached script files for more details about my configuration. Here is the backtrace when i enable TLS and ANTIFLOOD done. Loaded symbols for /usr/local/kamailio_proxy/lib64/kamailio/modules/ndb_mongodb.so Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00000000005fef6f in compat_old_handler (ti=2030104642, tl=0x7f9b68f7e130, data=0x7f9bffffffff) at timer.c:996 996 t->timer_f(TICKS_TO_S(*ticks), t->t_param); Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-20.el6_7.x86_64 glibc-2.12-1.166.el6_7.3.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00000000005fef6f in compat_old_handler (ti=2030104642, tl=0x7f9b68f7e130, data=0x7f9bffffffff) at timer.c:996 #1 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #2 0x00000000004a8676 in main_loop () at main.c:1628 #3 0x00000000004ae38e in main (argc=15, argv=0x7ffc9cc0ec78) at main.c:2533 (gdb) quit On Tue, Nov 3, 2015 at 10:04 AM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com > Hello, > > I tried to reproduce here and all seems fine -- I enabled tls and > registered a user using mongodb as backend. > > Can you give the parameters you set for usrloc module? > > Cheers, > Daniel > > > On 06/10/15 19:03, Jijo wrote: > > Thanks, I have opened the issue on github tracker. > > regards > jijo > > On Tue, Oct 6, 2015 at 9:53 AM, Daniel-Constantin Mierla < > <miconda@gmail.com>miconda@gmail.com> >> Hello, >> >> unfortunately not yet -- this requires compiling the libs and currently I >> am traveling to USA to attend few VoIP conferences, not having a good devel >> environment with me, access to internet is limited, so connecting to a >> server is not easy option as well. Can you open an issue on github tracker >> just not to forget about it? >> >> Cheers, >> Daniel >> >> >> On 06/10/15 15:49, realjijo(a)gmail.com wrote: >> >> Hi Daniel >> >> Thanks for your help . >> >> Did you get a chance to look at this problem. >> Please let me know how did you build the mongo-c-driver ? >> >> >> Regards, >> Jijo >> >> On Oct 2, 2015, at 7:21 AM, Jijo < <realjijo@gmail.com>realjijo@gmail.com> >>> >> I have rebuild the official version(4.3.2) again with the default scripts >> and now i'm getting diffrent core. >> >> This happens only when i enable TLS module in kamailio. So i believe >> there is some corruption causing when TLS module in Kamailio and SSL in >> Mongo Driver is enabled. >> >> I'm running with the offical kamailio script by just enabling mongo >> modules. It happens for NDB as well. >> >> How did you build the mongo driver when you implemented it? May be i can >> try the same. >> >> Please find the scripts used for testing. >> >> Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 >> -M 8'. >> >> Program terminated with signal 11, Segmentation fault. >> >> #0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, >> _aor=0x7fff6de7ef70) at udomain.c:1017 >> >> 1017 lock_get(_d->table[sl].lock); >> >> Missing separate debuginfos, use: debuginfo-install >> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 >> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 >> glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 >> krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 >> libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 >> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 >> >> (gdb) bt >> >> #0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, >> _aor=0x7fff6de7ef70) at udomain.c:1017 >> >> #1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, >> _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831 >> >> #2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, >> _cflags=0, _uri=0x0) at save.c:986 >> >> #3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 >> "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414 >> >> #4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, >> msg=0x7f85a3f054b8) at action.c:1059 >> >> #5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, >> a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548 >> >> #6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, >> a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613 >> >> #7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, >> msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at >> rvalue.c:912 >> >> #8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, >> msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906 >> >> #9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, >> msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914 >> >> #10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, >> msg=0x7f85a3f054b8) at action.c:1029 >> >> #11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, >> a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548 >> >> #12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, >> msg=0x7f85a3f054b8) at action.c:677 >> >> #13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, >> a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548 >> >> #14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, >> msg=0x7f85a3f054b8, c=0x0) at action.c:1634 >> >> #15 0x000000000050a9f4 in receive_msg ( >> >> buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP >> 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: >> 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., >> len=534, rcv_info=0x7fff6de80d00) at receive.c:196 >> >> #16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495 >> >> #17 0x00000000004a7fb3 in main_loop () at main.c:1573 >> >> #18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at >> main.c:2533 >> >> (gdb) quit >> >> [root@localhost / >> >> >> On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla < >> miconda(a)gmail.com>> >>> If you recompiled with different souce code or different flags then the >>> binary is no longer matching the corefile properly. >>> >>> Cheers, >>> Daniel >>> >>> >>> On 01/10/15 17:00, Jijo wrote: >>> >>> Why the backtrace doesn't show the timer function pointer? Am i missing >>> something? >>> >>> Im building the mongo-c-driver as below >>> get the tar file from >>> <https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz> >>> https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-… >>> run the rpm build using the spec file attached. >>> To disable the ssl, i have modified '--enable-ssl=no' in the configure >>> command in the spec file >>> %configure --disable-static --disable-silent-rules >>> --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl >>> --with-libbson=bundled --enable-optimizations >>> >>> In kamailio im using db_mongodb. I'm getting same error for >>> ndb_mongodb as well. >>> >>> Please find the kamailio config scripts. >>> >>> On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla < >>> <miconda@gmail.com>miconda@gmail.com>>> >>>> Hello, >>>> >>>> the bracktrace is not useful. >>>> >>>> How did you compile the mongo-c library before and were there any >>>> special parameters you set to the modules in kamailio config? Are you using >>>> db_mongodb or ndb_mongodb? >>>> >>>> Cheers, >>>> Daniel >>>> >>>> >>>> >>>> On 01/10/15 16:25, Jijo wrote: >>>> >>>> Hi Daniel, >>>> >>>> Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library >>>> then i don't have any crash.. >>>> >>>> Something changed in my environment. Im not able to get the full >>>> backtrace. This is what i'm getting now. >>>> >>>> >>>> >>>> aded symbols for /lib64/liblber-2.4.so.2 >>>> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from >>>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. >>>> done. >>>> Loaded symbols for /lib64/libnss_dns-2.12.so >>>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f >>>> /usr/local/kamailio_proxy/etc/kamail'. >>>> Program terminated with signal 11, Segmentation fault. >>>> #0 0x00007f69ffffffff in ?? () >>>> Missing separate debuginfos, use: debuginfo-install >>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 >>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 >>>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 >>>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 >>>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 >>>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 >>>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 >>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 >>>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 >>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 >>>> (gdb) bt >>>> #0 0x00007f69ffffffff in ?? () >>>> #1 0x00000000005fef8e in compat_old_handler (ti=69660042, >>>> tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 >>>> #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 >>>> #3 0x00000000004a8676 in main_loop () at main.c:1628 >>>> #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at >>>> main.c:2533 >>>> (gdb) >>>> >>>> >>>> >>>> On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla < >>>> <miconda@gmail.com>miconda@gmail.com>>>> >>>>> The first backtrace is related to pike. Can you get from gdb of fisrt >>>>> trace: >>>>> >>>>> frame 2 >>>>> p root->entries[b] >>>>> >>>>> Cheers, >>>>> Daniel >>>>> >>>>> >>>>> On 30/09/15 16:19, Jijo wrote: >>>>> >>>>> I'm observing a segmentation fault when mongodb & tls enabled. It >>>>> doesn't happen when one of the module is disabled. >>>>> OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10. >>>>> >>>>> It happens immediately after i register a subscriber. The location >>>>> table is updated with the new data and in parallel core is generated as >>>>> well. >>>>> >>>>> >>>>> >>>>> Loaded symbols for /lib64/libnss_dns-2.12.so >>>>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f >>>>> /usr/local/kamailio_proxy/etc/kamail'. >>>>> Program terminated with signal 11, Segmentation fault. >>>>> #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, >>>>> old=0, new_v=1) at ../../atomic/atomic_x86.h:233 >>>>> 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) >>>>> Missing separate debuginfos, use: debuginfo-install >>>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 >>>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 >>>>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 >>>>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 >>>>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 >>>>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 >>>>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 >>>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 >>>>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 >>>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 >>>>> (gdb) bt >>>>> #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, >>>>> old=0, new_v=1) at ../../atomic/atomic_x86.h:233 >>>>> #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at >>>>> ../../futexlock.h:99 >>>>> #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at >>>>> ip_tree.c:47 >>>>> #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at >>>>> ip_tree.c:66 >>>>> #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at >>>>> pike_funcs.c:293 >>>>> #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, >>>>> tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 >>>>> #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 >>>>> #7 0x00000000004a8676 in main_loop () at main.c:1628 >>>>> #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at >>>>> main.c:2533 >>>>> (gdb) >>>>> >>>>> >>>>> On Clean up after the core, mongo db does segmentation fault as well >>>>> >>>>> Loaded symbols for /lib64/libnss_files-2.12.so >>>>> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from >>>>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. >>>>> done. >>>>> Loaded symbols for /lib64/libnss_dns-2.12.so >>>>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f >>>>> /usr/local/kamailio_proxy/etc/kamail'. >>>>> Program terminated with signal 11, Segmentation fault. >>>>> #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at >>>>> pthread_mutex_destroy.c:28 >>>>> 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 >>>>> Missing separate debuginfos, use: debuginfo-install >>>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 >>>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 >>>>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 >>>>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 >>>>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 >>>>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 >>>>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 >>>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 >>>>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 >>>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 >>>>> (gdb) bt >>>>> #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at >>>>> pthread_mutex_destroy.c:28 >>>>> #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at >>>>> src/mongoc/mongoc-ssl.c:555 >>>>> #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 >>>>> #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at >>>>> src/mongoc/mongoc-init.c:127 >>>>> #4 0x000000344480cdb3 in pthread_once () at >>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 >>>>> #5 0x00007fd32c8523df in __do_global_dtors_aux () from >>>>> /usr/lib64/libmongoc-1.0.so.0.0.0 >>>>> #6 0x0000000000000000 in ?? () >>>>> (gdb) >>>>> >>>>> >>>>> >>>>> mongo-c-driver version 1.1.10 >>>>> >>>>> version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty >>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, >>>>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, >>>>> PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, >>>>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, >>>>> HAVE_RESOLV_RES >>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, >>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB >>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>>>> id: b5980b -dirty >>>>> compiled on 13:35:24 Sep 28 2015 with gcc 4. >>>>> >>>>> >>>>> _______________________________________________ >>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>>> >>>>> >>>>> -- >>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>>> Book: SIP Routing With Kamailio - http://www.asipto.com >>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >>>>> >>>>> >>>>> _______________________________________________ >>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list >>>>> <sr-users@lists.sip-router.org>sr-users@lists.sip-router.org >>>>> <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users> >>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>>> >>>>> >>>> >>>> -- >>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>> Book: SIP Routing With Kamailio - http://www.asipto.com >>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >>>> >>>> >>> >>> -- >>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>> Book: SIP Routing With Kamailio - http://www.asipto.com >>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >>> >>> >> <kamailio-local.cfg> >> >> <kamailio.cfg> >> >> >> -- >> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> Book: SIP Routing With Kamailio - http://www.asipto.com >> >> > > -- > Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > Book: SIP Routing With Kamailio - http://www.asipto.com > > Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat > >