Re: [SR-Users] segmentation fault when mongodb & tls enabled. ( kamailio 4.3.2)

Hi Daniel Thanks for your help . Did you get a chance to look at this problem. Please let me know how did you build the mongo-c-driver ? Regards, Jijo On Oct 2, 2015, at 7:21 AM, Jijo &lt;realjijo(a)gmail.com <mailto:realjijo@gmail.com>> wrote: > I have rebuild the official version(4.3.2) again with the > default scripts and now i'm getting diffrent core. > > This happens only when i enable TLS module in kamailio. So i > believe there is some corruption causing when TLS module in > Kamailio and SSL in Mongo Driver is enabled. > > I'm running with the offical kamailio script by just enabling > mongo modules. It happens for NDB as well. > > How did you build the mongo driver when you implemented it? May > be i can try the same. > > Please find the scripts used for testing. > > Core was generated by `/usr/sbin/kamailio -P > /var/run/kamailio.pid -m 64 -M 8'. > > Program terminated with signal 11, Segmentation fault. > > #0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, > _aor=0x7fff6de7ef70) at udomain.c:1017 > > 1017lock_get(_d->table[sl].lock); > > Missing separate debuginfos, use: debuginfo-install > cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 > cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 > db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 > keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 > libcom_err-1.41.12-22.el6.x86_64 > libselinux-2.0.94-5.8.el6.x86_64 > nss-softokn-freebl-3.14.3-22.el6_6.x86_64 > openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 > > (gdb) bt > > #0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, > _aor=0x7fff6de7ef70) at udomain.c:1017 > > #1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, > _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at > save.c:831 > > #2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, > _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986 > > #3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, > _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414 > > #4 0x000000000041decb in do_action (h=0x7fff6de7f720, > a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059 > > #5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, > a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548 > > #6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, > a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613 > > #7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, > msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, > cache=0x0) at rvalue.c:912 > > #8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, > msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at > rvalue.c:1906 > > #9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, > msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at > rvalue.c:1914 > > #10 0x000000000041d927 in do_action (h=0x7fff6de80a10, > a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029 > > #11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, > a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548 > > #12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, > a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677 > > #13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, > a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548 > > #14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, > msg=0x7f85a3f054b8, c=0x0) at action.c:1634 > > #15 0x000000000050a9f4 in receive_msg ( > > buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: > SIP/2.0/UDP > 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: > 70\r\nContact: > <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, > rcv_info=0x7fff6de80d00) at receive.c:196 > > #16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495 > > #17 0x00000000004a7fb3 in main_loop () at main.c:1573 > > #18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at > main.c:2533 > > (gdb) quit > > [root@localhost / > > > > On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla > &lt;miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote: > > If you recompiled with different souce code or different > flags then the binary is no longer matching the corefile > properly. > > Cheers, > Daniel > > > On 01/10/15 17:00, Jijo wrote: >> Why the backtrace doesn't show the timer function pointer? >> Am i missing something? >> >> Im building the mongo-c-driver as below >> get the tar file >> from https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-… >> run the rpm build using the spec file attached. >> To disable the ssl, i have modified '--enable-ssl=no' in >> the configure command in the spec file >> %configure --disable-static --disable-silent-rules >> --enable-debug-symbols --enable-man-pages --enable-ssl=yes >> --enable-sasl --with-libbson=bundled --enable-optimizations >> >> In kamailio im using db_mongodb. I'm getting same error >> for ndb_mongodb as well. >> >> Please find the kamailio config scripts. >> >> On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla >> &lt;miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote: >> >> Hello, >> >> the bracktrace is not useful. >> >> How did you compile the mongo-c library before and were >> there any special parameters you set to the modules in >> kamailio config? Are you using db_mongodb or ndb_mongodb? >> >> Cheers, >> Daniel >> >> >> >> On 01/10/15 16:25, Jijo wrote: >>> Hi Daniel, >>> >>> Thanks.. As a sidenote, If i disable ssl from >>> mongo-c-driver library then i don't have any crash.. >>> >>> Something changed in my environment. Im not able to >>> get the full backtrace. This is what i'm getting now. >>> >>> >>> >>> aded symbols for /lib64/liblber-2.4.so.2 >>> Reading symbols from >>> /lib64/libnss_dns-2.12.so...Reading symbols from >>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. >>> done. >>> Loaded symbols for /lib64/libnss_dns-2.12.so >>> <http://libnss_dns-2.12.so> >>> Core was generated by >>> `/usr/local/kamailio_proxy/sbin/kamailio -f >>> /usr/local/kamailio_proxy/etc/kamail'. >>> Program terminated with signal 11, Segmentation fault. >>> #0 0x00007f69ffffffff in ?? () >>> Missing separate debuginfos, use: debuginfo-install >>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 >>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 >>> db4-4.7.25-19.el6_6.x86_64 >>> keyutils-libs-1.4-5.el6.x86_64 >>> krb5-libs-1.10.3-42.el6.x86_64 >>> libcom_err-1.41.12-22.el6.x86_64 >>> libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 >>> libselinux-2.0.94-5.8.el6.x86_64 >>> libssh2-1.4.2-1.el6_6.1.x86_64 >>> libunistring-0.9.3-5.el6.x86_64 >>> libxml2-2.7.6-20.el6.x86_64 >>> mysql-libs-5.1.73-5.el6_6.x86_64 >>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 >>> nss-util-3.18.0-1.el6_6.x86_64 >>> openldap-2.4.40-5.el6.x86_64 >>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 >>> (gdb) bt >>> #0 0x00007f69ffffffff in ?? () >>> #1 0x00000000005fef8e in compat_old_handler >>> (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) >>> at timer.c:996 >>> #2 0x00000000005ff94c in slow_timer_main () at >>> timer.c:1130 >>> #3 0x00000000004a8676 in main_loop () at main.c:1628 >>> #4 0x00000000004ae38e in main (argc=15, >>> argv=0x7ffdd0dc8848) at main.c:2533 >>> (gdb) >>> >>> >>> >>> On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin >>> Mierla &lt;miconda(a)gmail.com <mailto:miconda@gmail.com>> >>> wrote: >>> >>> The first backtrace is related to pike. Can you >>> get from gdb of fisrt trace: >>> >>> frame 2 >>> p root->entries[b] >>> >>> Cheers, >>> Daniel >>> >>> >>> On 30/09/15 16:19, Jijo wrote: >>>> I'm observing a segmentation fault when mongodb & >>>> tls enabled. It doesn't happen when one of the >>>> module is disabled. >>>> OS: centos 6.7 kamailio 4.3.2 mongo-c-driver >>>> version 1.1.10. >>>> >>>> It happens immediately after i register a >>>> subscriber. The location table is updated with >>>> the new data and in parallel core is generated as >>>> well. >>>> >>>> >>>> >>>> Loaded symbols for /lib64/libnss_dns-2.12.so >>>> <http://libnss_dns-2.12.so> >>>> Core was generated by >>>> `/usr/local/kamailio_proxy/sbin/kamailio -f >>>> /usr/local/kamailio_proxy/etc/kamail'. >>>> Program terminated with signal 11, Segmentation >>>> fault. >>>> #0 0x00007fd3adf65e69 in atomic_cmpxchg_int >>>> (var=0x56444e3055445030, old=0, new_v=1) at >>>> ../../atomic/atomic_x86.h:233 >>>> 233ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, >>>> %1", int , int) >>>> Missing separate debuginfos, use: >>>> debuginfo-install >>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 >>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 >>>> db4-4.7.25-19.el6_6.x86_64 >>>> keyutils-libs-1.4-5.el6.x86_64 >>>> krb5-libs-1.10.3-42.el6.x86_64 >>>> libcom_err-1.41.12-22.el6.x86_64 >>>> libcurl-7.19.7-46.el6.x86_64 >>>> libidn-1.18-2.el6.x86_64 >>>> libselinux-2.0.94-5.8.el6.x86_64 >>>> libssh2-1.4.2-1.el6_6.1.x86_64 >>>> libunistring-0.9.3-5.el6.x86_64 >>>> libxml2-2.7.6-20.el6.x86_64 >>>> mysql-libs-5.1.73-5.el6_6.x86_64 >>>> nspr-4.10.8-1.el6_6.x86_64 >>>> nss-3.18.0-5.3.el6_6.x86_64 >>>> nss-util-3.18.0-1.el6_6.x86_64 >>>> openldap-2.4.40-5.el6.x86_64 >>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 >>>> (gdb) bt >>>> #0 0x00007fd3adf65e69 in atomic_cmpxchg_int >>>> (var=0x56444e3055445030, old=0, new_v=1) at >>>> ../../atomic/atomic_x86.h:233 >>>> #1 0x00007fd3adf65eb6 in futex_get >>>> (lock=0x56444e3055445030) at ../../futexlock.h:99 >>>> #2 0x00007fd3adf66468 in prv_lock_tree_branch >>>> (b=192 '\300') at ip_tree.c:47 >>>> #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 >>>> '\300') at ip_tree.c:66 >>>> #4 0x00007fd3adf6dda6 in swap_routine >>>> (ticks=123206032, param=0x0) at pike_funcs.c:293 >>>> #5 0x00000000005fef8e in compat_old_handler >>>> (ti=1971296512, tl=0x7fd32ccea430, >>>> data=0x7fd32ccea430) at timer.c:996 >>>> #6 0x00000000005ff94c in slow_timer_main () at >>>> timer.c:1130 >>>> #7 0x00000000004a8676 in main_loop () at main.c:1628 >>>> #8 0x00000000004ae38e in main (argc=11, >>>> argv=0x7fff1d97ec28) at main.c:2533 >>>> (gdb) >>>> >>>> >>>> On Clean up after the core, mongo db does >>>> segmentation fault as well >>>> >>>> Loaded symbols for /lib64/libnss_files-2.12.so >>>> <http://libnss_files-2.12.so> >>>> Reading symbols from >>>> /lib64/libnss_dns-2.12.so...Reading symbols from >>>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. >>>> done. >>>> Loaded symbols for /lib64/libnss_dns-2.12.so >>>> <http://libnss_dns-2.12.so> >>>> Core was generated by >>>> `/usr/local/kamailio_proxy/sbin/kamailio -f >>>> /usr/local/kamailio_proxy/etc/kamail'. >>>> Program terminated with signal 11, Segmentation >>>> fault. >>>> #0 __pthread_mutex_destroy >>>> (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 >>>> 28 if ((mutex->__data.__kind & >>>> PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 >>>> Missing separate debuginfos, use: >>>> debuginfo-install >>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 >>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 >>>> db4-4.7.25-19.el6_6.x86_64 >>>> keyutils-libs-1.4-5.el6.x86_64 >>>> krb5-libs-1.10.3-42.el6.x86_64 >>>> libcom_err-1.41.12-22.el6.x86_64 >>>> libcurl-7.19.7-46.el6.x86_64 >>>> libidn-1.18-2.el6.x86_64 >>>> libselinux-2.0.94-5.8.el6.x86_64 >>>> libssh2-1.4.2-1.el6_6.1.x86_64 >>>> libunistring-0.9.3-5.el6.x86_64 >>>> libxml2-2.7.6-20.el6.x86_64 >>>> mysql-libs-5.1.73-5.el6_6.x86_64 >>>> nspr-4.10.8-1.el6_6.x86_64 >>>> nss-3.18.0-5.3.el6_6.x86_64 >>>> nss-util-3.18.0-1.el6_6.x86_64 >>>> openldap-2.4.40-5.el6.x86_64 >>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 >>>> (gdb) bt >>>> #0 __pthread_mutex_destroy >>>> (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 >>>> #1 0x00007fd32c86f9e6 in >>>> _mongoc_ssl_thread_cleanup () at >>>> src/mongoc/mongoc-ssl.c:555 >>>> #2 _mongoc_ssl_cleanup () at >>>> src/mongoc/mongoc-ssl.c:106 >>>> #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () >>>> at src/mongoc/mongoc-init.c:127 >>>> #4 0x000000344480cdb3 in pthread_once () at >>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 >>>> #5 0x00007fd32c8523df in __do_global_dtors_aux >>>> () from /usr/lib64/libmongoc-1.0.so.0.0.0 >>>> #6 0x0000000000000000 in ?? () >>>> (gdb) >>>> >>>> >>>> >>>> mongo-c-driver version 1.1.10 >>>> >>>> version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty >>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, >>>> TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, >>>> USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, >>>> PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, >>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, >>>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, >>>> HAVE_RESOLV_RES >>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE >>>> 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, >>>> BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB >>>> poll method support: poll, epoll_lt, epoll_et, >>>> sigio_rt, select. >>>> id: b5980b -dirty >>>> compiled on 13:35:24 Sep 28 2015 with gcc 4. >>>> >>>> >>>> _______________________________________________ >>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list >>>> sr-users(a)lists.sip-router.org >>>> <mailto:sr-users@lists.sip-router.org> >>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>> >>> -- >>> Daniel-Constantin Mierla >>> http://twitter.com/#!/miconda >>> <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda >>> Book: SIP Routing With Kamailio - http://www.asipto.com >>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >>> >>> >>> _______________________________________________ >>> SIP Express Router (SER) and Kamailio (OpenSER) - >>> sr-users mailing list >>> sr-users(a)lists.sip-router.org >>> <mailto:sr-users@lists.sip-router.org> >>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>> >>> >> >> -- >> Daniel-Constantin Mierla >> http://twitter.com/#!/miconda >> <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda >> Book: SIP Routing With Kamailio - http://www.asipto.com >> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat >> >> > > -- > Daniel-Constantin Mierla > http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda > Book: SIP Routing With Kamailio - http://www.asipto.com > Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat > > > <kamailio-local.cfg> > <kamailio.cfg>