segmentation fault when mongodb & tls enabled. ( kamailio 4.3.2)
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10. It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb) On Clean up after the core, mongo db does segmentation fault as well Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb) mongo-c-driver version 1.1.10 version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
The first backtrace is related to pike. Can you get from gdb of fisrt trace: frame 2 p root->entries[b] Cheers, Daniel On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so <http://libnss_files-2.12.so> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hi Daniel, Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash.. Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now. aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb) On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello, the bracktrace is not useful. How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb? Cheers, Daniel On 01/10/15 16:25, Jijo wrote:
Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so <http://libnss_files-2.12.so> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Why the backtrace doesn't show the timer function pointer? Am i missing something? Im building the mongo-c-driver as below get the tar file from https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-d... run the rpm build using the spec file attached. To disable the ssl, i have modified '--enable-ssl=no' in the configure command in the spec file %configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb as well. Please find the kamailio config scripts. On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
the bracktrace is not useful.
How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?
Cheers, Daniel
On 01/10/15 16:25, Jijo wrote:
Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla < <miconda@gmail.com>miconda@gmail.com> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
If you recompiled with different souce code or different flags then the binary is no longer matching the corefile properly. Cheers, Daniel On 01/10/15 17:00, Jijo wrote:
Why the backtrace doesn't show the timer function pointer? Am i missing something?
Im building the mongo-c-driver as below get the tar file from https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-d... run the rpm build using the spec file attached. To disable the ssl, i have modified '--enable-ssl=no' in the configure command in the spec file %configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations
In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb as well.
Please find the kamailio config scripts.
On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
the bracktrace is not useful.
How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?
Cheers, Daniel
On 01/10/15 16:25, Jijo wrote:
Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so <http://libnss_files-2.12.so> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
I have rebuild the official version(4.3.2) again with the default scripts and now i'm getting diffrent core. This happens only when i enable TLS module in kamailio. So i believe there is some corruption causing when TLS module in Kamailio and SSL in Mongo Driver is enabled. I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well. How did you build the mongo driver when you implemented it? May be i can try the same. Please find the scripts used for testing. Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'. Program terminated with signal 11, Segmentation fault. #0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017 1017 lock_get(_d->table[sl].lock); Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017 #1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831 #2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986 #3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414 #4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059 #5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548 #6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613 #7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912 #8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906 #9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914 #10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029 #11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548 #12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677 #13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548 #14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634 #15 0x000000000050a9f4 in receive_msg ( buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196 #16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495 #17 0x00000000004a7fb3 in main_loop () at main.c:1573 #18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533 (gdb) quit [root@localhost / On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
If you recompiled with different souce code or different flags then the binary is no longer matching the corefile properly.
Cheers, Daniel
On 01/10/15 17:00, Jijo wrote:
Why the backtrace doesn't show the timer function pointer? Am i missing something?
Im building the mongo-c-driver as below get the tar file from https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-d... run the rpm build using the spec file attached. To disable the ssl, i have modified '--enable-ssl=no' in the configure command in the spec file %configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations
In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb as well.
Please find the kamailio config scripts.
On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla < <miconda@gmail.com>miconda@gmail.com> wrote:
Hello,
the bracktrace is not useful.
How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?
Cheers, Daniel
On 01/10/15 16:25, Jijo wrote:
Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
Hi Daniel Thanks for your help . Did you get a chance to look at this problem. Please let me know how did you build the mongo-c-driver ? Regards, Jijo
On Oct 2, 2015, at 7:21 AM, Jijo <realjijo@gmail.com> wrote:
I have rebuild the official version(4.3.2) again with the default scripts and now i'm getting diffrent core.
This happens only when i enable TLS module in kamailio. So i believe there is some corruption causing when TLS module in Kamailio and SSL in Mongo Driver is enabled.
I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well.
How did you build the mongo driver when you implemented it? May be i can try the same.
Please find the scripts used for testing.
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
1017 lock_get(_d->table[sl].lock);
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
#1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831
#2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986
#3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
#4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059
#5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548
#6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
#7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912
#8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906
#9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914
#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029
#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548
#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677
#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548
#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634
#15 0x000000000050a9f4 in receive_msg (
buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196
#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
#17 0x00000000004a7fb3 in main_loop () at main.c:1573
#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533
(gdb) quit
[root@localhost /
On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla <miconda@gmail.com> wrote: If you recompiled with different souce code or different flags then the binary is no longer matching the corefile properly.
Cheers, Daniel
On 01/10/15 17:00, Jijo wrote: Why the backtrace doesn't show the timer function pointer? Am i missing something?
Im building the mongo-c-driver as below get the tar file from https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-d... run the rpm build using the spec file attached. To disable the ssl, i have modified '--enable-ssl=no' in the configure command in the spec file %configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations
In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb as well.
Please find the kamailio config scripts.
On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote: Hello,
the bracktrace is not useful.
How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?
Cheers, Daniel
On 01/10/15 16:25, Jijo wrote: Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote: > I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. > OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10. > > It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well. > > > > Loaded symbols for /lib64/libnss_dns-2.12.so > Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 > 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) > Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 > (gdb) bt > #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 > #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 > #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 > #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 > #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 > #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 > #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 > #7 0x00000000004a8676 in main_loop () at main.c:1628 > #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 > (gdb) > > > On Clean up after the core, mongo db does segmentation fault as well > > Loaded symbols for /lib64/libnss_files-2.12.so > Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. > done. > Loaded symbols for /lib64/libnss_dns-2.12.so > Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. > Program terminated with signal 11, Segmentation fault. > #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 > 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 > Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 > (gdb) bt > #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 > #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 > #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 > #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 > #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 > #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 > #6 0x0000000000000000 in ?? () > (gdb) > > > > mongo-c-driver version 1.1.10 > > version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty > flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES > ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > id: b5980b -dirty > compiled on 13:35:24 Sep 28 2015 with gcc 4. > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list > sr-users@lists.sip-router.org > http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
<kamailio-local.cfg> <kamailio.cfg>
Hello, unfortunately not yet -- this requires compiling the libs and currently I am traveling to USA to attend few VoIP conferences, not having a good devel environment with me, access to internet is limited, so connecting to a server is not easy option as well. Can you open an issue on github tracker just not to forget about it? Cheers, Daniel On 06/10/15 15:49, realjijo@gmail.com wrote:
Hi Daniel
Thanks for your help .
Did you get a chance to look at this problem. Please let me know how did you build the mongo-c-driver ?
Regards, Jijo
On Oct 2, 2015, at 7:21 AM, Jijo <realjijo@gmail.com <mailto:realjijo@gmail.com>> wrote:
I have rebuild the official version(4.3.2) again with the default scripts and now i'm getting diffrent core.
This happens only when i enable TLS module in kamailio. So i believe there is some corruption causing when TLS module in Kamailio and SSL in Mongo Driver is enabled.
I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well.
How did you build the mongo driver when you implemented it? May be i can try the same.
Please find the scripts used for testing.
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
1017lock_get(_d->table[sl].lock);
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
#1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831
#2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986
#3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
#4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059
#5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548
#6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
#7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912
#8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906
#9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914
#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029
#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548
#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677
#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548
#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634
#15 0x000000000050a9f4 in receive_msg (
buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196
#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
#17 0x00000000004a7fb3 in main_loop () at main.c:1573
#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533
(gdb) quit
[root@localhost /
On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
If you recompiled with different souce code or different flags then the binary is no longer matching the corefile properly.
Cheers, Daniel
On 01/10/15 17:00, Jijo wrote:
Why the backtrace doesn't show the timer function pointer? Am i missing something?
Im building the mongo-c-driver as below get the tar file from https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-d... run the rpm build using the spec file attached. To disable the ssl, i have modified '--enable-ssl=no' in the configure command in the spec file %configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations
In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb as well.
Please find the kamailio config scripts.
On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
the bracktrace is not useful.
How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?
Cheers, Daniel
On 01/10/15 16:25, Jijo wrote:
Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so <http://libnss_files-2.12.so> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
<kamailio-local.cfg> <kamailio.cfg>
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
Thanks, I have opened the issue on github tracker. regards jijo On Tue, Oct 6, 2015 at 9:53 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,
unfortunately not yet -- this requires compiling the libs and currently I am traveling to USA to attend few VoIP conferences, not having a good devel environment with me, access to internet is limited, so connecting to a server is not easy option as well. Can you open an issue on github tracker just not to forget about it?
Cheers, Daniel
On 06/10/15 15:49, realjijo@gmail.com wrote:
Hi Daniel
Thanks for your help .
Did you get a chance to look at this problem. Please let me know how did you build the mongo-c-driver ?
Regards, Jijo
On Oct 2, 2015, at 7:21 AM, Jijo <realjijo@gmail.com> wrote:
I have rebuild the official version(4.3.2) again with the default scripts and now i'm getting diffrent core.
This happens only when i enable TLS module in kamailio. So i believe there is some corruption causing when TLS module in Kamailio and SSL in Mongo Driver is enabled.
I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well.
How did you build the mongo driver when you implemented it? May be i can try the same.
Please find the scripts used for testing.
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
1017 lock_get(_d->table[sl].lock);
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
#1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831
#2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986
#3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
#4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059
#5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548
#6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
#7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912
#8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906
#9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914
#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029
#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548
#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677
#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548
#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634
#15 0x000000000050a9f4 in receive_msg (
buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196
#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
#17 0x00000000004a7fb3 in main_loop () at main.c:1573
#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533
(gdb) quit
[root@localhost /
On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla < <miconda@gmail.com>miconda@gmail.com> wrote:
If you recompiled with different souce code or different flags then the binary is no longer matching the corefile properly.
Cheers, Daniel
On 01/10/15 17:00, Jijo wrote:
Why the backtrace doesn't show the timer function pointer? Am i missing something?
Im building the mongo-c-driver as below get the tar file from <https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz> https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-d... run the rpm build using the spec file attached. To disable the ssl, i have modified '--enable-ssl=no' in the configure command in the spec file %configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations
In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb as well.
Please find the kamailio config scripts.
On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
the bracktrace is not useful.
How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?
Cheers, Daniel
On 01/10/15 16:25, Jijo wrote:
Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla < <miconda@gmail.com>miconda@gmail.com> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
<kamailio-local.cfg>
<kamailio.cfg>
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
Hello, I tried to reproduce here and all seems fine -- I enabled tls and registered a user using mongodb as backend. Can you give the parameters you set for usrloc module? Cheers, Daniel On 06/10/15 19:03, Jijo wrote:
Thanks, I have opened the issue on github tracker.
regards jijo
On Tue, Oct 6, 2015 at 9:53 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
unfortunately not yet -- this requires compiling the libs and currently I am traveling to USA to attend few VoIP conferences, not having a good devel environment with me, access to internet is limited, so connecting to a server is not easy option as well. Can you open an issue on github tracker just not to forget about it?
Cheers, Daniel
On 06/10/15 15:49, realjijo@gmail.com <mailto:realjijo@gmail.com> wrote:
Hi Daniel
Thanks for your help .
Did you get a chance to look at this problem. Please let me know how did you build the mongo-c-driver ?
Regards, Jijo
On Oct 2, 2015, at 7:21 AM, Jijo <realjijo@gmail.com <mailto:realjijo@gmail.com>> wrote:
I have rebuild the official version(4.3.2) again with the default scripts and now i'm getting diffrent core.
This happens only when i enable TLS module in kamailio. So i believe there is some corruption causing when TLS module in Kamailio and SSL in Mongo Driver is enabled.
I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well.
How did you build the mongo driver when you implemented it? May be i can try the same.
Please find the scripts used for testing.
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
1017lock_get(_d->table[sl].lock);
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
#1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831
#2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986
#3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
#4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059
#5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548
#6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
#7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912
#8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906
#9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914
#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029
#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548
#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677
#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548
#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634
#15 0x000000000050a9f4 in receive_msg (
buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196
#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
#17 0x00000000004a7fb3 in main_loop () at main.c:1573
#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533
(gdb) quit
[root@localhost /
On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
If you recompiled with different souce code or different flags then the binary is no longer matching the corefile properly.
Cheers, Daniel
On 01/10/15 17:00, Jijo wrote:
Why the backtrace doesn't show the timer function pointer? Am i missing something?
Im building the mongo-c-driver as below get the tar file from https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-d... run the rpm build using the spec file attached. To disable the ssl, i have modified '--enable-ssl=no' in the configure command in the spec file %configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations
In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb as well.
Please find the kamailio config scripts.
On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
the bracktrace is not useful.
How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?
Cheers, Daniel
On 01/10/15 16:25, Jijo wrote:
Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so <http://libnss_files-2.12.so> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so <http://libnss_dns-2.12.so> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
<kamailio-local.cfg> <kamailio.cfg>
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
Hi, I tried with mongo-c-driver 1.2.1 and crash occurring with that library as well. So i took the kamailio basic script and able to reproduce it. So the issue is slightly diffrent now. Core occurs when i enable pike(ANTIFLOOD). Another thing which i noticed is, core doesn't occur If i disable TLS and enable pike. Please find the attached script files for more details about my configuration. Here is the backtrace when i enable TLS and ANTIFLOOD done. Loaded symbols for /usr/local/kamailio_proxy/lib64/kamailio/modules/ndb_mongodb.so Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00000000005fef6f in compat_old_handler (ti=2030104642, tl=0x7f9b68f7e130, data=0x7f9bffffffff) at timer.c:996 996 t->timer_f(TICKS_TO_S(*ticks), t->t_param); Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-20.el6_7.x86_64 glibc-2.12-1.166.el6_7.3.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00000000005fef6f in compat_old_handler (ti=2030104642, tl=0x7f9b68f7e130, data=0x7f9bffffffff) at timer.c:996 #1 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #2 0x00000000004a8676 in main_loop () at main.c:1628 #3 0x00000000004ae38e in main (argc=15, argv=0x7ffc9cc0ec78) at main.c:2533 (gdb) quit On Tue, Nov 3, 2015 at 10:04 AM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
I tried to reproduce here and all seems fine -- I enabled tls and registered a user using mongodb as backend.
Can you give the parameters you set for usrloc module?
Cheers, Daniel
On 06/10/15 19:03, Jijo wrote:
Thanks, I have opened the issue on github tracker.
regards jijo
On Tue, Oct 6, 2015 at 9:53 AM, Daniel-Constantin Mierla < <miconda@gmail.com>miconda@gmail.com> wrote:
Hello,
unfortunately not yet -- this requires compiling the libs and currently I am traveling to USA to attend few VoIP conferences, not having a good devel environment with me, access to internet is limited, so connecting to a server is not easy option as well. Can you open an issue on github tracker just not to forget about it?
Cheers, Daniel
On 06/10/15 15:49, realjijo@gmail.com wrote:
Hi Daniel
Thanks for your help .
Did you get a chance to look at this problem. Please let me know how did you build the mongo-c-driver ?
Regards, Jijo
On Oct 2, 2015, at 7:21 AM, Jijo < <realjijo@gmail.com>realjijo@gmail.com> wrote:
I have rebuild the official version(4.3.2) again with the default scripts and now i'm getting diffrent core.
This happens only when i enable TLS module in kamailio. So i believe there is some corruption causing when TLS module in Kamailio and SSL in Mongo Driver is enabled.
I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well.
How did you build the mongo driver when you implemented it? May be i can try the same.
Please find the scripts used for testing.
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
1017 lock_get(_d->table[sl].lock);
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
#1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831
#2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986
#3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
#4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059
#5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548
#6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
#7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912
#8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906
#9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914
#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029
#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548
#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677
#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548
#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634
#15 0x000000000050a9f4 in receive_msg (
buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196
#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
#17 0x00000000004a7fb3 in main_loop () at main.c:1573
#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533
(gdb) quit
[root@localhost /
On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:
If you recompiled with different souce code or different flags then the binary is no longer matching the corefile properly.
Cheers, Daniel
On 01/10/15 17:00, Jijo wrote:
Why the backtrace doesn't show the timer function pointer? Am i missing something?
Im building the mongo-c-driver as below get the tar file from <https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz> https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-d... run the rpm build using the spec file attached. To disable the ssl, i have modified '--enable-ssl=no' in the configure command in the spec file %configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations
In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb as well.
Please find the kamailio config scripts.
On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla < <miconda@gmail.com>miconda@gmail.com> wrote:
Hello,
the bracktrace is not useful.
How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?
Cheers, Daniel
On 01/10/15 16:25, Jijo wrote:
Hi Daniel,
Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..
Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.
aded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f69ffffffff in ?? () Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f69ffffffff in ?? () #1 0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996 #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #3 0x00000000004a8676 in main_loop () at main.c:1628 #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533 (gdb)
On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla < <miconda@gmail.com>miconda@gmail.com> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:
frame 2 p root->entries[b]
Cheers, Daniel
On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.
Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233 #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99 #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47 #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66 #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293 #5 0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996 #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130 #7 0x00000000004a8676 in main_loop () at main.c:1628 #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533 (gdb)
On Clean up after the core, mongo db does segmentation fault as well
Loaded symbols for /lib64/libnss_files-2.12.so Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done. done. Loaded symbols for /lib64/libnss_dns-2.12.so Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28 #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555 #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106 #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127 #4 0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104 #5 0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0 #6 0x0000000000000000 in ?? () (gdb)
mongo-c-driver version 1.1.10
version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: b5980b -dirty compiled on 13:35:24 Sep 28 2015 with gcc 4.
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list <sr-users@lists.sip-router.org>sr-users@lists.sip-router.org <http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
<kamailio-local.cfg>
<kamailio.cfg>
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
participants (3)
-
Daniel-Constantin Mierla -
Jijo -
realjijo＠gmail.com