Thanks, I have opened the issue on github tracker.

regards
jijo

On Tue, Oct 6, 2015 at 9:53 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,

unfortunately not yet -- this requires compiling the libs and currently I am traveling to USA to attend few VoIP conferences, not having a good devel environment with me, access to internet is limited, so connecting to a server is not easy option as well. Can you open an issue on github tracker just not to forget about it?

Cheers,
Daniel


On 06/10/15 15:49, realjijo@gmail.com wrote:
Hi Daniel 

Thanks for your help .

Did you get a chance to look at this problem. 
Please let me know how did you build  the mongo-c-driver ?


Regards,
Jijo

On Oct 2, 2015, at 7:21 AM, Jijo <realjijo@gmail.com> wrote:

I have rebuild the official version(4.3.2) again with the default scripts and now i'm getting diffrent core. 

This happens only when i enable TLS module in kamailio. So i believe there is some corruption causing when TLS module in Kamailio  and SSL in Mongo Driver  is enabled. 

I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well.

How did you build the mongo driver when you implemented it? May be i can try the same.

Please find the scripts used for testing.

Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'.

Program terminated with signal 11, Segmentation fault.

#0  0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017

1017 lock_get(_d->table[sl].lock);

Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64

(gdb) bt

#0  0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017

#1  0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831

#2  0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986

#3  0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414

#4  0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059

#5  0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548

#6  0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613

#7  0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912

#8  0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906

#9  0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914

#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029

#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548

#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677

#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548

#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634

#15 0x000000000050a9f4 in receive_msg (

    buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196

#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495

#17 0x00000000004a7fb3 in main_loop () at main.c:1573

#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533

(gdb) quit

[root@localhost /



On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
If you recompiled with different souce code or different flags then the binary is no longer matching the corefile properly.

Cheers,
Daniel


On 01/10/15 17:00, Jijo wrote:
Why the backtrace doesn't show the timer function pointer? Am i missing something?

Im building the mongo-c-driver as below
run the rpm build using the spec file attached. 
To disable the ssl, i have modified  '--enable-ssl=no' in the configure  command in the spec file
%configure --disable-static --disable-silent-rules --enable-debug-symbols --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled --enable-optimizations

In kamailio im using db_mongodb. I'm getting same error  for ndb_mongodb as well.

Please find the kamailio config scripts.

On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,

the bracktrace is not useful.

How did you compile the mongo-c library before and were there any special parameters you set to the modules in kamailio config? Are you using db_mongodb or ndb_mongodb?

Cheers,
Daniel



On 01/10/15 16:25, Jijo wrote:
Hi Daniel,

Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then i don't have any crash..

Something changed in my environment. Im not able to get the full backtrace. This is what i'm getting now.



aded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libnss_dns-2.12.so
Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f69ffffffff in ?? ()
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007f69ffffffff in ?? ()
#1  0x00000000005fef8e in compat_old_handler (ti=69660042, tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996
#2  0x00000000005ff94c in slow_timer_main () at timer.c:1130
#3  0x00000000004a8676 in main_loop () at main.c:1628
#4  0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at main.c:2533
(gdb) 



On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
The first backtrace is related to pike. Can you get from gdb of fisrt trace:

frame 2
p root->entries[b]

Cheers,
Daniel


On 30/09/15 16:19, Jijo wrote:
I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled.
OS: centos 6.7   kamailio 4.3.2  mongo-c-driver version 1.1.10.

It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well.



Loaded symbols for /lib64/libnss_dns-2.12.so
Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233
233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int)
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030, old=0, new_v=1) at ../../atomic/atomic_x86.h:233
#1  0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at ../../futexlock.h:99
#2  0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at ip_tree.c:47
#3  0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66
#4  0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at pike_funcs.c:293
#5  0x00000000005fef8e in compat_old_handler (ti=1971296512, tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996
#6  0x00000000005ff94c in slow_timer_main () at timer.c:1130
#7  0x00000000004a8676 in main_loop () at main.c:1628
#8  0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at main.c:2533
(gdb) 


On Clean up after the core, mongo db does segmentation fault as well

Loaded symbols for /lib64/libnss_files-2.12.so
Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
done.
Loaded symbols for /lib64/libnss_dns-2.12.so
Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'.
Program terminated with signal 11, Segmentation fault.
#0  __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28
28  if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at pthread_mutex_destroy.c:28
#1  0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at src/mongoc/mongoc-ssl.c:555
#2  _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106
#3  0x00007fd32c8606a9 in _mongoc_do_cleanup () at src/mongoc/mongoc-init.c:127
#4  0x000000344480cdb3 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104
#5  0x00007fd32c8523df in __do_global_dtors_aux () from /usr/lib64/libmongoc-1.0.so.0.0.0
#6  0x0000000000000000 in ?? ()
(gdb) 



mongo-c-driver version 1.1.10

version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: b5980b -dirty
compiled on 13:35:24 Sep 28 2015 with gcc 4.


_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat

_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users



-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat


-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat

<kamailio-local.cfg>
<kamailio.cfg>

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com