[SR-Users] segmentation fault when mongodb & tls enabled. ( kamailio 4.3.2)

Tue Oct 6 19:03:24 CEST 2015

Thanks, I have opened the issue on github tracker.

regards
jijo

On Tue, Oct 6, 2015 at 9:53 AM, Daniel-Constantin Mierla <miconda at gmail.com>
wrote:

> Hello,
>
> unfortunately not yet -- this requires compiling the libs and currently I
> am traveling to USA to attend few VoIP conferences, not having a good devel
> environment with me, access to internet is limited, so connecting to a
> server is not easy option as well. Can you open an issue on github tracker
> just not to forget about it?
>
> Cheers,
> Daniel
>
>
> On 06/10/15 15:49, realjijo at gmail.com wrote:
>
> Hi Daniel
>
> Thanks for your help .
>
> Did you get a chance to look at this problem.
> Please let me know how did you build  the mongo-c-driver ?
>
>
> Regards,
> Jijo
>
> On Oct 2, 2015, at 7:21 AM, Jijo <realjijo at gmail.com> wrote:
>
> I have rebuild the official version(4.3.2) again with the default scripts
> and now i'm getting diffrent core.
>
> This happens only when i enable TLS module in kamailio. So i believe there
> is some corruption causing when TLS module in Kamailio  and SSL in Mongo
> Driver  is enabled.
>
> I'm running with the offical kamailio script by just enabling mongo
> modules. It happens for NDB as well.
>
> How did you build the mongo driver when you implemented it? May be i can
> try the same.
>
> Please find the scripts used for testing.
>
> Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64
> -M 8'.
>
> Program terminated with signal 11, Segmentation fault.
>
> #0  0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08,
> _aor=0x7fff6de7ef70) at udomain.c:1017
>
> 1017 lock_get(_d->table[sl].lock);
>
> Missing separate debuginfos, use: debuginfo-install
> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
> glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64
> krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64
> libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64
> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>
> (gdb) bt
>
> #0  0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08,
> _aor=0x7fff6de7ef70) at udomain.c:1017
>
> #1  0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8,
> _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831
>
> #2  0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08,
> _cflags=0, _uri=0x0) at save.c:986
>
> #3  0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08
> "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
>
> #4  0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830,
> msg=0x7f85a3f054b8) at action.c:1059
>
> #5  0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830,
> msg=0x7f85a3f054b8) at action.c:1548
>
> #6  0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10,
> a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
>
> #7  0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10,
> msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at
> rvalue.c:912
>
> #8  0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10,
> msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906
>
> #9  0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10,
> msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914
>
> #10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270,
> msg=0x7f85a3f054b8) at action.c:1029
>
> #11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990,
> msg=0x7f85a3f054b8) at action.c:1548
>
> #12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98,
> msg=0x7f85a3f054b8) at action.c:677
>
> #13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08,
> msg=0x7f85a3f054b8) at action.c:1548
>
> #14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08,
> msg=0x7f85a3f054b8, c=0x0) at action.c:1634
>
> #15 0x000000000050a9f4 in receive_msg (
>
>     buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP
> 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards:
> 70\r\nContact: <sip:usera at 192.168.2.119:60887;rinstance=d9f6274d7"...,
> len=534, rcv_info=0x7fff6de80d00) at receive.c:196
>
> #16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
>
> #17 0x00000000004a7fb3 in main_loop () at main.c:1573
>
> #18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533
>
> (gdb) quit
>
> [root at localhost /
>
>
> On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla <
> <miconda at gmail.com>miconda at gmail.com> wrote:
>
>> If you recompiled with different souce code or different flags then the
>> binary is no longer matching the corefile properly.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 01/10/15 17:00, Jijo wrote:
>>
>> Why the backtrace doesn't show the timer function pointer? Am i missing
>> something?
>>
>> Im building the mongo-c-driver as below
>> get the tar file from
>> <https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz>
>> https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz
>> run the rpm build using the spec file attached.
>> To disable the ssl, i have modified  '--enable-ssl=no' in the configure
>>  command in the spec file
>> %configure --disable-static --disable-silent-rules --enable-debug-symbols
>> --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled
>> --enable-optimizations
>>
>> In kamailio im using db_mongodb. I'm getting same error  for ndb_mongodb
>> as well.
>>
>> Please find the kamailio config scripts.
>>
>> On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> the bracktrace is not useful.
>>>
>>> How did you compile the mongo-c library before and were there any
>>> special parameters you set to the modules in kamailio config? Are you using
>>> db_mongodb or ndb_mongodb?
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>>
>>> On 01/10/15 16:25, Jijo wrote:
>>>
>>> Hi Daniel,
>>>
>>> Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library
>>> then i don't have any crash..
>>>
>>> Something changed in my environment. Im not able to get the full
>>> backtrace. This is what i'm getting now.
>>>
>>>
>>>
>>> aded symbols for /lib64/liblber-2.4.so.2
>>> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
>>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
>>> done.
>>> Loaded symbols for /lib64/libnss_dns-2.12.so
>>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f
>>> /usr/local/kamailio_proxy/etc/kamail'.
>>> Program terminated with signal 11, Segmentation fault.
>>> #0  0x00007f69ffffffff in ?? ()
>>> Missing separate debuginfos, use: debuginfo-install
>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
>>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64
>>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64
>>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
>>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64
>>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64
>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64
>>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64
>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>>> (gdb) bt
>>> #0  0x00007f69ffffffff in ?? ()
>>> #1  0x00000000005fef8e in compat_old_handler (ti=69660042,
>>> tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996
>>> #2  0x00000000005ff94c in slow_timer_main () at timer.c:1130
>>> #3  0x00000000004a8676 in main_loop () at main.c:1628
>>> #4  0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at
>>> main.c:2533
>>> (gdb)
>>>
>>>
>>>
>>> On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <
>>> <miconda at gmail.com>miconda at gmail.com> wrote:
>>>
>>>> The first backtrace is related to pike. Can you get from gdb of fisrt
>>>> trace:
>>>>
>>>> frame 2
>>>> p root->entries[b]
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 30/09/15 16:19, Jijo wrote:
>>>>
>>>> I'm observing a segmentation fault when mongodb & tls enabled. It
>>>> doesn't happen when one of the module is disabled.
>>>> OS: centos 6.7   kamailio 4.3.2  mongo-c-driver version 1.1.10.
>>>>
>>>> It happens immediately after i register a subscriber. The location
>>>> table is updated with the new data and in parallel core is generated as
>>>> well.
>>>>
>>>>
>>>>
>>>> Loaded symbols for /lib64/libnss_dns-2.12.so
>>>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f
>>>> /usr/local/kamailio_proxy/etc/kamail'.
>>>> Program terminated with signal 11, Segmentation fault.
>>>> #0  0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030,
>>>> old=0, new_v=1) at ../../atomic/atomic_x86.h:233
>>>> 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int)
>>>> Missing separate debuginfos, use: debuginfo-install
>>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
>>>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64
>>>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64
>>>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
>>>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64
>>>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64
>>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64
>>>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64
>>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>>>> (gdb) bt
>>>> #0  0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030,
>>>> old=0, new_v=1) at ../../atomic/atomic_x86.h:233
>>>> #1  0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at
>>>> ../../futexlock.h:99
>>>> #2  0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at
>>>> ip_tree.c:47
>>>> #3  0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at
>>>> ip_tree.c:66
>>>> #4  0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at
>>>> pike_funcs.c:293
>>>> #5  0x00000000005fef8e in compat_old_handler (ti=1971296512,
>>>> tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996
>>>> #6  0x00000000005ff94c in slow_timer_main () at timer.c:1130
>>>> #7  0x00000000004a8676 in main_loop () at main.c:1628
>>>> #8  0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at
>>>> main.c:2533
>>>> (gdb)
>>>>
>>>>
>>>> On Clean up after the core, mongo db does segmentation fault as well
>>>>
>>>> Loaded symbols for /lib64/libnss_files-2.12.so
>>>> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
>>>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
>>>> done.
>>>> Loaded symbols for /lib64/libnss_dns-2.12.so
>>>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f
>>>> /usr/local/kamailio_proxy/etc/kamail'.
>>>> Program terminated with signal 11, Segmentation fault.
>>>> #0  __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at
>>>> pthread_mutex_destroy.c:28
>>>> 28  if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0
>>>> Missing separate debuginfos, use: debuginfo-install
>>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
>>>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64
>>>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64
>>>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
>>>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64
>>>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64
>>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64
>>>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64
>>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>>>> (gdb) bt
>>>> #0  __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at
>>>> pthread_mutex_destroy.c:28
>>>> #1  0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at
>>>> src/mongoc/mongoc-ssl.c:555
>>>> #2  _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106
>>>> #3  0x00007fd32c8606a9 in _mongoc_do_cleanup () at
>>>> src/mongoc/mongoc-init.c:127
>>>> #4  0x000000344480cdb3 in pthread_once () at
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104
>>>> #5  0x00007fd32c8523df in __do_global_dtors_aux () from
>>>> /usr/lib64/libmongoc-1.0.so.0.0.0
>>>> #6  0x0000000000000000 in ?? ()
>>>> (gdb)
>>>>
>>>>
>>>>
>>>> mongo-c-driver version 1.1.10
>>>>
>>>> version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty
>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
>>>> USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP,
>>>> PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
>>>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
>>>> HAVE_RESOLV_RES
>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>>> id: b5980b -dirty
>>>> compiled on 13:35:24 Sep 28 2015 with gcc 4.
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>> sr-users at lists.sip-router.org
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
> <kamailio-local.cfg>
>
> <kamailio.cfg>
>
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151006/8828e39b/attachment.html>