[SR-Users] segmentation fault when mongodb & tls enabled. ( kamailio 4.3.2)
Daniel-Constantin Mierla
miconda at gmail.com
Tue Oct 6 15:53:20 CEST 2015
Hello,
unfortunately not yet -- this requires compiling the libs and currently
I am traveling to USA to attend few VoIP conferences, not having a good
devel environment with me, access to internet is limited, so connecting
to a server is not easy option as well. Can you open an issue on github
tracker just not to forget about it?
Cheers,
Daniel
On 06/10/15 15:49, realjijo at gmail.com wrote:
> Hi Daniel
>
> Thanks for your help .
>
> Did you get a chance to look at this problem.
> Please let me know how did you build the mongo-c-driver ?
>
>
> Regards,
> Jijo
>
> On Oct 2, 2015, at 7:21 AM, Jijo <realjijo at gmail.com
> <mailto:realjijo at gmail.com>> wrote:
>
>> I have rebuild the official version(4.3.2) again with the default
>> scripts and now i'm getting diffrent core.
>>
>> This happens only when i enable TLS module in kamailio. So i believe
>> there is some corruption causing when TLS module in Kamailio and SSL
>> in Mongo Driver is enabled.
>>
>> I'm running with the offical kamailio script by just enabling mongo
>> modules. It happens for NDB as well.
>>
>> How did you build the mongo driver when you implemented it? May be i
>> can try the same.
>>
>> Please find the scripts used for testing.
>>
>> Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m
>> 64 -M 8'.
>>
>> Program terminated with signal 11, Segmentation fault.
>>
>> #0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08,
>> _aor=0x7fff6de7ef70) at udomain.c:1017
>>
>> 1017lock_get(_d->table[sl].lock);
>>
>> Missing separate debuginfos, use: debuginfo-install
>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
>> glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64
>> krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64
>> libselinux-2.0.94-5.8.el6.x86_64
>> nss-softokn-freebl-3.14.3-22.el6_6.x86_64
>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>>
>> (gdb) bt
>>
>> #0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08,
>> _aor=0x7fff6de7ef70) at udomain.c:1017
>>
>> #1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8,
>> _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at
>> save.c:831
>>
>> #2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08,
>> _cflags=0, _uri=0x0) at save.c:986
>>
>> #3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8,
>> _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
>>
>> #4 0x000000000041decb in do_action (h=0x7fff6de7f720,
>> a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059
>>
>> #5 0x000000000042a553 in run_actions (h=0x7fff6de7f720,
>> a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548
>>
>> #6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10,
>> a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
>>
>> #7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10,
>> msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0)
>> at rvalue.c:912
>>
>> #8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10,
>> msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at
>> rvalue.c:1906
>>
>> #9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10,
>> msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at
>> rvalue.c:1914
>>
>> #10 0x000000000041d927 in do_action (h=0x7fff6de80a10,
>> a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029
>>
>> #11 0x000000000042a553 in run_actions (h=0x7fff6de80a10,
>> a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548
>>
>> #12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10,
>> a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677
>>
>> #13 0x000000000042a553 in run_actions (h=0x7fff6de80a10,
>> a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548
>>
>> #14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08,
>> msg=0x7f85a3f054b8, c=0x0) at action.c:1634
>>
>> #15 0x000000000050a9f4 in receive_msg (
>>
>> buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia:
>> SIP/2.0/UDP
>> 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards:
>> 70\r\nContact:
>> <sip:usera at 192.168.2.119:60887;rinstance=d9f6274d7"..., len=534,
>> rcv_info=0x7fff6de80d00) at receive.c:196
>>
>> #16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
>>
>> #17 0x00000000004a7fb3 in main_loop () at main.c:1573
>>
>> #18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at
>> main.c:2533
>>
>> (gdb) quit
>>
>> [root at localhost /
>>
>>
>>
>> On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> If you recompiled with different souce code or different flags
>> then the binary is no longer matching the corefile properly.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 01/10/15 17:00, Jijo wrote:
>>> Why the backtrace doesn't show the timer function pointer? Am i
>>> missing something?
>>>
>>> Im building the mongo-c-driver as below
>>> get the tar file
>>> from https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz
>>> run the rpm build using the spec file attached.
>>> To disable the ssl, i have modified '--enable-ssl=no' in the
>>> configure command in the spec file
>>> %configure --disable-static --disable-silent-rules
>>> --enable-debug-symbols --enable-man-pages --enable-ssl=yes
>>> --enable-sasl --with-libbson=bundled --enable-optimizations
>>>
>>> In kamailio im using db_mongodb. I'm getting same error for
>>> ndb_mongodb as well.
>>>
>>> Please find the kamailio config scripts.
>>>
>>> On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> the bracktrace is not useful.
>>>
>>> How did you compile the mongo-c library before and were
>>> there any special parameters you set to the modules in
>>> kamailio config? Are you using db_mongodb or ndb_mongodb?
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>>
>>> On 01/10/15 16:25, Jijo wrote:
>>>> Hi Daniel,
>>>>
>>>> Thanks.. As a sidenote, If i disable ssl from
>>>> mongo-c-driver library then i don't have any crash..
>>>>
>>>> Something changed in my environment. Im not able to get the
>>>> full backtrace. This is what i'm getting now.
>>>>
>>>>
>>>>
>>>> aded symbols for /lib64/liblber-2.4.so.2
>>>> Reading symbols from /lib64/libnss_dns-2.12.so...Reading
>>>> symbols from
>>>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
>>>> done.
>>>> Loaded symbols for /lib64/libnss_dns-2.12.so
>>>> <http://libnss_dns-2.12.so>
>>>> Core was generated by
>>>> `/usr/local/kamailio_proxy/sbin/kamailio -f
>>>> /usr/local/kamailio_proxy/etc/kamail'.
>>>> Program terminated with signal 11, Segmentation fault.
>>>> #0 0x00007f69ffffffff in ?? ()
>>>> Missing separate debuginfos, use: debuginfo-install
>>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64
>>>> db4-4.7.25-19.el6_6.x86_64 keyutils-libs-1.4-5.el6.x86_64
>>>> krb5-libs-1.10.3-42.el6.x86_64
>>>> libcom_err-1.41.12-22.el6.x86_64
>>>> libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64
>>>> libselinux-2.0.94-5.8.el6.x86_64
>>>> libssh2-1.4.2-1.el6_6.1.x86_64
>>>> libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64
>>>> mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64
>>>> nss-3.18.0-5.3.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64
>>>> openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64
>>>> zlib-1.2.3-29.el6.x86_64
>>>> (gdb) bt
>>>> #0 0x00007f69ffffffff in ?? ()
>>>> #1 0x00000000005fef8e in compat_old_handler (ti=69660042,
>>>> tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996
>>>> #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130
>>>> #3 0x00000000004a8676 in main_loop () at main.c:1628
>>>> #4 0x00000000004ae38e in main (argc=15,
>>>> argv=0x7ffdd0dc8848) at main.c:2533
>>>> (gdb)
>>>>
>>>>
>>>>
>>>> On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla
>>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>>
>>>> The first backtrace is related to pike. Can you get
>>>> from gdb of fisrt trace:
>>>>
>>>> frame 2
>>>> p root->entries[b]
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>> On 30/09/15 16:19, Jijo wrote:
>>>>> I'm observing a segmentation fault when mongodb & tls
>>>>> enabled. It doesn't happen when one of the module is
>>>>> disabled.
>>>>> OS: centos 6.7 kamailio 4.3.2 mongo-c-driver
>>>>> version 1.1.10.
>>>>>
>>>>> It happens immediately after i register a subscriber.
>>>>> The location table is updated with the new data and in
>>>>> parallel core is generated as well.
>>>>>
>>>>>
>>>>>
>>>>> Loaded symbols for /lib64/libnss_dns-2.12.so
>>>>> <http://libnss_dns-2.12.so>
>>>>> Core was generated by
>>>>> `/usr/local/kamailio_proxy/sbin/kamailio -f
>>>>> /usr/local/kamailio_proxy/etc/kamail'.
>>>>> Program terminated with signal 11, Segmentation fault.
>>>>> #0 0x00007fd3adf65e69 in atomic_cmpxchg_int
>>>>> (var=0x56444e3055445030, old=0, new_v=1) at
>>>>> ../../atomic/atomic_x86.h:233
>>>>> 233ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int
>>>>> , int)
>>>>> Missing separate debuginfos, use: debuginfo-install
>>>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>>>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64
>>>>> db4-4.7.25-19.el6_6.x86_64
>>>>> keyutils-libs-1.4-5.el6.x86_64
>>>>> krb5-libs-1.10.3-42.el6.x86_64
>>>>> libcom_err-1.41.12-22.el6.x86_64
>>>>> libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64
>>>>> libselinux-2.0.94-5.8.el6.x86_64
>>>>> libssh2-1.4.2-1.el6_6.1.x86_64
>>>>> libunistring-0.9.3-5.el6.x86_64
>>>>> libxml2-2.7.6-20.el6.x86_64
>>>>> mysql-libs-5.1.73-5.el6_6.x86_64
>>>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64
>>>>> nss-util-3.18.0-1.el6_6.x86_64
>>>>> openldap-2.4.40-5.el6.x86_64
>>>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>>>>> (gdb) bt
>>>>> #0 0x00007fd3adf65e69 in atomic_cmpxchg_int
>>>>> (var=0x56444e3055445030, old=0, new_v=1) at
>>>>> ../../atomic/atomic_x86.h:233
>>>>> #1 0x00007fd3adf65eb6 in futex_get
>>>>> (lock=0x56444e3055445030) at ../../futexlock.h:99
>>>>> #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192
>>>>> '\300') at ip_tree.c:47
>>>>> #3 0x00007fd3adf664e5 in lock_tree_branch (b=192
>>>>> '\300') at ip_tree.c:66
>>>>> #4 0x00007fd3adf6dda6 in swap_routine
>>>>> (ticks=123206032, param=0x0) at pike_funcs.c:293
>>>>> #5 0x00000000005fef8e in compat_old_handler
>>>>> (ti=1971296512, tl=0x7fd32ccea430,
>>>>> data=0x7fd32ccea430) at timer.c:996
>>>>> #6 0x00000000005ff94c in slow_timer_main () at
>>>>> timer.c:1130
>>>>> #7 0x00000000004a8676 in main_loop () at main.c:1628
>>>>> #8 0x00000000004ae38e in main (argc=11,
>>>>> argv=0x7fff1d97ec28) at main.c:2533
>>>>> (gdb)
>>>>>
>>>>>
>>>>> On Clean up after the core, mongo db does segmentation
>>>>> fault as well
>>>>>
>>>>> Loaded symbols for /lib64/libnss_files-2.12.so
>>>>> <http://libnss_files-2.12.so>
>>>>> Reading symbols from
>>>>> /lib64/libnss_dns-2.12.so...Reading symbols from
>>>>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
>>>>> done.
>>>>> Loaded symbols for /lib64/libnss_dns-2.12.so
>>>>> <http://libnss_dns-2.12.so>
>>>>> Core was generated by
>>>>> `/usr/local/kamailio_proxy/sbin/kamailio -f
>>>>> /usr/local/kamailio_proxy/etc/kamail'.
>>>>> Program terminated with signal 11, Segmentation fault.
>>>>> #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at
>>>>> pthread_mutex_destroy.c:28
>>>>> 28 if ((mutex->__data.__kind &
>>>>> PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0
>>>>> Missing separate debuginfos, use: debuginfo-install
>>>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>>>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64
>>>>> db4-4.7.25-19.el6_6.x86_64
>>>>> keyutils-libs-1.4-5.el6.x86_64
>>>>> krb5-libs-1.10.3-42.el6.x86_64
>>>>> libcom_err-1.41.12-22.el6.x86_64
>>>>> libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64
>>>>> libselinux-2.0.94-5.8.el6.x86_64
>>>>> libssh2-1.4.2-1.el6_6.1.x86_64
>>>>> libunistring-0.9.3-5.el6.x86_64
>>>>> libxml2-2.7.6-20.el6.x86_64
>>>>> mysql-libs-5.1.73-5.el6_6.x86_64
>>>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64
>>>>> nss-util-3.18.0-1.el6_6.x86_64
>>>>> openldap-2.4.40-5.el6.x86_64
>>>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>>>>> (gdb) bt
>>>>> #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at
>>>>> pthread_mutex_destroy.c:28
>>>>> #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup
>>>>> () at src/mongoc/mongoc-ssl.c:555
>>>>> #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106
>>>>> #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at
>>>>> src/mongoc/mongoc-init.c:127
>>>>> #4 0x000000344480cdb3 in pthread_once () at
>>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104
>>>>> #5 0x00007fd32c8523df in __do_global_dtors_aux ()
>>>>> from /usr/lib64/libmongoc-1.0.so.0.0.0
>>>>> #6 0x0000000000000000 in ?? ()
>>>>> (gdb)
>>>>>
>>>>>
>>>>>
>>>>> mongo-c-driver version 1.1.10
>>>>>
>>>>> version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty
>>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP,
>>>>> TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
>>>>> DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC,
>>>>> DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
>>>>> USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
>>>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
>>>>> MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535,
>>>>> DEFAULT PKG_SIZE 8MB
>>>>> poll method support: poll, epoll_lt, epoll_et,
>>>>> sigio_rt, select.
>>>>> id: b5980b -dirty
>>>>> compiled on 13:35:24 Sep 28 2015 with gcc 4.
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>> sr-users at lists.sip-router.org
>>>>> <mailto:sr-users at lists.sip-router.org>
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierla
>>>> http://twitter.com/#!/miconda
>>>> <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) -
>>>> sr-users mailing list
>>>> sr-users at lists.sip-router.org
>>>> <mailto:sr-users at lists.sip-router.org>
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>
>>> --
>>> Daniel-Constantin Mierla
>>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierla
>> http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
>> <kamailio-local.cfg>
>> <kamailio.cfg>
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151006/bcdba22b/attachment.html>
More information about the sr-users
mailing list