[SR-Users] segmentation fault when mongodb & tls enabled. ( kamailio 4.3.2)
Jijo
realjijo at gmail.com
Fri Oct 2 13:21:34 CEST 2015
I have rebuild the official version(4.3.2) again with the default scripts
and now i'm getting diffrent core.
This happens only when i enable TLS module in kamailio. So i believe there
is some corruption causing when TLS module in Kamailio and SSL in Mongo
Driver is enabled.
I'm running with the offical kamailio script by just enabling mongo
modules. It happens for NDB as well.
How did you build the mongo driver when you implemented it? May be i can
try the same.
Please find the scripts used for testing.
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M
8'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08,
_aor=0x7fff6de7ef70) at udomain.c:1017
1017 lock_get(_d->table[sl].lock);
Missing separate debuginfos, use: debuginfo-install
cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64
krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64
libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64
openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08,
_aor=0x7fff6de7ef70) at udomain.c:1017
#1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8,
_d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831
#2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08,
_cflags=0, _uri=0x0) at save.c:986
#3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08
"h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
#4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830,
msg=0x7f85a3f054b8) at action.c:1059
#5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830,
msg=0x7f85a3f054b8) at action.c:1548
#6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10,
a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
#7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10,
msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at
rvalue.c:912
#8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10,
msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906
#9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10,
msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914
#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270,
msg=0x7f85a3f054b8) at action.c:1029
#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990,
msg=0x7f85a3f054b8) at action.c:1548
#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98,
msg=0x7f85a3f054b8) at action.c:677
#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08,
msg=0x7f85a3f054b8) at action.c:1548
#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08,
msg=0x7f85a3f054b8, c=0x0) at action.c:1634
#15 0x000000000050a9f4 in receive_msg (
buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP
192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards:
70\r\nContact: <sip:usera at 192.168.2.119:60887;rinstance=d9f6274d7"...,
len=534, rcv_info=0x7fff6de80d00) at receive.c:196
#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
#17 0x00000000004a7fb3 in main_loop () at main.c:1573
#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533
(gdb) quit
[root at localhost /
On Thu, Oct 1, 2015 at 12:29 PM, Daniel-Constantin Mierla <miconda at gmail.com
> wrote:
> If you recompiled with different souce code or different flags then the
> binary is no longer matching the corefile properly.
>
> Cheers,
> Daniel
>
>
> On 01/10/15 17:00, Jijo wrote:
>
> Why the backtrace doesn't show the timer function pointer? Am i missing
> something?
>
> Im building the mongo-c-driver as below
> get the tar file from
> https://github.com/mongodb/mongo-c-driver/releases/download/1.1.11/mongo-c-driver-1.1.11.tar.gz
> run the rpm build using the spec file attached.
> To disable the ssl, i have modified '--enable-ssl=no' in the configure
> command in the spec file
> %configure --disable-static --disable-silent-rules --enable-debug-symbols
> --enable-man-pages --enable-ssl=yes --enable-sasl --with-libbson=bundled
> --enable-optimizations
>
> In kamailio im using db_mongodb. I'm getting same error for ndb_mongodb
> as well.
>
> Please find the kamailio config scripts.
>
> On Thu, Oct 1, 2015 at 10:43 AM, Daniel-Constantin Mierla <
> <miconda at gmail.com>miconda at gmail.com> wrote:
>
>> Hello,
>>
>> the bracktrace is not useful.
>>
>> How did you compile the mongo-c library before and were there any special
>> parameters you set to the modules in kamailio config? Are you using
>> db_mongodb or ndb_mongodb?
>>
>> Cheers,
>> Daniel
>>
>>
>>
>> On 01/10/15 16:25, Jijo wrote:
>>
>> Hi Daniel,
>>
>> Thanks.. As a sidenote, If i disable ssl from mongo-c-driver library then
>> i don't have any crash..
>>
>> Something changed in my environment. Im not able to get the full
>> backtrace. This is what i'm getting now.
>>
>>
>>
>> aded symbols for /lib64/liblber-2.4.so.2
>> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
>> done.
>> Loaded symbols for /lib64/libnss_dns-2.12.so
>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f
>> /usr/local/kamailio_proxy/etc/kamail'.
>> Program terminated with signal 11, Segmentation fault.
>> #0 0x00007f69ffffffff in ?? ()
>> Missing separate debuginfos, use: debuginfo-install
>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64
>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64
>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64
>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64
>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64
>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64
>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>> (gdb) bt
>> #0 0x00007f69ffffffff in ?? ()
>> #1 0x00000000005fef8e in compat_old_handler (ti=69660042,
>> tl=0x7f693e736d50, data=0x7f693e736d50) at timer.c:996
>> #2 0x00000000005ff94c in slow_timer_main () at timer.c:1130
>> #3 0x00000000004a8676 in main_loop () at main.c:1628
>> #4 0x00000000004ae38e in main (argc=15, argv=0x7ffdd0dc8848) at
>> main.c:2533
>> (gdb)
>>
>>
>>
>> On Thu, Oct 1, 2015 at 2:37 AM, Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> The first backtrace is related to pike. Can you get from gdb of fisrt
>>> trace:
>>>
>>> frame 2
>>> p root->entries[b]
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 30/09/15 16:19, Jijo wrote:
>>>
>>> I'm observing a segmentation fault when mongodb & tls enabled. It
>>> doesn't happen when one of the module is disabled.
>>> OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
>>>
>>> It happens immediately after i register a subscriber. The location table
>>> is updated with the new data and in parallel core is generated as well.
>>>
>>>
>>>
>>> Loaded symbols for /lib64/libnss_dns-2.12.so
>>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f
>>> /usr/local/kamailio_proxy/etc/kamail'.
>>> Program terminated with signal 11, Segmentation fault.
>>> #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030,
>>> old=0, new_v=1) at ../../atomic/atomic_x86.h:233
>>> 233 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int)
>>> Missing separate debuginfos, use: debuginfo-install
>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
>>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64
>>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64
>>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
>>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64
>>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64
>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64
>>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64
>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>>> (gdb) bt
>>> #0 0x00007fd3adf65e69 in atomic_cmpxchg_int (var=0x56444e3055445030,
>>> old=0, new_v=1) at ../../atomic/atomic_x86.h:233
>>> #1 0x00007fd3adf65eb6 in futex_get (lock=0x56444e3055445030) at
>>> ../../futexlock.h:99
>>> #2 0x00007fd3adf66468 in prv_lock_tree_branch (b=192 '\300') at
>>> ip_tree.c:47
>>> #3 0x00007fd3adf664e5 in lock_tree_branch (b=192 '\300') at ip_tree.c:66
>>> #4 0x00007fd3adf6dda6 in swap_routine (ticks=123206032, param=0x0) at
>>> pike_funcs.c:293
>>> #5 0x00000000005fef8e in compat_old_handler (ti=1971296512,
>>> tl=0x7fd32ccea430, data=0x7fd32ccea430) at timer.c:996
>>> #6 0x00000000005ff94c in slow_timer_main () at timer.c:1130
>>> #7 0x00000000004a8676 in main_loop () at main.c:1628
>>> #8 0x00000000004ae38e in main (argc=11, argv=0x7fff1d97ec28) at
>>> main.c:2533
>>> (gdb)
>>>
>>>
>>> On Clean up after the core, mongo db does segmentation fault as well
>>>
>>> Loaded symbols for /lib64/libnss_files-2.12.so
>>> Reading symbols from /lib64/libnss_dns-2.12.so...Reading symbols from
>>> /usr/lib/debug/lib64/libnss_dns-2.12.so.debug...done.
>>> done.
>>> Loaded symbols for /lib64/libnss_dns-2.12.so
>>> Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f
>>> /usr/local/kamailio_proxy/etc/kamail'.
>>> Program terminated with signal 11, Segmentation fault.
>>> #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at
>>> pthread_mutex_destroy.c:28
>>> 28 if ((mutex->__data.__kind & PTHREAD_MUTEX_ROBUST_NORMAL_NP) == 0
>>> Missing separate debuginfos, use: debuginfo-install
>>> cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
>>> cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64
>>> keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64
>>> libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64
>>> libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64
>>> libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64
>>> libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64
>>> nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64
>>> nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64
>>> openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
>>> (gdb) bt
>>> #0 __pthread_mutex_destroy (mutex=0x7fd32cce7be8) at
>>> pthread_mutex_destroy.c:28
>>> #1 0x00007fd32c86f9e6 in _mongoc_ssl_thread_cleanup () at
>>> src/mongoc/mongoc-ssl.c:555
>>> #2 _mongoc_ssl_cleanup () at src/mongoc/mongoc-ssl.c:106
>>> #3 0x00007fd32c8606a9 in _mongoc_do_cleanup () at
>>> src/mongoc/mongoc-init.c:127
>>> #4 0x000000344480cdb3 in pthread_once () at
>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:104
>>> #5 0x00007fd32c8523df in __do_global_dtors_aux () from
>>> /usr/lib64/libmongoc-1.0.so.0.0.0
>>> #6 0x0000000000000000 in ?? ()
>>> (gdb)
>>>
>>>
>>>
>>> mongo-c-driver version 1.1.10
>>>
>>> version: kamailio 4.3.2 (x86_64/linux) b5980b-dirty
>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS,
>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
>>> F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
>>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
>>> id: b5980b -dirty
>>> compiled on 13:35:24 Sep 28 2015 with gcc 4.
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users at lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>> --
>>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>> sr-users at lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>> --
>> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>>
>>
>
> --
> Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> Book: SIP Routing With Kamailio - http://www.asipto.com
> Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151002/53cda78d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kamailio-local.cfg
Type: application/octet-stream
Size: 782 bytes
Desc: not available
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151002/53cda78d/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kamailio.cfg
Type: application/octet-stream
Size: 22658 bytes
Desc: not available
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151002/53cda78d/attachment-0001.obj>
More information about the sr-users
mailing list