### Description
I'm having crashes on the Http worker process. I didn't find a way to reproduce the crash yet. Crashes are happening on two servers after a couple of days.
#### Debugging Data
Not very useful
``` Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __GI__IO_wfile_underflow (fp=0x7f45900008c0) at wfileops.c:280 280 wfileops.c: No such file or directory. (gdb) bt #0 __GI__IO_wfile_underflow (fp=0x7f45900008c0) at wfileops.c:280 #1 0x00000000000004b0 in ?? () #2 0x0000000000000428 in ?? () #3 0x00007f459d9e2640 in ?? () #4 0x0000000000000000 in ?? () (gdb) ```
#### Log Messages I have debug enabled for http_async_client module, and this is the last messages before the crash ``` Jan 3 11:02:11 vox-kamailio-1887-0 kamailio[2685]: DEBUG: http_async_client [http_multi.c:67]: timer_cb(): timeout on socket -1 Jan 3 11:02:11 vox-kamailio-1887-0 kamailio[2685]: DEBUG: http_async_client [http_multi.c:53]: multi_timer_cb(): multi_timer_cb: Setting timeout to 1 ms Jan 3 11:02:11 vox-kamailio-1887-0 kamailio[2685]: DEBUG: http_async_client [http_multi.c:561]: check_multi_info(): REMAINING: 2 [...] Jan 3 11:02:15 vox-kamailio-1887-0 kamailio[2702]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 89 Jan 3 11:02:15 vox-kamailio-1887-0 kamailio[2618]: ALERT: <core> [main.c:743]: handle_sigs(): child process 2685 exited by a signal 11 Jan 3 11:02:15 vox-kamailio-1887-0 kamailio[2618]: ALERT: <core> [main.c:746]: handle_sigs(): core was generated ```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.0.5 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 4.8.2 ```
This is 5.0.5 + https://github.com/kamailio/kamailio/pull/1386/commits/388ba3642338992a04ca1...
Module parameters are: ``` modparam("http_async_client", "connection_timeout", 10000) modparam("http_async_client", "tls_verify_host", 0) modparam("http_async_client", "tls_verify_peer", 0) modparam("http_async_client", "hash_size", 1024) ```
The use of the module is like this: ``` $var(url) = $avp(s:deviceStateURL)+"ari/deviceStates/Custom:" + $avp(s:queueDevice); $http_req(timeout) = 100; $http_req(suspend) = 0; $http_req(authmethod) = 1; $http_req(username) = $sht(lcr=>ari_user); $http_req(password) = $sht(lcr=>ari_pass); http_async_query("$var(url)", "QUEUE_CHECK_STATUS_OR_SET"); ```
* **Operating System**:
Ubuntu Trusty
Is this something happening recently? If yes did anything change on that server? Which version of libcurl are you using?
Is this something happening recently? If yes did anything change on that server?
We upgraded from 5.0.0 to 5.0.4 and We changed the kamailio config too. If you need the config, I can send it to you in private.
Which version of libcurl are you using?
``` # ldd /usr/lib/x86_64-linux-gnu/kamailio/modules/http_async_client.so linux-vdso.so.1 => (0x00007ffe69bb4000) libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007efe31a30000) libevent-2.0.so.5 => /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 (0x00007efe317ec000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efe31422000) libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007efe311ef000) librtmp.so.0 => /usr/lib/x86_64-linux-gnu/librtmp.so.0 (0x00007efe30fd5000) libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007efe30d75000) libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007efe30999000) libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007efe30752000) liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007efe30542000) libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007efe302f1000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007efe300d8000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efe2feb9000) /lib64/ld-linux-x86-64.so.2 (0x0000560728482000) libgnutls.so.26 => /usr/lib/x86_64-linux-gnu/libgnutls.so.26 (0x00007efe2fbfb000) libgcrypt.so.11 => /lib/x86_64-linux-gnu/libgcrypt.so.11 (0x00007efe2f97b000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007efe2f776000) libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007efe2f4ab000) libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007efe2f27c000) libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007efe2f077000) libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007efe2ee6c000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007efe2ec51000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007efe2ea35000) libgssapi.so.3 => /usr/lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007efe2e7f7000) libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007efe2e5e3000) libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007efe2e3a0000) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007efe2e19b000) libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007efe2df97000) libheimntlm.so.0 => /usr/lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007efe2dd8d000) libkrb5.so.26 => /usr/lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007efe2db05000) libasn1.so.8 => /usr/lib/x86_64-linux-gnu/libasn1.so.8 (0x00007efe2d863000) libhcrypto.so.4 => /usr/lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007efe2d630000) libroken.so.18 => /usr/lib/x86_64-linux-gnu/libroken.so.18 (0x00007efe2d41b000) libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007efe2d212000) libwind.so.0 => /usr/lib/x86_64-linux-gnu/libwind.so.0 (0x00007efe2cfe9000) libheimbase.so.1 => /usr/lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007efe2cddb000) libhx509.so.5 => /usr/lib/x86_64-linux-gnu/libhx509.so.5 (0x00007efe2cb91000) libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007efe2c8d8000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007efe2c69f000) ``` ``` # dpkg -S /usr/lib/x86_64-linux-gnu/libcurl.so.4 libcurl3:amd64: /usr/lib/x86_64-linux-gnu/libcurl.so.4 # apt-cache policy libcurl3 libcurl3: Installed: 7.35.0-1ubuntu2.13 ```
At first look I don't see any change in http_async_client between 5.0.0 and 5.0.4 that could be cause of the crash. Could you please send the kamailio config changes?
Sent config changes on private email.
I detected a memory leak and it seems that libcurl3 is the one to blame. The version on trusty is too old. I'm going to create a backport and check the results with a newer version.
No luck at all, I'm having cores even with ``` # apt-cache policy libcurl3 libcurl3: Installed: 7.57.0-1trusty1 ```
``` # gdb kamailio CoreDump GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from kamailio...Reading symbols from /usr/lib/debug/.build-id/6d/4bb62f73fd6cf4572deb923b7ace1f75c419be.debug...done. done. [New LWP 37737] [New LWP 37738] [New LWP 37739] [New LWP 37736] [New LWP 5784] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ff92aa76b6a in malloc_printerr (ptr=0x7ff9100008c0, str=0x7ff92ab79441 "corrupted double-linked list", action=<optimised out>) at malloc.c:4998 4998 malloc.c: No such file or directory. Traceback (most recent call last): File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in <module> from libstdcxx.v6.printers import register_libstdcxx_printers ImportError: No module named 'libstdcxx' (gdb) bt #0 0x00007ff92aa76b6a in malloc_printerr (ptr=0x7ff9100008c0, str=0x7ff92ab79441 "corrupted double-linked list", action=<optimised out>) at malloc.c:4998 #1 malloc_consolidate (av=0x7ff91571a678) at malloc.c:4159 #2 0x00007ff91571a650 in ?? () #3 0x00007ff91571ad10 in ?? () #4 0x00007ff91a34bbd0 in _nss_files_gethostbyname3_r (name=0x400000002 <error: Cannot access memory at address 0x400000002>, af=715561765, result=0x7ff91571a66a, buffer=0x7ff91571ac00 "\001", buflen=140707803718936, errnop=0x7ff9100008c0, herrnop=0x400000002, ttlp=0x7ff91571a650, canonp=0x7ff91571ad10) at nss_files/files-hosts.c:149 Backtrace stopped: previous frame inner to this frame (corrupt stack?) (gdb) ```
No luck at all, I'm still getting cores ``` [1618945.161599] kamailio[30353]: segfault at 0 ip 00007f7b9c4e5a00 sp 00007f7b88eec4e0 error 4 in libc-2.19.so[7f7b9c472000+1bb000] [1739973.051705] kamailio[15255]: segfault at 206c9 ip 00007f23c5c190bd sp 00007f23b2e22580 error 4 in libc-2.19.so[7f23c5ba7000+1bb000] [1884997.863041] kamailio[37737]: segfault at 206c9 ip 00007ff92aa76b6a sp 00007ff91571a530 error 6 in libc-2.19.so[7ff92a9fc000+1bb000] [2127899.985297] kamailio[11410]: segfault at 206c9 ip 00007fe9021110bd sp 00007fe8ed5be580 error 4 in libc-2.19.so[7fe90209f000+1bb000] ```
Will try to upgrade the servers to latest versions of trusty
Upgrading libc6 on Trusty to ``` $ apt-cache policy libc6 libc6: Installed: 2.19-0ubuntu6.14 Candidate: 2.19-0ubuntu6.14 ``` seems to help with the crash. No crashes so far. The memory leak was related to creating $avp variables on the Http worker, they were never released. I had to delete them explicitly.
To cleanup the avps/xavps, the function in `http_async_client` that executes the `event_route` block has to do `ksr_msg_env_reset()` after the execution of the `event_route` -- something like done in the rtimer module:
* https://github.com/kamailio/kamailio/blob/master/src/modules/rtimer/rtimer_m...
Closed #1391.
-
Daniel-Constantin Mierla -
Federico Cabiddu
-
Victor Seva