[kamailio/kamailio] q_malloc crash on Kamailio 5.2.3 with http_async_query function (#2091) - sr-dev

3 Oct 2019


      ### Description
We added http_async_client module to our configuration. We want to send HTTP POST request after the call is ended (in event_route[dialog_end]).
We used the function **http_async_query** and right after calling the function Kamailio child exited by a signal 6 (more in the log down).
### Troubleshooting
#### Reproduction
In our system we just added http_async_module and its function http_async_query. It is located in own route HTTP_TEST_CALL_END called from event_route[dialog:end]
 Somewhat like this:
```
route[HTTP_TEST_CALL_END] {
t_newtran();
$http_req(body) = "TEST";
http_async_query("http://192.168.6.23:2080/cdr_http", "HTTP_TEST_REPLY");
}
```
<!--
If the issue can be reproduced, describe how it can be done.
-->
#### Debugging Data
<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile
  bt full
  info locals
  list
If you are familiar with gdb, feel free to attach more of what you consider to
be relevant.
-->
```
Core was generated by `/opt/ipgate/kamailio/sbin/kamailio -f /opt/ipgate/kamailio/etc/kamailio/kamaili'.
Program terminated with signal 6, Aborted.
#0  0xb7791424 in __kernel_vsyscall ()
(gdb) bt full
#0  0xb7791424 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7633661 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#2  0xb7636a92 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6
No symbol table info available.
#3  0x0829fed6 in qm_debug_check_frag (qm=0x97607008, f=0x97a7ff3c, 
    file=0x966f22d6 "http_async_client: hm_hash.c", line=82, efile=0x83f5093 "core/mem/q_malloc.c", 
    eline=384) at core/mem/q_malloc.c:151
        __FUNCTION__ = "qm_debug_check_frag"
#4  0x082a16aa in qm_malloc (qmp=0x97607008, size=8, file=0x966f22d6 "http_async_client: hm_hash.c", 
    func=0x966f2bfb "build_hash_key", line=82, mname=0x966f22c4 "http_async_client")
    at core/mem/q_malloc.c:384
        qm = 0x97607008
        f = 0x97a7ff3c
        hash = 4
        list_cntr = 1
        __FUNCTION__ = "qm_malloc"
#5  0x966d5323 in build_hash_key (p=0x567d9620) at hm_hash.c:82
        hash_str = 0x966f2be9
        pointer_str = 0x97a7ff1c "0x567d9620"
        len = 10
        hash = 360
        __FUNCTION__ = "build_hash_key"
#6  0x966d5d94 in build_http_m_cell (p=0x567d9620) at hm_hash.c:116
        cell = 0x567cab3c
        len = 360
        __FUNCTION__ = "build_http_m_cell"
#7  0x966eafac in new_request (query=0xbfe65718, query_params=0xbfe65720, 
    cb=0x966c9125 <async_http_cb>, param=0x567be804) at http_multi.c:442
        __FUNCTION__ = "new_request"
        easy = 0x567d9620
        rc = -1217109748
        cell = 0x0
#8  0x966ce473 in notification_socket_cb (fd=16, event=2, arg=0x5679c988) at async_http.c:390
        worker = 0x5679c988
        received = 4
        i = 0
        len = -1075423144
        aq = 0x567be804
        query_params = {timeout = 500, tls_verify_host = 1, tls_verify_peer = 1, headers = 0x0, 
          method = 0, tls_client_cert = 0x0, tls_client_key = 0x0, tls_ca_path = 0x0, body = {
            s = 0x567cb650 "TEST\300\300\300\300\355\357ͫ", len = 4}, authmethod = 3, 
          username = 0x0, password = 0x0}
        query = {s = 0x567bf75c "http://192.168.6.23:2080/cdr_httpt_", len = 33}
        __FUNCTION__ = "notification_socket_cb"
#9  0x9661d522 in event_base_loop () from /usr/lib/i386-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#10 0x9661e683 in event_base_dispatch () from /usr/lib/i386-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#11 0x966c8a4c in async_http_run_worker (worker=0x5679c988) at async_http.c:92
No locals.
#12 0x966dabcd in child_init (rank=0) at http_async_client_mod.c:352
---Type <return> to continue, or q <return> to quit---
        pid = 0
        i = 0
        __FUNCTION__ = "child_init"
#13 0x08200daa in init_mod_child (m=0x976a286c, rank=0) at core/sr_module.c:846
        __FUNCTION__ = "init_mod_child"
#14 0x082011ad in init_child (rank=0) at core/sr_module.c:874
        ret = 23
#15 0x0806b409 in main_loop () at main.c:1737
        i = 3
        pid = 5318
        si = 0x0
        si_desc = "udp receiver child=2 sock=192.168.5.67:5060\000]\330g\267\360\243v\267X\037\000\000\300\243v\267\364\217v\267\300\243v\267\b\376R\t`\\346\277]\330g\267\251\000\000\000\b\376R\t\364\217v\267\025\000\000\000\000]\346\277\367\273m\267\060+Z\t0+Z\t\\v\232\227\000@\000\000@\200v\267\000\000\000"
        nrprocs = 3
        woneinit = 1
        __FUNCTION__ = "main_loop"
#16 0x08071eb4 in main (argc=13, argv=0xbfe660b4) at main.c:2696
        cfg_stream = 0x93f3008
        c = -1
        r = 0
        tmp = 0xbfe6678c ""
        tmp_len = -1216949440
        port = 2210
        proto = 1
        options = 0x8376690 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
        ret = -1
        seed = 2661251522
        rfd = 4
        debug_save = 0
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = 0xbfe65fe0
        p = 0x805e4a0 "[\201\303p\374?"
        st = {st_dev = 14, __pad1 = 0, st_ino = 5047, st_mode = 16877, st_nlink = 2, st_uid = 109, 
          st_gid = 111, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, 
          st_atim = {tv_sec = 1561470746, tv_nsec = 145357408}, st_mtim = {tv_sec = 1570111971, 
            tv_nsec = 819339961}, st_ctim = {tv_sec = 1570111972, tv_nsec = 31337425}, __unused4 = 0, 
          __unused5 = 0}
        __FUNCTION__ = "main"
(gdb) info locals
No symbol table info available.
(gdb) list
1876		int proto;
1877		char *options;
1878		int ret;
1879		unsigned int seed;
1880		int rfd;
1881		int debug_save, debug_flag;
1882		int dont_fork_cnt;
1883		struct name_lst* n_lst;
1884		char *p;
1885		struct stat st = {0};
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
Oct  3 16:13:37 sip_server /opt/ipgate/kamailio/sbin/kamailio[5331]: CRITICAL: <core> [core/mem/q_malloc.c:149]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(c0003032, abcdefed)[0x97a7ff3c:0x97a7ff58]! Memory allocator was called from http_async_client: hm_hash.c:82. Fragment marked by http_async_client: hm_hash.c:71. Exec from core/mem/q_malloc.c:384.
Oct  3 16:13:37 sip_server /opt/ipgate/kamailio/sbin/kamailio[5315]: INFO: <script>: XLOG: 93ffd680-1f94-4f60-b3b0-0fd599a46677 [onsend_route] Request BYE sent to sip:FreeSWITCH@192.168.5.94:5060;transport=udp;gw=balancer1
Oct  3 16:13:42 sip_server call-control[5004]: Starting factory <callcontrol.rating.RatingEngineFactory instance at 0xb6e8114c>
Oct  3 16:13:42 sip_server call-control[5004]: <twisted.internet.tcp.Connector instance at 0xb6e8116c> will retry in 14 seconds
Oct  3 16:13:42 sip_server call-control[5004]: Stopping factory <callcontrol.rating.RatingEngineFactory instance at 0xb6e8114c>
Oct  3 16:13:44 sip_server /opt/ipgate/kamailio/sbin/kamailio[5312]: ALERT: <core> [main.c:756]: handle_sigs(): child process 5331 exited by a signal 6
Oct  3 16:13:44 sip_server /opt/ipgate/kamailio/sbin/kamailio[5312]: ALERT: <core> [main.c:759]: handle_sigs(): core was generated
Oct  3 16:13:44 sip_server /opt/ipgate/kamailio/sbin/kamailio[5312]: INFO: <core> [main.c:781]: handle_sigs(): terminating due to SIGCHLD
Oct  3 16:13:44 sip_server /opt/ipgate/kamailio/sbin/kamailio[5321]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received
...
```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.2.3 (i386/linux) dcce68
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: dcce68 
compiled on 17:08:25 Jun 20 2019 with gcc 4.7.2
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `uname -a`)
-->
```
Debian GNU/Linux 7.1 (wheezy)
Linux munda 3.2.0-4-686-pae #1 SMP Debian 3.2.46-1+deb7u1 i686 GNU/Linux
```
-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2091