### Description
I am experimenting with fuzzing on Kamailio SIP. A malformed INVITE (with a long tag) crashes the server, raised by qm_debug_check_frag().
### Troubleshooting
The error message:
``` qm_debug_check_frag(): BUG: qm: fragm. 0x7ffff03642e8 (address 0x7ffff0364320) end overwritten (9191919191919191, 9191919191919191)! Memory allocator was called from tm: t_reply.c:2410. Fragment marked by tm: t_msgbuilder.c:327. Exec from core/mem/q_malloc.c:511. ```
Output from GDB: ``` (gdb) watch *0x7ffff0364320 Hardware watchpoint 1: *0x7ffff0364320 (gdb) run -f kamailio-basic.cfg -L src/modules -Y runtime_dir/ -n 1 -D -E Starting program: /home/rnatella/workdir-sip/kamailio/src/kamailio -f kamailio-basic.cfg -L src/modules -Y runtime_dir/ -n 1 -D -E [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0(29450) INFO: <core> [core/sctp_core.c:75]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module Listening on udp: 127.0.0.1 [127.0.0.1]:5060 Aliases:
WARNING: no fork mode 0(29450) INFO: rr [./../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded 0(29450) INFO: rr [rr_mod.c:185]: mod_init(): outbound module not available 0(29450) INFO: <core> [main.c:2841]: main(): processes (at least): 4 - shm size: 67108864 - pkg size: 8388608 0(29450) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 0(29450) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 0(29450) ERROR: {1 2 INVITE 1-670@127.0.0.1} <core> [core/parser/parse_rr.c:78]: do_parse_rr_body(): Failed parsing name-addr (<sip:127.0"0tttttttttttttttttttttttttttK-670-1-7) 0(29450) ERROR: {1 2 INVITE 1-670@127.0.0.1} <core> [core/parser/parse_rr.c:140]: do_parse_rr_body(): Failed parsing rr header body [<sip:127.0"0tttttttttttttttttttttttttttK-670-1-7] 0(29450) ERROR: {1 2 INVITE 1-670@127.0.0.1} rr [loose.c:468]: find_rem_target(): failed to parse last Route HF 0(29450) ERROR: {1 2 INVITE 1-670@127.0.0.1} rr [loose.c:700]: after_strict(): searching for last Route URI failed
Hardware watchpoint 1: *0x7ffff0364320
Old value = <unreadable> New value = 4932352 __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:316 316 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory. (gdb) bt #0 __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:316 #1 0x00007ffff602e4ff in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>, method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:336 #2 0x00007ffff607eaff in build_ack (rpl=<optimized out>, trans=<optimized out>, branch=0, ret_len=<optimized out>) at t_reply.c:360 #3 reply_received (p_msg=<optimized out>) at t_reply.c:2398 #4 0x00000000005a0af3 in do_forward_reply (msg=0x7ffff68eed28, mode=<optimized out>) at core/forward.c:757 #5 0x0000000000688cae in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at core/receive.c:509 #6 0x00000000004a6b39 in udp_rcv_loop () at core/udp_server.c:543 #7 0x000000000042c938 in main_loop () at main.c:1480 #8 0x000000000043c574 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2863 (gdb) frame \ Quit (gdb) frame No symbol "frame" in current context. (gdb) frame frame 1 No symbol "frame" in current context. (gdb) frame 1 #1 0x00007ffff602e4ff in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>, method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:336 336 append_str(d, method, method_len); (gdb) print d $1 = 0x7ffff0364320 "" (gdb) c Continuing.
Hardware watchpoint 1: *0x7ffff0364320
Old value = 4932352 New value = 4932417 __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317 317 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory. (gdb) bt #0 __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317 #1 0x00007ffff602e4ff in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>, method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:336 #2 0x00007ffff607eaff in build_ack (rpl=<optimized out>, trans=<optimized out>, branch=0, ret_len=<optimized out>) at t_reply.c:360 #3 reply_received (p_msg=<optimized out>) at t_reply.c:2398 #4 0x00000000005a0af3 in do_forward_reply (msg=0x7ffff68eed28, mode=<optimized out>) at core/forward.c:757 #5 0x0000000000688cae in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at core/receive.c:509 #6 0x00000000004a6b39 in udp_rcv_loop () at core/udp_server.c:543 #7 0x000000000042c938 in main_loop () at main.c:1480 #8 0x000000000043c574 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2863 (gdb) c Continuing.
Hardware watchpoint 1: *0x7ffff0364320
Old value = 4932417 New value = 541803329 0x00007ffff602e504 in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>, method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:337 337 *d = ' '; (gdb) bt #0 0x00007ffff602e504 in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>, method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:337 #1 0x00007ffff607eaff in build_ack (rpl=<optimized out>, trans=<optimized out>, branch=0, ret_len=<optimized out>) at t_reply.c:360 #2 reply_received (p_msg=<optimized out>) at t_reply.c:2398 #3 0x00000000005a0af3 in do_forward_reply (msg=0x7ffff68eed28, mode=<optimized out>) at core/forward.c:757 #4 0x0000000000688cae in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at core/receive.c:509 #5 0x00000000004a6b39 in udp_rcv_loop () at core/udp_server.c:543 #6 0x000000000042c938 in main_loop () at main.c:1480 #7 0x000000000043c574 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2863 (gdb) c Continuing. 0(29450) CRITICAL: {2 2 INVITE 1-670@127.0.0.1} <core> [core/mem/q_malloc.c:138]: qm_debug_check_frag(): BUG: qm: fragm. 0x7ffff03642e8 (address 0x7ffff0364320) end overwritten (9191919191919191, 9191919191919191)! Memory allocator was called from tm: t_reply.c:2410. Fragment marked by tm: t_msgbuilder.c:327. Exec from core/mem/q_malloc.c:511.
Program received signal SIGSEGV, Segmentation fault. 0x000000000082f45f in qm_status (qmp=<optimized out>) at core/mem/q_malloc.c:902 902 f!=&(qm->free_hash[h].head); f=f->u.nxt_free, i++, j++){ ```
#### Reproduction
I am running the server with a basic configuration (attached kamailio-basic.cfg), using the command:
``` ./src/kamailio -f kamailio-basic.cfg -L src/modules -Y runtime_dir/ -n 1 -D -E ```
[kamailio-basic.cfg.txt](https://github.com/kamailio/kamailio/files/5354439/kamailio-basic.cfg.txt)
On the same machine, I am sending the malformed message (also attached):
``` cat sip-crash.txt | nc -4u -w1 localhost 5060 ```
[sip-crash.txt](https://github.com/kamailio/kamailio/files/5354468/sip-crash.txt)
You can find more information about my fuzzing setup at: [](https://github.com/rnatella/aflnet-kamailio-sip)
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
See previous section
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
See previous section
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
See previos section
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.5.0-dev2 (x86_64/linux) 6049a1-dirty flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 6049a1 -dirty compiled on 10:12:13 Oct 9 2020 with /home/rnatella/aflnet/afl-clang-fast 6.0 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) -->
``` Ubuntu 18.04.2 LTS
Linux dockertest1 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ```
Thanks for the report. If you wanted to include some description about the fuzzing setup, it is missing. In case you find more crashes with your fuzzing, please send a report of them to the e-mail address mentioned in our [wiki](https://www.kamailio.org/wiki/security/policy) instead of opening an issue on the public tracker.
Fixed the missing link about the fuzzing setup. Thanks for the pointer!
This happens when the INVITE is forwarded and not answered with 200ok, isn't it? Does it happen every time? If you get a core file, can you get the `gdb` output for `bt full`?
The crash happens every time, regardless that another SIP client ("33" in my setup) is registered or not. You can find attached `bt full`. Despite I compiled with "make mode=debug all", most variables are optimized out. [gdb.txt](https://github.com/kamailio/kamailio/files/5356287/gdb.txt)
I pushed some commits to master branch to catch it, can you try and see if it is fixed in your tests?
Seems fixed, no more crashes!
The output: ``` Call-I+: 1-670@127.0.0.1 CSeq: 2 INVITE Contact.0.1:5061 Max-Forward] 0(26164) WARNING: <core> [core/receive.c:317]: receive_msg(): parsing relevant headers failed 0(26164) ERROR: <core> [core/parser/msg_parser.c:397]: parse_headers(): duplicate From header field [From: sip:30@127.0.0.1;tag=: To: sip:33@127.0.0.1;tag=gggggggg���������������������������������] 0(26164) ERROR: pv [pv_core.c:1965]: pv_get_hdr(): error parsing headers 0(26164) ERROR: <core> [core/parser/msg_parser.c:397]: parse_headers(): duplicate From header field [From: sip:30@127.0.0.1;tag=: To: sip:33@127.0.0.1;tag=gggggggg���������������������������������] 0(26164) ERROR: pv [pv_core.c:731]: pv_get_callid(): cannot parse Call-Id header 0(26164) ERROR: {1 <null> <null>} <core> [core/parser/msg_parser.c:397]: parse_headers(): duplicate From header field [From: sip:30@127.0.0.1;tag=: To: sip:33@127.0.0.1;tag=gggggggg���������������������������������] 0(26164) ERROR: {1 <null> <null>} maxfwd [mf_funcs.c:51]: is_maxfwd_present(): parsing MAX_FORWARD header failed! 0(26164) ERROR: {1 <null> <null>} <core> [core/parser/msg_parser.c:397]: parse_headers(): duplicate From header field [From: sip:30@127.0.0.1;tag=: To: sip:33@127.0.0.1;tag=gggggggg���������������������������������] 0(26164) ERROR: {1 <null> <null>} <core> [core/msg_translator.c:2394]: build_res_buf_from_sip_req(): alas, parse_headers failed ```
Thanks for testing and feedback. Related commits were backported to stable branches.
Closed #2503.