On 02/10/2013 19:18, Daniel-Constantin Mierla wrote:
Hello,
can you give bt full as well as kamailio -v output? Any log error messages?
Also, it would be good to recompile with MEMDBG=1 and watch for errors in the logs to see if there is a buffer overflow.
Cheers, Daniel
On 10/2/13 7:19 PM, Hugh Waite wrote:
Hi, We've had some more crashes on the current master build. (gdb) bt #0 qm_insert_free (qm=0x7fc1e1b9e010, p=<value optimized out>) at mem/q_malloc.c:181 #1 qm_free (qm=0x7fc1e1b9e010, p=<value optimized out>) at mem/q_malloc.c:527 #2 0x000000000055027f in reset_ruid (msg=0x7fc1e1c35360) at parser/msg_parser.c:911 #3 free_sip_msg (msg=0x7fc1e1c35360) at parser/msg_parser.c:730 #4 0x00000000004a4012 in receive_msg (buf=<value optimized out>, len=<value optimized out>, rcv_info=<value optimized out>) at receive.c:297 #5 0x000000000052a251 in tcp_read_req (con=0x7fc1ca4c6e00, bytes_read=0x7fff041b327c, read_flags=0x7fff041b3274) at tcp_read.c:1387 #6 0x000000000052c53b in handle_io (fm=<value optimized out>, events=1, idx=-1) at tcp_read.c:1617 #7 0x000000000052eb69 in io_wait_loop_epoll (unix_sock=<value optimized out>) at io_wait.h:1092 #8 tcp_receive_loop (unix_sock=<value optimized out>) at tcp_read.c:1728 #9 0x00000000004fc0eb in tcp_init_children () at tcp_main.c:4959 #10 0x000000000046c3d5 in main_loop () at main.c:1702 #11 0x000000000046dec9 in main (argc=<value optimized out>, argv=<value optimized out>) at main.c:2533
(gdb) frame 2 #2 0x000000000055027f in reset_ruid (msg=0x7fc1e1c35360) at parser/msg_parser.c:911 911 pkg_free(msg->ruid.s); (gdb) p msg->ruid $7 = {s = 0x845d20 "", len = 20}
Might this be related to the changes made on Sept 19th to the free_sip_msg functions?
Regards, Hugh
Extra output below. Nothing was printed in the logs (WARNING or ERROR level) before the crash. It seemed to be quite reproduceable when there was traffic being sent to registered websocket clients, but there is no-one online now. We'll have multiple people logged on tomorrow morning.
Regards, Hugh
kamailio -v version: kamailio 4.1.0-dev9 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 13:35:36 Oct 2 2013 with gcc 4.4.7
(gdb) bt full #0 qm_insert_free (qm=0x7fc1e1b9e010, p=<value optimized out>) at mem/q_malloc.c:181 f = 0x845d10 prev = 0x65332d3231653163 hash = 0 #1 qm_free (qm=0x7fc1e1b9e010, p=<value optimized out>) at mem/q_malloc.c:527 f = 0x845d10 size = <value optimized out> next = <value optimized out> prev = <value optimized out> __FUNCTION__ = "qm_free" #2 0x000000000055027f in reset_ruid (msg=0x7fc1e1c35360) at parser/msg_parser.c:911 No locals. #3 free_sip_msg (msg=0x7fc1e1c35360) at parser/msg_parser.c:730 No locals. #4 0x00000000004a4012 in receive_msg (buf=<value optimized out>, len=<value optimized out>, rcv_info=<value optimized out>) at receive.c:297 msg = 0x7fc1e1c35360 ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {1048575, -3596212518023615478, 140470693039152, 140470299422208, 140470299422208, 140733262279292, 140733262279292, 140733262279284}, __mask_was_saved = -900960744, __saved_mask = {__val = {5406222, 532575944923, 541165879417, 4294967297, 1042, 140470299422952, 5409666, 65535, 140733262279284, 140470300510264, 140470697171281, 1042, 140470300510984, 18446744072809678880, 16, 17179869210}}}}} ret = <value optimized out> inb = { s = 0x23d7cc0 "OPTIONS sip:gavin.llewellyn@crocodiletalk.com SIP/2.0\r\nVia: SIP/2.0/TCP edge00-int.crocodilertc.net:5080;branch=z9hG4bKab92.bb8249afcf13f20080f25121e49865b8.0\r\nVia: SIP/2.0/WSS qvis2mie4gas.invalid;rp"..., len = 1028} __FUNCTION__ = "receive_msg" #5 0x000000000052a251 in tcp_read_req (con=0x7fc1ca4c6e00, bytes_read=0x7fff041b327c, read_flags=0x7fff041b3274) at tcp_read.c:1387 bytes = <value optimized out> total_bytes = 1028 resp = 1 size = <value optimized out> req = 0x7fc1ca4c6e80 dst = {send_sock = 0x14, to = {s = {sa_family = 1, sa_data = "\000\000\001\000\000\000\001 \000\000x\313\306", <incomplete sequence \341>}, sin = {sin_family = 1, sin_port = 0, sin_addr = {s_addr = 1}, sin_zero = "\001 \000\000x\313\306", <incomplete sequence \341>}, sin6 = {sin6_family = 1, sin6_port = 0, sin6_flowinfo = 1, sin6_addr = {__in6_u = { __u6_addr8 = "\001 \000\000x\313\306\341\301\177\000\000\000\000\000", __u6_addr16 = {8193, 0, 52088, 57798, 32705, 0, 0, 0}, __u6_addr32 = {8193, 3787901816, 32705, 0}}}, sin6_scope_id = 68891240}}, id = 32767, proto = 8 '\b', send_flags = { f = 0 '\000', blst_imask = 0 '\000'}} c = 13 '\r' ret = <value optimized out> __FUNCTION__ = "tcp_read_req" #6 0x000000000052c53b in handle_io (fm=<value optimized out>, events=1, idx=-1) at tcp_read.c:1617 ret = <value optimized out> n = <value optimized out> read_flags = 1 con = 0x7fc1ca4c6e00 s = <value optimized out> resp = <value optimized out> t = <value optimized out> __FUNCTION__ = "handle_io" #7 0x000000000052eb69 in io_wait_loop_epoll (unix_sock=<value optimized out>) at io_wait.h:1092