<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports. If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list: * http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list: * http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue. If there is no content to be filled in a section, the entire section can be removed. You can delete the comments from the template sections when filling. You can delete next line and everything above before submitting (it is a comment). --> ### Description I have a segfault in an average of 3 to 6 per day on tm module always on the t_should_relay_response. I'm using kamailio 5.2.0 (x86_64/linux) 535e13 on CentOS Linux release 7.6.1810 (Core) x86_64 runing on a XenServer. ### Troubleshooting No troubleshooting was done, since it happened on a production server. We simply restarted the server. #### Reproduction The problem is random and has happened a couple of times per day. My kamailio uses tm, dialog, htable. All calls is using topoh. The server run with an average of 1000-1200 concurrent calls. I've seen this segfault happen with less than 400 concurrent calls too. One very curious thing, when the fifth or sixth time the segfault occurs, it does not happen again, even reaching 3000 concurrent calls. #### Debugging Data <!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location. gdb /path/to/kamailio /path/to/corefile bt full info locals list If you are familiar with gdb, feel free to attach more of what you consider to be relevant. --> ``` (gdb) bt #0 0x00007f685674a84d in t_should_relay_response (Trans=0x7f683d86a3e0, new_code=503, branch=0, should_store=0x7fffed6c323c, should_relay=0x7fffed6c3240, cancel_data=0x7fffed6c3490, reply=0x7f685b6cafa0) at t_reply.c:1279 #1 0x00007f685674ee6b in relay_reply (t=0x7f683d86a3e0, p_msg=0x7f685b6cafa0, branch=0, msg_status=503, cancel_data=0x7fffed6c3490, do_put_on_wait=1) at t_reply.c:1804 #2 0x00007f6856754ac3 in reply_received (p_msg=0x7f685b6cafa0) at t_reply.c:2563 #3 0x00000000005291a9 in do_forward_reply (msg=0x7f685b6cafa0, mode=0) at core/forward.c:747 #4 0x000000000052ad21 in forward_reply (msg=0x7f685b6cafa0) at core/forward.c:852 #5 0x000000000059e89e in receive_msg ( buf=0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len=565, rcv_info=0x7fffed6c3c30) at core/receive.c:433 #6 0x0000000000481690 in udp_rcv_loop () at core/udp_server.c:541 #7 0x0000000000424a27 in main_loop () at main.c:1621 #8 0x000000000042c078 in main (argc=9, argv=0x7fffed6c4178) at main.c:2645 (gdb) list 1274 /* except the exception above, too late messages will be discarded */ 1275 goto discard; 1276 } 1277 1278 /* if final response received at this branch, allow only INVITE 2xx */ 1279 if (Trans->uac[branch].last_received>=200 1280 && !(inv_through && Trans->uac[branch].last_received<300)) { 1281 /* don't report on retransmissions */ 1282 if (Trans->uac[branch].last_received==new_code) { 1283 LM_DBG("final reply retransmission\n"); 1284 goto discard; (gdb) info locals branch_cnt = 32767 picked_code = 0 new_branch = -311676496 inv_through = 0 extra_flags = 0 i = 32616 replies_dropped = 1143247665 __FUNCTION__ = "t_should_relay_response" (gdb) bt full #0 0x00007f685674a84d in t_should_relay_response (Trans=0x7f683d86a3e0, new_code=503, branch=0, should_store=0x7fffed6c323c, should_relay=0x7fffed6c3240, cancel_data=0x7fffed6c3490, reply=0x7f685b6cafa0) at t_reply.c:1279 branch_cnt = 32767 picked_code = 0 new_branch = -311676496 inv_through = 0 extra_flags = 0 i = 32616 replies_dropped = 1143247665 __FUNCTION__ = "t_should_relay_response" #1 0x00007f685674ee6b in relay_reply (t=0x7f683d86a3e0, p_msg=0x7f685b6cafa0, branch=0, msg_status=503, cancel_data=0x7fffed6c3490, do_put_on_wait=1) at t_reply.c:1804 relay = -311676288 save_clone = 0 buf = 0x0 res_len = 0 relayed_code = 0 relayed_msg = 0x0 reply_bak = 0x7fffed6c3290 bm = {to_tag_val = {s = 0x1 <Address 0x1 out of bounds>, len = 1032234408}} totag_retr = 0 reply_status = RPS_ERROR uas_rb = 0x68924f <futex_release+29> to_tag = 0x7f685e8ffed0 <__syslog> reason = {s = 0x0, len = 0} onsend_params = {req = 0x7f685b6cafa0, rpl = 0x7f685b597870, param = 0x7f68567e0c50, code = 0, flags = 2288, branch = 0, t_rbuf = 0x7f68567e6edb <__FUNCTION__.12468>, dst = 0x7f68567e23ab, send_buf = { s = 0x7f68440006f0 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP X.X.X.35:5060;received=X.X.X.35;TH=div;rport=5060;branch=z9hG4bK-97cc84683b6011e980ec0cc47a0ad35a;sig=7970278d\r\nVia: SIP/2.0/UDP X.X.X.35:"..., len = 841211904}} ip = {af = 3983291088, len = 32767, u = {addrl = {5867152, 11367808}, addr32 = {5867152, 0, 11367808, 0}, addr16 = {34448, 89, 0, 0, 30080, 173, 0, 0}, addr = "\220\206Y\000\000\000\000\000\200u\255\000\000\000\000"}} __FUNCTION__ = "relay_reply" #2 0x00007f6856754ac3 in reply_received (p_msg=0x7f685b6cafa0) at t_reply.c:2563 msg_status = 503 last_uac_status = 408 ack = 0x7f68440006f0 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP X.X.X.35:5060;received=X.X.X.35;TH=div;rport=5060;branch=z9hG4bK-97cc84683b6011e980ec0cc47a0ad35a;sig=7970278d\r\nVia: SIP/2.0/UDP X.X.X.35:"... ack_len = 406 branch = 0 reply_status = 67108864 onreply_route = 3 cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 1586495184}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 1586495184}}}} uac = 0x7f683d86a5f0 t = 0x7f683d86a3e0 lack_dst = {send_sock = 0x20000000, to = {s = {sa_family = 0, sa_data = "\000\004\000\000\000\000p5l\355\377\177\000"}, sin = {sin_family = 0, sin_port = 1024, sin_addr = {s_addr = 0}, sin_zero = "p5l\355\377\177\000"}, sin6 = {sin6_family = 0, sin6_port = 1024, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "p5l\355\377\177\000\000\201\020\342Uh\177\000", __u6_addr16 = {13680, 60780, 32767, 0, 4225, 21986, 32616, 0}, __u6_addr32 = {3983291760, 32767, 1440878721, 32616}}}, sin6_scope_id = 11693840}}, id = 0, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}} backup_user_from = 0xad7390 <def_list+16> backup_user_to = 0xad7398 <def_list+24> backup_domain_from = 0xad73a0 <def_list+32> backup_domain_to = 0xad73a8 <def_list+40> backup_uri_from = 0xad7380 <def_list> backup_uri_to = 0xad7388 <def_list+8> backup_xavps = 0xa6b250 <_xavp_list_head> replies_locked = 1 branch_ret = 536870912 prev_branch = 0 blst_503_timeout = 90 hf = 0x7c7213 onsend_params = {req = 0x7f685b3af890, rpl = 0xa6c4d5 <buf.6868+565>, param = 0x0, code = 10929363, flags = 0, branch = 0, t_rbuf = 0xa6c4d5 <buf.6868+565>, dst = 0x7f685e8ffed0 <__syslog>, send_buf = {s = 0x7c7213 "INFO", len = 90}} ctx = {rec_lev = 0, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {140086239821520, -3528413356315156285, 8155667, 90, 536870912, 67108864, -3528413356269018941, 3528382109948141763}, __mask_was_saved = 0, __saved_mask = {__val = {67108864, 140737176679536, 7233135, 140737176679600, 6865256, 140086183963912, 140086183963896, 15482808, 8, 18446744073709551615, 140086187175840, 8155667, 90, 140737176679668, 140737176679672, 140086183985368}}}}} bctx = 0x7f685b6cafa0 keng = 0x0 __FUNCTION__ = "reply_received" #3 0x00000000005291a9 in do_forward_reply (msg=0x7f685b6cafa0, mode=0) at core/forward.c:747 new_buf = 0x0 dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}} new_len = 0 r = 1 ip = {af = 3983291952, len = 32767, u = {addrl = {140086109025369, 8278245420}, addr32 = {1455699033, 32616, 3983278124, 1}, addr16 = {13401, 22212, 32616, 0, 44, 60780, 1, 0}, addr = "Y4\304Vh\177\000\000,\000l\355\001\000\000"}} s = 0x6f8 <Address 0x6f8 out of bounds> len = 0 __FUNCTION__ = "do_forward_reply" #4 0x000000000052ad21 in forward_reply (msg=0x7f685b6cafa0) at core/forward.c:852 No locals. #5 0x000000000059e89e in receive_msg ( buf=0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len=565, rcv_info=0x7fffed6c3c30) at core/receive.c:433 msg = 0x7f685b6cafa0 ctx = {rec_lev = 70, run_flags = 0, last_retcode = 8, jmp_env = {{__jmpbuf = {2314885530818453536, 2314885530818453536, 0, 0, 0, 0, 0, 0}, __mask_was_saved = 0, __saved_mask = {__val = {65280, 1099807431579733077, 3183208866038363301, 17755792806188831655, 4262298206711962970, 11302123416292502927, 212649150131803653, 8808588017067063289, 17812784738947474131, 170888629502, 10994336, 140737176681528, 0, 55834574851, 140086239821520, 8155667}}}}} bctx = 0x3b26b9a2e9d9dd5a ret = 0 stats_on = 0 tvb = {tv_sec = 9, tv_usec = 26651291} tve = {tv_sec = -16777216, tv_usec = 0} tz = {tz_minuteswest = 0, tz_dsttime = 0} diff = 0 inb = {s = 0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len = 565} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0} keng = 0x0 evp = {data = 0x7fffed6c37b0, rcv = 0x7fffed6c3c30, dst = 0x0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 __FUNCTION__ = "receive_msg" #6 0x0000000000481690 in udp_rcv_loop () at core/udp_server.c:541 len = 628 buf = "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"... tmp = 0xa7c2a0 <buff.5595> "177.99.230.99" from = 0x7f685b3bdba8 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {39569941286, 140737176681744}, addr32 = {915235622, 9, 3983293712, 32767}, addr16 = {25382, 13965, 9, 0, 15632, 60780, 32767, 0}, addr = "&c\215\066\t\000\000\000\020=l\355\377\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {646919601, 0}, addr32 = {646919601, 0, 0, 0}, addr16 = {13745, 9871, 0, 0, 0, 0, 0, 0}, addr = "\261\065\217&", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = { sa_family = 2, sa_data = "\023\304&c\215\066\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 915235622}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 915235622, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f685a89bb28, proto = 1 '\001'} evp = {data = 0x0, rcv = 0x0, dst = 0x0} printbuf = "\023r|\000\b\000\000\000\326*\250Rh\177\000\000\000\000\000 \000\000\000\000\020\266\215\001\000\000\000\000\000\000\000\000\002\000\000\000\002", '\000' <repeats 39 times>, "\300\214\001\062h\177\000\000\210V\213Z(\a\000\000\336=\002\062h\177\000\000\373\252\001\062h\177\000\000\360", '\000' <repeats 11 times>, "h\177\000\000\250\333;[h\177\000\000\001\000\000\000\003", '\000' <repeats 19 times>, "\266\333;[h\177\000\000\020p\200Z\000\000\000\000\177\000\000\000\000\000\000\000\204\020\204\062h\177\000\000\023r|\000\000\000\000\000"... i = -1 j = 4579542 l = 258781216 __FUNCTION__ = "udp_rcv_loop" #7 0x0000000000424a27 in main_loop () at main.c:1621 i = 8 pid = 0 si = 0x7f685a89bb28 si_desc = "udp receiver child=8 sock=X.X.X.38:5060\000\177\000\000\300=l\355\377\177\000\000\320\376\217^h\177\000\000\220@l\355\377\177\000\000\023r|\000\000\000\000\000Z\000\000\000\000\000\000\000\000\000\000 \000\000\000\000\000\000\000\004\000\000\000\000_\377\217^h\177\000\000\200bx\000\000\000\000\000@\032F[h\177\000" nrprocs = 32 woneinit = 1 __FUNCTION__ = "main_loop" #8 0x000000000042c078 in main (argc=9, argv=0x7fffed6c4178) at main.c:2645 cfg_stream = 0x16f8010 c = -1 r = 0 tmp = 0x7fffed6c5f66 "" tmp_len = 0 port = 0 proto = 0 options = 0x768aa0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1705952777 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0x0 st = {st_dev = 20, st_ino = 32456, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 2, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1551072083, tv_nsec = 812037328}, st_mtim = {tv_sec = 1551360053, tv_nsec = 958825627}, st_ctim = {tv_sec = 1551360053, tv_nsec = 958825627}, __unused = {0, 0, 0}} __FUNCTION__ = "main" ======================================================================= (gdb) p Trans $1 = (struct cell *) 0x7f683d86a3e0 (gdb) p *Trans $2 = {next_c = 0x7f68323e7b50, prev_c = 0x7f68323e7b50, hash_index = 36580, label = 151969688, flags = 321, nr_of_outgoings = 1, fcount = 0, ref_count = {val = 1}, from = { s = 0x7f683558283b "From: \"6654588489\" <sip:6654588489@X.X.X.28>;tag=11-C6F1DC43\r\nMax-Forwards: 67\r\nSession-ID: 4d22ebbbfcedf4c1da9aff86b1c1b1ae\r\nSupported: 100rel,timer,replaces\r\nTo: <sip:5566996495302@X.X.X.28:"..., len = 66}, callid = { s = 0x7f68355827b3 "Call-ID: 9BkwYi6R3gjI6VRy(a)X.X.X.28\r\nContact: <sip:X.X.X.19;did=e5b.89a103e>\r\nContent-Type: application/sdp\r\nCSeq: 19054 INVITE\r\nFrom: \"6654588489\" <sip:6654588489@X.X.X.28>;tag=11-C6F1DC43"..., len = 40}, cseq_n = { s = 0x7f6835582827 "CSeq: 19054 INVITE\r\nFrom: \"6654588489\" <sip:6654588489@X.X.X.28>;tag=11-C6F1DC43\r\nMax-Forwards: 67\r\nSession-ID: 4d22ebbbfcedf4c1da9aff86b1c1b1ae\r\nSupported: 100rel,timer,replaces\r\nTo: <sip:5566996"..., len = 11}, to = { s = 0x7f68355828df "To: <sip:5566996495302@X.X.X.28:5060>\r\nVia: SIP/2.0/UDP X.X.X.19:5060;TH=div;branch=z9hG4bK4ee8.13eb8d24.0\r\nContent-Length: 294\r\nTH: dih\r\n\r\nv=0\r\no=- 1012292 0 IN IP4 177.220.255.123\r\ns=-\r\nc=IN"..., len = 43}, method = { s = 0x7f6835582780 "INVITE sip:11#5566996495302@X.X.X.38 SIP/2.0\r\nCall-ID: 9BkwYi6R3gjI6VRy(a)X.X.X.28\r\nContact: <sip:X.X.X.19;did=e5b.89a103e>\r\nContent-Type: application/sdp\r\nCSeq: 19054 INVITE\r\nFrom: \"665458"..., len = 6}, tmcb_hl = { first = 0x7f68422880a8, reg_types = 1048738}, wait_timer = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 0, data = 0x7f683d86a3e0, f = 0x7f685679d50b <wait_handler>, flags = 1, slow_idx = 0}, uas = {request = 0x7f6835582060, end_request = 0x7f6835583318 "\300\300\300\300", response = {rbtype = 100, flags = 0, t_active = 0, branch = 0, buffer_len = 334, buffer = 0x7f683b50fc38 "SIP/2.0 100 Trying\r\nCall-ID: 9BkwYi6R3gjI6VRy(a)X.X.X.28\r\nCSeq: 19054 INVITE\r\nFrom: \"6654588489\" <sip:6654588489@X.X.X.28>;tag=11-C6F1DC43\r\nTo: <sip:5566996495302@X.X.X.28:5060>\r\nVia: SIP/2."..., my_T = 0x7f683d86a3e0, timer = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 0, data = 0x0, f = 0x7f685679cfda <retr_buf_handler>, flags = 0, slow_idx = 0}, dst = {send_sock = 0x7f685a89bb28, to = {s = {sa_family = 2, sa_data = "\023\304\310GM\023\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 323831752}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 323831752, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 1 '\001', send_flags = {f = 0, blst_imask = 0}}, retr_expire = 0, fr_expire = 0}, local_totag = {s = 0x7f683b50fcf3 "\r\nVia: SIP/2.0/UDP X.X.X.19:5060;TH=div;branch=z9hG4bK4ee8.13eb8d24.0;rport=5060\r\nServer: NextRouter SoftSWITCH Pro v4.1\r\nContent-Length: 0\r\n\r\n956a5b272923f34-7a86\r\nCall-ID: 2ff99aa726a3a3082660e2"..., len = 0}, cancel_reas = 0x0, status = 100}, uac = 0x7f683d86a5f0, async_backup = {backup_route = 0, backup_branch = 0, blind_uac = 0, ruri_new = 0}, fwded_totags = 0x0, uri_avps_from = 0x7f6834c993b0, uri_avps_to = 0x0, user_avps_from = 0x0, user_avps_to = 0x0, domain_avps_from = 0x0, domain_avps_to = 0x0, xavps_list = 0x0, reply_mutex = {val = 0}, reply_locker_pid = {val = 0}, reply_rec_lock_level = 0, fr_timeout = 32, fr_inv_timeout = 112, rt_t1_timeout_ms = 500, rt_t2_timeout_ms = 4000, end_of_life = 474292144, relayed_reply_branch = -2, on_failure = 3, on_branch_failure = 0, on_reply = 3, on_branch = 0, on_branch_delayed = 3, md5 = 0x7f683d86a5d0 "9bc5c0acdc87d6654d5368cad5906e3e"} (gdb) p *should_store $3 = 0 (gdb) p should_store $4 = (int *) 0x7fffed6c323c (gdb) p cancel_data $5 = (struct cancel_info *) 0x7fffed6c3490 (gdb) p *cancel_data $6 = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 1586495184}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 1586495184}}}} (gdb) p *reply $7 = {id = 55770, pid = 121295, tval = {tv_sec = 1551362168, tv_usec = 692071}, fwd_send_flags = {f = 0, blst_imask = 0}, rpl_send_flags = {f = 0, blst_imask = 0}, first_line = {type = 2, flags = 1, len = 33, u = {request = {method = { s = 0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len = 7}, uri = { s = 0xa6c2a8 <buf.6868+8> "503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;branch=z9hG"..., len = 3}, version = { s = 0xa6c2ac <buf.6868+12> "Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;branch=z9hG4bK4"..., len = 19}, method_value = 503}, reply = {version = { s = 0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len = 7}, status = { s = 0xa6c2a8 <buf.6868+8> "503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;branch=z9hG"..., len = 3}, reason = { s = 0xa6c2ac <buf.6868+12> "Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;branch=z9hG4bK4"..., len = 19}, statuscode = 503}}}, via1 = 0x7f685b3af740, via2 = 0x7f685b3af9e0, headers = 0x7f685b3b1ec8, last_header = 0x7f685b3adf68, parsed_flag = 18446744073709551615, h_via1 = 0x7f685b3b1ec8, h_via2 = 0x7f685b3bbf98, callid = 0x7f685b3b9c50, to = 0x7f685b3b9ba8, cseq = 0x7f685b3adec0, from = 0x7f685b3b2f28, contact = 0x0, maxforwards = 0x0, route = 0x0, record_route = 0x0, content_type = 0x0, content_length = 0x7f685b3b7468, authorization = 0x0, expires = 0x0, proxy_auth = 0x0, supported = 0x0, require = 0x0, proxy_require = 0x0, unsupported = 0x0, allow = 0x0, event = 0x0, accept = 0x0, accept_language = 0x0, organization = 0x0, priority = 0x0, subject = 0x0, user_agent = 0x0, server = 0x7f685b3bdf70, content_disposition = 0x0, diversion = 0x0, rpid = 0x0, refer_to = 0x0, session_expires = 0x0, min_se = 0x0, sipifmatch = 0x0, subscription_state = 0x0, date = 0x0, identity = 0x0, identity_info = 0x0, pai = 0x0, ppi = 0x0, path = 0x0, privacy = 0x0, min_expires = 0x0, body = 0x0, eoh = 0xa6c4d3 <buf.6868+563> "\r\n", unparsed = 0xa6c4d3 <buf.6868+563> "\r\n", rcv = {src_ip = {af = 2, len = 4, u = {addrl = {39569941286, 140737176681744}, addr32 = {915235622, 9, 3983293712, 32767}, addr16 = {25382, 13965, 9, 0, 15632, 60780, 32767, 0}, addr = "&c\215\066\t\000\000\000\020=l\355\377\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {646919601, 0}, addr32 = {646919601, 0, 0, 0}, addr16 = {13745, 9871, 0, 0, 0, 0, 0, 0}, addr = "\261\065\217&", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304&c\215\066\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 915235622}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 915235622, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f685a89bb28, proto = 1 '\001'}, buf = 0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len = 565, new_uri = {s = 0x0, len = 0}, dst_uri = {s = 0x0, len = 0}, parsed_uri_ok = 0, parsed_uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}}, parsed_orig_ruri_ok = 0, parsed_orig_ruri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}}, add_rm = 0x7f685b3abb88, body_lumps = 0x0, reply_lump = 0x0, add_to_branch_s = '\000' <repeats 57 times>, add_to_branch_len = 0, hash_index = 0, msg_flags = 32768, flags = 2692743168, xflags = {0, 0}, set_global_address = {s = 0x0, len = 0}, set_global_port = {s = 0x0, len = 0}, force_send_socket = 0x0, path_vec = {s = 0x0, len = 0}, instance = {s = 0x0, len = 0}, reg_id = 0, ruid = {s = 0x0, len = 0}, location_ua = {s = 0x0, len = 0}, ldv = {flow = {decoded = 0, rcv = {src_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}}, dst_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}}, src_port = 0, dst_port = 0, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x0, proto = 0 '\000'}}}} ``` #### Log Messages <!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> No logs available. ``` [29069.581044] kamailio[12322]: segfault at 190 ip 00007f5985ad884d sp 00007ffcea744f00 error 4 in tm.so[7f5985a61000+135000] [49907.558580] kamailio[84693]: segfault at 1a0 ip 00007fc37b75ef90 sp 00007fff954b5880 error 4 in tm.so[7fc37b742000+135000] [86413.184058] sh (123233): drop_caches: 3 [172811.546070] sh (99955): drop_caches: 3 [201459.115194] kamailio[106801]: segfault at 190 ip 00007f3528cac84d sp 00007ffdf8d27c50 error 4 in tm.so[7f3528c35000+135000] [259248.603313] sh (34334): drop_caches: 3 [287619.734155] kamailio[40824]: segfault at 190 ip 00007f03ce01984d sp 00007ffe869650f0 error 4 in tm.so[7f03cdfa2000+135000] [287870.066030] kamailio[119402]: segfault at 1a0 ip 00007fbf4ff34f90 sp 00007ffd1f8cc190 error 4 in tm.so[7fbf4ff18000+135000] [287952.567590] kamailio[120541]: segfault at 190 ip 00007f20f616684d sp 00007ffc9c760630 error 4 in tm.so[7f20f60ef000+135000] [288012.343497] kamailio[120947]: segfault at 340 ip 00007fa90abac84d sp 00007ffe8ca01160 error 4 in tm.so[7fa90ab35000+135000] [290128.570063] kamailio[121295]: segfault at 190 ip 00007f685674a84d sp 00007fffed6c3050 error 4 in tm.so[7f68566d3000+135000] ``` #### SIP Traffic <!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> No SIP traffic available. ``` (paste your sip traffic here) ``` ### Possible Solutions <!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. --> ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.2.0 (x86_64/linux) 535e13 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 535e13 compiled on 22:16:55 Feb 20 2019 with gcc 4.8.5 ``` * **Operating System**: <!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) --> ``` LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch Distributor ID: CentOS Description: CentOS Linux release 7.6.1810 (Core) Release: 7.6.1810 Codename: Core ----------------- Linux sip 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1875