<h3>Description</h3>
<p>I have a segfault in an average of 3 to 6 per day on tm module always on the t_should_relay_response.<br>
I'm using kamailio 5.2.0 (x86_64/linux) 535e13 on CentOS Linux release 7.6.1810 (Core) x86_64 runing on a XenServer.</p>
<h3>Troubleshooting</h3>
<p>No troubleshooting was done, since it happened on a production server. We simply restarted the server.</p>
<h4>Reproduction</h4>
<p>The problem is random and has happened a couple of times per day. My kamailio uses tm, dialog, htable. All calls is using topoh. The server run with an average of 1000-1200 concurrent calls. I've seen this segfault happen with less than 400 concurrent calls too.<br>
One very curious thing, when the fifth or sixth time the segfault occurs, it does not happen again, even reaching 3000 concurrent calls.</p>
<h4>Debugging Data</h4>
<pre><code>(gdb) bt
#0 0x00007f685674a84d in t_should_relay_response (Trans=0x7f683d86a3e0, new_code=503, branch=0, should_store=0x7fffed6c323c, should_relay=0x7fffed6c3240, cancel_data=0x7fffed6c3490, reply=0x7f685b6cafa0) at t_reply.c:1279
#1 0x00007f685674ee6b in relay_reply (t=0x7f683d86a3e0, p_msg=0x7f685b6cafa0, branch=0, msg_status=503, cancel_data=0x7fffed6c3490, do_put_on_wait=1) at t_reply.c:1804
#2 0x00007f6856754ac3 in reply_received (p_msg=0x7f685b6cafa0) at t_reply.c:2563
#3 0x00000000005291a9 in do_forward_reply (msg=0x7f685b6cafa0, mode=0) at core/forward.c:747
#4 0x000000000052ad21 in forward_reply (msg=0x7f685b6cafa0) at core/forward.c:852
#5 0x000000000059e89e in receive_msg (
buf=0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len=565,
rcv_info=0x7fffed6c3c30) at core/receive.c:433
#6 0x0000000000481690 in udp_rcv_loop () at core/udp_server.c:541
#7 0x0000000000424a27 in main_loop () at main.c:1621
#8 0x000000000042c078 in main (argc=9, argv=0x7fffed6c4178) at main.c:2645
(gdb) list
1274 /* except the exception above, too late messages will be discarded */
1275 goto discard;
1276 }
1277
1278 /* if final response received at this branch, allow only INVITE 2xx */
1279 if (Trans->uac[branch].last_received>=200
1280 && !(inv_through && Trans->uac[branch].last_received<300)) {
1281 /* don't report on retransmissions */
1282 if (Trans->uac[branch].last_received==new_code) {
1283 LM_DBG("final reply retransmission\n");
1284 goto discard;
(gdb) info locals
branch_cnt = 32767
picked_code = 0
new_branch = -311676496
inv_through = 0
extra_flags = 0
i = 32616
replies_dropped = 1143247665
__FUNCTION__ = "t_should_relay_response"
(gdb) bt full
#0 0x00007f685674a84d in t_should_relay_response (Trans=0x7f683d86a3e0, new_code=503, branch=0, should_store=0x7fffed6c323c, should_relay=0x7fffed6c3240, cancel_data=0x7fffed6c3490, reply=0x7f685b6cafa0) at t_reply.c:1279
branch_cnt = 32767
picked_code = 0
new_branch = -311676496
inv_through = 0
extra_flags = 0
i = 32616
replies_dropped = 1143247665
__FUNCTION__ = "t_should_relay_response"
#1 0x00007f685674ee6b in relay_reply (t=0x7f683d86a3e0, p_msg=0x7f685b6cafa0, branch=0, msg_status=503, cancel_data=0x7fffed6c3490, do_put_on_wait=1) at t_reply.c:1804
relay = -311676288
save_clone = 0
buf = 0x0
res_len = 0
relayed_code = 0
relayed_msg = 0x0
reply_bak = 0x7fffed6c3290
bm = {to_tag_val = {s = 0x1 <Address 0x1 out of bounds>, len = 1032234408}}
totag_retr = 0
reply_status = RPS_ERROR
uas_rb = 0x68924f <futex_release+29>
to_tag = 0x7f685e8ffed0 <__syslog>
reason = {s = 0x0, len = 0}
onsend_params = {req = 0x7f685b6cafa0, rpl = 0x7f685b597870, param = 0x7f68567e0c50, code = 0, flags = 2288, branch = 0, t_rbuf = 0x7f68567e6edb <__FUNCTION__.12468>, dst = 0x7f68567e23ab, send_buf = {
s = 0x7f68440006f0 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP X.X.X.35:5060;received=X.X.X.35;TH=div;rport=5060;branch=z9hG4bK-97cc84683b6011e980ec0cc47a0ad35a;sig=7970278d\r\nVia: SIP/2.0/UDP X.X.X.35:"..., len = 841211904}}
ip = {af = 3983291088, len = 32767, u = {addrl = {5867152, 11367808}, addr32 = {5867152, 0, 11367808, 0}, addr16 = {34448, 89, 0, 0, 30080, 173, 0, 0}, addr = "\220\206Y\000\000\000\000\000\200u\255\000\000\000\000"}}
__FUNCTION__ = "relay_reply"
#2 0x00007f6856754ac3 in reply_received (p_msg=0x7f685b6cafa0) at t_reply.c:2563
msg_status = 503
last_uac_status = 408
ack = 0x7f68440006f0 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP X.X.X.35:5060;received=X.X.X.35;TH=div;rport=5060;branch=z9hG4bK-97cc84683b6011e980ec0cc47a0ad35a;sig=7970278d\r\nVia: SIP/2.0/UDP X.X.X.35:"...
ack_len = 406
branch = 0
reply_status = 67108864
onreply_route = 3
cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 1586495184}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 1586495184}}}}
uac = 0x7f683d86a5f0
t = 0x7f683d86a3e0
lack_dst = {send_sock = 0x20000000, to = {s = {sa_family = 0, sa_data = "\000\004\000\000\000\000p5l\355\377\177\000"}, sin = {sin_family = 0, sin_port = 1024, sin_addr = {s_addr = 0}, sin_zero = "p5l\355\377\177\000"}, sin6 = {sin6_family = 0, sin6_port = 1024,
sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "p5l\355\377\177\000\000\201\020\342Uh\177\000", __u6_addr16 = {13680, 60780, 32767, 0, 4225, 21986, 32616, 0}, __u6_addr32 = {3983291760, 32767, 1440878721, 32616}}}, sin6_scope_id = 11693840}},
id = 0, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}}
backup_user_from = 0xad7390 <def_list+16>
backup_user_to = 0xad7398 <def_list+24>
backup_domain_from = 0xad73a0 <def_list+32>
backup_domain_to = 0xad73a8 <def_list+40>
backup_uri_from = 0xad7380 <def_list>
backup_uri_to = 0xad7388 <def_list+8>
backup_xavps = 0xa6b250 <_xavp_list_head>
replies_locked = 1
branch_ret = 536870912
prev_branch = 0
blst_503_timeout = 90
hf = 0x7c7213
onsend_params = {req = 0x7f685b3af890, rpl = 0xa6c4d5 <buf.6868+565>, param = 0x0, code = 10929363, flags = 0, branch = 0, t_rbuf = 0xa6c4d5 <buf.6868+565>, dst = 0x7f685e8ffed0 <__syslog>, send_buf = {s = 0x7c7213 "INFO", len = 90}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {140086239821520, -3528413356315156285, 8155667, 90, 536870912, 67108864, -3528413356269018941, 3528382109948141763}, __mask_was_saved = 0, __saved_mask = {__val = {67108864,
140737176679536, 7233135, 140737176679600, 6865256, 140086183963912, 140086183963896, 15482808, 8, 18446744073709551615, 140086187175840, 8155667, 90, 140737176679668, 140737176679672, 140086183985368}}}}}
bctx = 0x7f685b6cafa0
keng = 0x0
__FUNCTION__ = "reply_received"
#3 0x00000000005291a9 in do_forward_reply (msg=0x7f685b6cafa0, mode=0) at core/forward.c:747
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}}
new_len = 0
r = 1
ip = {af = 3983291952, len = 32767, u = {addrl = {140086109025369, 8278245420}, addr32 = {1455699033, 32616, 3983278124, 1}, addr16 = {13401, 22212, 32616, 0, 44, 60780, 1, 0}, addr = "Y4\304Vh\177\000\000,\000l\355\001\000\000"}}
s = 0x6f8 <Address 0x6f8 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#4 0x000000000052ad21 in forward_reply (msg=0x7f685b6cafa0) at core/forward.c:852
No locals.
#5 0x000000000059e89e in receive_msg (
buf=0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len=565,
rcv_info=0x7fffed6c3c30) at core/receive.c:433
msg = 0x7f685b6cafa0
ctx = {rec_lev = 70, run_flags = 0, last_retcode = 8, jmp_env = {{__jmpbuf = {2314885530818453536, 2314885530818453536, 0, 0, 0, 0, 0, 0}, __mask_was_saved = 0, __saved_mask = {__val = {65280, 1099807431579733077, 3183208866038363301, 17755792806188831655,
4262298206711962970, 11302123416292502927, 212649150131803653, 8808588017067063289, 17812784738947474131, 170888629502, 10994336, 140737176681528, 0, 55834574851, 140086239821520, 8155667}}}}}
bctx = 0x3b26b9a2e9d9dd5a
ret = 0
stats_on = 0
tvb = {tv_sec = 9, tv_usec = 26651291}
tve = {tv_sec = -16777216, tv_usec = 0}
tz = {tz_minuteswest = 0, tz_dsttime = 0}
diff = 0
inb = {s = 0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len = 565}
netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0}
keng = 0x0
evp = {data = 0x7fffed6c37b0, rcv = 0x7fffed6c3c30, dst = 0x0}
cidlockidx = 0
cidlockset = 0
errsipmsg = 0
__FUNCTION__ = "receive_msg"
#6 0x0000000000481690 in udp_rcv_loop () at core/udp_server.c:541
len = 628
buf = "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"...
tmp = 0xa7c2a0 <buff.5595> "177.99.230.99"
from = 0x7f685b3bdba8
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {39569941286, 140737176681744}, addr32 = {915235622, 9, 3983293712, 32767}, addr16 = {25382, 13965, 9, 0, 15632, 60780, 32767, 0}, addr = "&c\215\066\t\000\000\000\020=l\355\377\177\000"}}, dst_ip = {af = 2, len = 4,
u = {addrl = {646919601, 0}, addr32 = {646919601, 0, 0, 0}, addr16 = {13745, 9871, 0, 0, 0, 0, 0, 0}, addr = "\261\065\217&", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data = "\023\304&c\215\066\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 915235622}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 915235622, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f685a89bb28, proto = 1 '\001'}
evp = {data = 0x0, rcv = 0x0, dst = 0x0}
printbuf = "\023r|\000\b\000\000\000\326*\250Rh\177\000\000\000\000\000 \000\000\000\000\020\266\215\001\000\000\000\000\000\000\000\000\002\000\000\000\002", '\000' <repeats 39 times>, "\300\214\001\062h\177\000\000\210V\213Z(\a\000\000\336=\002\062h\177\000\000\373\252\001\062h\177\000\000\360", '\000' <repeats 11 times>, "h\177\000\000\250\333;[h\177\000\000\001\000\000\000\003", '\000' <repeats 19 times>, "\266\333;[h\177\000\000\020p\200Z\000\000\000\000\177\000\000\000\000\000\000\000\204\020\204\062h\177\000\000\023r|\000\000\000\000\000"...
i = -1
j = 4579542
l = 258781216
__FUNCTION__ = "udp_rcv_loop"
#7 0x0000000000424a27 in main_loop () at main.c:1621
i = 8
pid = 0
si = 0x7f685a89bb28
si_desc = "udp receiver child=8 sock=X.X.X.38:5060\000\177\000\000\300=l\355\377\177\000\000\320\376\217^h\177\000\000\220@l\355\377\177\000\000\023r|\000\000\000\000\000Z\000\000\000\000\000\000\000\000\000\000 \000\000\000\000\000\000\000\004\000\000\000\000_\377\217^h\177\000\000\200bx\000\000\000\000\000@\032F[h\177\000"
nrprocs = 32
woneinit = 1
__FUNCTION__ = "main_loop"
#8 0x000000000042c078 in main (argc=9, argv=0x7fffed6c4178) at main.c:2645
cfg_stream = 0x16f8010
c = -1
r = 0
tmp = 0x7fffed6c5f66 ""
tmp_len = 0
port = 0
proto = 0
options = 0x768aa0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 1705952777
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x0
p = 0x0
st = {st_dev = 20, st_ino = 32456, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 2, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1551072083, tv_nsec = 812037328}, st_mtim = {tv_sec = 1551360053,
tv_nsec = 958825627}, st_ctim = {tv_sec = 1551360053, tv_nsec = 958825627}, __unused = {0, 0, 0}}
__FUNCTION__ = "main"
=======================================================================
(gdb) p Trans
$1 = (struct cell *) 0x7f683d86a3e0
(gdb) p *Trans
$2 = {next_c = 0x7f68323e7b50, prev_c = 0x7f68323e7b50, hash_index = 36580, label = 151969688, flags = 321, nr_of_outgoings = 1, fcount = 0, ref_count = {val = 1}, from = {
s = 0x7f683558283b "From: \"6654588489\" <sip:6654588489@X.X.X.28>;tag=11-C6F1DC43\r\nMax-Forwards: 67\r\nSession-ID: 4d22ebbbfcedf4c1da9aff86b1c1b1ae\r\nSupported: 100rel,timer,replaces\r\nTo: <sip:5566996495302@X.X.X.28:"..., len = 66}, callid = {
s = 0x7f68355827b3 "Call-ID: 9BkwYi6R3gjI6VRy@X.X.X.28\r\nContact: <sip:X.X.X.19;did=e5b.89a103e>\r\nContent-Type: application/sdp\r\nCSeq: 19054 INVITE\r\nFrom: \"6654588489\" <sip:6654588489@X.X.X.28>;tag=11-C6F1DC43"..., len = 40}, cseq_n = {
s = 0x7f6835582827 "CSeq: 19054 INVITE\r\nFrom: \"6654588489\" <sip:6654588489@X.X.X.28>;tag=11-C6F1DC43\r\nMax-Forwards: 67\r\nSession-ID: 4d22ebbbfcedf4c1da9aff86b1c1b1ae\r\nSupported: 100rel,timer,replaces\r\nTo: <sip:5566996"..., len = 11}, to = {
s = 0x7f68355828df "To: <sip:5566996495302@X.X.X.28:5060>\r\nVia: SIP/2.0/UDP X.X.X.19:5060;TH=div;branch=z9hG4bK4ee8.13eb8d24.0\r\nContent-Length: 294\r\nTH: dih\r\n\r\nv=0\r\no=- 1012292 0 IN IP4 177.220.255.123\r\ns=-\r\nc=IN"..., len = 43}, method = {
s = 0x7f6835582780 "INVITE sip:11#5566996495302@X.X.X.38 SIP/2.0\r\nCall-ID: 9BkwYi6R3gjI6VRy@X.X.X.28\r\nContact: <sip:X.X.X.19;did=e5b.89a103e>\r\nContent-Type: application/sdp\r\nCSeq: 19054 INVITE\r\nFrom: \"665458"..., len = 6}, tmcb_hl = {
first = 0x7f68422880a8, reg_types = 1048738}, wait_timer = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 0, data = 0x7f683d86a3e0, f = 0x7f685679d50b <wait_handler>, flags = 1, slow_idx = 0}, uas = {request = 0x7f6835582060,
end_request = 0x7f6835583318 "\300\300\300\300", response = {rbtype = 100, flags = 0, t_active = 0, branch = 0, buffer_len = 334,
buffer = 0x7f683b50fc38 "SIP/2.0 100 Trying\r\nCall-ID: 9BkwYi6R3gjI6VRy@X.X.X.28\r\nCSeq: 19054 INVITE\r\nFrom: \"6654588489\" <sip:6654588489@X.X.X.28>;tag=11-C6F1DC43\r\nTo: <sip:5566996495302@X.X.X.28:5060>\r\nVia: SIP/2."...,
my_T = 0x7f683d86a3e0, timer = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 0, data = 0x0, f = 0x7f685679cfda <retr_buf_handler>, flags = 0, slow_idx = 0}, dst = {send_sock = 0x7f685a89bb28, to = {s = {sa_family = 2,
sa_data = "\023\304\310GM\023\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 323831752}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 323831752,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 1 '\001', send_flags = {f = 0, blst_imask = 0}}, retr_expire = 0, fr_expire = 0},
local_totag = {s = 0x7f683b50fcf3 "\r\nVia: SIP/2.0/UDP X.X.X.19:5060;TH=div;branch=z9hG4bK4ee8.13eb8d24.0;rport=5060\r\nServer: NextRouter SoftSWITCH Pro v4.1\r\nContent-Length: 0\r\n\r\n956a5b272923f34-7a86\r\nCall-ID: 2ff99aa726a3a3082660e2"..., len = 0},
cancel_reas = 0x0, status = 100}, uac = 0x7f683d86a5f0, async_backup = {backup_route = 0, backup_branch = 0, blind_uac = 0, ruri_new = 0}, fwded_totags = 0x0, uri_avps_from = 0x7f6834c993b0, uri_avps_to = 0x0, user_avps_from = 0x0, user_avps_to = 0x0,
domain_avps_from = 0x0, domain_avps_to = 0x0, xavps_list = 0x0, reply_mutex = {val = 0}, reply_locker_pid = {val = 0}, reply_rec_lock_level = 0, fr_timeout = 32, fr_inv_timeout = 112, rt_t1_timeout_ms = 500, rt_t2_timeout_ms = 4000, end_of_life = 474292144,
relayed_reply_branch = -2, on_failure = 3, on_branch_failure = 0, on_reply = 3, on_branch = 0, on_branch_delayed = 3, md5 = 0x7f683d86a5d0 "9bc5c0acdc87d6654d5368cad5906e3e"}
(gdb) p *should_store
$3 = 0
(gdb) p should_store
$4 = (int *) 0x7fffed6c323c
(gdb) p cancel_data
$5 = (struct cancel_info *) 0x7fffed6c3490
(gdb) p *cancel_data
$6 = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 1586495184}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 1586495184}}}}
(gdb) p *reply
$7 = {id = 55770, pid = 121295, tval = {tv_sec = 1551362168, tv_usec = 692071}, fwd_send_flags = {f = 0, blst_imask = 0}, rpl_send_flags = {f = 0, blst_imask = 0}, first_line = {type = 2, flags = 1, len = 33, u = {request = {method = {
s = 0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len = 7}, uri = {
s = 0xa6c2a8 <buf.6868+8> "503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;branch=z9hG"..., len = 3}, version = {
s = 0xa6c2ac <buf.6868+12> "Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;branch=z9hG4bK4"..., len = 19},
method_value = 503}, reply = {version = {
s = 0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len = 7}, status = {
s = 0xa6c2a8 <buf.6868+8> "503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;branch=z9hG"..., len = 3}, reason = {
s = 0xa6c2ac <buf.6868+12> "Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;branch=z9hG4bK4"..., len = 19},
statuscode = 503}}}, via1 = 0x7f685b3af740, via2 = 0x7f685b3af9e0, headers = 0x7f685b3b1ec8, last_header = 0x7f685b3adf68, parsed_flag = 18446744073709551615, h_via1 = 0x7f685b3b1ec8, h_via2 = 0x7f685b3bbf98, callid = 0x7f685b3b9c50, to = 0x7f685b3b9ba8,
cseq = 0x7f685b3adec0, from = 0x7f685b3b2f28, contact = 0x0, maxforwards = 0x0, route = 0x0, record_route = 0x0, content_type = 0x0, content_length = 0x7f685b3b7468, authorization = 0x0, expires = 0x0, proxy_auth = 0x0, supported = 0x0, require = 0x0,
proxy_require = 0x0, unsupported = 0x0, allow = 0x0, event = 0x0, accept = 0x0, accept_language = 0x0, organization = 0x0, priority = 0x0, subject = 0x0, user_agent = 0x0, server = 0x7f685b3bdf70, content_disposition = 0x0, diversion = 0x0, rpid = 0x0, refer_to = 0x0,
session_expires = 0x0, min_se = 0x0, sipifmatch = 0x0, subscription_state = 0x0, date = 0x0, identity = 0x0, identity_info = 0x0, pai = 0x0, ppi = 0x0, path = 0x0, privacy = 0x0, min_expires = 0x0, body = 0x0, eoh = 0xa6c4d3 <buf.6868+563> "\r\n",
unparsed = 0xa6c4d3 <buf.6868+563> "\r\n", rcv = {src_ip = {af = 2, len = 4, u = {addrl = {39569941286, 140737176681744}, addr32 = {915235622, 9, 3983293712, 32767}, addr16 = {25382, 13965, 9, 0, 15632, 60780, 32767, 0},
addr = "&c\215\066\t\000\000\000\020=l\355\377\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {646919601, 0}, addr32 = {646919601, 0, 0, 0}, addr16 = {13745, 9871, 0, 0, 0, 0, 0, 0}, addr = "\261\065\217&", '\000' <repeats 11 times>}}, src_port = 5060,
dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304&c\215\066\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 915235622},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 915235622, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}, bind_address = 0x7f685a89bb28, proto = 1 '\001'},
buf = 0xa6c2a0 <buf.6868> "SIP/2.0 503 Service Unavailable\r\nVia: SIP/2.0/UDP X.X.X.38:5060;TH=ucv;branch=z9hG4bKf581.18dae6287b86c3ef95ba52c12f282865.0\r\nVia: SIP/2.0/UDP X.X.X.69:5060;received=X.X.X.69;TH=div;bra"..., len = 565, new_uri = {s = 0x0,
len = 0}, dst_uri = {s = 0x0, len = 0}, parsed_uri_ok = 0, parsed_uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0,
len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0,
len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0},
gr_val = {s = 0x0, len = 0}}, parsed_orig_ruri_ok = 0, parsed_orig_ruri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0,
len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0,
len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0},
gr_val = {s = 0x0, len = 0}}, add_rm = 0x7f685b3abb88, body_lumps = 0x0, reply_lump = 0x0, add_to_branch_s = '\000' <repeats 57 times>, add_to_branch_len = 0, hash_index = 0, msg_flags = 32768, flags = 2692743168, xflags = {0, 0}, set_global_address = {s = 0x0,
len = 0}, set_global_port = {s = 0x0, len = 0}, force_send_socket = 0x0, path_vec = {s = 0x0, len = 0}, instance = {s = 0x0, len = 0}, reg_id = 0, ruid = {s = 0x0, len = 0}, location_ua = {s = 0x0, len = 0}, ldv = {flow = {decoded = 0, rcv = {src_ip = {af = 0,
len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}}, dst_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
addr = '\000' <repeats 15 times>}}, src_port = 0, dst_port = 0, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x0, proto = 0 '\000'}}}}
</code></pre>
<h4>Log Messages</h4>
<p>No logs available.</p>
<pre><code>[29069.581044] kamailio[12322]: segfault at 190 ip 00007f5985ad884d sp 00007ffcea744f00 error 4 in tm.so[7f5985a61000+135000]
[49907.558580] kamailio[84693]: segfault at 1a0 ip 00007fc37b75ef90 sp 00007fff954b5880 error 4 in tm.so[7fc37b742000+135000]
[86413.184058] sh (123233): drop_caches: 3
[172811.546070] sh (99955): drop_caches: 3
[201459.115194] kamailio[106801]: segfault at 190 ip 00007f3528cac84d sp 00007ffdf8d27c50 error 4 in tm.so[7f3528c35000+135000]
[259248.603313] sh (34334): drop_caches: 3
[287619.734155] kamailio[40824]: segfault at 190 ip 00007f03ce01984d sp 00007ffe869650f0 error 4 in tm.so[7f03cdfa2000+135000]
[287870.066030] kamailio[119402]: segfault at 1a0 ip 00007fbf4ff34f90 sp 00007ffd1f8cc190 error 4 in tm.so[7fbf4ff18000+135000]
[287952.567590] kamailio[120541]: segfault at 190 ip 00007f20f616684d sp 00007ffc9c760630 error 4 in tm.so[7f20f60ef000+135000]
[288012.343497] kamailio[120947]: segfault at 340 ip 00007fa90abac84d sp 00007ffe8ca01160 error 4 in tm.so[7fa90ab35000+135000]
[290128.570063] kamailio[121295]: segfault at 190 ip 00007f685674a84d sp 00007fffed6c3050 error 4 in tm.so[7f68566d3000+135000]
</code></pre>
<h4>SIP Traffic</h4>
<p>No SIP traffic available.</p>
<pre><code>(paste your sip traffic here)
</code></pre>
<h3>Possible Solutions</h3>
<h3>Additional Information</h3>
<ul>
<li><strong>Kamailio Version</strong> - output of <code>kamailio -v</code></li>
</ul>
<pre><code>version: kamailio 5.2.0 (x86_64/linux) 535e13
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 535e13
compiled on 22:16:55 Feb 20 2019 with gcc 4.8.5
</code></pre>
<ul>
<li><strong>Operating System</strong>:</li>
</ul>
<pre><code>LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.6.1810 (Core)
Release: 7.6.1810
Codename: Core
-----------------
Linux sip 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
</code></pre>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/kamailio/kamailio/issues/1875">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/AF36ZV6yw1qn8mP193VM77d4wRTMqDQ2ks5vSBYAgaJpZM4bXYUz">mute the thread</a>.<img src="https://github.com/notifications/beacon/AF36ZWrDzpUOp2opxNmT4_IjQyxanOUvks5vSBYAgaJpZM4bXYUz.gif" height="1" width="1" alt="" /></p>
<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kamailio/kamailio","title":"kamailio/kamailio","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/kamailio/kamailio"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Segmentation fault on tm:t_should_relay_response (#1875)"}],"action":{"name":"View Issue","url":"https://github.com/kamailio/kamailio/issues/1875"}}}</script>
<script type="application/ld+json">[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "https://github.com/kamailio/kamailio/issues/1875",
"url": "https://github.com/kamailio/kamailio/issues/1875",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script>