[sr-dev] [kamailio/kamailio] segfault during avp destroy after t_continue / branching (#2421)
malcolmohare
notifications at github.com
Fri Jul 31 22:19:58 CEST 2020
### Description
I have an occasional segfault while executing this line.
``` $avp(set_to_a2a_reply) = $null; ```
The steps that occur up to this point are
1. host gets invite
2. host sees no online entries in location table
3. host does t_suspend
4. ... waiting ...
5. host receives xhttp request with transaction details to perform t_continue
``` $vn(result) = t_continue("$var(tx_hash_index)", "$var(tx_label)", "PUSHRESUME"); ```
6. the pushresume route eventually does a lookup() finding an entry in the location table
7. we enter the branch route
8. the avp gets cleared with the null assignment
9. signal 11 *boom*
This happens a few times a week, but the code path executes 800,000 times in a week so its pretty rare.
### Troubleshooting
I've looked through the core dumps, looks to me like a memory issue but not 100% sure.
#### Reproduction
Unable to reproduce myself
#### Debugging Data
So the code at this failure is looking through the avp list found in crt_list and looking for the avp with a matching name to destroy. You can see at frame 0 that the avp address its trying to check against is not valid.
I can't access ANY of the memory referenced by crt_list. The memory location looks pretty close to where the transaction cell is addressed. If you look at the sip message I've put under the stack trace, you may notice several "address out of bounds" fields, and if you look at the ua_instance field its printing the ua and then some other data appended (not sure if this is some gdb printing bug or its actually messed up). Is this indicative of running out of shm?
```
(gdb) print crt_list
$1 = {0x7f590c6273c0, 0x7f590c6273c8, 0x7f590c6273d0, 0x7f590c6273d8, 0x7f590c6273e0, 0x7f590c6273e8}
(gdb) print crt_list[0]
$2 = (avp_list_t *) 0x7f590c6273c0
(gdb) print *crt_list[0]
Cannot access memory at address 0x7f590c6273c0
(gdb) print t
$3 = (struct cell *) 0x7f590c627238
(gdb) print *t
Cannot access memory at address 0x7f590c627238
```
If you need something specific from the bt please let me know. I have to double check the output to make sure there is no customer data in it before I paste it, so bt full is a bit tricky to provide.
```
#0 0x00000000009d2ab9 in match_by_name (avp=0x300f317375020c06, id=102, name=0x7ffe8098fd08) at usr_avp.c:378
#1 0x00000000009d510e in search_next_avp (s=0x7ffe8098fd00, val=0x0) at usr_avp.c:494
#2 0x00000000009d3ea2 in search_avp (ident=..., val=0x0, state=0x7ffe8098fd00) at usr_avp.c:462
#3 0x00000000009d2c1a in search_first_avp (flags=1, name=..., val=0x0, s=0x7ffe8098fd00) at usr_avp.c:414
#4 0x00000000009e7c70 in destroy_avps (flags=1, name=..., all=0) at usr_avp.c:1157
#5 0x00007f5da4f65f06 in pv_set_avp (msg=0x7ffe8099a250, param=0x7f5da6321060, op=254, val=0x7ffe8098ff20) at pv_core.c:1983
#6 0x0000000000589689 in lval_pvar_assign (h=0x7ffe80993280, msg=0x7ffe8099a250, lv=0x7f5da6321108, rv=0x7f5da6321248) at lvalue.c:351
#7 0x000000000058b6b7 in lval_assign (h=0x7ffe80993280, msg=0x7ffe8099a250, lv=0x7f5da6321108, rve=0x7f5da6321240) at lvalue.c:399
#8 0x000000000044662a in do_action (h=0x7ffe80993280, a=0x7f5da6321950, msg=0x7ffe8099a250) at action.c:1430
#9 0x000000000044b7ad in run_actions (h=0x7ffe80993280, a=0x7f5da6320e38, msg=0x7ffe8099a250) at action.c:1557
#10 0x000000000042743a in do_action (h=0x7ffe80993280, a=0x7f5da73203e0, msg=0x7ffe8099a250) at action.c:678
#11 0x000000000044b7ad in run_actions (h=0x7ffe80993280, a=0x7f5da73203e0, msg=0x7ffe8099a250) at action.c:1557
#12 0x0000000000430110 in do_action (h=0x7ffe80993280, a=0x7f5da7320510, msg=0x7ffe8099a250) at action.c:1045
#13 0x000000000044b7ad in run_actions (h=0x7ffe80993280, a=0x7f5da7320510, msg=0x7ffe8099a250) at action.c:1557
#14 0x000000000044c80c in run_top_route (a=0x7f5da7320510, msg=0x7ffe8099a250, c=0x7ffe80993280) at action.c:1643
#15 0x00007f5da523176e in prepare_new_uac (t=0x7f590c627238, i_req=0x7ffe8099a250, branch=1, uri=0x7ffe809933d0, path=0x7ffe809933b0, next_hop=0x7ffe809933c0, fsocket=0x0, snd_flags=..., fproto=0, flags=0, instance=0x7ffe809933a0,
ruid=0x7ffe80993390, location_ua=0x7ffe80993380) at t_fwd.c:343
#16 0x00007f5da523c283 in add_uac (t=0x7f590c627238, request=0x7ffe8099a250, uri=0x7ffe8099a4c0, next_hop=0x7ffe8099a4d0, path=0x7ffe8099a888, proxy=0x0, fsocket=0x0, snd_flags=..., proto=0, flags=0, instance=0x7ffe8099a8a8,
ruid=0x7ffe8099a8c0, location_ua=0x7ffe8099a8d0) at t_fwd.c:788
#17 0x00007f5da5250df1 in t_forward_nonack (t=0x7f590c627238, p_msg=0x7ffe8099a250, proxy=0x0, proto=0) at t_fwd.c:1668
#18 0x00007f5da537b879 in _w_t_relay_to (p_msg=0x7ffe8099a250, proxy=0x0, force_proto=0) at tm.c:1416
#19 0x00007f5da537f062 in w_t_relay (p_msg=0x7ffe8099a250, _foo=0x0, _bar=0x0) at tm.c:1630
#20 0x00000000004301a1 in do_action (h=0x7ffe80994b00, a=0x7f5da737d788, msg=0x7ffe8099a250) at action.c:1054
#21 0x000000000044b7ad in run_actions (h=0x7ffe80994b00, a=0x7f5da737d788, msg=0x7ffe8099a250) at action.c:1557
#22 0x000000000044c744 in run_actions_safe (h=0x7ffe8099a090, a=0x7f5da737d788, msg=0x7ffe8099a250) at action.c:1622
#23 0x0000000000776f29 in rval_get_int (h=0x7ffe8099a090, msg=0x7ffe8099a250, i=0x7ffe809950c8, rv=0x7f5da737d8c0, cache=0x0) at rvalue.c:912
#24 0x00000000007813c1 in rval_expr_eval_int (h=0x7ffe8099a090, msg=0x7ffe8099a250, res=0x7ffe809950c8, rve=0x7f5da737d8b8) at rvalue.c:1910
#25 0x000000000078204a in rval_expr_eval_int (h=0x7ffe8099a090, msg=0x7ffe8099a250, res=0x7ffe80995640, rve=0x7f5da737dfc8) at rvalue.c:1918
#26 0x000000000042f442 in do_action (h=0x7ffe8099a090, a=0x7f5da738ff88, msg=0x7ffe8099a250) at action.c:1030
#27 0x000000000044b7ad in run_actions (h=0x7ffe8099a090, a=0x7f5da737b458, msg=0x7ffe8099a250) at action.c:1557
#28 0x000000000042743a in do_action (h=0x7ffe8099a090, a=0x7f5da70ad478, msg=0x7ffe8099a250) at action.c:678
#29 0x000000000044b7ad in run_actions (h=0x7ffe8099a090, a=0x7f5da70acb58, msg=0x7ffe8099a250) at action.c:1557
#30 0x000000000043015e in do_action (h=0x7ffe8099a090, a=0x7f5da70ad5a8, msg=0x7ffe8099a250) at action.c:1049
#31 0x000000000044b7ad in run_actions (h=0x7ffe8099a090, a=0x7f5da70ad5a8, msg=0x7ffe8099a250) at action.c:1557
#32 0x000000000042743a in do_action (h=0x7ffe8099a090, a=0x7f5da7281608, msg=0x7ffe8099a250) at action.c:678
#33 0x000000000044b7ad in run_actions (h=0x7ffe8099a090, a=0x7f5da727d458, msg=0x7ffe8099a250) at action.c:1557
#34 0x0000000000430110 in do_action (h=0x7ffe8099a090, a=0x7f5da7294e18, msg=0x7ffe8099a250) at action.c:1045
#35 0x000000000044b7ad in run_actions (h=0x7ffe8099a090, a=0x7f5da7232a50, msg=0x7ffe8099a250) at action.c:1557
#36 0x000000000044c80c in run_top_route (a=0x7f5da7232a50, msg=0x7ffe8099a250, c=0x0) at action.c:1643
#37 0x00007f5da533d8ca in t_continue (hash_index=47761, label=1485553300, route=0x7f5da7232a50) at t_suspend.c:278
#38 0x00007f5da5158e8e in w_t_continue (msg=0x7ffe809a0050, idx=0x7f5da6da3b98 "0\264ڦ]\177", lbl=0x7f5da6da3cb8 "\230\330ڦ]\177", rtn=0x7f5da6da2f58 "\370\310ڦ]\177") at tmx_mod.c:658
#39 0x0000000000430350 in do_action (h=0x7ffe8099bc40, a=0x7f5da6dace70, msg=0x7ffe809a0050) at action.c:1072
#40 0x000000000044b7ad in run_actions (h=0x7ffe8099bc40, a=0x7f5da6dace70, msg=0x7ffe809a0050) at action.c:1557
#41 0x000000000044c744 in run_actions_safe (h=0x7ffe8099ff70, a=0x7f5da6dace70, msg=0x7ffe809a0050) at action.c:1622
#42 0x00000000005882ed in lval_pvar_assign (h=0x7ffe8099ff70, msg=0x7ffe809a0050, lv=0x7f5da6dacd38, rv=0x7f5da6dae780) at lvalue.c:282
#43 0x000000000058b6b7 in lval_assign (h=0x7ffe8099ff70, msg=0x7ffe809a0050, lv=0x7f5da6dacd38, rve=0x7f5da6dae778) at lvalue.c:399
#44 0x000000000044662a in do_action (h=0x7ffe8099ff70, a=0x7f5da6dab490, msg=0x7ffe809a0050) at action.c:1430
#45 0x000000000044b7ad in run_actions (h=0x7ffe8099ff70, a=0x7f5da6dab490, msg=0x7ffe809a0050) at action.c:1557
#46 0x000000000043015e in do_action (h=0x7ffe8099ff70, a=0x7f5da6dab5c0, msg=0x7ffe809a0050) at action.c:1049
#47 0x000000000044b7ad in run_actions (h=0x7ffe8099ff70, a=0x7f5da6dab5c0, msg=0x7ffe809a0050) at action.c:1557
#63 0x00000000005c4a87 in main (argc=16, argv=0x7ffe809a1f58) at main.c:2628
(gdb) frame 5
#5 0x00007f5da4f65f06 in pv_set_avp (msg=0x7ffe8099a250, param=0x7f5da6321060, op=254, val=0x7ffe8098ff20) at pv_core.c:1983
1983 pv_core.c: No such file or directory.
(gdb) print *msg
$6 = {id = 7307, pid = 12151, tval = {tv_sec = 1595776279, tv_usec = 968094}, fwd_send_flags = {f = 0 '\000', blst_imask = 0 '\000'}, rpl_send_flags = {f = 0 '\000', blst_imask = 0 '\000'}, first_line = {type = 1, flags = 1, len = 97,
u = {request = {method = {s = 0x7f590a4f9958 <Address 0x7f590a4f9958 out of bounds>, len = 6}, uri = {s = 0x7f590a4f995f <Address 0x7f590a4f995f out of bounds>, len = 80}, version = {
s = 0x7f590a4f99b0 <Address 0x7f590a4f99b0 out of bounds>, len = 7}, method_value = 1}, reply = {version = {s = 0x7f590a4f9958 <Address 0x7f590a4f9958 out of bounds>, len = 6}, status = {
s = 0x7f590a4f995f <Address 0x7f590a4f995f out of bounds>, len = 80}, reason = {s = 0x7f590a4f99b0 <Address 0x7f590a4f99b0 out of bounds>, len = 7}, statuscode = 1}}}, via1 = 0x7f590a4fb528, via2 = 0x7f590a4fb6d0,
headers = 0x7f590a4fb3e8, last_header = 0x7f590a4fc4f0, parsed_flag = 18446744073709551615, h_via1 = 0x7f590a4fb4e8, h_via2 = 0x7f590a4fb690, callid = 0x7f590a4fbf80, to = 0x7f590a4fbd28, cseq = 0x7f590a4fbfc0,
from = 0x7f590a4fbae0, contact = 0x7f590a4fbf40, maxforwards = 0x7f590a4fbaa0, route = 0x0, record_route = 0x7f590a4fb4a8, content_type = 0x7f590a4fc430, content_length = 0x7f590a4fc470, authorization = 0x0, expires = 0x0,
proxy_auth = 0x0, supported = 0x7f590a4fc070, require = 0x0, proxy_require = 0x0, unsupported = 0x0, allow = 0x7f590a4fc030, event = 0x0, accept = 0x0, accept_language = 0x0, organization = 0x0, priority = 0x0, subject = 0x0,
user_agent = 0x7f590a4fc130, server = 0x0, content_disposition = 0x0, diversion = 0x0, rpid = 0x0, refer_to = 0x0, session_expires = 0x7f590a4fc0b0, min_se = 0x7f590a4fc0f0, sipifmatch = 0x0, subscription_state = 0x0, date = 0x0,
identity = 0x0, identity_info = 0x0, pai = 0x0, ppi = 0x0, path = 0x0, privacy = 0x0, body = 0x0, eoh = 0x7f590a4fa57c <Address 0x7f590a4fa57c out of bounds>, unparsed = 0x7f590a4fa57c <Address 0x7f590a4fa57c out of bounds>, rcv = {
src_ip = {af = 2, len = 4, u = {addrl = {2122129324, 0}, addr32 = {2122129324, 0, 0, 0}, addr16 = {8108, 32381, 0, 0, 0, 0, 0, 0}, addr = "\254\037}~", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {
2122129324, 0}, addr32 = {2122129324, 0, 0, 0}, addr16 = {8108, 32381, 0, 0, 0, 0, 0, 0}, addr = "\254\037}~", '\000' <repeats 11 times>}}, src_port = 49204, dst_port = 5061, proto_reserved1 = 41848, proto_reserved2 = 0,
src_su = {s = {sa_family = 2, sa_data = "\300\064\254\037}~\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 13504, sin_addr = {s_addr = 2122129324}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
sin6_family = 2, sin6_port = 13504, sin6_flowinfo = 2122129324, sin6_addr = {in6_u = {u6_addr8 = "\000\000\000\000\000\000\000\000`\236A\000\000\000\000", u6_addr16 = {0, 0, 0, 0, 40544, 65, 0, 0}, u6_addr32 = {0, 0, 4300384,
0}}}, sin6_scope_id = 2157584208}}, bind_address = 0x7f5da6139788, proto = 3 '\003'}, buf = 0x7f590a4f9958 <Address 0x7f590a4f9958 out of bounds>, len = 6799, new_uri = {
s = 0x7f5da73ffe30 "sips:[redacted]@192.168.1.45:5061;transport=TLS;ob", len = 98}, dst_uri = {
s = 0x7f5da7401a80 "sip:2Eo2OHMVoyb69QMH38d5rB/cFwQlrB/GAbSU at 54.172.127.136:5062;transport=tls;lr;ob", len = 80}, parsed_uri_ok = 0, parsed_uri = {user = {s = 0x7f590a4f9964 <Address 0x7f590a4f9964 out of bounds>, len = 58},
passwd = {s = 0x0, len = 0}, host = {s = 0x7f590a4f999f <Address 0x7f590a4f999f out of bounds>, len = 16}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0},
port_no = 0, proto = 0, type = SIPS_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0,
len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0},
lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}}, parsed_orig_ruri_ok = 0, parsed_orig_ruri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {
s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0,
len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {
s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}},
add_rm = 0x7f5da6fd52b8, body_lumps = 0x0, reply_lump = 0x7f590558ec20, add_to_branch_s = "z9hG4bK19ab.785259c573052806570cfa5c8301c629.1", '\000' <repeats 11 times>, add_to_branch_len = 46, hash_index = 47761, msg_flags = 144,
flags = 75497472, set_global_address = {s = 0x0, len = 0}, set_global_port = {s = 0x0, len = 0}, force_send_socket = 0x0, force_tcp_connid = 0, path_vec = {
s = 0x7f5da740d678 "<sip:2Eo2OHMVoyb69QMH38d5rB/cFwQlrB/GAbSU at 54.172.127.136:5062;transport=tls;lr;ob>", len = 82}, claim = {s = 0x0, len = 0}, instance = {s = 0x7f5da73fd998 "1101D657-7037-4BB8-82F1-978CA802A249", len = 36},
reg_id = 1, ruid = {s = 0x7f5da73756a8 "uloc-d73b4347-5ed7bb86-14fe-b76cf179", len = 35}, location_ua = {
s = 0x7f5da73fa9c8 "iPhone/iPhone10,2 ****MobileiOSComms/1.4.9696.0 PJSUA/iOS-2.5.5 WebRTC/******/v60.3112 iOS/13.5.1 ****** *****/2.2.3558562.5.5-svnCE_ACCOUNT_ID%g1A00866562DRS71QASCMV9", len = 131}, ldv = {flow = {decoded = 0,
rcv = {src_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0}, addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}}, dst_ip = {af = 0, len = 0, u = {addrl = {0, 0}, addr32 = {0, 0, 0, 0},
addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, addr = '\000' <repeats 15 times>}}, src_port = 0, dst_port = 0, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {
sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {in6_u = {u6_addr8 = '\000' <repeats 15 times>,
u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x0, proto = 0 '\000'}}}}
```
#### Log Messages
None
#### SIP Traffic
Not a specific message, this is random
### Possible Solutions
None I can find
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 4.4.2 (x86_64/linux) 892ad6
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 892ad6
compiled on 17:55:07 Jul 2 2020 with x86_64-unknown-linux-gnu-gcc 4.9.4
```
* **Operating System**:
```
Linux ip-172-31-114-254 4.14.181-108.257.amzn1.x86_64 #1 SMP Wed May 27 02:43:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2421
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20200731/26673659/attachment-0001.htm>
More information about the sr-dev
mailing list