[sr-dev] [kamailio/kamailio] Kamailio 5.3.0 crashed when exec module tried to execute external program (#2165)
Konstantin
notifications at github.com
Thu Dec 5 14:47:03 CET 2019
### Description
<!--
Explain what you did, what you expected to happen, and what actually happened.
-->
In some cases we need to call zabbix_sender utility to notify that some failure happens.
At this time we move our configs from Kamailio 4.4.7 to Kamailio 5.3.0 and during testing we meet this issue.
We configured exec module to kill the program after 20 seconds
```
modparam("exec", "time_to_kill", 20)
....
exec_avp("/usr/bin/zabbix_sender --config ZABBIX_AGENT_CONFIG_PATH --key ecsp_no_telephony --value 1");
```
zabbix_sender could not connect to server and were hanged up.
Kamailio probably tried to kill it and crashed.
### Troubleshooting
#### Reproduction
<!--
If the issue can be reproduced, describe how it can be done.
-->
#### Debugging Data
<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile
bt full
```
(gdb) bt full
#0 0x00007f57a4397e4b in run_failure_handlers (t=0x7f579d31b8c8, rpl=0xffffffffffffffff, code=408, extra_flags=32) at t_reply.c:1020
faked_req = 0x7f579d31f4a0
faked_req_len = 7264
shmem_msg = 0x7f579d31cf88
on_failure = 1
keng = 0x0
__FUNCTION__ = "run_failure_handlers"
#1 0x00007f57a439b6ff in t_should_relay_response (Trans=0x7f579d31b8c8, new_code=408, branch=0, should_store=0x7ffdc1d48efc, should_relay=0x7ffdc1d48f00, cancel_data=0x7ffdc1d48fb0,
reply=0xffffffffffffffff) at t_reply.c:1390
branch_cnt = 1
picked_code = 408
new_branch = 0
inv_through = 0
extra_flags = 32
i = 0
replies_dropped = 0
__FUNCTION__ = "t_should_relay_response"
#2 0x00007f57a43a04ae in relay_reply (t=0x7f579d31b8c8, p_msg=0xffffffffffffffff, branch=0, msg_status=408, cancel_data=0x7ffdc1d48fb0, do_put_on_wait=0) at t_reply.c:1821
relay = -1043034112
save_clone = 0
buf = 0x0
res_len = 0
relayed_code = 0
relayed_msg = 0x0
reply_bak = 0x0
bm = {to_tag_val = {s = 0x419140 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\020yw", len = -1043032080}}
totag_retr = 0
reply_status = RPS_ERROR
uas_rb = 0x7f579d31cc10
to_tag = 0x7f579d31ba98
reason = {s = 0x0, len = -2058737957}
onsend_params = {req = 0x419140 <_start>, rpl = 0x7ffdc1d497f0, param = 0x0, code = 0, flags = 0, branch = 0, t_rbuf = 0x7ffdc1d48fd0, dst = 0x7f57a1935bb9 <db_timer_udomain+1213>,
send_buf = {s = 0x50ffffffff <Address 0x50ffffffff out of bounds>, len = -1657680744}}
ip = {af = 26069008, len = 0, u = {addrl = {12898834416, 0}, addr32 = {13932528, 3, 0, 0}, addr16 = {38896, 212, 3, 0, 0, 0, 0, 0},
addr = "\360\227\324\000\003\000\000\000\000\000\000\000\000\000\000"}}
__FUNCTION__ = "relay_reply"
#3 0x00007f57a43f0b80 in fake_reply (t=0x7f579d31b8c8, branch=0, code=408) at timer.c:294
cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = -1657685816}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = -1657685816}}}}
do_cancel_branch = 0
reply_status = 12458
#4 0x00007f57a43f100d in final_response_handler (r_buf=0x7f579d31baf0, t=0x7f579d31b8c8) at timer.c:455
silent = 0
branch_ret = 0
prev_branch = 0
now = 0
#5 0x00007f57a43f10bc in retr_buf_handler (ticks=1687494363, tl=0x7f579d31bb10, p=0x1f40) at timer.c:511
rbuf = 0x7f579d31baf0
fr_remainder = 1
retr_remainder = 32599
retr_interval = 2634899704
new_retr_interval_ms = 7546900672
crt_retr_interval_ms = 140014273782008
t = 0x7f579d31b8c8
__FUNCTION__ = "retr_buf_handler"
#6 0x00000000004bb4f9 in slow_timer_main () at core/timer.c:1104
n = 12
---Type <return> to continue, or q <return> to quit---
ret = 4294967295
tl = 0x7f579d31bb10
i = 951
__FUNCTION__ = "slow_timer_main"
#7 0x000000000042655f in main_loop () at main.c:1727
i = 20
pid = 0
si = 0x0
si_desc = "udp receiver child=19 sock=127.0.0.1:16060\000\066\060 (5.17.138.141:16060)\000\000\000\000\000\000\310\026\t\235W\177", '\000' <repeats 14 times>, "\001\000\000\000`\222\324\301\375\177\000\000\061\212j\000\000\000\000\000\300\352z\000\000\000\000\000Xv<\250W\177\000"
nrprocs = 20
woneinit = 1
__FUNCTION__ = "main_loop"
#8 0x000000000042e56a in main (argc=15, argv=0x7ffdc1d497f8) at main.c:2802
cfg_stream = 0x17e0010
c = -1
r = 0
tmp = 0x7ffdc1d49f44 ""
tmp_len = 0
port = 0
proto = 0
ahost = 0x0
aport = 0
options = 0x77b510 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 4246587393
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x2f2f2f2f2f2f2f2f
p = 0x0
st = {st_dev = 19, st_ino = 22551, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {
tv_sec = 1575037545, tv_nsec = 784790655}, st_mtim = {tv_sec = 1571941656, tv_nsec = 613426287}, st_ctim = {tv_sec = 1571941656, tv_nsec = 613426287}, __unused = {0, 0, 0}}
tbuf = "\000\000\000\000\000\000\000\000\240\223\324\301\375\177\000\000\210+\251\250W\177\000\000\350\363\251\250W\177\000\000\260\224\324\301\375\177\000\000\240\224\324\301\375\177\000\000.", '\000' <repeats 23 times>, "\300\365H\251W\177\000\000\000\360H\251W\177\000\000\270\366@\000\000\000\000\000\320\377\251\250W\177\000\000\230=@\000\000\000\000\000\000\000\000\000\001\000\000\000D\b\000\000\001", '\000' <repeats 11 times>, "h\225\324\301\375\177\000\000@\225\324\301\375\177\000\000\001\000\000\000\000\000\000\000\300\365H\251W\177\000\000\250\324I\251W\177\000\000P\321I\251W\177\000\000O@(\251W\177\000\000\000\000\000\000\000\000\000\000"...
option_index = 0
long_options = {{name = 0x77e08a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x777dac "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x77e08f "alias", has_arg = 1,
flag = 0x0, val = 1024}, {name = 0x77e095 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x77e09b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {
name = 0x77e0a4 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x77e0ae "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x0, has_arg = 0, flag = 0x0,
val = 0}}
__FUNCTION__ = "main"
```
info locals
```
(gdb) info locals
cfg_stream = 0x17e0010
c = -1
r = 0
tmp = 0x7ffdc1d49f44 ""
tmp_len = 0
port = 0
proto = 0
ahost = 0x0
aport = 0
options = 0x77b510 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 4246587393
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x2f2f2f2f2f2f2f2f
p = 0x0
st = {st_dev = 19, st_ino = 22551, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {
tv_sec = 1575037545, tv_nsec = 784790655}, st_mtim = {tv_sec = 1571941656, tv_nsec = 613426287}, st_ctim = {tv_sec = 1571941656, tv_nsec = 613426287}, __unused = {0, 0, 0}}
tbuf = "\000\000\000\000\000\000\000\000\240\223\324\301\375\177\000\000\210+\251\250W\177\000\000\350\363\251\250W\177\000\000\260\224\324\301\375\177\000\000\240\224\324\301\375\177\000\000.", '\000' <repeats 23 times>, "\300\365H\251W\177\000\000\000\360H\251W\177\000\000\270\366@\000\000\000\000\000\320\377\251\250W\177\000\000\230=@\000\000\000\000\000\000\000\000\000\001\000\000\000D\b\000\000\001", '\000' <repeats 11 times>, "h\225\324\301\375\177\000\000@\225\324\301\375\177\000\000\001\000\000\000\000\000\000\000\300\365H\251W\177\000\000\250\324I\251W\177\000\000P\321I\251W\177\000\000O@(\251W\177\000\000\000\000\000\000\000\000\000\000"...
option_index = 0
long_options = {{name = 0x77e08a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x777dac "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x77e08f "alias", has_arg = 1,
flag = 0x0, val = 1024}, {name = 0x77e095 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x77e09b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x77e0a4 "substdefs",
has_arg = 1, flag = 0x0, val = 1027}, {name = 0x77e0ae "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__FUNCTION__ = "main"
```
list
```
(gdb) list
1015 }
1016 exec_post_script_cb(faked_req, FAILURE_CB_TYPE);
1017 }
1018 log_prefix_set(NULL);
1019 /* update message flags, if changed in failure route */
1020 t->uas.request->flags = faked_req->flags;
1021 }
1022
1023 /* restore original environment */
1024 faked_env( t, 0, 0);
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
Dec 5 12:25:08 sip-proxy01 /usr/ecsp/sbin/kamailio[12458]: NOTICE: <script>: zabbix_sender (ecsp_no_telephony)
Dec 5 12:25:09 sip-proxy01 /usr/ecsp/sbin/kamailio[12417]: ALERT: <core> [main.c:767]: handle_sigs(): child process 12458 exited by a signal 11
Dec 5 12:25:09 sip-proxy01 /usr/ecsp/sbin/kamailio[12466]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 50
Dec 5 12:25:09 sip-proxy01 /usr/ecsp/sbin/kamailio[12417]: ALERT: <core> [main.c:770]: handle_sigs(): core was generated
```
#### SIP Traffic
<!--
If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
(paste your sip traffic here)
```
### Possible Solutions
<!--
If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix.
-->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.3.0 (x86_64/linux) 4cc67a
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 4cc67a
compiled on 14:01:24 Nov 28 2019 with gcc 4.8.5
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `uname -a`)
-->
```
Linux sip-proxy01 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.7.1908 (Core)
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2165
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20191205/10695be0/attachment-0001.html>
More information about the sr-dev
mailing list