[sr-dev] [kamailio/kamailio] Kamailio-5.4.0 : Crashes from keepalive module (#2448)
sagarmalam
notifications at github.com
Wed Aug 19 16:22:20 CEST 2020
<!--
Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment).
-->
### Description
<!--
Explain what you did, what you expected to happen, and what actually happened.
-->
I am using keepalive module to send keepalives to all the destinations registered with an application server which sits behind kamailio.
When Kamailio recieves 200 Ok. I take URI contact from 200 OK and pass it as a param to ka_add_destination function.Most of the time it works fine but sometimes kamailio crashes when it recieves 200 OK of OPTIONS.
### Troubleshooting
#### Reproduction
I am able to reproduce it easily in below scenario :
All the URIs which we are feeding to KA module has this syntax :
sip:2003 at 10.50.8.11:9090;alias=10.50.8.1~5060~1;pb-ip=35.185.177.49;pb-pt=5060;tp=udp
1. So request will be generate by KA module and it will be send to 10.50.8.11:9090 ( which is another port of same kamailio instance ) via UDP protocol.
2. On recieving OPTION on 9090 port ,
- We are calling handle_ruri_alias() function which will set $du =sip:10.50.8.1:5060;transport=udp
- Modify RURI from sip:2003 at 10.50.8.11:9090;alias=10.50.8.1~5060~1;pb-ip=35.185.177.49;pb-pt=5060;tp=udp to
sip:2003@**35.185.177.49:5060**;pb-ip=35.185.177.49;pb-pt=5060;**transport**=udp
- and relay to 10.50.8.1
3. 10.50.8.1 is another instance of Kamailio which will relay OPTION packet to final destination
<!--
If the issue can be reproduced, describe how it can be done.
-->
#### Debugging Data
<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile
bt full
info locals
list
If you are familiar with gdb, feel free to attach more of what you consider to
be relevant.
-->
```
[root at SBC-4-2 /]# gdb kamailio core.67771
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/sbin/kamailio...done.
[New LWP 67771]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/kamailio -m 5000 -M 500 -P /run/kamailio/kamailio.pid'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000abcdefed in ?? ()
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.el7.x86_64 glibc-2.17-222.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64 libcurl-7.29.0-51.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-36.el7_6.2.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-12.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-36.el7_6.2.x86_64 libuuid-2.23.2-52.el7_5.1.x86_64 mariadb-libs-5.5.64-1.el7.x86_64 nspr-4.19.0-1.el7_5.x86_64 nss-3.36.0-5.el7_5.x86_64 nss-softokn-freebl-3.36.0-5.el7_5.x86_64 nss-util-3.36.0-1.el7_5.x86_64 openldap-2.4.44-15.el7_5.x86_64 openssl-libs-1.0.2k-16.el7_6.1.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-18.el7.x86_64
(gdb) bt full
#0 0x00000000abcdefed in ?? ()
No symbol table info available.
#1 0x00007f931dc69c7e in ka_options_callback (t=0x7f91e58f05c0, type=1024, ps=0x7ffda0977bd0) at keepalive_core.c:126
uri = {
s = 0x7f91e58f63f0 "sip:2003 at 10.50.8.11:9090;alias=10.50.8.1~5060~1;pb-ip=35.185.177.49;pb-pt=5060;tp=udp>\r\nFrom: <sip:keepalive at fromsbc>;tag=c7bfd876be500fff196414658218fc27-d6989f6d\r\nCSeq: 10 OPTIONS\r\nCall-ID: 3f2c3c7c"..., len = 85}
msg = 0x0
state = 2
state_routes = {0x7f931dc6f844 "", 0x7f931dc6f93d "keepalive:dst-up", 0x7f931dc6f94e "keepalive:dst-down"}
ka_dest = 0x7f91e5898988
__FUNCTION__ = "ka_options_callback"
#2 0x00007f9324bfddd8 in run_trans_callbacks_internal (cb_lst=0x7f91e58f0638, type=1024, trans=0x7f91e58f05c0, params=0x7ffda0977bd0)
at t_hooks.c:258
cbp = 0x7f91e58f6590
backup_from = 0xb2a750 <def_list+16>
backup_to = 0xb2a758 <def_list+24>
backup_dom_from = 0xb2a760 <def_list+32>
backup_dom_to = 0xb2a768 <def_list+40>
backup_uri_from = 0xb2a740 <def_list>
backup_uri_to = 0xb2a748 <def_list+8>
backup_xavps = 0xb29fd0 <_xavp_list_head>
backup_xavus = 0xb29fd8 <_xavu_list_head>
backup_xavis = 0xb29fe0 <_xavi_list_head>
__FUNCTION__ = "run_trans_callbacks_internal"
#3 0x00007f9324bfdf0a in run_trans_callbacks (type=1024, trans=0x7f91e58f05c0, req=0x0, rpl=0xffffffffffffffff, code=408) at t_hooks.c:285
params = {req = 0x0, rpl = 0xffffffffffffffff, param = 0x7f91e58f65a0, code = 408, flags = 0, branch = 0, t_rbuf = 0x0, dst = 0x0,
send_buf = {s = 0x0, len = 0}}
#4 0x00007f9324b943f1 in local_reply (t=0x7f91e58f05c0, p_msg=0xffffffffffffffff, branch=0, msg_status=408, cancel_data=0x7ffda0977d50)
at t_reply.c:2265
local_store = 0
local_winner = 0
reply_status = RPS_COMPLETED
winning_msg = 0xffffffffffffffff
winning_code = 408
totag_retr = 0
---Type <return> to continue, or q <return> to quit---
__FUNCTION__ = "local_reply"
#5 0x00007f9324bcc9fd in fake_reply (t=0x7f91e58f05c0, branch=0, code=408) at timer.c:290
cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = -443611712}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0,
len = -443611712}}}}
do_cancel_branch = 0
reply_status = 67771
#6 0x00007f9324bcce95 in final_response_handler (r_buf=0x7f91e58f0860, t=0x7f91e58f05c0) at timer.c:462
silent = 0
branch_ret = 0
prev_branch = 0
now = 0
#7 0x00007f9324bccf56 in retr_buf_handler (ticks=112894435, tl=0x7f91e58f0880, p=0xfa0) at timer.c:518
rbuf = 0x7f91e58f0860
fr_remainder = 1
retr_remainder = 32657
retr_interval = 3847088576
new_retr_interval_ms = 6989250208
crt_retr_interval_ms = 140264594074048
t = 0x7f91e58f05c0
__FUNCTION__ = "retr_buf_handler"
#8 0x00000000004b7bcd in slow_timer_main () at core/timer.c:1105
n = 12
ret = 4294967295
tl = 0x7f91e58f0880
i = 700
__FUNCTION__ = "slow_timer_main"
#9 0x000000000042a4c3 in main_loop () at main.c:1737
i = 16
pid = 0
si = 0x0
si_desc = "udp receiver child=15 sock=10.50.8.11:9090\000:0:0:11]:5060\000\061:5060)\000\000\000\000\000\000\000\000\000\000F\345\221\177\000\000\000\000\000\000\000\000\000\000\bLv\345\221\177\000\000P\201\227\240\375\177\000\000\210\212K\000\000\000\000\000\260\270A\000\000\000\000\000\b{\017(\223\177\000"
nrprocs = 16
woneinit = 1
---Type <return> to continue, or q <return> to quit---
__FUNCTION__ = "main_loop"
#10 0x0000000000433a66 in main (argc=7, argv=0x7ffda09786b8) at main.c:2856
cfg_stream = 0x27e7010
c = -1
r = 0
tmp = 0x7ffda0979f26 ""
tmp_len = 0
port = 0
proto = 0
ahost = 0x0
aport = 0
options = 0x7d2498 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 791137056
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x7f934718ea00 <intel_02_known>
p = 0x0
st = {st_dev = 23, st_ino = 30676, st_nlink = 2, st_mode = 16877, st_uid = 0, st_gid = 5001, __pad0 = 0, st_rdev = 0, st_size = 40,
st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1597841333, tv_nsec = 176009291}, st_mtim = {tv_sec = 1597841333,
tv_nsec = 166009247}, st_ctim = {tv_sec = 1597841333, tv_nsec = 176009291}, __unused = {0, 0, 0}}
tbuf = "\377\377\377\377", '\000' <repeats 12 times>, "\340\263\001G\223\177\000\000\310T\320G\223\177", '\000' <repeats 90 times>, "p\342\252\000\000\000\000\000\260\270A\000\000\000\000\000\260\206\227\240\375\177", '\000' <repeats 26 times>, "\036_\260G\223\177\000\000\001", '\000' <repeats 23 times>...
option_index = 0
long_options = {{name = 0x7d468f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7cfc94 "version", has_arg = 0, flag = 0x0,
val = 118}, {name = 0x7d4694 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7d469a "subst", has_arg = 1, flag = 0x0,
val = 1025}, {name = 0x7d46a0 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7d46a9 "substdefs", has_arg = 1, flag = 0x0,
val = 1027}, {name = 0x7d46b3 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7d46bd "loadmodule", has_arg = 1, flag = 0x0,
val = 1029}, {name = 0x7d46c8 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7d46d1 "log-engine", has_arg = 1, flag = 0x0,
val = 1031}, {name = 0x7d46dc "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__FUNCTION__ = "main"
(gdb)
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
Aug 19 12:58:26 SBC-4-2 /usr/local/sbin/kamailio[67772]: CRITICAL: <core> [core/mem/q_malloc.c:138]: qm_debug_check_frag(): BUG: qm: fragm. 0x7f91e5898950 (address 0x7f91e5898988) end overwritten (c0c0c0c1, abcdefed)! Memory allocator was called from core: core/usr_avp.c:626. Fragment marked by core: core/usr_avp.c:175. Exec from core/mem/q_malloc.c:511.
Aug 19 12:58:38 SBC-4-2 /usr/local/sbin/kamailio[67793]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 94
```
#### SIP Traffic
<!--
If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
(paste your sip traffic here)
```
### Possible Solutions
<!--
If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix.
-->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
[root at SBC-4-2 /]# kamailio -v
version: kamailio 5.4.0 (x86_64/linux) 6c4fce
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 6c4fce
compiled on 14:16:47 Aug 14 2020 with gcc 4.8.5
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `uname -a`)
-->
```
[root at SBC-4-2 /]# uname -a
Linux SBC-4-2.netcarrier.net 5.1.0 #1 SMP Mon May 6 13:44:38 GMT 2019 x86_64 x86_64 x86_64 GNU/Linux
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2448
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20200819/7bf68814/attachment.htm>
More information about the sr-dev
mailing list