[SR-Users] Kamailio 5.3.2 crashing continuously
harneet singh
hbilling at gmail.com
Fri Apr 10 10:19:51 CEST 2020
Hi Daniel,
Please see the requested details below:
SETUP: Kamailio Version 5.3.3 being used. Similar issues were seen in
Kamailio 5.3.2 as well so we upgraded anyhow.
Kamailio is acting as a Call Stateful Proxy with Dispatcher module
enabled with Algorith-10 being used(Call Load based Routing) though there
is only one UAS in our case for now.
The load is a very simple one.
UAC(Sipp) --------- Kamailio ----------- UAS(Sipp)
Initial INVITE-- 200OK -- ACK happens properly. Then UAS sends an UPDATE
after 1 minute which is passed properly to UAC which sends a 200OK and that
reaches UAS. After another minute, the UAS sends another refresh UPDATE to
Kamailio and at the same time UAC sends BYE for the call. Please note that
this is a very time sensitive issue and happens only when such a glare
happens at Kamailio. Crash is not seen with a single call. However, at 50
cps, the crashes appears at a regular interval of 4-5 minutes.
Crash-1 and Crash-2 are seen almost at the same timestamp whereas the
Crash-3 is seen on the new restarted kamailio instance that is handling
calls and the in-dialog messages of the prevoious kamailio instance.
[root at localhost tmp]# ll
total 3161744
-rw------- 1 kamailio kamailio 1111052288 Apr 9 18:57
core.kamailio.995.1586438821.869 ----> CRASH-1
-rw------- 1 kamailio kamailio 1110773760 Apr 9 18:57
core.kamailio.995.1586438824.867 ----> CRASH-2
-rw------- 1 kamailio kamailio 1111052288 Apr 9 18:59
core.kamailio.995.1586438991.3775 ----> CRASH-3
CRASH-1:
---------
Logs:
Apr 9 18:57:01 localhost /usr/local/sbin/kamailio[869]: CRITICAL: {1 10648
INVITE NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND} <core> [core/mem/q_malloc.c:150]:
qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(78,
abcdefed)[0x7f7a5bd1e5d8:0x7f7a5bd1e610]! Memory allocator was called from
dialog: dlg_hash.c:544. Fragment marked by dialog: dlg_handlers.c:308. Exec
from core/mem/q_malloc.c:391.
Apr 9 18:57:01 localhost /usr/local/sbin/kamailio[869]: CRITICAL: {1 10648
INVITE NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND} <core> [core/mem/q_malloc.c:155]:
qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten
[0x7f7a5bd1e568:0x7f7a5bd1e5a0] - fragment marked by dialog:
dlg_handlers.c:308
Apr 9 18:57:03 localhost mysqld: 2020-04-09 18:57:03 276 [Warning] Aborted
connection 276 to db: 'kamailio' user: 'kamailio' host: 'localhost' (Got an
error reading communication packets)
Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: ALERT: <core>
[main.c:767]: handle_sigs(): child process 869 exited by a signal 6
Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: ALERT: <core>
[main.c:770]: handle_sigs(): core was generated
Full BackTrace:
[root at localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb
/usr/local/sbin/kamailio core.kamailio.995.1586438821.869
GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/kamailio...done.
[New LWP 869]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f7aa3a29337 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt full
#0 0x00007f7aa3a29337 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:55
resultvar = 0
pid = 869
selftid = 869
#1 0x00007f7aa3a2aa28 in __GI_abort () at abort.c:90
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x7cafe4, sa_sigaction =
0x7cafe4}, sa_mask = {__val = {50, 11474112, 0, 851352, 1301680, 1304944,
8, 2,
21474836483, 140163493761024, 9015141347359, 98784247808,
140163493996064, 32252848, 140164708874416, 8171492}}, sa_flags = 50,
sa_restorer = 0x7f7aa3aeb8b0 <__syslog>}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x000000000069deac in qm_debug_check_frag (qm=0x7f7a5b419000,
f=0x7f7a5bd1e5d8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", line=544,
efile=0x8094ff "core/mem/q_malloc.c", eline=391) at
core/mem/q_malloc.c:158
p = 0x7f7a5bd1e568
__FUNCTION__ = "qm_debug_check_frag"
#3 0x000000000069f9b9 in qm_malloc (qmp=0x7f7a5b419000, size=8,
file=0x7f7a9b8e2187 "dialog: dlg_hash.c",
func=0x7f7a9b8e4f30 <__FUNCTION__.13318> "dlg_set_leg_info", line=544,
mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:391
qm = 0x7f7a5b419000
f = 0x7f7a5bd1e5d8
hash = 2
list_cntr = 1
__FUNCTION__ = "qm_malloc"
#4 0x00000000006aaa6d in qm_shm_malloc (qmp=0x7f7a5b419000, size=5,
file=0x7f7a9b8e2187 "dialog: dlg_hash.c",
func=0x7f7a9b8e4f30 <__FUNCTION__.13318> "dlg_set_leg_info", line=544,
mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:1226
r = 0x7f7a5b9b7050
#5 0x00007f7a9b850fcd in dlg_set_leg_info (dlg=0x7f7a5b9ba288,
tag=0x7f7aa1b2f210, rr=0x7ffc64edf810, contact=0x7ffc64edf820,
cseq=0x7ffc64edf830, leg=0)
at dlg_hash.c:544
cs = {
s = 0xa955fc <buf+348> "10648 INVITE\r\nAllow: PRACK, INVITE,
ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE,
OPTIONS\r\nSupported: replaces, 100rel, timer,
norefersub\r\nSession-Expires: 120\r\nMin-SE: 120\r\nUser-"..., len = 5}
__FUNCTION__ = "dlg_set_leg_info"
#6 0x00007f7a9b87bd78 in populate_leg_info (dlg=0x7f7a5b9ba288,
msg=0x7f7aa1b2df10, t=0x0, leg=0, tag=0x7f7aa1b2f210) at dlg_handlers.c:266
skip_recs = 0
own_rr = 0
---Type <return> to continue, or q <return> to quit---
cseq = {
s = 0xa955fc <buf+348> "10648 INVITE\r\nAllow: PRACK, INVITE,
ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE,
OPTIONS\r\nSupported: replaces, 100rel, timer,
norefersub\r\nSession-Expires: 120\r\nMin-SE: 120\r\nUser-"..., len = 5}
contact = {
s = 0xa955b2 <buf+274> "sip:10.201.8.5:5060;ob>\r\nCall-ID:
NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK,
INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE,
OPTIONS\r\nSupported: rep"..., len = 22}
rr_set = {s = 0x0, len = 0}
__FUNCTION__ = "populate_leg_info"
#7 0x00007f7a9b883f38 in dlg_new_dialog (req=0x7f7aa1b2df10, t=0x0,
run_initial_cbs=1) at dlg_handlers.c:949
dlg = 0x7f7a5b9ba288
s = {s = 0xf614f80000000000 <error: Cannot access memory at address
0xf614f80000000000>, len = -1582559920}
callid = {
s = 0xa955d4 <buf+308> "NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq:
10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO,
SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: replaces, 100rel,
timer, norefersub\r\n"..., len = 32}
ftag = {
s = 0xa95564 <buf+196> "9s2l4B7KJz0pm8tKXwR39PGRVnJm8sK3\r\nTo:
sip:+918123088839 at 172.27.6.5\r\nContact: <sip:10.201.8.5:5060;ob>\r\nCall-ID:
NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK,
INVITE, ACK, BYE, CA"..., len = 32}
ttag = {s = 0x0, len = 0}
req_uri = {s = 0x7f7aa1a5d910
"sip:+918123088839 at 172.27.6.5:5060;transport=udp",
len = 47}
dir = 0
__FUNCTION__ = "dlg_new_dialog"
#8 0x00007f7a9b890325 in dlg_manage (msg=0x7f7aa1b2df10) at
dlg_handlers.c:1864
tag = {s = 0x0, len = 0}
backup_mode = 0
dlg = 0x0
t = 0x0
__FUNCTION__ = "dlg_manage"
#9 0x00007f7a9b8c869c in w_dlg_manage (msg=0x7f7aa1b2df10, s1=0x0, s2=0x0)
at dialog.c:1094
No locals.
#10 0x0000000000458b36 in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ac3fb0,
msg=0x7f7aa1b2df10) at core/action.c:1071
ret = -5
v = 1
dst = {send_sock = 0x419210 <_start>, to = {s = {sa_family = 7776,
sa_data = "\356d\374\177\000\000\000\000\000\000\000\000\000"}, sin =
{sin_family = 7776,
---Type <return> to continue, or q <return> to quit---
sin_port = 25838, sin_addr = {s_addr = 32764}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 7776, sin6_port =
25838,
sin6_flowinfo = 32764, sin6_addr = {__in6_u = {__u6_addr8 =
'\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 1693319264}}, id = 32764,
send_flags = {f = 56144, blst_imask = 102}, proto = 0 '\000', proto_pad0 =
0 '\000',
proto_pad1 = 0}
tmp = 0x0
new_uri = 0x0
end = 0x7f7a00000000 <error: Cannot access memory at address
0x7f7a00000000>
crt = 0x0
cmd = 0x7f7aa1a9a230
len = 32634
user = -1638375131
uri = {user = {s = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
1693326944}, passwd = {s = 0x0, len = 0}, host = {
s = 0x7ffc64edfee0 "`", len = 6739658}, port = {s = 0x2 <error:
Cannot access memory at address 0x2>, len = 1300448}, params = {
s = 0x7ffc64edfdb0 "\310\375\253\241z\177", len = 4993055},
sip_params = {s = 0x0, len = -1583406912}, headers = {
s = 0x100000000 <error: Cannot access memory at address
0x100000000>, len = -1582113008}, port_no = 64968, proto = 41387, type =
32634,
flags = (unknown: 1693318768), transport = {s = 0x7d4950 "core",
len = 1}, ttl = {s = 0x7ffc64edfee0 "`", len = 6721818}, user_param = {
s = 0x7f7aa1ac2938 "\001", len = 1693319192}, maddr = {s =
0x7f7aa1b2df10 ">\330\001", len = 1693323440}, method = {
s = 0x1000cf960 <error: Cannot access memory at address
0x1000cf960>, len = 0}, lr = {s = 0x13e970 <error: Cannot access memory at
address 0x13e970>,
len = 0}, r2 = {s = 0x7f7aa1ac1fb8 "\004", len = 1693319576},
gr = {s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, transport_val = {
s = 0x7ffc64edff50 "`", len = 4611518}, ttl_val = {s =
0x7ffc64edfed0 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1abfdc8
"\316\002",
len = 1693323440}, maddr_val = {s = 0x0, len = 1}, method_val =
{s = 0x60bb3a29 <error: Cannot access memory at address 0x60bb3a29>, len =
-1251003067},
lr_val = {s = 0x400000000 <error: Cannot access memory at address
0x400000000>, len = 1693319572}, r2_val = {s = 0x0, len = 0}, gr_val = {
s = 0x46dc132b38f3545 <error: Cannot access memory at address
0x46dc132b38f3545>, len = 1622882857}}
next_hop = {user = {s = 0x0, len = 1693319040}, passwd = {s =
0x7ffc64ee0000 "`", len = 6729767}, host = {
s = 0x1 <error: Cannot access memory at address 0x1>, len =
45273232}, port = {s = 0x7ffc64edfdd0 "\340\376\355d\374\177", len = 4},
params = {
s = 0x7f7aa1a934e0 "\364.7\235z\177", len = -1548974973},
sip_params = {s = 0x7f7aa1ac5320 "\002", len = 1}, headers = {
s = 0x7f7a00000000 <error: Cannot access memory at address
0x7f7a00000000>, len = -1582113008}, port_no = 6760, proto = 41388, type =
32634,
flags = (unknown: 1693318384), transport = {s = 0x7f7aa1b2df10
">\330\001", len = -1}, ttl = {s = 0x7ffc64edfd60 "\340\376\355d\374\177",
len = 6721818},
user_param = {s = 0x0, len = 0}, maddr = {s = 0x7f7aa1aa62b8 "",
len = -1582113008}, method = {
s = 0x1ffffffff <error: Cannot access memory at address
0x1ffffffff>, len = 0}, lr = {
s = 0xf612d00000000000 <error: Cannot access memory at address
0xf612d00000000000>, len = 0}, r2 = {s = 0x7f7aa1ac2940 "\004", len =
1693319192}, gr = {
s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440},
transport_val = {s = 0x7ffc64edfdd0 "\340\376\355d\374\177", len =
4611518}, ttl_val = {
s = 0x7f7aa1a2a520 "", len = -1582113008}, user_param_val = {s
= 0x7f7aa1ac1a68 "\333\002", len = 1693323440}, maddr_val = {s = 0x0, len =
1},
---Type <return> to continue, or q <return> to quit---
method_val = {s = 0x60bb3a29 <error: Cannot access memory at
address 0x60bb3a29>, len = -1334889147}, lr_val = {
s = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
1693326944}, r2_val = {s = 0x0, len = 0}, gr_val = {
s = 0x46dc132be8f3545 <error: Cannot access memory at address
0x46dc132be8f3545>, len = 1622882857}}
u = 0x7f7aa1b2f4b0
port = 0
dst_host = 0x0
i = 6741152
flags = -74119067
avp = 0x7ffc64ee01e0
st = {flags = 4297232, id = 0, name = {n = 1693326944, s = {s =
0x7ffc64ee1e60 "\r", len = 0}, re = 0x7ffc64ee1e60}, avp = 0x0}
sct = 0x0
sjt = 0x7f7aa1b2df10
rve = 0x0
mct = 0x4000000000
rv = 0x46dc132bf6f3545
rv1 = 0x0
c1 = {cache_type = 1693319040, val_type = 32764, c = {avp_val = {n
= -1582113008, s = {s = 0x7f7aa1b2df10 ">\330\001", len = -1582553112},
re = 0x7f7aa1b2df10}, pval = {rs = {s = 0x7f7aa1b2df10
">\330\001", len = -1582553112}, ri = 1693323440, flags = 32764}},
i2s =
"\000\000\000\000\000\000\000\000\377\377\377\377\000\000\000\000):\273`\000"}
s = {s = 0x7ffc64edfc50 "`\375\355d\374\177", len = 4611518}
srevp = {0x0, 0x0}
evp = {data = 0x0, rcv = 0x0, dst = 0x0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string
= 0x0, str = {
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
= {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#11 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1ac3fb0,
msg=0x7f7aa1b2df10) at core/action.c:1576
t = 0x7f7aa1ac3fb0
ret = -1
---Type <return> to continue, or q <return> to quit---
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = 0, tz_dsttime = 4}
tdiff = 1
__FUNCTION__ = "run_actions"
#12 0x0000000000458aa5 in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ac4100,
msg=0x7f7aa1b2df10) at core/action.c:1062
ret = 1
v = 1
dst = {send_sock = 0x0, to = {s = {sa_family = 1856, sa_data =
"\356d\374\177\000\000\360\026\253\241z\177\000"}, sin = {sin_family =
1856, sin_port = 25838,
sin_addr = {s_addr = 32764}, sin_zero =
"\360\026\253\241z\177\000"}, sin6 = {sin6_family = 1856, sin6_port =
25838, sin6_flowinfo = 32764, sin6_addr = {
__in6_u = {__u6_addr8 =
"\360\026\253\241z\177\000\000\020\337\262\241z\177\000", __u6_addr16 =
{5872, 41387, 32634, 0, 57104, 41394, 32634, 0},
__u6_addr32 = {2712344304, 32634, 2712854288, 32634}}},
sin6_scope_id = 1693321136}}, id = 32764, send_flags = {f = 54986,
blst_imask = 102},
proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}
tmp = 0x7f7a9c5ecc83 <acc_log_request+8047> "\270\001"
new_uri = 0x7f7aa1b2df10 ">\330\001"
end = 0x7f7aa1ab16d0 "\a"
crt = 0x7ffc64ee08e0 "\300\t\356d\374\177"
cmd = 0x7f7aa1a949c8
len = 0
user = 0
uri = {user = {s = 0x0, len = 1559864800}, passwd = {s =
0x2f00000000 <error: Cannot access memory at address 0x2f00000000>, len =
1536887583}, host = {
s = 0x0, len = 0}, port = {s = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
1693326944}, params = {s = 0x0,
len = 0}, sip_params = {s = 0x7ffc64ee0550
"\360\026\253\241z\177", len = 7393852}, headers = {s = 0x7f7a5b41bb90 "",
len = -1582113008}, port_no = 43232,
proto = 41386, type = 32634, flags = (unknown: 1693320640),
transport = {s = 0x100000000 <error: Cannot access memory at address
0x100000000>, len = -1},
ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr
= {s = 0x7f7aa1b2e1b0 "\253T\251", len = 1693321024}, method = {s =
0x7f7aa1ab16f0 "",
len = -1582113008}, lr = {s = 0x7ffc64ee05f0
"p\006\356d\374\177", len = -1647784851}, r2 = {
s = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
1693326944}, gr = {s = 0x0, len = 0}, transport_val = {
s = 0x7ffc64ee0740 "", len = -1582622992}, ttl_val = {s =
0x7f7aa1b2e1b0 "\253T\251", len = -1582113008}, user_param_val = {s =
0x7f7aa1aaa8e0 "q\002",
len = 1693323440}, maddr_val = {s = 0x0, len = 1622882857},
method_val = {
s = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
1693326944}, lr_val = {s = 0x0, len = 0}, r2_val = {
s = 0x7ffc64ee0670 "\360\006\356d\374\177", len = -1647782719},
gr_val = {
s = 0x46dc13540ef3545 <error: Cannot access memory at address
0x46dc13540ef3545>, len = -282512059}}
next_hop = {user = {s = 0xa8428197 <error: Cannot access memory at
address 0xa8428197>, len = -1541505685}, passwd = {
---Type <return> to continue, or q <return> to quit---
s = 0x19 <error: Cannot access memory at address 0x19>, len =
1693320032}, host = {
s = 0xf611b40000000000 <error: Cannot access memory at address
0xf611b40000000000>, len = 6982157}, port = {s = 0x7ffc64ee0380
"\310i\252\241z\177",
len = -1583407088}, params = {s = 0x7f7aa1a934e0
"\364.7\235z\177", len = -1583407088}, sip_params = {s = 0x7ffc64ee03c0
"\360\003\356d\374\177",
len = 6995517}, headers = {s = 0x7f7aa1a2a520 "", len =
-1582113008}, port_no = 27080, proto = 41386, type = 32634, flags =
(unknown: 1693320256),
transport = {s = 0xcfff0 <error: Cannot access memory at address
0xcfff0>, len = 1301656}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0,
len = 0},
maddr = {s = 0x7ffc64ee03f0 "\020\222A", len = 4993055}, method =
{s = 0xffffffff00000000 <error: Cannot access memory at address
0xffffffff00000000>,
len = 0}, lr = {s = 0x1ec23d0 <error: Cannot access memory at
address 0x1ec23d0>, len = 1622882857}, r2 = {
s = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
1693326944}, gr = {s = 0x0, len = 0}, transport_val = {
s = 0x7ffc64ee0520 "", len = 4611518}, ttl_val = {s =
0x7ffc64ee0520 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1aa69c8
"X\002",
len = 1693323440}, maddr_val = {s = 0x0, len = 1622882857},
method_val = {
s = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
1693326944}, lr_val = {s = 0x0, len = 0}, r2_val = {
s = 0x7ffc64ee0590 "@\a\356d\374\177", len = 4609789}, gr_val =
{s = 0x46dc1354fef3545 <error: Cannot access memory at address
0x46dc1354fef3545>,
len = -282512059}}
u = 0x0
port = 0
dst_host = 0x7ffc64ee07b0
i = 6744853
flags = 0
avp = 0x7ffc64ee0930
st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len =
1693320208}, re = 0x0}, avp = 0x4656fd <run_actions+1860>}
sct = 0x7f7a9c616b90 <__FUNCTION__.11588>
sjt = 0x419c61f7fc
rve = 0x7f7aa1ac3798
mct = 0x7f7a9c615c60
rv = 0x7ffc64ee1e60
rv1 = 0x7ffc64ee10b0
c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val
= {n = 0, s = {s = 0x0, len = 1}, re = 0x0}, pval = {rs = {s = 0x0, len =
1}, ri = 0,
flags = 0}}, i2s =
"\260Q\260\241z\177\000\000):\273`\000\000\000\000\020\222A\000\000"}
s = {s = 0x0, len = 0}
srevp = {0x7ffc64ee0860, 0x7ffc64ee0418}
evp = {data = 0x0, rcv = 0x0, dst = 0x0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string
= 0x0, str = {
---Type <return> to continue, or q <return> to quit---
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
= {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#13 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1abd308,
msg=0x7f7aa1b2df10) at core/action.c:1576
t = 0x7f7aa1ac4100
ret = 1
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = -1671324464, tz_dsttime = 32634}
tdiff = 2712291872
__FUNCTION__ = "run_actions"
#14 0x000000000045555c in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ab2d40,
msg=0x7f7aa1b2df10) at core/action.c:695
ret = -5
v = 0
dst = {send_sock = 0x823590, to = {s = {sa_family = 33536, sa_data
= "\200\000V\001\000\000\260d\202\000\000\000\000"}, sin = {sin_family =
33536,
sin_port = 128, sin_addr = {s_addr = 342}, sin_zero =
"\260d\202\000\000\000\000"}, sin6 = {sin6_family = 33536, sin6_port = 128,
sin6_flowinfo = 342,
sin6_addr = {__in6_u = {__u6_addr8 =
"\260d\202\000\000\000\000\000\030:\202\000\000\000\000", __u6_addr16 =
{25776, 130, 0, 0, 14872, 130, 0, 0},
__u6_addr32 = {8545456, 0, 8534552, 0}}}, sin6_scope_id =
2712859096}}, id = 32634, send_flags = {f = 8208, blst_imask = 41375},
proto = 122 'z',
proto_pad0 = 127 '\177', proto_pad1 = 0}
tmp = 0x7ffc64ee10d0 "\020\222A"
new_uri = 0x7ffc64ee0f64 "\374\177"
end = 0x1ec3308 <error: Cannot access memory at address 0x1ec3308>
crt = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w"
cmd = 0x7f7aa1a931d0
len = 0
user = 1622882857
uri = {user = {s = 0x8163a0 "core", len = 0}, passwd = {s = 0x0,
len = 0}, host = {s = 0xc300000000 <error: Cannot access memory at address
0xc300000000>,
len = 11097831}, port = {s = 0x164ee0c50 <error: Cannot access
memory at address 0x164ee0c50>, len = 11098026}, params = {s =
0x7ffc64ee0cb0 "\001",
len = 7299836}, sip_params = {s = 0x7ffc64ee0d42 "\251", len =
6982157}, headers = {s = 0x7ffc64ee0c80 "", len = -1583407088}, port_no =
0, proto = 0,
type = 4128681984, flags = (unknown: 2711560208), transport = {s
= 0x7ffc64ee0cc0 "\360\f\356d\374\177", len = -1632797901}, ttl = {
---Type <return> to continue, or q <return> to quit---
s = 0x69def9 <qm_insert_free+59> "\005\363\a", len = 0},
user_param = {s = 0x2000000 <error: Cannot access memory at address
0x2000000>, len = 32255496},
maddr = {s = 0xcf960 <error: Cannot access memory at address
0xcf960>, len = 1298936}, method = {
s = 0x13e970 <error: Cannot access memory at address 0x13e970>,
len = 8}, lr = {s = 0x1 <error: Cannot access memory at address 0x1>, len =
6938361},
r2 = {s = 0x7ffc64ee0cf0 "\020\016\356d\374\177", len = 4993055},
gr = {s = 0x0, len = -1583176416}, transport_val = {
s = 0x80ba1a2a520 <error: Cannot access memory at address
0x80ba1a2a520>, len = -1583176416}, ttl_val = {s = 0x7ffc64ee0e10
"\340\016\356d\374\177",
len = 6955649}, user_param_val = {s = 0x1ec2dc8 <error: Cannot
access memory at address 0x1ec2dc8>, len = 1622882857}, maddr_val = {
s = 0x7ffc64ee0e10 "\340\016\356d\374\177", len = 6946539},
method_val = {s = 0x823590 "core", len = 11097317}, lr_val = {
s = 0x13e970 <error: Cannot access memory at address 0x13e970>,
len = -1582108200}, r2_val = {s = 0xa957aa <buf+778> "", len = 11097873},
gr_val = {
s = 0x7ffc64ee0d80 "\330\361\262\241z\177", len = 4993055}}
next_hop = {user = {s = 0x2 <error: Cannot access memory at address
0x2>, len = 4959780}, passwd = {s = 0x7ffc64ee0aa0 "\300\v\356d\374\177",
len = 4993055},
host = {s = 0x0, len = -1583397840}, port = {s = 0x5200000000
<error: Cannot access memory at address 0x5200000000>, len = -1583397840},
params = {
s = 0x7ffc64ee0bc0 "H", len = 6955649}, sip_params = {s =
0x2009e6676b0 <error: Cannot access memory at address 0x2009e6676b0>, len =
1548742352},
headers = {s = 0x7f7a9e66e830 <__FUNCTION__.12289>
"update_totag_set", len = -1637445885}, port_no = 4976, proto = 175, type =
ERROR_URI_T,
flags = (unknown: 11475272), transport = {s = 0xaf1940
<def_list+32> "", len = 11475256}, ttl = {
s = 0x5000af1930 <error: Cannot access memory at address
0x5000af1930>, len = 1548742368}, user_param = {s = 0xaf1920 <def_list>
"@t\233[z\177", len = 0},
maddr = {s = 0x7f7a9e6676b0 "tm", len = 6982157}, method = {s =
0x7ffc64ee0b60 "", len = -1583407088}, lr = {s = 0x7f7aa1b2f780 "\300
\237\241z\177",
len = -1583407088}, r2 = {s = 0x7ffc64ee0ba0
"\320\f\356d\374\177", len = -1632797901}, gr = {s = 0x7f7aa19f27c0 "", len
= 0}, transport_val = {
s = 0x2000000 <error: Cannot access memory at address
0x2000000>, len = 32258408}, ttl_val = {
s = 0xcf418 <error: Cannot access memory at address 0xcf418>,
len = 1296024}, user_param_val = {
s = 0x40013e970 <error: Cannot access memory at address
0x40013e970>, len = 8479648}, maddr_val = {s = 0x1 <error: Cannot access
memory at address 0x1>,
len = -1582235496}, method_val = {s = 0x7ffc64ee0cd0 "", len =
7110832}, lr_val = {s = 0x0, len = 8422144}, r2_val = {
s = 0x48 <error: Cannot access memory at address 0x48>, len =
1622882857}, gr_val = {s = 0x7ffc64ee0cd0 "", len = 6946539}}
u = 0xf6199c0000000000
port = 0
dst_host = 0x0
i = 2
flags = 0
avp = 0xa957aa <buf+778>
st = {flags = 33554432, id = 0, name = {n = 32253432, s = {s =
0x1ec25f8 <error: Cannot access memory at address 0x1ec25f8>, len =
851296}, re = 0x1ec25f8},
avp = 0x13da08}
sct = 0x7ffc64ee0e40
sjt = 0x7f7aa1b2f1a0
rve = 0x7f7aa1ab2528
---Type <return> to continue, or q <return> to quit---
mct = 0x1ec2e08
rv = 0x0
rv1 = 0x0
c1 = {cache_type = 1693321776, val_type = 32764, c = {avp_val = {n
= -1583407088, s = {s = 0x7f7aa19f2010 "\001", len = 391}, re =
0x7f7aa19f2010}, pval = {
rs = {s = 0x7f7aa19f2010 "\001", len = 391}, ri =
-1583407088, flags = 32634}}, i2s =
"p\n\356d\374\177\000\000\063{\255\236z\177\000\000HF\267\\z\177"}
s = {s = 0x300000002 <error: Cannot access memory at address
0x300000002>, len = 6982157}
srevp = {0x0, 0x8094ff}
evp = {data = 0x0, rcv = 0x0, dst = 0x0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
NOSUBTYPE, u = {
number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string
= 0x0, str = {
s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
data = 0x0,
attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
0x0}}, {
type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
= {number = 0,
string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
= {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__FUNCTION__ = "do_action"
#15 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1aa48d0,
msg=0x7f7aa1b2df10) at core/action.c:1576
t = 0x7f7aa1ab2d40
ret = 1
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = 11098026, tz_dsttime = 0}
tdiff = 1622882857
__FUNCTION__ = "run_actions"
#16 0x0000000000465e86 in run_top_route (a=0x7f7aa1aa48d0,
msg=0x7f7aa1b2df10, c=0x0) at core/action.c:1661
ctx = {rec_lev = 3, run_flags = 0, last_retcode = -1, jmp_env =
{{__jmpbuf = {1622882857, 319123583751107909, 4297232, 140722001813088, 0,
0,
319123583451215173, -318338964762577595}, __mask_was_saved
= 0, __saved_mask = {__val = {0, 4611686022722355200, 140164675591952,
140722001809824,
5326593, 4297232, 140722001813088, 0, 0, 5988290944,
140164675591952, 1073741825, 0, 140164675389112, 1622882857, 4297232}}}}}
p = 0x7ffc64ee10b0
ret = 0
sfbk = 0
#17 0x000000000059344c in receive_msg (
buf=0xa954a0 <buf> "INVITE sip:+918123088839 at 172.27.6.5:5060;transport=udp
SIP/2.0\r\nVia: SIP/2.0/UDP
10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZ---Type
<return> to continue, or q <return> to quit---
H0UFZ5Xj2\r\nMax-Forwards: 69\r\nFrom: sip:10.201.8.5;tag=9s2l"...,
len=778, rcv_info=0x7ffc64ee16f0) at core/receive.c:424
msg = 0x7f7aa1b2df10
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env =
{{__jmpbuf = {1622882857, 319123583751107909, 4297232, 140722001813088, 0,
0,
319123583451215173, -318338964762577595}, __mask_was_saved
= 0, __saved_mask = {__val = {8, 2, 140164675468656, 140722001810256,
4993055, 0,
21483258624, 912, 1622882857, 140722001810512, 6946539,
7946256, 8589934592, 140722001810480, 5483931, 0}}}}}
bctx = 0x0
ret = -1
tvb = {tv_sec = 140722001810144, tv_usec = 140164674297872}
tve = {tv_sec = 4297232, tv_usec = 6982157}
tz = {tz_minuteswest = 0, tz_dsttime = -127}
diff = 0
inb = {
s = 0xa954a0 <buf> "INVITE
sip:+918123088839 at 172.27.6.5:5060;transport=udp
SIP/2.0\r\nVia: SIP/2.0/UDP
10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZH0UFZ5Xj2\r\nMax-Forwards:
69\r\nFrom: sip:10.201.8.5;tag=9s2l"..., len = 778}
netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0}
keng = 0x0
evp = {data = 0x7ffc64ee1270, rcv = 0x7ffc64ee16f0, dst = 0x0}
cidlockidx = 0
cidlockset = 0
errsipmsg = 0
exectime = 0
__FUNCTION__ = "receive_msg"
#18 0x0000000000498ca7 in udp_rcv_loop () at core/udp_server.c:548
len = 778
buf = "INVITE sip:+918123088839 at 172.27.6.5:5060;transport=udp
SIP/2.0\r\nVia: SIP/2.0/UDP
10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZH0UFZ5Xj2\r\nMax-Forwards:
69\r\nFrom: sip:10.201.8.5;tag=9s2l"...
tmp = 0x0
from = 0x7f7aa1b0fda8
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {84461834,
140722001811280}, addr32 = {84461834, 0, 1693325136, 32764}, addr16 =
{51466, 1288, 0, 0, 5968,
25838, 32764, 0}, addr =
"\n\311\b\005\000\000\000\000P\027\356d\374\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {84286380, 0}, addr32 = {
84286380, 0, 0, 0}, addr16 = {7084, 1286, 0, 0, 0, 0, 0,
0}, addr = "\254\033\006\005", '\000' <repeats 11 times>}}, src_port =
5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s =
{sa_family = 2, sa_data =
"\023\304\n\311\b\005\000\000\000\000\000\000\000"}, sin = {
---Type <return> to continue, or q <return> to quit---
sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
84461834}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family
= 2,
sin6_port = 50195, sin6_flowinfo = 84461834, sin6_addr =
{__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0,
0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7f7aa1a83510, proto = 1 '\001', proto_pad0 = 24 '\030',
proto_pad1 = 25838}
evp = {data = 0x0, rcv = 0x0, dst = 0x0}
printbuf = "\020\222A\000\000\000\000\000`\036\356d\374\177",
'\000' <repeats 18 times>, "\260\025\356d\374\177\000\000\233\255S", '\000'
<repeats 21 times>,
"\320\025\356d\374\177\000\000\233\255S\000\000\000\000\000\023\225\200\000\000\000\000\000\bz}\000\000\000\000\000
", '\000' <repeats 13 times>,
"\200\300\000\000\000\000@~J\377\037\060L\000\017\000\000\000\000\026\356d\035\000\000\061\240\025\356d\374\177\000\000
\030\356d\002\000\000\000`\241\251\241z\177\000\000\000\325w\000\000\000\000\000\b\027\356d\374\177\000\000\177\000\000\000\000\000\000\000\060\nm[z\177\000\000\020\222A\000\000\000\000\000"...
i = -1
j = 49280
l = 1194642238
__FUNCTION__ = "udp_rcv_loop"
#19 0x0000000000425f31 in main_loop () at main.c:1673
i = 1
pid = 0
si = 0x7f7aa1a83510
si_desc = "udp receiver child=1 sock=172.27.6.5:5060 (
172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310&E[z\177",
'\000' <repeats 14 times>,
"\001\000\000\000\320\030\356d\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\352\260\241z\177\000"
nrprocs = 8
woneinit = 1
__FUNCTION__ = "main_loop"
#20 0x000000000042e63a in main (argc=13, argv=0x7ffc64ee1e68) at main.c:2802
cfg_stream = 0x2a68010
c = -1
r = 0
tmp = 0x7ffc64ee2ed8 ""
tmp_len = 2496
port = 2496
proto = 2496
ahost = 0x0
aport = 0
options = 0x77db90
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
---Type <return> to continue, or q <return> to quit---
seed = 2887616840
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x4000000100
p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff>
st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode =
16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 40,
st_blksize = 4096,
st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec =
629166543}, st_mtim = {tv_sec = 1586435764, tv_nsec = 286216393}, st_ctim =
{tv_sec = 1586435764,
tv_nsec = 286216393}, __unused = {0, 0, 0}}
tbuf = '\000' <repeats 376 times>...
option_index = 0
long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0,
val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val =
118}, {
name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024},
{name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {
name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val =
1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {
name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val =
1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__FUNCTION__ = "main"
(gdb)
==========================================================================================================================================
CRASH-2:
-------
Logs:
Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: CRITICAL: <core>
[core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev. fragm.
tail overwritten(78, abcdefed)[0x7f7a5bd1e5d8:0x7f7a5bd1e610]! Memory
allocator was called from dialog: dlg_hash.c:392. Fragment marked by
dialog: dlg_handlers.c:308. Exec from core/mem/q_malloc.c:566.
Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: CRITICAL: <core>
[core/mem/q_malloc.c:155]: qm_debug_check_frag(): BUG: qm: prev. fragm.
tail overwritten [0x7f7a5bd1e568:0x7f7a5bd1e5a0] - fragment marked by
dialog: dlg_handlers.c:308
Apr 9 18:57:06 localhost systemd: kamailio.service: main process exited,
code=dumped, status=6/ABRT
Full BackTrace:
[root at localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb
/usr/local/sbin/kamailio core.kamailio.995.1586438824.867
GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/kamailio...done.
[New LWP 867]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f7aa3a29337 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt full
#0 0x00007f7aa3a29337 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:55
resultvar = 0
pid = 867
selftid = 867
#1 0x00007f7aa3a2aa28 in __GI_abort () at abort.c:90
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x7cafe4, sa_sigaction =
0x7cafe4}, sa_mask = {__val = {8424376, 8442432, 140161962737664, 0, 0, 0,
0, 0,
21474836483, 140163493761024, 9015136354304, 64355789963264,
140163493996064, 0, 140164708874416, 8171492}}, sa_flags = -1685147642,
sa_restorer = 0x7f7aa3aeb8b0 <__syslog>}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x000000000069deac in qm_debug_check_frag (qm=0x7f7a5b419000,
f=0x7f7a5bd1e5d8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", line=392,
efile=0x8094ff "core/mem/q_malloc.c", eline=566) at
core/mem/q_malloc.c:158
p = 0x7f7a5bd1e568
__FUNCTION__ = "qm_debug_check_frag"
#3 0x00000000006a2119 in qm_free (qmp=0x7f7a5b419000, p=0x7f7a5bd1e688,
file=0x7f7a9b8e2187 "dialog: dlg_hash.c",
func=0x7f7a9b8e4ef9 <__FUNCTION__.13276> "destroy_dlg", line=392,
mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:566
qm = 0x7f7a5b419000
f = 0x7f7a5bd1e650
size = 24
next = 0x7f7a5bd1e6d0
prev = 0x7f7a5bd1e5d8
__FUNCTION__ = "qm_free"
#4 0x00000000006aacba in qm_shm_free (qmp=0x7f7a5b419000,
p=0x7f7a5bd1e688, file=0x7f7a9b8e2187 "dialog: dlg_hash.c",
func=0x7f7a9b8e4ef9 <__FUNCTION__.13276> "destroy_dlg", line=392,
mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:1275
No locals.
#5 0x00007f7a9b84fe73 in destroy_dlg (dlg=0x7f7a5b9c3f80) at dlg_hash.c:392
ret = 0
var = 0x7f7aa1b2a4f8
__FUNCTION__ = "destroy_dlg"
#6 0x00007f7a9b850162 in destroy_dlg_table () at dlg_hash.c:438
dlg = 0x7f7a5b747498
l_dlg = 0x7f7a5b9c3f80
i = 3854
---Type <return> to continue, or q <return> to quit---
__FUNCTION__ = "destroy_dlg_table"
#7 0x00007f7a9b8c4f17 in mod_destroy () at dialog.c:806
No locals.
#8 0x000000000053a94a in destroy_modules () at core/sr_module.c:746
t = 0x7f7aa1a9a160
foo = 0x7f7aa1a99c40
__FUNCTION__ = "destroy_modules"
#9 0x000000000041a0c7 in cleanup (show_status=1) at main.c:555
memlog = 0
__FUNCTION__ = "cleanup"
#10 0x000000000041b7f6 in shutdown_children (sig=15, show_status=1) at
main.c:696
__FUNCTION__ = "shutdown_children"
#11 0x000000000041e422 in handle_sigs () at main.c:796
chld = 0
chld_status = 134
any_chld_stopped = 1
memlog = 1533928056
__FUNCTION__ = "handle_sigs"
#12 0x0000000000427814 in main_loop () at main.c:1806
i = 8
pid = 901
si = 0x0
si_desc = "udp receiver child=7 sock=172.27.6.5:5090\000(
172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310&E[z\177",
'\000' <repeats 14 times>,
"\001\000\000\000\320\030\356d\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\352\260\241z\177\000"
nrprocs = 8
woneinit = 1
__FUNCTION__ = "main_loop"
#13 0x000000000042e63a in main (argc=13, argv=0x7ffc64ee1e68) at main.c:2802
cfg_stream = 0x2a68010
c = -1
r = 0
tmp = 0x7ffc64ee2ed8 ""
tmp_len = 2496
---Type <return> to continue, or q <return> to quit---
port = 2496
proto = 2496
ahost = 0x0
aport = 0
options = 0x77db90
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 2887616840
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x4000000100
p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff>
st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode =
16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 40,
st_blksize = 4096,
st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec =
629166543}, st_mtim = {tv_sec = 1586435764, tv_nsec = 286216393}, st_ctim =
{tv_sec = 1586435764,
tv_nsec = 286216393}, __unused = {0, 0, 0}}
tbuf = '\000' <repeats 376 times>...
option_index = 0
long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0,
val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val =
118}, {
name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024},
{name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {
name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val =
1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {
name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val =
1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__FUNCTION__ = "main"
(gdb)
===========================================================================================================================================
CRASH-3:
---------
Secnario:
This crash is seen after the previous crashes happened and kamailio
restarted quickly and was handling new calls and the in-dialog messages
from prior to the restart.
Logs:
Apr 9 18:59:51 localhost kernel: kamailio[3775]: segfault at 7f4500000078
ip 00007f4500000078 sp 00007ffcd4255688 error 14 in
libbz2.so.1.0.6[7f452f812000+f000]
Apr 9 18:59:53 localhost mysqld: 2020-04-09 18:59:53 308 [Warning] Aborted
connection 308 to db: 'kamailio' user: 'kamailio' host: 'localhost' (Got an
error reading communication packets)
Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3774]: ALERT: <core>
[main.c:767]: handle_sigs(): child process 3775 exited by a signal 11
Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3774]: ALERT: <core>
[main.c:770]: handle_sigs(): core was generated
Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3808]: CRITICAL: <core>
[core/pass_fd.c:277]: receive_fd(): EOF on 13
Full BackTrace:
[root at localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb
/usr/local/sbin/kamailio core.kamailio.995.1586438991.3775
GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/kamailio...done.
[New LWP 3775]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/kamailio -P
/var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f4500000078 in ?? ()
(gdb) bt full
#0 0x00007f4500000078 in ?? ()
No symbol table info available.
#1 0x00007f4573f11d7b in run_trans_callbacks_internal
(cb_lst=0x7f45329421e0, type=512, trans=0x7f4532942168,
params=0x7ffcd4255790) at t_hooks.c:254
cbp = 0x7f45322f9b40
backup_from = 0xaf1930 <def_list+16>
backup_to = 0xaf1938 <def_list+24>
backup_dom_from = 0xaf1940 <def_list+32>
backup_dom_to = 0xaf1948 <def_list+40>
backup_uri_from = 0xaf1920 <def_list>
backup_uri_to = 0xaf1928 <def_list+8>
backup_xavps = 0xaf1370 <_xavp_list_head>
__FUNCTION__ = "run_trans_callbacks_internal"
#2 0x00007f4573f11f7b in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f4532942238, req=0x7f4532a03d88, repl=0x7f4577407f10, flags=0) at
t_hooks.c:297
params = {req = 0x7f4532a03d88, rpl = 0x7f4577407f10, param =
0x7f45322f9b50, code = 200, flags = 0, branch = 0, t_rbuf = 0x7f4532942238,
dst = 0x7f4532942288, send_buf = {
s = 0x7f45318d3f00 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91hCISeVu.MVjk\r\nFrom:
sip:10.201.8.5;tag=9CgNCj5xs3ih1R-o4eMnHS3RcqY2k0P-\r\nTo: sip:+91"..., len
= 376}}
trans = 0x7f4532942168
#3 0x00007f4573eaabe1 in relay_reply (t=0x7f4532942168,
p_msg=0x7f4577407f10, branch=0, msg_status=200, cancel_data=0x7ffcd4255b60,
do_put_on_wait=1)
at t_reply.c:2021
relay = 0
save_clone = 0
buf = 0x7f4577409540 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91hCISeVu.MVjk\r\nFrom:
sip:10.201.8.5;tag=9CgNCj5xs3ih1R-o4eMnHS3RcqY2k0P-\r\nTo: sip:+91"...
res_len = 376
relayed_code = 200
relayed_msg = 0x7f4577407f10
reply_bak = 0x1ec2dd8
bm = {to_tag_val = {s = 0x7ffcd4255910 "\220Y%\324\374\177", len =
4959780}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f4532942238
to_tag = 0x4b77854c
---Type <return> to continue, or q <return> to quit---
reason = {s = 0x1d4255910 <error: Cannot access memory at address
0x1d4255910>, len = 819120656}
onsend_params = {req = 0x69def9 <qm_insert_free+59>, rpl = 0x0,
param = 0x2000000, code = 32255512, flags = 0, branch = 0, t_rbuf = 0x1,
dst = 0x7f4530d2ca10,
send_buf = {s = 0x13eac0 <error: Cannot access memory at address
0x13eac0>, len = 8}}
ip = {af = 0, len = 4128684032, u = {addrl = {139936328892432,
140723867703520}, addr32 = {1999421456, 32581, 3559217376, 32764}, addr16 =
{49168, 30508,
32581, 0, 22752, 54309, 32764, 0}, addr =
"\020\300,wE\177\000\000\340X%\324\374\177\000"}}
__FUNCTION__ = "relay_reply"
#4 0x00007f4573eaf0c0 in reply_received (p_msg=0x7f4577407f10) at
t_reply.c:2540
msg_status = 200
last_uac_status = 0
ack = 0x0
ack_len = 3559218416
branch = 0
reply_status = 0
onreply_route = 0
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text
= {s = 0x0, len = 0}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 0}}}}
uac = 0x7f4532942380
t = 0x7f4532942168
lack_dst = {send_sock = 0xf618000000000000, to = {s = {sa_family =
34124, sa_data = "wK\000\000\000\000\020\222A\000\000\000\000"}, sin =
{sin_family = 34124,
sin_port = 19319, sin_addr = {s_addr = 0}, sin_zero =
"\020\222A\000\000\000\000"}, sin6 = {sin6_family = 34124, sin6_port =
19319, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 =
"\020\222A\000\000\000\000\000\360\\%\324\374\177\000", __u6_addr16 =
{37392, 65, 0, 0, 23792, 54309, 32764, 0},
__u6_addr32 = {4297232, 0, 3559218416, 32764}}},
sin6_scope_id = 2000513248}}, id = 32581, send_flags = {f = 34124,
blst_imask = 19319},
proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}
backup_user_from = 0x0
backup_user_to = 0x7f4577408f38
backup_domain_from = 0x7ffcd4255c00
backup_domain_to = 0x7f4577407f10
backup_uri_from = 0x419210 <_start>
backup_uri_to = 0x4b77854c
backup_xavps = 0x7f4577389c38
replies_locked = 1
branch_ret = 0
prev_branch = 0
blst_503_timeout = 0
---Type <return> to continue, or q <return> to quit---
hf = 0x0
onsend_params = {req = 0x0, rpl = 0x0, param = 0x7ffcd4255c00, code
= 1935113626, flags = 32581, branch = 0, t_rbuf = 0x7f45773d71c8, dst =
0x0, send_buf = {
s = 0xa9566b <buf+459> "", len = 11097626}}
ctx = {rec_lev = 4297232, run_flags = 0, last_retcode = -735745408,
jmp_env = {{__jmpbuf = {0, 0, 140723867704064, 7325262, 1305280, 8, 1,
139936330190760},
__mask_was_saved = -735749488, __saved_mask = {__val =
{4993055, 18446744073709551615, 139936330186512, 32255952, 1266124108,
140723867704196,
140723867704200, 139936329984224, 139936330186512,
139936330191304, 11097707, 0, 11097705, 4297232, 1266124108, 4297232}}}}}
bctx = 0x7ffcd4255cf0
keng = 0x0
ret = 32764
evname = {s = 0x7f4573f46ee2 "on_sl_reply", len = 11}
__FUNCTION__ = "reply_received"
#5 0x000000000052edf7 in do_forward_reply (msg=0x7f4577407f10, mode=0) at
core/forward.c:745
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000'
<repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0,
0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id
= 0,
send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000',
proto_pad0 = 0 '\000', proto_pad1 = 0}
new_len = 0
r = 0
ip = {af = 3559218448, len = 32764, u = {addrl = {139936279475385,
4306042945}, addr32 = {1950004409, 32581, 11075649, 1}, addr16 = {46265,
29754, 32581, 0,
65, 169, 1, 0}, addr =
"\271\264:tE\177\000\000A\000\251\000\001\000\000"}}
s = 0x419210 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w"
len = 0
__FUNCTION__ = "do_forward_reply"
#6 0x000000000053096f in forward_reply (msg=0x7f4577407f10) at
core/forward.c:846
No locals.
#7 0x000000000059444d in receive_msg (
buf=0xa954a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0,
SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"...,
len=459, rcv_info=0x7ffcd4256310) at core/receive.c:510
msg = 0x7f4577407f10
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env =
{{__jmpbuf = {1266124108, 3199900600670044426, 4297232, 140723867708032, 0,
0,
3199900600563089674, -3201500360552660726},
__mask_was_saved = 0, __saved_mask = {__val = {8, 2, 139936330063368,
140723867705200, 4993055, 0,
21483258624, 760, 1266124108, 140723867705456, 6946539,
7946256, 4294967296, 140723867705424, 5483931, 0}}}}}
---Type <return> to continue, or q <return> to quit---
bctx = 0x0
ret = 1
tvb = {tv_sec = 140723867705088, tv_usec = 139936328892432}
tve = {tv_sec = 4297232, tv_usec = 6982157}
tz = {tz_minuteswest = 0, tz_dsttime = -127}
diff = 0
inb = {
s = 0xa954a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0,
SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"...,
len = 459}
netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0}
keng = 0x0
evp = {data = 0x7ffcd4255e90, rcv = 0x7ffcd4256310, dst = 0x0}
cidlockidx = 0
cidlockset = 0
errsipmsg = 0
exectime = 0
__FUNCTION__ = "receive_msg"
#8 0x0000000000498ca7 in udp_rcv_loop () at core/udp_server.c:548
len = 459
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0,
SIP/2.0/UDP 10.201.8.5:5060
;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"...
tmp = 0x0
from = 0x7f45773e9e40
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {2032933804,
140723867706224}, addr32 = {2032933804, 0, 3559220080, 32764}, addr16 =
{7084, 31020, 0, 0, 25456,
54309, 32764, 0}, addr =
"\254\033,y\000\000\000\000pc%\324\374\177\000"}}, dst_ip = {af = 2, len =
4, u = {addrl = {84286380, 0}, addr32 = {84286380,
0, 0, 0}, addr16 = {7084, 1286, 0, 0, 0, 0, 0, 0}, addr =
"\254\033\006\005", '\000' <repeats 11 times>}}, src_port = 5090, dst_port
= 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s =
{sa_family = 2, sa_data =
"\023\342\254\033,y\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 57875, sin_addr = {s_addr = 2032933804}, sin_zero
= "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
57875,
sin6_flowinfo = 2032933804, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0}, __u6_addr32 = {0, 0,
0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f457735d510, proto = 1 '\001', proto_pad0 = 100 'd', proto_pad1 = -11227}
evp = {data = 0x0, rcv = 0x0, dst = 0x0}
printbuf =
"\020\222A\000\000\000\000\000\200j%\324\374\177\000\000\060\004~\000\000\000\000\000\000\000\000\000|\000\000\000Dg~\000\000\000\000\000\275\006~\00---Type
<return> to continue, or q <return> to quit---
0\000\000\000\000\060\000\000\000\000\000\000\000\020\300,wE\177\000\000\360a%\324\024\000\000\000\060\262x\000\000\000\000\000\023\225\200\000\000\000\000\000D\262x\000\000\000\000\000L\205wK\000\000\000\000\020\222A\000\000\000\000\000\220a%\324\374\177\000\000\032\204_\000\000\000\000\000\250\235>wE\177\000\000\060z\265",
'\000' <repeats 17 times>,
"\004\000\000\000\060b%\324\374\177\000\000~\214_\000\000\000\000\000\177\000\000\000\000\000\000\000\234\251\372\060E\177\000\000\020\222A\000\000\000\000\000"...
i = 6
j = 49280
l = 1242268245
__FUNCTION__ = "udp_rcv_loop"
#9 0x0000000000425f31 in main_loop () at main.c:1673
i = 0
pid = 0
si = 0x7f457735d510
si_desc = "udp receiver child=0 sock=172.27.6.5:5060 (
172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310\306\322\060E\177",
'\000' <repeats 14 times>,
"\001\000\000\000\360d%\324\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\212>wE\177\000"
nrprocs = 8
woneinit = 0
__FUNCTION__ = "main_loop"
#10 0x000000000042e63a in main (argc=13, argv=0x7ffcd4256a88) at main.c:2802
cfg_stream = 0x16dd010
c = -1
r = 0
tmp = 0x7ffcd4257ed8 ""
tmp_len = 2496
port = 2496
proto = 2496
ahost = 0x0
aport = 0
options = 0x77db90
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 637115892
rfd = 4
debug_save = 0
debug_flag = 0
---Type <return> to continue, or q <return> to quit---
dont_fork_cnt = 0
n_lst = 0x4000000100
p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff>
st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode =
16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 100,
st_blksize = 4096,
st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec =
629166543}, st_mtim = {tv_sec = 1586438826, tv_nsec = 779661371}, st_ctim =
{tv_sec = 1586438826,
tv_nsec = 779661371}, __unused = {0, 0, 0}}
tbuf = '\000' <repeats 376 times>...
option_index = 0
long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0,
val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val =
118}, {
name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024},
{name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {
name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val =
1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {
name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val =
1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__FUNCTION__ = "main"
(gdb)
=================================================================================================================================================
Regards,
Harneet
On Fri, Apr 3, 2020 at 11:06 PM Daniel-Constantin Mierla <miconda at gmail.com>
wrote:
> Hello,
>
> If it is something that it repeats, try to get the core file, the log
> message says core was not generated.
>
> You have to start kamailio as user root in order to have all privileges,
> then before starting kamailio, in the terminal, run:
>
> ulimit -c unlimited
>
> Now start kamailio and then reproduce the case. From the corefile using
> gdb get the output of "bt full". The corefile can be located in / folder or
> where you set the working directory via kamailio's -w command line options.
> It would be good to enable core file per process as well.
>
> Cheers,
> Daniel
> On 03.04.20 16:42, harneet singh wrote:
>
> Hi All,
>
> As part of our very low cps load at 50 calls per second through Kamailio
> Version 5.3.2 (Dispatcher Module used with Call Load based Routing -
> Algorithm 10. Machine is a Centos: 7.7 with kernel version 3.10), we are
> seeing the kamailio continuously crashing. Please see the data below for 3
> types of segfaults seen:
>
> Crash-1:
> --------
> Apr 3 19:32:25 FE-A07-34-VM6 kernel: kamailio[10382]: segfault at
> 7ff300000078 ip 00007ff300000078 sp 00007ffe5f2eafc8 error 14 in
> libbz2.so.1.0.6[7ff331582000+f000]
> Apr 3 19:32:25 FE-A07-34-VM6 mysqld: 2020-04-03 19:32:25 6131 [Warning]
> Aborted connection 6131 to db: 'kamailio' user: 'kamailio' host:
> 'localhost' (Got an error reading communication packets)
> Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10423]: CRITICAL: <core>
> [core/pass_fd.c:277]: receive_fd(): EOF on 13
> Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10381]: ALERT: <core>
> [main.c:767]: handle_sigs(): child process 10382 exited by a signal 11
> Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10381]: ALERT: <core>
> [main.c:770]: handle_sigs(): core was not generated
> ===============================
> Crash-2:
> -------
> Apr 3 19:36:11 FE-A07-34-VM6 kernel: kamailio[12838]: segfault at
> 7efe00000078 ip 00007efe00000078 sp 00007ffdc876b868 error 15 in zero
> (deleted)[7efdc407f000+40000000]
> Apr 3 19:36:11 FE-A07-34-VM6 mysqld: 2020-04-03 19:36:11 6175 [Warning]
> Aborted connection 6175 to db: 'kamailio' user: 'kamailio' host:
> 'localhost' (Got an error reading communication packets)
> Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12874]: CRITICAL: <core>
> [core/pass_fd.c:277]: receive_fd(): EOF on 18
> Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12826]: ALERT: <core>
> [main.c:767]: handle_sigs(): child process 12838 exited by a signal 11
> Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12826]: ALERT: <core>
> [main.c:770]: handle_sigs(): core was not generated
> ============================
> Crash-3:
> --------
> Apr 3 19:40:53 FE-A07-34-VM6 kernel: kamailio[13542]: segfault at 80 ip
> 000000000065e193 sp 00007ffdae10c1f0 error 4 in kamailio[400000+476000]
> Apr 3 19:40:53 FE-A07-34-VM6 mysqld: 2020-04-03 19:40:53 6222 [Warning]
> Aborted connection 6222 to db: 'kamailio' user: 'kamailio' host:
> 'localhost' (Got an error reading communication packets)
> Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13516]: ALERT: <core>
> [main.c:767]: handle_sigs(): child process 13542 exited by a signal 11
> Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13558]: CRITICAL: <core>
> [core/pass_fd.c:277]: receive_fd(): EOF on 38
> Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13516]: ALERT: <core>
> [main.c:770]: handle_sigs(): core was not generated
>
> Any pointers for resolution are most welcome as we need to quickly resolve
> this.
>
> Regards,
> Harneet Singh
> --
> "Once you eliminate the impossible, whatever remains, no matter how
> improbable, must be the truth" - Sir Arthur Conan Doyle
>
> _______________________________________________
> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>
>
--
"Once you eliminate the impossible, whatever remains, no matter how
improbable, must be the truth" - Sir Arthur Conan Doyle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200410/a55b435d/attachment.html>
More information about the sr-users
mailing list