[sr-dev] [kamailio/kamailio] kamailio Crash on qm_debug_check_frag() (#2607)
aatif218
notifications at github.com
Mon Jan 25 10:09:42 CET 2021
**Description**
facing crashes the server, raised by qm_debug_check_frag().
**Troubleshooting**
The error message:
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbdc31f360, fd 31166
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbcbf232a8, fd 31326
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbd990b9e8, fd 31327
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbd89cd1d8, fd 31360
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbcf0bab28, fd 31434
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbc6e031b0, fd 31471
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbdb7fd598, fd 31472
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbdeb334c8, fd 31494
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbddee6de8, fd 31524
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbdb91ef70, fd 31576
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbc76a1da8, fd 31690
Jan 23 12:27:31 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 31327, not in [0, 2054)
Jan 23 12:27:31 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 31360, not in [0, 2054)
Jan 23 12:27:31 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 31434, not in [0, 2054)
Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: ALERT: <core> [main.c:777]: handle_sigs(): child process 2858 exited by a signal 6
Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: ALERT: <core> [main.c:780]: handle_sigs(): core was generated
Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: CRITICAL: <core> [core/mem/q_malloc.c:138]: qm_debug_check_frag(): BUG: qm: fragm. 0x7fdbd7e8a0e8 (address 0x7fdbd7e8a120) end overwritten (5fd4cd2e, abcdefed)! Memory allocator was called from core: core/usr_avp.c:626. Fragment marked by core: core/usr_avp.c:175. Exec from core/mem/q_malloc.c:511.
The output from GDB:
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/fep-kamailio/sbin/kamailio...done.
[New LWP 2809]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/fep-kamailio/sbin/kamailio -f /usr/local/fep-kamailio/etc/kamailio/k'.
Program terminated with signal 6, Aborted.
#0 0x00007fde7acbf277 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-21.el7.x86_64 glibc-2.17-222.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64 libcurl-7.29.0-54.el7.x86_64 libgcc-4.8.5-36.el7_6.1.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-12.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-36.el7_6.1.x86_64 libunistring-0.9.3-9.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-4.el7.x86_64 nss-softokn-freebl-3.44.0-5.el7.x86_64 nss-util-3.44.0-3.el7.x86_64 openldap-2.4.44-5.el7.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x00007fde7acbf277 in raise () from /lib64/libc.so.6
#1 0x00007fde7acc0968 in abort () from /lib64/libc.so.6
#2 0x00000000006d7c95 in qm_debug_check_frag (qm=0x7fdbc25ae000, f=0x7fdbd7e8a0e8, file=0x817155 "core: core/usr_avp.c", line=626, efile=0x83f7c5 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:140
#3 0x00000000006db99e in qm_free (qmp=0x7fdbc25ae000, p=0x7fdbd7e8a120, file=0x817155 "core: core/usr_avp.c", func=0x8189a0 <__FUNCTION__.8526> "destroy_avp_list_unsafe", line=626, mname=0x817150 "core") at core/mem/q_malloc.c:511
#4 0x00000000005e7ede in destroy_avp_list_unsafe (list=0x7fdbd7e8a950) at core/usr_avp.c:626
#5 0x00007fde38d4bd8f in free_cell_helper (dead_cell=0x7fdbd7e8a750, silent=1, fname=0x7fde38e4267f "h_table.c", fline=466) at h_table.c:255
#6 0x00007fde38d4c9e1 in free_hash_table () at h_table.c:466
#7 0x00007fde38df9cd0 in tm_shutdown () at t_funcs.c:88
#8 0x0000000000581960 in destroy_modules () at core/sr_module.c:746
#9 0x000000000041cda7 in cleanup (show_status=1) at main.c:563
#10 0x000000000041e682 in shutdown_children (sig=15, show_status=1) at main.c:706
#11 0x0000000000421715 in handle_sigs () at main.c:806
#12 0x000000000042b9a7 in main_loop () at main.c:1817
#13 0x0000000000433b96 in main (argc=9, argv=0x7ffea05cd2c8) at main.c:2856
(gdb) bt full
#0 0x00007fde7acbf277 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007fde7acc0968 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00000000006d7c95 in qm_debug_check_frag (qm=0x7fdbc25ae000, f=0x7fdbd7e8a0e8, file=0x817155 "core: core/usr_avp.c", line=626, efile=0x83f7c5 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:140
p = 0x7ffea05cc3b0
__FUNCTION__ = "qm_debug_check_frag"
#3 0x00000000006db99e in qm_free (qmp=0x7fdbc25ae000, p=0x7fdbd7e8a120, file=0x817155 "core: core/usr_avp.c", func=0x8189a0 <__FUNCTION__.8526> "destroy_avp_list_unsafe", line=626, mname=0x817150 "core") at core/mem/q_malloc.c:511
qm = 0x7fdbc25ae000
f = 0x7fdbd7e8a0e8
size = 408
next = 0x7fdbd7e8a5c8
prev = 0x7fdbd7e8a1d8
__FUNCTION__ = "qm_free"
#4 0x00000000005e7ede in destroy_avp_list_unsafe (list=0x7fdbd7e8a950) at core/usr_avp.c:626
avp = 0x7fdbd7e8a080
foo = 0x7fdbd7e8a120
__FUNCTION__ = "destroy_avp_list_unsafe"
#5 0x00007fde38d4bd8f in free_cell_helper (dead_cell=0x7fdbd7e8a750, silent=1, fname=0x7fde38e4267f "h_table.c", fline=466) at h_table.c:255
b = 0x0
i = 1
rpl = 0x0
tt = 0x0
foo = 0x7fde34a5df80 <__FUNCTION__.7259>
cbs = 0x0
cbs_tmp = 0x7fde388fe570
__FUNCTION__ = "free_cell_helper"
#6 0x00007fde38d4c9e1 in free_hash_table () at h_table.c:466
p_cell = 0x7fdbd7e8a750
tmp_cell = 0x7fdbc27e3060
i = 40845
__FUNCTION__ = "free_hash_table"
#7 0x00007fde38df9cd0 in tm_shutdown () at t_funcs.c:88
__FUNCTION__ = "tm_shutdown"
#8 0x0000000000581960 in destroy_modules () at core/sr_module.c:746
t = 0x7fde3c51e4c0
foo = 0x7fde3c51e020
__FUNCTION__ = "destroy_modules"
#9 0x000000000041cda7 in cleanup (show_status=1) at main.c:563
memlog = 0
__FUNCTION__ = "cleanup"
#10 0x000000000041e682 in shutdown_children (sig=15, show_status=1) at main.c:706
__FUNCTION__ = "shutdown_children"
#11 0x0000000000421715 in handle_sigs () at main.c:806
chld = 0
chld_status = 134
any_chld_stopped = 1
memlog = -1027733624
__FUNCTION__ = "handle_sigs"
#12 0x000000000042b9a7 in main_loop () at main.c:1817
i = 10
pid = 2893
si = 0x0
si_desc = "udp receiver child=9 sock=10.50.7.18:5060\000:0:0:1]:5060\000:5060)\000\000\000\004\000\000\000\000\000\000\000\000\340Z\302\333\177\000\000\000\000\000\000\000\000\000\000 \006\276\302\333\177\000\000`\315\\\240\376\177\000\000\220\006\276\302\333\177\000\000r\fb3\336\177\000\000P\246\222<\336\177\000"
nrprocs = 10
woneinit = 1
__FUNCTION__ = "main_loop"
#13 0x0000000000433b96 in main (argc=9, argv=0x7ffea05cd2c8) at main.c:2856
**Additional Information**
Kamailio Version - output of Kamailio -v
version: kamailio 5.4.2 (x86_64/linux) f8885c
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: f8885c
compiled on 04:30:53 Dec 16 2020 with gcc 4.8.5
**Operating System:**
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
**Note:**
It looks similar to the following issue
https://github.com/kamailio/kamailio/issues/2503
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2607
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-dev/attachments/20210125/840b2ff6/attachment-0001.htm>
More information about the sr-dev
mailing list