Re: [SR-Users] Kamailio 5.3.2 crashing continuously

Hi Daniel, Please see the requested details below: SETUP: Kamailio Version 5.3.3 being used. Similar issues were seen in Kamailio 5.3.2 as well so we upgraded anyhow. Kamailio is acting as a Call Stateful Proxy with Dispatcher module enabled with Algorith-10 being used(Call Load based Routing) though there is only one UAS in our case for now. The load is a very simple one. UAC(Sipp) --------- Kamailio ----------- UAS(Sipp) Initial INVITE-- 200OK -- ACK happens properly. Then UAS sends an UPDATE after 1 minute which is passed properly to UAC which sends a 200OK and that reaches UAS. After another minute, the UAS sends another refresh UPDATE to Kamailio and at the same time UAC sends BYE for the call. Please note that this is a very time sensitive issue and happens only when such a glare happens at Kamailio. Crash is not seen with a single call. However, at 50 cps, the crashes appears at a regular interval of 4-5 minutes. Crash-1 and Crash-2 are seen almost at the same timestamp whereas the Crash-3 is seen on the new restarted kamailio instance that is handling calls and the in-dialog messages of the prevoious kamailio instance. [root@localhost tmp]# ll total 3161744 -rw------- 1 kamailio kamailio 1111052288 Apr 9 18:57 core.kamailio.995.1586438821.869 ----> CRASH-1 -rw------- 1 kamailio kamailio 1110773760 Apr 9 18:57 core.kamailio.995.1586438824.867 ----> CRASH-2 -rw------- 1 kamailio kamailio 1111052288 Apr 9 18:59 core.kamailio.995.1586438991.3775 ----> CRASH-3 CRASH-1: --------- Logs: Apr 9 18:57:01 localhost /usr/local/sbin/kamailio[869]: CRITICAL: {1 10648 INVITE NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND} <core> [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(78, abcdefed)[0x7f7a5bd1e5d8:0x7f7a5bd1e610]! Memory allocator was called from dialog: dlg_hash.c:544. Fragment marked by dialog: dlg_handlers.c:308. Exec from core/mem/q_malloc.c:391. Apr 9 18:57:01 localhost /usr/local/sbin/kamailio[869]: CRITICAL: {1 10648 INVITE NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND} <core> [core/mem/q_malloc.c:155]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten [0x7f7a5bd1e568:0x7f7a5bd1e5a0] - fragment marked by dialog: dlg_handlers.c:308 Apr 9 18:57:03 localhost mysqld: 2020-04-09 18:57:03 276 [Warning] Aborted connection 276 to db: 'kamailio' user: 'kamailio' host: 'localhost' (Got an error reading communication packets) Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: ALERT: <core> [main.c:767]: handle_sigs(): child process 869 exited by a signal 6 Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: ALERT: <core> [main.c:770]: handle_sigs(): core was generated Full BackTrace: [root@localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb /usr/local/sbin/kamailio core.kamailio.995.1586438821.869 GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/kamailio...done. [New LWP 869] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f7aa3a29337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) bt full #0 0x00007f7aa3a29337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 resultvar = 0 pid = 869 selftid = 869 #1 0x00007f7aa3a2aa28 in __GI_abort () at abort.c:90 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x7cafe4, sa_sigaction = 0x7cafe4}, sa_mask = {__val = {50, 11474112, 0, 851352, 1301680, 1304944, 8, 2, 21474836483, 140163493761024, 9015141347359, 98784247808, 140163493996064, 32252848, 140164708874416, 8171492}}, sa_flags = 50, sa_restorer = 0x7f7aa3aeb8b0 <__syslog>} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x000000000069deac in qm_debug_check_frag (qm=0x7f7a5b419000, f=0x7f7a5bd1e5d8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", line=544, efile=0x8094ff "core/mem/q_malloc.c", eline=391) at core/mem/q_malloc.c:158 p = 0x7f7a5bd1e568 __FUNCTION__ = "qm_debug_check_frag" #3 0x000000000069f9b9 in qm_malloc (qmp=0x7f7a5b419000, size=8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", func=0x7f7a9b8e4f30 <__FUNCTION__.13318> "dlg_set_leg_info", line=544, mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:391 qm = 0x7f7a5b419000 f = 0x7f7a5bd1e5d8 hash = 2 list_cntr = 1 __FUNCTION__ = "qm_malloc" #4 0x00000000006aaa6d in qm_shm_malloc (qmp=0x7f7a5b419000, size=5, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", func=0x7f7a9b8e4f30 <__FUNCTION__.13318> "dlg_set_leg_info", line=544, mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:1226 r = 0x7f7a5b9b7050 #5 0x00007f7a9b850fcd in dlg_set_leg_info (dlg=0x7f7a5b9ba288, tag=0x7f7aa1b2f210, rr=0x7ffc64edf810, contact=0x7ffc64edf820, cseq=0x7ffc64edf830, leg=0) at dlg_hash.c:544 cs = { s = 0xa955fc <buf+348> "10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: replaces, 100rel, timer, norefersub\r\nSession-Expires: 120\r\nMin-SE: 120\r\nUser-"..., len = 5} __FUNCTION__ = "dlg_set_leg_info" #6 0x00007f7a9b87bd78 in populate_leg_info (dlg=0x7f7a5b9ba288, msg=0x7f7aa1b2df10, t=0x0, leg=0, tag=0x7f7aa1b2f210) at dlg_handlers.c:266 skip_recs = 0 own_rr = 0 ---Type <return> to continue, or q <return> to quit--- cseq = { s = 0xa955fc <buf+348> "10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: replaces, 100rel, timer, norefersub\r\nSession-Expires: 120\r\nMin-SE: 120\r\nUser-"..., len = 5} contact = { s = 0xa955b2 <buf+274> "sip:10.201.8.5:5060;ob>\r\nCall-ID: NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: rep"..., len = 22} rr_set = {s = 0x0, len = 0} __FUNCTION__ = "populate_leg_info" #7 0x00007f7a9b883f38 in dlg_new_dialog (req=0x7f7aa1b2df10, t=0x0, run_initial_cbs=1) at dlg_handlers.c:949 dlg = 0x7f7a5b9ba288 s = {s = 0xf614f80000000000 <error: Cannot access memory at address 0xf614f80000000000>, len = -1582559920} callid = { s = 0xa955d4 <buf+308> "NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: replaces, 100rel, timer, norefersub\r\n"..., len = 32} ftag = { s = 0xa95564 <buf+196> "9s2l4B7KJz0pm8tKXwR39PGRVnJm8sK3\r\nTo: sip:+918123088839@172.27.6.5\r\nContact: <sip:10.201.8.5:5060;ob>\r\nCall-ID: NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CA"..., len = 32} ttag = {s = 0x0, len = 0} req_uri = {s = 0x7f7aa1a5d910 "sip:+918123088839@172.27.6.5:5060;transport=udp", len = 47} dir = 0 __FUNCTION__ = "dlg_new_dialog" #8 0x00007f7a9b890325 in dlg_manage (msg=0x7f7aa1b2df10) at dlg_handlers.c:1864 tag = {s = 0x0, len = 0} backup_mode = 0 dlg = 0x0 t = 0x0 __FUNCTION__ = "dlg_manage" #9 0x00007f7a9b8c869c in w_dlg_manage (msg=0x7f7aa1b2df10, s1=0x0, s2=0x0) at dialog.c:1094 No locals. #10 0x0000000000458b36 in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ac3fb0, msg=0x7f7aa1b2df10) at core/action.c:1071 ret = -5 v = 1 dst = {send_sock = 0x419210 <_start>, to = {s = {sa_family = 7776, sa_data = "\356d\374\177\000\000\000\000\000\000\000\000\000"}, sin = {sin_family = 7776, ---Type <return> to continue, or q <return> to quit--- sin_port = 25838, sin_addr = {s_addr = 32764}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 7776, sin6_port = 25838, sin6_flowinfo = 32764, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 1693319264}}, id = 32764, send_flags = {f = 56144, blst_imask = 102}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} tmp = 0x0 new_uri = 0x0 end = 0x7f7a00000000 <error: Cannot access memory at address 0x7f7a00000000> crt = 0x0 cmd = 0x7f7aa1a9a230 len = 32634 user = -1638375131 uri = {user = {s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, passwd = {s = 0x0, len = 0}, host = { s = 0x7ffc64edfee0 "`", len = 6739658}, port = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 1300448}, params = { s = 0x7ffc64edfdb0 "\310\375\253\241z\177", len = 4993055}, sip_params = {s = 0x0, len = -1583406912}, headers = { s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = -1582113008}, port_no = 64968, proto = 41387, type = 32634, flags = (unknown: 1693318768), transport = {s = 0x7d4950 "core", len = 1}, ttl = {s = 0x7ffc64edfee0 "`", len = 6721818}, user_param = { s = 0x7f7aa1ac2938 "\001", len = 1693319192}, maddr = {s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, method = { s = 0x1000cf960 <error: Cannot access memory at address 0x1000cf960>, len = 0}, lr = {s = 0x13e970 <error: Cannot access memory at address 0x13e970>, len = 0}, r2 = {s = 0x7f7aa1ac1fb8 "\004", len = 1693319576}, gr = {s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, transport_val = { s = 0x7ffc64edff50 "`", len = 4611518}, ttl_val = {s = 0x7ffc64edfed0 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1abfdc8 "\316\002", len = 1693323440}, maddr_val = {s = 0x0, len = 1}, method_val = {s = 0x60bb3a29 <error: Cannot access memory at address 0x60bb3a29>, len = -1251003067}, lr_val = {s = 0x400000000 <error: Cannot access memory at address 0x400000000>, len = 1693319572}, r2_val = {s = 0x0, len = 0}, gr_val = { s = 0x46dc132b38f3545 <error: Cannot access memory at address 0x46dc132b38f3545>, len = 1622882857}} next_hop = {user = {s = 0x0, len = 1693319040}, passwd = {s = 0x7ffc64ee0000 "`", len = 6729767}, host = { s = 0x1 <error: Cannot access memory at address 0x1>, len = 45273232}, port = {s = 0x7ffc64edfdd0 "\340\376\355d\374\177", len = 4}, params = { s = 0x7f7aa1a934e0 "\364.7\235z\177", len = -1548974973}, sip_params = {s = 0x7f7aa1ac5320 "\002", len = 1}, headers = { s = 0x7f7a00000000 <error: Cannot access memory at address 0x7f7a00000000>, len = -1582113008}, port_no = 6760, proto = 41388, type = 32634, flags = (unknown: 1693318384), transport = {s = 0x7f7aa1b2df10 ">\330\001", len = -1}, ttl = {s = 0x7ffc64edfd60 "\340\376\355d\374\177", len = 6721818}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x7f7aa1aa62b8 "", len = -1582113008}, method = { s = 0x1ffffffff <error: Cannot access memory at address 0x1ffffffff>, len = 0}, lr = { s = 0xf612d00000000000 <error: Cannot access memory at address 0xf612d00000000000>, len = 0}, r2 = {s = 0x7f7aa1ac2940 "\004", len = 1693319192}, gr = { s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, transport_val = {s = 0x7ffc64edfdd0 "\340\376\355d\374\177", len = 4611518}, ttl_val = { s = 0x7f7aa1a2a520 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1ac1a68 "\333\002", len = 1693323440}, maddr_val = {s = 0x0, len = 1}, ---Type <return> to continue, or q <return> to quit--- method_val = {s = 0x60bb3a29 <error: Cannot access memory at address 0x60bb3a29>, len = -1334889147}, lr_val = { s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, r2_val = {s = 0x0, len = 0}, gr_val = { s = 0x46dc132be8f3545 <error: Cannot access memory at address 0x46dc132be8f3545>, len = 1622882857}} u = 0x7f7aa1b2f4b0 port = 0 dst_host = 0x0 i = 6741152 flags = -74119067 avp = 0x7ffc64ee01e0 st = {flags = 4297232, id = 0, name = {n = 1693326944, s = {s = 0x7ffc64ee1e60 "\r", len = 0}, re = 0x7ffc64ee1e60}, avp = 0x0} sct = 0x0 sjt = 0x7f7aa1b2df10 rve = 0x0 mct = 0x4000000000 rv = 0x46dc132bf6f3545 rv1 = 0x0 c1 = {cache_type = 1693319040, val_type = 32764, c = {avp_val = {n = -1582113008, s = {s = 0x7f7aa1b2df10 ">\330\001", len = -1582553112}, re = 0x7f7aa1b2df10}, pval = {rs = {s = 0x7f7aa1b2df10 ">\330\001", len = -1582553112}, ri = 1693323440, flags = 32764}}, i2s = "\000\000\000\000\000\000\000\000\377\377\377\377\000\000\000\000):\273`\000"} s = {s = 0x7ffc64edfc50 "`\375\355d\374\177", len = 4611518} srevp = {0x0, 0x0} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = { s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #11 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1ac3fb0, msg=0x7f7aa1b2df10) at core/action.c:1576 t = 0x7f7aa1ac3fb0 ret = -1 ---Type <return> to continue, or q <return> to quit--- tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} tz = {tz_minuteswest = 0, tz_dsttime = 4} tdiff = 1 __FUNCTION__ = "run_actions" #12 0x0000000000458aa5 in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ac4100, msg=0x7f7aa1b2df10) at core/action.c:1062 ret = 1 v = 1 dst = {send_sock = 0x0, to = {s = {sa_family = 1856, sa_data = "\356d\374\177\000\000\360\026\253\241z\177\000"}, sin = {sin_family = 1856, sin_port = 25838, sin_addr = {s_addr = 32764}, sin_zero = "\360\026\253\241z\177\000"}, sin6 = {sin6_family = 1856, sin6_port = 25838, sin6_flowinfo = 32764, sin6_addr = { __in6_u = {__u6_addr8 = "\360\026\253\241z\177\000\000\020\337\262\241z\177\000", __u6_addr16 = {5872, 41387, 32634, 0, 57104, 41394, 32634, 0}, __u6_addr32 = {2712344304, 32634, 2712854288, 32634}}}, sin6_scope_id = 1693321136}}, id = 32764, send_flags = {f = 54986, blst_imask = 102}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} tmp = 0x7f7a9c5ecc83 <acc_log_request+8047> "\270\001" new_uri = 0x7f7aa1b2df10 ">\330\001" end = 0x7f7aa1ab16d0 "\a" crt = 0x7ffc64ee08e0 "\300\t\356d\374\177" cmd = 0x7f7aa1a949c8 len = 0 user = 0 uri = {user = {s = 0x0, len = 1559864800}, passwd = {s = 0x2f00000000 <error: Cannot access memory at address 0x2f00000000>, len = 1536887583}, host = { s = 0x0, len = 0}, port = {s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, params = {s = 0x0, len = 0}, sip_params = {s = 0x7ffc64ee0550 "\360\026\253\241z\177", len = 7393852}, headers = {s = 0x7f7a5b41bb90 "", len = -1582113008}, port_no = 43232, proto = 41386, type = 32634, flags = (unknown: 1693320640), transport = {s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = -1}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x7f7aa1b2e1b0 "\253T\251", len = 1693321024}, method = {s = 0x7f7aa1ab16f0 "", len = -1582113008}, lr = {s = 0x7ffc64ee05f0 "p\006\356d\374\177", len = -1647784851}, r2 = { s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, gr = {s = 0x0, len = 0}, transport_val = { s = 0x7ffc64ee0740 "", len = -1582622992}, ttl_val = {s = 0x7f7aa1b2e1b0 "\253T\251", len = -1582113008}, user_param_val = {s = 0x7f7aa1aaa8e0 "q\002", len = 1693323440}, maddr_val = {s = 0x0, len = 1622882857}, method_val = { s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, lr_val = {s = 0x0, len = 0}, r2_val = { s = 0x7ffc64ee0670 "\360\006\356d\374\177", len = -1647782719}, gr_val = { s = 0x46dc13540ef3545 <error: Cannot access memory at address 0x46dc13540ef3545>, len = -282512059}} next_hop = {user = {s = 0xa8428197 <error: Cannot access memory at address 0xa8428197>, len = -1541505685}, passwd = { ---Type <return> to continue, or q <return> to quit--- s = 0x19 <error: Cannot access memory at address 0x19>, len = 1693320032}, host = { s = 0xf611b40000000000 <error: Cannot access memory at address 0xf611b40000000000>, len = 6982157}, port = {s = 0x7ffc64ee0380 "\310i\252\241z\177", len = -1583407088}, params = {s = 0x7f7aa1a934e0 "\364.7\235z\177", len = -1583407088}, sip_params = {s = 0x7ffc64ee03c0 "\360\003\356d\374\177", len = 6995517}, headers = {s = 0x7f7aa1a2a520 "", len = -1582113008}, port_no = 27080, proto = 41386, type = 32634, flags = (unknown: 1693320256), transport = {s = 0xcfff0 <error: Cannot access memory at address 0xcfff0>, len = 1301656}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x7ffc64ee03f0 "\020\222A", len = 4993055}, method = {s = 0xffffffff00000000 <error: Cannot access memory at address 0xffffffff00000000>, len = 0}, lr = {s = 0x1ec23d0 <error: Cannot access memory at address 0x1ec23d0>, len = 1622882857}, r2 = { s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, gr = {s = 0x0, len = 0}, transport_val = { s = 0x7ffc64ee0520 "", len = 4611518}, ttl_val = {s = 0x7ffc64ee0520 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1aa69c8 "X\002", len = 1693323440}, maddr_val = {s = 0x0, len = 1622882857}, method_val = { s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, lr_val = {s = 0x0, len = 0}, r2_val = { s = 0x7ffc64ee0590 "@\a\356d\374\177", len = 4609789}, gr_val = {s = 0x46dc1354fef3545 <error: Cannot access memory at address 0x46dc1354fef3545>, len = -282512059}} u = 0x0 port = 0 dst_host = 0x7ffc64ee07b0 i = 6744853 flags = 0 avp = 0x7ffc64ee0930 st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len = 1693320208}, re = 0x0}, avp = 0x4656fd <run_actions+1860>} sct = 0x7f7a9c616b90 <__FUNCTION__.11588> sjt = 0x419c61f7fc rve = 0x7f7aa1ac3798 mct = 0x7f7a9c615c60 rv = 0x7ffc64ee1e60 rv1 = 0x7ffc64ee10b0 c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 0, s = {s = 0x0, len = 1}, re = 0x0}, pval = {rs = {s = 0x0, len = 1}, ri = 0, flags = 0}}, i2s = "\260Q\260\241z\177\000\000):\273`\000\000\000\000\020\222A\000\000"} s = {s = 0x0, len = 0} srevp = {0x7ffc64ee0860, 0x7ffc64ee0418} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = { ---Type <return> to continue, or q <return> to quit--- s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #13 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1abd308, msg=0x7f7aa1b2df10) at core/action.c:1576 t = 0x7f7aa1ac4100 ret = 1 tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} tz = {tz_minuteswest = -1671324464, tz_dsttime = 32634} tdiff = 2712291872 __FUNCTION__ = "run_actions" #14 0x000000000045555c in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ab2d40, msg=0x7f7aa1b2df10) at core/action.c:695 ret = -5 v = 0 dst = {send_sock = 0x823590, to = {s = {sa_family = 33536, sa_data = "\200\000V\001\000\000\260d\202\000\000\000\000"}, sin = {sin_family = 33536, sin_port = 128, sin_addr = {s_addr = 342}, sin_zero = "\260d\202\000\000\000\000"}, sin6 = {sin6_family = 33536, sin6_port = 128, sin6_flowinfo = 342, sin6_addr = {__in6_u = {__u6_addr8 = "\260d\202\000\000\000\000\000\030:\202\000\000\000\000", __u6_addr16 = {25776, 130, 0, 0, 14872, 130, 0, 0}, __u6_addr32 = {8545456, 0, 8534552, 0}}}, sin6_scope_id = 2712859096}}, id = 32634, send_flags = {f = 8208, blst_imask = 41375}, proto = 122 'z', proto_pad0 = 127 '\177', proto_pad1 = 0} tmp = 0x7ffc64ee10d0 "\020\222A" new_uri = 0x7ffc64ee0f64 "\374\177" end = 0x1ec3308 <error: Cannot access memory at address 0x1ec3308> crt = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w" cmd = 0x7f7aa1a931d0 len = 0 user = 1622882857 uri = {user = {s = 0x8163a0 "core", len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0xc300000000 <error: Cannot access memory at address 0xc300000000>, len = 11097831}, port = {s = 0x164ee0c50 <error: Cannot access memory at address 0x164ee0c50>, len = 11098026}, params = {s = 0x7ffc64ee0cb0 "\001", len = 7299836}, sip_params = {s = 0x7ffc64ee0d42 "\251", len = 6982157}, headers = {s = 0x7ffc64ee0c80 "", len = -1583407088}, port_no = 0, proto = 0, type = 4128681984, flags = (unknown: 2711560208), transport = {s = 0x7ffc64ee0cc0 "\360\f\356d\374\177", len = -1632797901}, ttl = { ---Type <return> to continue, or q <return> to quit--- s = 0x69def9 <qm_insert_free+59> "\005\363\a", len = 0}, user_param = {s = 0x2000000 <error: Cannot access memory at address 0x2000000>, len = 32255496}, maddr = {s = 0xcf960 <error: Cannot access memory at address 0xcf960>, len = 1298936}, method = { s = 0x13e970 <error: Cannot access memory at address 0x13e970>, len = 8}, lr = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 6938361}, r2 = {s = 0x7ffc64ee0cf0 "\020\016\356d\374\177", len = 4993055}, gr = {s = 0x0, len = -1583176416}, transport_val = { s = 0x80ba1a2a520 <error: Cannot access memory at address 0x80ba1a2a520>, len = -1583176416}, ttl_val = {s = 0x7ffc64ee0e10 "\340\016\356d\374\177", len = 6955649}, user_param_val = {s = 0x1ec2dc8 <error: Cannot access memory at address 0x1ec2dc8>, len = 1622882857}, maddr_val = { s = 0x7ffc64ee0e10 "\340\016\356d\374\177", len = 6946539}, method_val = {s = 0x823590 "core", len = 11097317}, lr_val = { s = 0x13e970 <error: Cannot access memory at address 0x13e970>, len = -1582108200}, r2_val = {s = 0xa957aa <buf+778> "", len = 11097873}, gr_val = { s = 0x7ffc64ee0d80 "\330\361\262\241z\177", len = 4993055}} next_hop = {user = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 4959780}, passwd = {s = 0x7ffc64ee0aa0 "\300\v\356d\374\177", len = 4993055}, host = {s = 0x0, len = -1583397840}, port = {s = 0x5200000000 <error: Cannot access memory at address 0x5200000000>, len = -1583397840}, params = { s = 0x7ffc64ee0bc0 "H", len = 6955649}, sip_params = {s = 0x2009e6676b0 <error: Cannot access memory at address 0x2009e6676b0>, len = 1548742352}, headers = {s = 0x7f7a9e66e830 <__FUNCTION__.12289> "update_totag_set", len = -1637445885}, port_no = 4976, proto = 175, type = ERROR_URI_T, flags = (unknown: 11475272), transport = {s = 0xaf1940 <def_list+32> "", len = 11475256}, ttl = { s = 0x5000af1930 <error: Cannot access memory at address 0x5000af1930>, len = 1548742368}, user_param = {s = 0xaf1920 <def_list> "@t\233[z\177", len = 0}, maddr = {s = 0x7f7a9e6676b0 "tm", len = 6982157}, method = {s = 0x7ffc64ee0b60 "", len = -1583407088}, lr = {s = 0x7f7aa1b2f780 "\300 \237\241z\177", len = -1583407088}, r2 = {s = 0x7ffc64ee0ba0 "\320\f\356d\374\177", len = -1632797901}, gr = {s = 0x7f7aa19f27c0 "", len = 0}, transport_val = { s = 0x2000000 <error: Cannot access memory at address 0x2000000>, len = 32258408}, ttl_val = { s = 0xcf418 <error: Cannot access memory at address 0xcf418>, len = 1296024}, user_param_val = { s = 0x40013e970 <error: Cannot access memory at address 0x40013e970>, len = 8479648}, maddr_val = {s = 0x1 <error: Cannot access memory at address 0x1>, len = -1582235496}, method_val = {s = 0x7ffc64ee0cd0 "", len = 7110832}, lr_val = {s = 0x0, len = 8422144}, r2_val = { s = 0x48 <error: Cannot access memory at address 0x48>, len = 1622882857}, gr_val = {s = 0x7ffc64ee0cd0 "", len = 6946539}} u = 0xf6199c0000000000 port = 0 dst_host = 0x0 i = 2 flags = 0 avp = 0xa957aa <buf+778> st = {flags = 33554432, id = 0, name = {n = 32253432, s = {s = 0x1ec25f8 <error: Cannot access memory at address 0x1ec25f8>, len = 851296}, re = 0x1ec25f8}, avp = 0x13da08} sct = 0x7ffc64ee0e40 sjt = 0x7f7aa1b2f1a0 rve = 0x7f7aa1ab2528 ---Type <return> to continue, or q <return> to quit--- mct = 0x1ec2e08 rv = 0x0 rv1 = 0x0 c1 = {cache_type = 1693321776, val_type = 32764, c = {avp_val = {n = -1583407088, s = {s = 0x7f7aa19f2010 "\001", len = 391}, re = 0x7f7aa19f2010}, pval = { rs = {s = 0x7f7aa19f2010 "\001", len = 391}, ri = -1583407088, flags = 32634}}, i2s = "p\n\356d\374\177\000\000\063{\255\236z\177\000\000HF\267\\z\177"} s = {s = 0x300000002 <error: Cannot access memory at address 0x300000002>, len = 6982157} srevp = {0x0, 0x8094ff} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = { s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #15 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1aa48d0, msg=0x7f7aa1b2df10) at core/action.c:1576 t = 0x7f7aa1ab2d40 ret = 1 tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} tz = {tz_minuteswest = 11098026, tz_dsttime = 0} tdiff = 1622882857 __FUNCTION__ = "run_actions" #16 0x0000000000465e86 in run_top_route (a=0x7f7aa1aa48d0, msg=0x7f7aa1b2df10, c=0x0) at core/action.c:1661 ctx = {rec_lev = 3, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {1622882857, 319123583751107909, 4297232, 140722001813088, 0, 0, 319123583451215173, -318338964762577595}, __mask_was_saved = 0, __saved_mask = {__val = {0, 4611686022722355200, 140164675591952, 140722001809824, 5326593, 4297232, 140722001813088, 0, 0, 5988290944, 140164675591952, 1073741825, 0, 140164675389112, 1622882857, 4297232}}}}} p = 0x7ffc64ee10b0 ret = 0 sfbk = 0 #17 0x000000000059344c in receive_msg ( buf=0xa954a0 <buf> "INVITE sip:+918123088839@172.27.6.5:5060;transport=udp SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZ---Type <return> to continue, or q <return> to quit--- H0UFZ5Xj2\r\nMax-Forwards: 69\r\nFrom: sip:10.201.8.5;tag=9s2l"..., len=778, rcv_info=0x7ffc64ee16f0) at core/receive.c:424 msg = 0x7f7aa1b2df10 ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {1622882857, 319123583751107909, 4297232, 140722001813088, 0, 0, 319123583451215173, -318338964762577595}, __mask_was_saved = 0, __saved_mask = {__val = {8, 2, 140164675468656, 140722001810256, 4993055, 0, 21483258624, 912, 1622882857, 140722001810512, 6946539, 7946256, 8589934592, 140722001810480, 5483931, 0}}}}} bctx = 0x0 ret = -1 tvb = {tv_sec = 140722001810144, tv_usec = 140164674297872} tve = {tv_sec = 4297232, tv_usec = 6982157} tz = {tz_minuteswest = 0, tz_dsttime = -127} diff = 0 inb = { s = 0xa954a0 <buf> "INVITE sip:+918123088839@172.27.6.5:5060;transport=udp SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZH0UFZ5Xj2\r\nMax-Forwards: 69\r\nFrom: sip:10.201.8.5;tag=9s2l"..., len = 778} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0} keng = 0x0 evp = {data = 0x7ffc64ee1270, rcv = 0x7ffc64ee16f0, dst = 0x0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 exectime = 0 __FUNCTION__ = "receive_msg" #18 0x0000000000498ca7 in udp_rcv_loop () at core/udp_server.c:548 len = 778 buf = "INVITE sip:+918123088839@172.27.6.5:5060;transport=udp SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZH0UFZ5Xj2\r\nMax-Forwards: 69\r\nFrom: sip:10.201.8.5;tag=9s2l"... tmp = 0x0 from = 0x7f7aa1b0fda8 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {84461834, 140722001811280}, addr32 = {84461834, 0, 1693325136, 32764}, addr16 = {51466, 1288, 0, 0, 5968, 25838, 32764, 0}, addr = "\n\311\b\005\000\000\000\000P\027\356d\374\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {84286380, 0}, addr32 = { 84286380, 0, 0, 0}, addr16 = {7084, 1286, 0, 0, 0, 0, 0, 0}, addr = "\254\033\006\005", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\n\311\b\005\000\000\000\000\000\000\000"}, sin = { ---Type <return> to continue, or q <return> to quit--- sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 84461834}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 84461834, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f7aa1a83510, proto = 1 '\001', proto_pad0 = 24 '\030', proto_pad1 = 25838} evp = {data = 0x0, rcv = 0x0, dst = 0x0} printbuf = "\020\222A\000\000\000\000\000`\036\356d\374\177", '\000' <repeats 18 times>, "\260\025\356d\374\177\000\000\233\255S", '\000' <repeats 21 times>, "\320\025\356d\374\177\000\000\233\255S\000\000\000\000\000\023\225\200\000\000\000\000\000\bz}\000\000\000\000\000 ", '\000' <repeats 13 times>, "\200\300\000\000\000\000@~J\377\037\060L\000\017\000\000\000\000\026\356d\035\000\000\061\240\025\356d\374\177\000\000 \030\356d\002\000\000\000`\241\251\241z\177\000\000\000\325w\000\000\000\000\000\b\027\356d\374\177\000\000\177\000\000\000\000\000\000\000\060\nm[z\177\000\000\020\222A\000\000\000\000\000"... i = -1 j = 49280 l = 1194642238 __FUNCTION__ = "udp_rcv_loop" #19 0x0000000000425f31 in main_loop () at main.c:1673 i = 1 pid = 0 si = 0x7f7aa1a83510 si_desc = "udp receiver child=1 sock=172.27.6.5:5060 ( 172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310&E[z\177", '\000' <repeats 14 times>, "\001\000\000\000\320\030\356d\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\352\260\241z\177\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #20 0x000000000042e63a in main (argc=13, argv=0x7ffc64ee1e68) at main.c:2802 cfg_stream = 0x2a68010 c = -1 r = 0 tmp = 0x7ffc64ee2ed8 "" tmp_len = 2496 port = 2496 proto = 2496 ahost = 0x0 aport = 0 options = 0x77db90 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 ---Type <return> to continue, or q <return> to quit--- seed = 2887616840 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x4000000100 p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode = 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec = 629166543}, st_mtim = {tv_sec = 1586435764, tv_nsec = 286216393}, st_ctim = {tv_sec = 1586435764, tv_nsec = 286216393}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 376 times>... option_index = 0 long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val = 118}, { name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, { name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) ========================================================================================================================================== CRASH-2: ------- Logs: Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: CRITICAL: <core> [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(78, abcdefed)[0x7f7a5bd1e5d8:0x7f7a5bd1e610]! Memory allocator was called from dialog: dlg_hash.c:392. Fragment marked by dialog: dlg_handlers.c:308. Exec from core/mem/q_malloc.c:566. Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: CRITICAL: <core> [core/mem/q_malloc.c:155]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten [0x7f7a5bd1e568:0x7f7a5bd1e5a0] - fragment marked by dialog: dlg_handlers.c:308 Apr 9 18:57:06 localhost systemd: kamailio.service: main process exited, code=dumped, status=6/ABRT Full BackTrace: [root@localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb /usr/local/sbin/kamailio core.kamailio.995.1586438824.867 GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/kamailio...done. [New LWP 867] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f7aa3a29337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) bt full #0 0x00007f7aa3a29337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 resultvar = 0 pid = 867 selftid = 867 #1 0x00007f7aa3a2aa28 in __GI_abort () at abort.c:90 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x7cafe4, sa_sigaction = 0x7cafe4}, sa_mask = {__val = {8424376, 8442432, 140161962737664, 0, 0, 0, 0, 0, 21474836483, 140163493761024, 9015136354304, 64355789963264, 140163493996064, 0, 140164708874416, 8171492}}, sa_flags = -1685147642, sa_restorer = 0x7f7aa3aeb8b0 <__syslog>} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x000000000069deac in qm_debug_check_frag (qm=0x7f7a5b419000, f=0x7f7a5bd1e5d8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", line=392, efile=0x8094ff "core/mem/q_malloc.c", eline=566) at core/mem/q_malloc.c:158 p = 0x7f7a5bd1e568 __FUNCTION__ = "qm_debug_check_frag" #3 0x00000000006a2119 in qm_free (qmp=0x7f7a5b419000, p=0x7f7a5bd1e688, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", func=0x7f7a9b8e4ef9 <__FUNCTION__.13276> "destroy_dlg", line=392, mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:566 qm = 0x7f7a5b419000 f = 0x7f7a5bd1e650 size = 24 next = 0x7f7a5bd1e6d0 prev = 0x7f7a5bd1e5d8 __FUNCTION__ = "qm_free" #4 0x00000000006aacba in qm_shm_free (qmp=0x7f7a5b419000, p=0x7f7a5bd1e688, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", func=0x7f7a9b8e4ef9 <__FUNCTION__.13276> "destroy_dlg", line=392, mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:1275 No locals. #5 0x00007f7a9b84fe73 in destroy_dlg (dlg=0x7f7a5b9c3f80) at dlg_hash.c:392 ret = 0 var = 0x7f7aa1b2a4f8 __FUNCTION__ = "destroy_dlg" #6 0x00007f7a9b850162 in destroy_dlg_table () at dlg_hash.c:438 dlg = 0x7f7a5b747498 l_dlg = 0x7f7a5b9c3f80 i = 3854 ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "destroy_dlg_table" #7 0x00007f7a9b8c4f17 in mod_destroy () at dialog.c:806 No locals. #8 0x000000000053a94a in destroy_modules () at core/sr_module.c:746 t = 0x7f7aa1a9a160 foo = 0x7f7aa1a99c40 __FUNCTION__ = "destroy_modules" #9 0x000000000041a0c7 in cleanup (show_status=1) at main.c:555 memlog = 0 __FUNCTION__ = "cleanup" #10 0x000000000041b7f6 in shutdown_children (sig=15, show_status=1) at main.c:696 __FUNCTION__ = "shutdown_children" #11 0x000000000041e422 in handle_sigs () at main.c:796 chld = 0 chld_status = 134 any_chld_stopped = 1 memlog = 1533928056 __FUNCTION__ = "handle_sigs" #12 0x0000000000427814 in main_loop () at main.c:1806 i = 8 pid = 901 si = 0x0 si_desc = "udp receiver child=7 sock=172.27.6.5:5090\000( 172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310&E[z\177", '\000' <repeats 14 times>, "\001\000\000\000\320\030\356d\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\352\260\241z\177\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #13 0x000000000042e63a in main (argc=13, argv=0x7ffc64ee1e68) at main.c:2802 cfg_stream = 0x2a68010 c = -1 r = 0 tmp = 0x7ffc64ee2ed8 "" tmp_len = 2496 ---Type <return> to continue, or q <return> to quit--- port = 2496 proto = 2496 ahost = 0x0 aport = 0 options = 0x77db90 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2887616840 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x4000000100 p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode = 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec = 629166543}, st_mtim = {tv_sec = 1586435764, tv_nsec = 286216393}, st_ctim = {tv_sec = 1586435764, tv_nsec = 286216393}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 376 times>... option_index = 0 long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val = 118}, { name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, { name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) =========================================================================================================================================== CRASH-3: --------- Secnario: This crash is seen after the previous crashes happened and kamailio restarted quickly and was handling new calls and the in-dialog messages from prior to the restart. Logs: Apr 9 18:59:51 localhost kernel: kamailio[3775]: segfault at 7f4500000078 ip 00007f4500000078 sp 00007ffcd4255688 error 14 in libbz2.so.1.0.6[7f452f812000+f000] Apr 9 18:59:53 localhost mysqld: 2020-04-09 18:59:53 308 [Warning] Aborted connection 308 to db: 'kamailio' user: 'kamailio' host: 'localhost' (Got an error reading communication packets) Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3774]: ALERT: <core> [main.c:767]: handle_sigs(): child process 3775 exited by a signal 11 Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3774]: ALERT: <core> [main.c:770]: handle_sigs(): core was generated Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3808]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 13 Full BackTrace: [root@localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb /usr/local/sbin/kamailio core.kamailio.995.1586438991.3775 GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7 Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/kamailio...done. [New LWP 3775] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f4500000078 in ?? () (gdb) bt full #0 0x00007f4500000078 in ?? () No symbol table info available. #1 0x00007f4573f11d7b in run_trans_callbacks_internal (cb_lst=0x7f45329421e0, type=512, trans=0x7f4532942168, params=0x7ffcd4255790) at t_hooks.c:254 cbp = 0x7f45322f9b40 backup_from = 0xaf1930 <def_list+16> backup_to = 0xaf1938 <def_list+24> backup_dom_from = 0xaf1940 <def_list+32> backup_dom_to = 0xaf1948 <def_list+40> backup_uri_from = 0xaf1920 <def_list> backup_uri_to = 0xaf1928 <def_list+8> backup_xavps = 0xaf1370 <_xavp_list_head> __FUNCTION__ = "run_trans_callbacks_internal" #2 0x00007f4573f11f7b in run_trans_callbacks_with_buf (type=512, rbuf=0x7f4532942238, req=0x7f4532a03d88, repl=0x7f4577407f10, flags=0) at t_hooks.c:297 params = {req = 0x7f4532a03d88, rpl = 0x7f4577407f10, param = 0x7f45322f9b50, code = 200, flags = 0, branch = 0, t_rbuf = 0x7f4532942238, dst = 0x7f4532942288, send_buf = { s = 0x7f45318d3f00 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91hCISeVu.MVjk\r\nFrom: sip:10.201.8.5;tag=9CgNCj5xs3ih1R-o4eMnHS3RcqY2k0P-\r\nTo: sip:+91"..., len = 376}} trans = 0x7f4532942168 #3 0x00007f4573eaabe1 in relay_reply (t=0x7f4532942168, p_msg=0x7f4577407f10, branch=0, msg_status=200, cancel_data=0x7ffcd4255b60, do_put_on_wait=1) at t_reply.c:2021 relay = 0 save_clone = 0 buf = 0x7f4577409540 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91hCISeVu.MVjk\r\nFrom: sip:10.201.8.5;tag=9CgNCj5xs3ih1R-o4eMnHS3RcqY2k0P-\r\nTo: sip:+91"... res_len = 376 relayed_code = 200 relayed_msg = 0x7f4577407f10 reply_bak = 0x1ec2dd8 bm = {to_tag_val = {s = 0x7ffcd4255910 "\220Y%\324\374\177", len = 4959780}} totag_retr = 0 reply_status = RPS_COMPLETED uas_rb = 0x7f4532942238 to_tag = 0x4b77854c ---Type <return> to continue, or q <return> to quit--- reason = {s = 0x1d4255910 <error: Cannot access memory at address 0x1d4255910>, len = 819120656} onsend_params = {req = 0x69def9 <qm_insert_free+59>, rpl = 0x0, param = 0x2000000, code = 32255512, flags = 0, branch = 0, t_rbuf = 0x1, dst = 0x7f4530d2ca10, send_buf = {s = 0x13eac0 <error: Cannot access memory at address 0x13eac0>, len = 8}} ip = {af = 0, len = 4128684032, u = {addrl = {139936328892432, 140723867703520}, addr32 = {1999421456, 32581, 3559217376, 32764}, addr16 = {49168, 30508, 32581, 0, 22752, 54309, 32764, 0}, addr = "\020\300,wE\177\000\000\340X%\324\374\177\000"}} __FUNCTION__ = "relay_reply" #4 0x00007f4573eaf0c0 in reply_received (p_msg=0x7f4577407f10) at t_reply.c:2540 msg_status = 200 last_uac_status = 0 ack = 0x0 ack_len = 3559218416 branch = 0 reply_status = 0 onreply_route = 0 cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text = {s = 0x0, len = 0}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 0}}}} uac = 0x7f4532942380 t = 0x7f4532942168 lack_dst = {send_sock = 0xf618000000000000, to = {s = {sa_family = 34124, sa_data = "wK\000\000\000\000\020\222A\000\000\000\000"}, sin = {sin_family = 34124, sin_port = 19319, sin_addr = {s_addr = 0}, sin_zero = "\020\222A\000\000\000\000"}, sin6 = {sin6_family = 34124, sin6_port = 19319, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\020\222A\000\000\000\000\000\360\\%\324\374\177\000", __u6_addr16 = {37392, 65, 0, 0, 23792, 54309, 32764, 0}, __u6_addr32 = {4297232, 0, 3559218416, 32764}}}, sin6_scope_id = 2000513248}}, id = 32581, send_flags = {f = 34124, blst_imask = 19319}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} backup_user_from = 0x0 backup_user_to = 0x7f4577408f38 backup_domain_from = 0x7ffcd4255c00 backup_domain_to = 0x7f4577407f10 backup_uri_from = 0x419210 <_start> backup_uri_to = 0x4b77854c backup_xavps = 0x7f4577389c38 replies_locked = 1 branch_ret = 0 prev_branch = 0 blst_503_timeout = 0 ---Type <return> to continue, or q <return> to quit--- hf = 0x0 onsend_params = {req = 0x0, rpl = 0x0, param = 0x7ffcd4255c00, code = 1935113626, flags = 32581, branch = 0, t_rbuf = 0x7f45773d71c8, dst = 0x0, send_buf = { s = 0xa9566b <buf+459> "", len = 11097626}} ctx = {rec_lev = 4297232, run_flags = 0, last_retcode = -735745408, jmp_env = {{__jmpbuf = {0, 0, 140723867704064, 7325262, 1305280, 8, 1, 139936330190760}, __mask_was_saved = -735749488, __saved_mask = {__val = {4993055, 18446744073709551615, 139936330186512, 32255952, 1266124108, 140723867704196, 140723867704200, 139936329984224, 139936330186512, 139936330191304, 11097707, 0, 11097705, 4297232, 1266124108, 4297232}}}}} bctx = 0x7ffcd4255cf0 keng = 0x0 ret = 32764 evname = {s = 0x7f4573f46ee2 "on_sl_reply", len = 11} __FUNCTION__ = "reply_received" #5 0x000000000052edf7 in do_forward_reply (msg=0x7f4577407f10, mode=0) at core/forward.c:745 new_buf = 0x0 dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} new_len = 0 r = 0 ip = {af = 3559218448, len = 32764, u = {addrl = {139936279475385, 4306042945}, addr32 = {1950004409, 32581, 11075649, 1}, addr16 = {46265, 29754, 32581, 0, 65, 169, 1, 0}, addr = "\271\264:tE\177\000\000A\000\251\000\001\000\000"}} s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w" len = 0 __FUNCTION__ = "do_forward_reply" #6 0x000000000053096f in forward_reply (msg=0x7f4577407f10) at core/forward.c:846 No locals. #7 0x000000000059444d in receive_msg ( buf=0xa954a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0, SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"..., len=459, rcv_info=0x7ffcd4256310) at core/receive.c:510 msg = 0x7f4577407f10 ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {1266124108, 3199900600670044426, 4297232, 140723867708032, 0, 0, 3199900600563089674, -3201500360552660726}, __mask_was_saved = 0, __saved_mask = {__val = {8, 2, 139936330063368, 140723867705200, 4993055, 0, 21483258624, 760, 1266124108, 140723867705456, 6946539, 7946256, 4294967296, 140723867705424, 5483931, 0}}}}} ---Type <return> to continue, or q <return> to quit--- bctx = 0x0 ret = 1 tvb = {tv_sec = 140723867705088, tv_usec = 139936328892432} tve = {tv_sec = 4297232, tv_usec = 6982157} tz = {tz_minuteswest = 0, tz_dsttime = -127} diff = 0 inb = { s = 0xa954a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0, SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"..., len = 459} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0} keng = 0x0 evp = {data = 0x7ffcd4255e90, rcv = 0x7ffcd4256310, dst = 0x0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 exectime = 0 __FUNCTION__ = "receive_msg" #8 0x0000000000498ca7 in udp_rcv_loop () at core/udp_server.c:548 len = 459 buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0, SIP/2.0/UDP 10.201.8.5:5060 ;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"... tmp = 0x0 from = 0x7f45773e9e40 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {2032933804, 140723867706224}, addr32 = {2032933804, 0, 3559220080, 32764}, addr16 = {7084, 31020, 0, 0, 25456, 54309, 32764, 0}, addr = "\254\033,y\000\000\000\000pc%\324\374\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {84286380, 0}, addr32 = {84286380, 0, 0, 0}, addr16 = {7084, 1286, 0, 0, 0, 0, 0, 0}, addr = "\254\033\006\005", '\000' <repeats 11 times>}}, src_port = 5090, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\342\254\033,y\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 57875, sin_addr = {s_addr = 2032933804}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 57875, sin6_flowinfo = 2032933804, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f457735d510, proto = 1 '\001', proto_pad0 = 100 'd', proto_pad1 = -11227} evp = {data = 0x0, rcv = 0x0, dst = 0x0} printbuf = "\020\222A\000\000\000\000\000\200j%\324\374\177\000\000\060\004~\000\000\000\000\000\000\000\000\000|\000\000\000Dg~\000\000\000\000\000\275\006~\00---Type <return> to continue, or q <return> to quit--- 0\000\000\000\000\060\000\000\000\000\000\000\000\020\300,wE\177\000\000\360a%\324\024\000\000\000\060\262x\000\000\000\000\000\023\225\200\000\000\000\000\000D\262x\000\000\000\000\000L\205wK\000\000\000\000\020\222A\000\000\000\000\000\220a%\324\374\177\000\000\032\204_\000\000\000\000\000\250\235>wE\177\000\000\060z\265", '\000' <repeats 17 times>, "\004\000\000\000\060b%\324\374\177\000\000~\214_\000\000\000\000\000\177\000\000\000\000\000\000\000\234\251\372\060E\177\000\000\020\222A\000\000\000\000\000"... i = 6 j = 49280 l = 1242268245 __FUNCTION__ = "udp_rcv_loop" #9 0x0000000000425f31 in main_loop () at main.c:1673 i = 0 pid = 0 si = 0x7f457735d510 si_desc = "udp receiver child=0 sock=172.27.6.5:5060 ( 172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310\306\322\060E\177", '\000' <repeats 14 times>, "\001\000\000\000\360d%\324\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\212>wE\177\000" nrprocs = 8 woneinit = 0 __FUNCTION__ = "main_loop" #10 0x000000000042e63a in main (argc=13, argv=0x7ffcd4256a88) at main.c:2802 cfg_stream = 0x16dd010 c = -1 r = 0 tmp = 0x7ffcd4257ed8 "" tmp_len = 2496 port = 2496 proto = 2496 ahost = 0x0 aport = 0 options = 0x77db90 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 637115892 rfd = 4 debug_save = 0 debug_flag = 0 ---Type <return> to continue, or q <return> to quit--- dont_fork_cnt = 0 n_lst = 0x4000000100 p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode = 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 100, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec = 629166543}, st_mtim = {tv_sec = 1586438826, tv_nsec = 779661371}, st_ctim = {tv_sec = 1586438826, tv_nsec = 779661371}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 376 times>... option_index = 0 long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val = 118}, { name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, { name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) ================================================================================================================================================= Regards, Harneet On Fri, Apr 3, 2020 at 11:06 PM Daniel-Constantin Mierla < miconda(a)gmail.com&gt; wrote: -- "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth" - Sir Arthur Conan Doyle