<div dir="ltr">Hi,<div><br></div><div>Looks like the Crash-1 is happening when the previous fragment's end check1 pattern seems to have been overwritten and does not match with the expected '0xc0c0c0c0'. I am not sure though how to go about debugging further on this issue. </div><div>I have verified that the previous fragment's check and the previous fragment's end check 2 are both fine. </div><div><br></div><div>Can someone help out here on how to go about troubleshooting this?</div><div><br></div><div>Regards,</div><div>Harneet Singh</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 10, 2020 at 1:49 PM harneet singh <<a href="mailto:hbilling@gmail.com">hbilling@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Daniel,<div><br></div><div>Please see the requested details below:</div><div><br></div><div>SETUP: Kamailio Version 5.3.3 being used. Similar issues were seen in Kamailio 5.3.2 as well so we upgraded anyhow.<br> Kamailio is acting as a Call Stateful Proxy with Dispatcher module enabled with Algorith-10 being used(Call Load based Routing) though there is only one UAS in our case for now.<br> The load is a very simple one. <br> <br> UAC(Sipp) --------- Kamailio ----------- UAS(Sipp)<br> <br> Initial INVITE-- 200OK -- ACK happens properly. Then UAS sends an UPDATE after 1 minute which is passed properly to UAC which sends a 200OK and that reaches UAS. After another minute, the UAS sends another refresh UPDATE to Kamailio and at the same time UAC sends BYE for the call. Please note that this is a very time sensitive issue and happens only when such a glare happens at Kamailio. Crash is not seen with a single call. However, at 50 cps, the crashes appears at a regular interval of 4-5 minutes.<br><br>Crash-1 and Crash-2 are seen almost at the same timestamp whereas the Crash-3 is seen on the new restarted kamailio instance that is handling calls and the in-dialog messages of the prevoious kamailio instance.<br><br>[root@localhost tmp]# ll<br>total 3161744<br>-rw------- 1 kamailio kamailio 1111052288 Apr 9 18:57 core.kamailio.995.1586438821.869 ----> CRASH-1<br>-rw------- 1 kamailio kamailio 1110773760 Apr 9 18:57 core.kamailio.995.1586438824.867 ----> CRASH-2<br>-rw------- 1 kamailio kamailio 1111052288 Apr 9 18:59 core.kamailio.995.1586438991.3775 ----> CRASH-3<br><br>CRASH-1:<br>---------<br><br>Logs:<br><br>Apr 9 18:57:01 localhost /usr/local/sbin/kamailio[869]: CRITICAL: {1 10648 INVITE NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND} <core> [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(78, abcdefed)[0x7f7a5bd1e5d8:0x7f7a5bd1e610]! Memory allocator was called from dialog: dlg_hash.c:544. Fragment marked by dialog: dlg_handlers.c:308. Exec from core/mem/q_malloc.c:391.<br>Apr 9 18:57:01 localhost /usr/local/sbin/kamailio[869]: CRITICAL: {1 10648 INVITE NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND} <core> [core/mem/q_malloc.c:155]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten [0x7f7a5bd1e568:0x7f7a5bd1e5a0] - fragment marked by dialog: dlg_handlers.c:308<br>Apr 9 18:57:03 localhost mysqld: 2020-04-09 18:57:03 276 [Warning] Aborted connection 276 to db: 'kamailio' user: 'kamailio' host: 'localhost' (Got an error reading communication packets)<br>Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: ALERT: <core> [main.c:767]: handle_sigs(): child process 869 exited by a signal 6<br>Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: ALERT: <core> [main.c:770]: handle_sigs(): core was generated<br><br>Full BackTrace:<br><br>[root@localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb /usr/local/sbin/kamailio core.kamailio.995.1586438821.869<br>GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7<br>Copyright (C) 2017 Free Software Foundation, Inc.<br>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>This is free software: you are free to change and redistribute it.<br>There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>and "show warranty" for details.<br>This GDB was configured as "x86_64-redhat-linux-gnu".<br>Type "show configuration" for configuration details.<br>For bug reporting instructions, please see:<br><<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>.<br>Find the GDB manual and other documentation resources online at:<br><<a href="http://www.gnu.org/software/gdb/documentation/" target="_blank">http://www.gnu.org/software/gdb/documentation/</a>>.<br>For help, type "help".<br>Type "apropos word" to search for commands related to "word"...<br>Reading symbols from /usr/local/sbin/kamailio...done.<br>[New LWP 869]<br>[Thread debugging using libthread_db enabled]<br>Using host libthread_db library "/lib64/libthread_db.so.1".<br>Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.<br>Program terminated with signal SIGABRT, Aborted.<br>#0 0x00007f7aa3a29337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55<br>55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);<br>(gdb) bt full<br>#0 0x00007f7aa3a29337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55<br> resultvar = 0<br> pid = 869<br> selftid = 869<br>#1 0x00007f7aa3a2aa28 in __GI_abort () at abort.c:90<br> save_stage = 2<br> act = {__sigaction_handler = {sa_handler = 0x7cafe4, sa_sigaction = 0x7cafe4}, sa_mask = {__val = {50, 11474112, 0, 851352, 1301680, 1304944, 8, 2,<br> 21474836483, 140163493761024, 9015141347359, 98784247808, 140163493996064, 32252848, 140164708874416, 8171492}}, sa_flags = 50,<br> sa_restorer = 0x7f7aa3aeb8b0 <__syslog>}<br> sigs = {__val = {32, 0 <repeats 15 times>}}<br>#2 0x000000000069deac in qm_debug_check_frag (qm=0x7f7a5b419000, f=0x7f7a5bd1e5d8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", line=544,<br> efile=0x8094ff "core/mem/q_malloc.c", eline=391) at core/mem/q_malloc.c:158<br> p = 0x7f7a5bd1e568<br> __FUNCTION__ = "qm_debug_check_frag"<br>#3 0x000000000069f9b9 in qm_malloc (qmp=0x7f7a5b419000, size=8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c",<br> func=0x7f7a9b8e4f30 <__FUNCTION__.13318> "dlg_set_leg_info", line=544, mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:391<br> qm = 0x7f7a5b419000<br> f = 0x7f7a5bd1e5d8<br> hash = 2<br> list_cntr = 1<br> __FUNCTION__ = "qm_malloc"<br>#4 0x00000000006aaa6d in qm_shm_malloc (qmp=0x7f7a5b419000, size=5, file=0x7f7a9b8e2187 "dialog: dlg_hash.c",<br> func=0x7f7a9b8e4f30 <__FUNCTION__.13318> "dlg_set_leg_info", line=544, mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:1226<br> r = 0x7f7a5b9b7050<br>#5 0x00007f7a9b850fcd in dlg_set_leg_info (dlg=0x7f7a5b9ba288, tag=0x7f7aa1b2f210, rr=0x7ffc64edf810, contact=0x7ffc64edf820, cseq=0x7ffc64edf830, leg=0)<br> at dlg_hash.c:544<br> cs = {<br> s = 0xa955fc <buf+348> "10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: replaces, 100rel, timer, norefersub\r\nSession-Expires: 120\r\nMin-SE: 120\r\nUser-"..., len = 5}<br> __FUNCTION__ = "dlg_set_leg_info"<br>#6 0x00007f7a9b87bd78 in populate_leg_info (dlg=0x7f7a5b9ba288, msg=0x7f7aa1b2df10, t=0x0, leg=0, tag=0x7f7aa1b2f210) at dlg_handlers.c:266<br> skip_recs = 0<br> own_rr = 0<br>---Type <return> to continue, or q <return> to quit---<br> cseq = {<br> s = 0xa955fc <buf+348> "10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: replaces, 100rel, timer, norefersub\r\nSession-Expires: 120\r\nMin-SE: 120\r\nUser-"..., len = 5}<br> contact = {<br> s = 0xa955b2 <buf+274> "sip:10.201.8.5:5060;ob>\r\nCall-ID: NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: rep"..., len = 22}<br> rr_set = {s = 0x0, len = 0}<br> __FUNCTION__ = "populate_leg_info"<br>#7 0x00007f7a9b883f38 in dlg_new_dialog (req=0x7f7aa1b2df10, t=0x0, run_initial_cbs=1) at dlg_handlers.c:949<br> dlg = 0x7f7a5b9ba288<br> s = {s = 0xf614f80000000000 <error: Cannot access memory at address 0xf614f80000000000>, len = -1582559920}<br> callid = {<br> s = 0xa955d4 <buf+308> "NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS\r\nSupported: replaces, 100rel, timer, norefersub\r\n"..., len = 32}<br> ftag = {<br> s = 0xa95564 <buf+196> "9s2l4B7KJz0pm8tKXwR39PGRVnJm8sK3\r\nTo: <a href="mailto:sip%3A%2B918123088839@172.27.6.5" target="_blank">sip:+918123088839@172.27.6.5</a>\r\nContact: <sip:10.201.8.5:5060;ob>\r\nCall-ID: NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK, INVITE, ACK, BYE, CA"..., len = 32}<br> ttag = {s = 0x0, len = 0}<br> req_uri = {s = 0x7f7aa1a5d910 "sip:+918123088839@172.27.6.5:5060;transport=udp", len = 47}<br> dir = 0<br> __FUNCTION__ = "dlg_new_dialog"<br>#8 0x00007f7a9b890325 in dlg_manage (msg=0x7f7aa1b2df10) at dlg_handlers.c:1864<br> tag = {s = 0x0, len = 0}<br> backup_mode = 0<br> dlg = 0x0<br> t = 0x0<br> __FUNCTION__ = "dlg_manage"<br>#9 0x00007f7a9b8c869c in w_dlg_manage (msg=0x7f7aa1b2df10, s1=0x0, s2=0x0) at dialog.c:1094<br>No locals.<br>#10 0x0000000000458b36 in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ac3fb0, msg=0x7f7aa1b2df10) at core/action.c:1071<br> ret = -5<br> v = 1<br> dst = {send_sock = 0x419210 <_start>, to = {s = {sa_family = 7776, sa_data = "\356d\374\177\000\000\000\000\000\000\000\000\000"}, sin = {sin_family = 7776,<br>---Type <return> to continue, or q <return> to quit---<br> sin_port = 25838, sin_addr = {s_addr = 32764}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 7776, sin6_port = 25838,<br> sin6_flowinfo = 32764, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,<br> 0}}}, sin6_scope_id = 1693319264}}, id = 32764, send_flags = {f = 56144, blst_imask = 102}, proto = 0 '\000', proto_pad0 = 0 '\000',<br> proto_pad1 = 0}<br> tmp = 0x0<br> new_uri = 0x0<br> end = 0x7f7a00000000 <error: Cannot access memory at address 0x7f7a00000000><br> crt = 0x0<br> cmd = 0x7f7aa1a9a230<br> len = 32634<br> user = -1638375131<br> uri = {user = {s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, passwd = {s = 0x0, len = 0}, host = {<br> s = 0x7ffc64edfee0 "`", len = 6739658}, port = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 1300448}, params = {<br> s = 0x7ffc64edfdb0 "\310\375\253\241z\177", len = 4993055}, sip_params = {s = 0x0, len = -1583406912}, headers = {<br> s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = -1582113008}, port_no = 64968, proto = 41387, type = 32634,<br> flags = (unknown: 1693318768), transport = {s = 0x7d4950 "core", len = 1}, ttl = {s = 0x7ffc64edfee0 "`", len = 6721818}, user_param = {<br> s = 0x7f7aa1ac2938 "\001", len = 1693319192}, maddr = {s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, method = {<br> s = 0x1000cf960 <error: Cannot access memory at address 0x1000cf960>, len = 0}, lr = {s = 0x13e970 <error: Cannot access memory at address 0x13e970>,<br> len = 0}, r2 = {s = 0x7f7aa1ac1fb8 "\004", len = 1693319576}, gr = {s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, transport_val = {<br> s = 0x7ffc64edff50 "`", len = 4611518}, ttl_val = {s = 0x7ffc64edfed0 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1abfdc8 "\316\002",<br> len = 1693323440}, maddr_val = {s = 0x0, len = 1}, method_val = {s = 0x60bb3a29 <error: Cannot access memory at address 0x60bb3a29>, len = -1251003067},<br> lr_val = {s = 0x400000000 <error: Cannot access memory at address 0x400000000>, len = 1693319572}, r2_val = {s = 0x0, len = 0}, gr_val = {<br> s = 0x46dc132b38f3545 <error: Cannot access memory at address 0x46dc132b38f3545>, len = 1622882857}}<br> next_hop = {user = {s = 0x0, len = 1693319040}, passwd = {s = 0x7ffc64ee0000 "`", len = 6729767}, host = {<br> s = 0x1 <error: Cannot access memory at address 0x1>, len = 45273232}, port = {s = 0x7ffc64edfdd0 "\340\376\355d\374\177", len = 4}, params = {<br> s = 0x7f7aa1a934e0 "\364.7\235z\177", len = -1548974973}, sip_params = {s = 0x7f7aa1ac5320 "\002", len = 1}, headers = {<br> s = 0x7f7a00000000 <error: Cannot access memory at address 0x7f7a00000000>, len = -1582113008}, port_no = 6760, proto = 41388, type = 32634,<br> flags = (unknown: 1693318384), transport = {s = 0x7f7aa1b2df10 ">\330\001", len = -1}, ttl = {s = 0x7ffc64edfd60 "\340\376\355d\374\177", len = 6721818},<br> user_param = {s = 0x0, len = 0}, maddr = {s = 0x7f7aa1aa62b8 "", len = -1582113008}, method = {<br> s = 0x1ffffffff <error: Cannot access memory at address 0x1ffffffff>, len = 0}, lr = {<br> s = 0xf612d00000000000 <error: Cannot access memory at address 0xf612d00000000000>, len = 0}, r2 = {s = 0x7f7aa1ac2940 "\004", len = 1693319192}, gr = {<br> s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, transport_val = {s = 0x7ffc64edfdd0 "\340\376\355d\374\177", len = 4611518}, ttl_val = {<br> s = 0x7f7aa1a2a520 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1ac1a68 "\333\002", len = 1693323440}, maddr_val = {s = 0x0, len = 1},<br>---Type <return> to continue, or q <return> to quit---<br> method_val = {s = 0x60bb3a29 <error: Cannot access memory at address 0x60bb3a29>, len = -1334889147}, lr_val = {<br> s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, r2_val = {s = 0x0, len = 0}, gr_val = {<br> s = 0x46dc132be8f3545 <error: Cannot access memory at address 0x46dc132be8f3545>, len = 1622882857}}<br> u = 0x7f7aa1b2f4b0<br> port = 0<br> dst_host = 0x0<br> i = 6741152<br> flags = -74119067<br> avp = 0x7ffc64ee01e0<br> st = {flags = 4297232, id = 0, name = {n = 1693326944, s = {s = 0x7ffc64ee1e60 "\r", len = 0}, re = 0x7ffc64ee1e60}, avp = 0x0}<br> sct = 0x0<br> sjt = 0x7f7aa1b2df10<br> rve = 0x0<br> mct = 0x4000000000<br> rv = 0x46dc132bf6f3545<br> rv1 = 0x0<br> c1 = {cache_type = 1693319040, val_type = 32764, c = {avp_val = {n = -1582113008, s = {s = 0x7f7aa1b2df10 ">\330\001", len = -1582553112},<br> re = 0x7f7aa1b2df10}, pval = {rs = {s = 0x7f7aa1b2df10 ">\330\001", len = -1582553112}, ri = 1693323440, flags = 32764}},<br> i2s = "\000\000\000\000\000\000\000\000\377\377\377\377\000\000\000\000):\273`\000"}<br> s = {s = 0x7ffc64edfc50 "`\375\355d\374\177", len = 4611518}<br> srevp = {0x0, 0x0}<br> evp = {data = 0x0, rcv = 0x0, dst = 0x0}<br> mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {<br> number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {<br> s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,<br> attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {<br> type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,<br> string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,<br> len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}<br> __FUNCTION__ = "do_action"<br>#11 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1ac3fb0, msg=0x7f7aa1b2df10) at core/action.c:1576<br> t = 0x7f7aa1ac3fb0<br> ret = -1<br>---Type <return> to continue, or q <return> to quit---<br> tvb = {tv_sec = 0, tv_usec = 0}<br> tve = {tv_sec = 0, tv_usec = 0}<br> tz = {tz_minuteswest = 0, tz_dsttime = 4}<br> tdiff = 1<br> __FUNCTION__ = "run_actions"<br>#12 0x0000000000458aa5 in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ac4100, msg=0x7f7aa1b2df10) at core/action.c:1062<br> ret = 1<br> v = 1<br> dst = {send_sock = 0x0, to = {s = {sa_family = 1856, sa_data = "\356d\374\177\000\000\360\026\253\241z\177\000"}, sin = {sin_family = 1856, sin_port = 25838,<br> sin_addr = {s_addr = 32764}, sin_zero = "\360\026\253\241z\177\000"}, sin6 = {sin6_family = 1856, sin6_port = 25838, sin6_flowinfo = 32764, sin6_addr = {<br> __in6_u = {__u6_addr8 = "\360\026\253\241z\177\000\000\020\337\262\241z\177\000", __u6_addr16 = {5872, 41387, 32634, 0, 57104, 41394, 32634, 0},<br> __u6_addr32 = {2712344304, 32634, 2712854288, 32634}}}, sin6_scope_id = 1693321136}}, id = 32764, send_flags = {f = 54986, blst_imask = 102},<br> proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}<br> tmp = 0x7f7a9c5ecc83 <acc_log_request+8047> "\270\001"<br> new_uri = 0x7f7aa1b2df10 ">\330\001"<br> end = 0x7f7aa1ab16d0 "\a"<br> crt = 0x7ffc64ee08e0 "\300\t\356d\374\177"<br> cmd = 0x7f7aa1a949c8<br> len = 0<br> user = 0<br> uri = {user = {s = 0x0, len = 1559864800}, passwd = {s = 0x2f00000000 <error: Cannot access memory at address 0x2f00000000>, len = 1536887583}, host = {<br> s = 0x0, len = 0}, port = {s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, params = {s = 0x0,<br> len = 0}, sip_params = {s = 0x7ffc64ee0550 "\360\026\253\241z\177", len = 7393852}, headers = {s = 0x7f7a5b41bb90 "", len = -1582113008}, port_no = 43232,<br> proto = 41386, type = 32634, flags = (unknown: 1693320640), transport = {s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = -1},<br> ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x7f7aa1b2e1b0 "\253T\251", len = 1693321024}, method = {s = 0x7f7aa1ab16f0 "",<br> len = -1582113008}, lr = {s = 0x7ffc64ee05f0 "p\006\356d\374\177", len = -1647784851}, r2 = {<br> s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, gr = {s = 0x0, len = 0}, transport_val = {<br> s = 0x7ffc64ee0740 "", len = -1582622992}, ttl_val = {s = 0x7f7aa1b2e1b0 "\253T\251", len = -1582113008}, user_param_val = {s = 0x7f7aa1aaa8e0 "q\002",<br> len = 1693323440}, maddr_val = {s = 0x0, len = 1622882857}, method_val = {<br> s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, lr_val = {s = 0x0, len = 0}, r2_val = {<br> s = 0x7ffc64ee0670 "\360\006\356d\374\177", len = -1647782719}, gr_val = {<br> s = 0x46dc13540ef3545 <error: Cannot access memory at address 0x46dc13540ef3545>, len = -282512059}}<br> next_hop = {user = {s = 0xa8428197 <error: Cannot access memory at address 0xa8428197>, len = -1541505685}, passwd = {<br>---Type <return> to continue, or q <return> to quit---<br> s = 0x19 <error: Cannot access memory at address 0x19>, len = 1693320032}, host = {<br> s = 0xf611b40000000000 <error: Cannot access memory at address 0xf611b40000000000>, len = 6982157}, port = {s = 0x7ffc64ee0380 "\310i\252\241z\177",<br> len = -1583407088}, params = {s = 0x7f7aa1a934e0 "\364.7\235z\177", len = -1583407088}, sip_params = {s = 0x7ffc64ee03c0 "\360\003\356d\374\177",<br> len = 6995517}, headers = {s = 0x7f7aa1a2a520 "", len = -1582113008}, port_no = 27080, proto = 41386, type = 32634, flags = (unknown: 1693320256),<br> transport = {s = 0xcfff0 <error: Cannot access memory at address 0xcfff0>, len = 1301656}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0},<br> maddr = {s = 0x7ffc64ee03f0 "\020\222A", len = 4993055}, method = {s = 0xffffffff00000000 <error: Cannot access memory at address 0xffffffff00000000>,<br> len = 0}, lr = {s = 0x1ec23d0 <error: Cannot access memory at address 0x1ec23d0>, len = 1622882857}, r2 = {<br> s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, gr = {s = 0x0, len = 0}, transport_val = {<br> s = 0x7ffc64ee0520 "", len = 4611518}, ttl_val = {s = 0x7ffc64ee0520 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1aa69c8 "X\002",<br> len = 1693323440}, maddr_val = {s = 0x0, len = 1622882857}, method_val = {<br> s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len = 1693326944}, lr_val = {s = 0x0, len = 0}, r2_val = {<br> s = 0x7ffc64ee0590 "@\a\356d\374\177", len = 4609789}, gr_val = {s = 0x46dc1354fef3545 <error: Cannot access memory at address 0x46dc1354fef3545>,<br> len = -282512059}}<br> u = 0x0<br> port = 0<br> dst_host = 0x7ffc64ee07b0<br> i = 6744853<br> flags = 0<br> avp = 0x7ffc64ee0930<br> st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len = 1693320208}, re = 0x0}, avp = 0x4656fd <run_actions+1860>}<br> sct = 0x7f7a9c616b90 <__FUNCTION__.11588><br> sjt = 0x419c61f7fc<br> rve = 0x7f7aa1ac3798<br> mct = 0x7f7a9c615c60<br> rv = 0x7ffc64ee1e60<br> rv1 = 0x7ffc64ee10b0<br> c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 0, s = {s = 0x0, len = 1}, re = 0x0}, pval = {rs = {s = 0x0, len = 1}, ri = 0,<br> flags = 0}}, i2s = "\260Q\260\241z\177\000\000):\273`\000\000\000\000\020\222A\000\000"}<br> s = {s = 0x0, len = 0}<br> srevp = {0x7ffc64ee0860, 0x7ffc64ee0418}<br> evp = {data = 0x0, rcv = 0x0, dst = 0x0}<br> mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {<br> number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {<br>---Type <return> to continue, or q <return> to quit---<br> s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,<br> attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {<br> type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,<br> string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,<br> len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}<br> __FUNCTION__ = "do_action"<br>#13 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1abd308, msg=0x7f7aa1b2df10) at core/action.c:1576<br> t = 0x7f7aa1ac4100<br> ret = 1<br> tvb = {tv_sec = 0, tv_usec = 0}<br> tve = {tv_sec = 0, tv_usec = 0}<br> tz = {tz_minuteswest = -1671324464, tz_dsttime = 32634}<br> tdiff = 2712291872<br> __FUNCTION__ = "run_actions"<br>#14 0x000000000045555c in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ab2d40, msg=0x7f7aa1b2df10) at core/action.c:695<br> ret = -5<br> v = 0<br> dst = {send_sock = 0x823590, to = {s = {sa_family = 33536, sa_data = "\200\000V\001\000\000\260d\202\000\000\000\000"}, sin = {sin_family = 33536,<br> sin_port = 128, sin_addr = {s_addr = 342}, sin_zero = "\260d\202\000\000\000\000"}, sin6 = {sin6_family = 33536, sin6_port = 128, sin6_flowinfo = 342,<br> sin6_addr = {__in6_u = {__u6_addr8 = "\260d\202\000\000\000\000\000\030:\202\000\000\000\000", __u6_addr16 = {25776, 130, 0, 0, 14872, 130, 0, 0},<br> __u6_addr32 = {8545456, 0, 8534552, 0}}}, sin6_scope_id = 2712859096}}, id = 32634, send_flags = {f = 8208, blst_imask = 41375}, proto = 122 'z',<br> proto_pad0 = 127 '\177', proto_pad1 = 0}<br> tmp = 0x7ffc64ee10d0 "\020\222A"<br> new_uri = 0x7ffc64ee0f64 "\374\177"<br> end = 0x1ec3308 <error: Cannot access memory at address 0x1ec3308><br> crt = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w"<br> cmd = 0x7f7aa1a931d0<br> len = 0<br> user = 1622882857<br> uri = {user = {s = 0x8163a0 "core", len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0xc300000000 <error: Cannot access memory at address 0xc300000000>,<br> len = 11097831}, port = {s = 0x164ee0c50 <error: Cannot access memory at address 0x164ee0c50>, len = 11098026}, params = {s = 0x7ffc64ee0cb0 "\001",<br> len = 7299836}, sip_params = {s = 0x7ffc64ee0d42 "\251", len = 6982157}, headers = {s = 0x7ffc64ee0c80 "", len = -1583407088}, port_no = 0, proto = 0,<br> type = 4128681984, flags = (unknown: 2711560208), transport = {s = 0x7ffc64ee0cc0 "\360\f\356d\374\177", len = -1632797901}, ttl = {<br>---Type <return> to continue, or q <return> to quit---<br> s = 0x69def9 <qm_insert_free+59> "\005\363\a", len = 0}, user_param = {s = 0x2000000 <error: Cannot access memory at address 0x2000000>, len = 32255496},<br> maddr = {s = 0xcf960 <error: Cannot access memory at address 0xcf960>, len = 1298936}, method = {<br> s = 0x13e970 <error: Cannot access memory at address 0x13e970>, len = 8}, lr = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 6938361},<br> r2 = {s = 0x7ffc64ee0cf0 "\020\016\356d\374\177", len = 4993055}, gr = {s = 0x0, len = -1583176416}, transport_val = {<br> s = 0x80ba1a2a520 <error: Cannot access memory at address 0x80ba1a2a520>, len = -1583176416}, ttl_val = {s = 0x7ffc64ee0e10 "\340\016\356d\374\177",<br> len = 6955649}, user_param_val = {s = 0x1ec2dc8 <error: Cannot access memory at address 0x1ec2dc8>, len = 1622882857}, maddr_val = {<br> s = 0x7ffc64ee0e10 "\340\016\356d\374\177", len = 6946539}, method_val = {s = 0x823590 "core", len = 11097317}, lr_val = {<br> s = 0x13e970 <error: Cannot access memory at address 0x13e970>, len = -1582108200}, r2_val = {s = 0xa957aa <buf+778> "", len = 11097873}, gr_val = {<br> s = 0x7ffc64ee0d80 "\330\361\262\241z\177", len = 4993055}}<br> next_hop = {user = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 4959780}, passwd = {s = 0x7ffc64ee0aa0 "\300\v\356d\374\177", len = 4993055},<br> host = {s = 0x0, len = -1583397840}, port = {s = 0x5200000000 <error: Cannot access memory at address 0x5200000000>, len = -1583397840}, params = {<br> s = 0x7ffc64ee0bc0 "H", len = 6955649}, sip_params = {s = 0x2009e6676b0 <error: Cannot access memory at address 0x2009e6676b0>, len = 1548742352},<br> headers = {s = 0x7f7a9e66e830 <__FUNCTION__.12289> "update_totag_set", len = -1637445885}, port_no = 4976, proto = 175, type = ERROR_URI_T,<br> flags = (unknown: 11475272), transport = {s = 0xaf1940 <def_list+32> "", len = 11475256}, ttl = {<br> s = 0x5000af1930 <error: Cannot access memory at address 0x5000af1930>, len = 1548742368}, user_param = {s = 0xaf1920 <def_list> "@t\233[z\177", len = 0},<br> maddr = {s = 0x7f7a9e6676b0 "tm", len = 6982157}, method = {s = 0x7ffc64ee0b60 "", len = -1583407088}, lr = {s = 0x7f7aa1b2f780 "\300 \237\241z\177",<br> len = -1583407088}, r2 = {s = 0x7ffc64ee0ba0 "\320\f\356d\374\177", len = -1632797901}, gr = {s = 0x7f7aa19f27c0 "", len = 0}, transport_val = {<br> s = 0x2000000 <error: Cannot access memory at address 0x2000000>, len = 32258408}, ttl_val = {<br> s = 0xcf418 <error: Cannot access memory at address 0xcf418>, len = 1296024}, user_param_val = {<br> s = 0x40013e970 <error: Cannot access memory at address 0x40013e970>, len = 8479648}, maddr_val = {s = 0x1 <error: Cannot access memory at address 0x1>,<br> len = -1582235496}, method_val = {s = 0x7ffc64ee0cd0 "", len = 7110832}, lr_val = {s = 0x0, len = 8422144}, r2_val = {<br> s = 0x48 <error: Cannot access memory at address 0x48>, len = 1622882857}, gr_val = {s = 0x7ffc64ee0cd0 "", len = 6946539}}<br> u = 0xf6199c0000000000<br> port = 0<br> dst_host = 0x0<br> i = 2<br> flags = 0<br> avp = 0xa957aa <buf+778><br> st = {flags = 33554432, id = 0, name = {n = 32253432, s = {s = 0x1ec25f8 <error: Cannot access memory at address 0x1ec25f8>, len = 851296}, re = 0x1ec25f8},<br> avp = 0x13da08}<br> sct = 0x7ffc64ee0e40<br> sjt = 0x7f7aa1b2f1a0<br> rve = 0x7f7aa1ab2528<br>---Type <return> to continue, or q <return> to quit---<br> mct = 0x1ec2e08<br> rv = 0x0<br> rv1 = 0x0<br> c1 = {cache_type = 1693321776, val_type = 32764, c = {avp_val = {n = -1583407088, s = {s = 0x7f7aa19f2010 "\001", len = 391}, re = 0x7f7aa19f2010}, pval = {<br> rs = {s = 0x7f7aa19f2010 "\001", len = 391}, ri = -1583407088, flags = 32634}}, i2s = "p\n\356d\374\177\000\000\063{\255\236z\177\000\000HF\267\\z\177"}<br> s = {s = 0x300000002 <error: Cannot access memory at address 0x300000002>, len = 6982157}<br> srevp = {0x0, 0x8094ff}<br> evp = {data = 0x0, rcv = 0x0, dst = 0x0}<br> mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {<br> number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {<br> s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0,<br> attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {<br> type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0,<br> string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,<br> len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}<br> __FUNCTION__ = "do_action"<br>#15 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1aa48d0, msg=0x7f7aa1b2df10) at core/action.c:1576<br> t = 0x7f7aa1ab2d40<br> ret = 1<br> tvb = {tv_sec = 0, tv_usec = 0}<br> tve = {tv_sec = 0, tv_usec = 0}<br> tz = {tz_minuteswest = 11098026, tz_dsttime = 0}<br> tdiff = 1622882857<br> __FUNCTION__ = "run_actions"<br>#16 0x0000000000465e86 in run_top_route (a=0x7f7aa1aa48d0, msg=0x7f7aa1b2df10, c=0x0) at core/action.c:1661<br> ctx = {rec_lev = 3, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {1622882857, 319123583751107909, 4297232, 140722001813088, 0, 0,<br> 319123583451215173, -318338964762577595}, __mask_was_saved = 0, __saved_mask = {__val = {0, 4611686022722355200, 140164675591952, 140722001809824,<br> 5326593, 4297232, 140722001813088, 0, 0, 5988290944, 140164675591952, 1073741825, 0, 140164675389112, 1622882857, 4297232}}}}}<br> p = 0x7ffc64ee10b0<br> ret = 0<br> sfbk = 0<br>#17 0x000000000059344c in receive_msg (<br> buf=0xa954a0 <buf> "INVITE sip:+918123088839@172.27.6.5:5060;transport=udp SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZ---Type <return> to continue, or q <return> to quit---<br>H0UFZ5Xj2\r\nMax-Forwards: 69\r\nFrom: sip:10.201.8.5;tag=9s2l"..., len=778, rcv_info=0x7ffc64ee16f0) at core/receive.c:424<br> msg = 0x7f7aa1b2df10<br> ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {1622882857, 319123583751107909, 4297232, 140722001813088, 0, 0,<br> 319123583451215173, -318338964762577595}, __mask_was_saved = 0, __saved_mask = {__val = {8, 2, 140164675468656, 140722001810256, 4993055, 0,<br> 21483258624, 912, 1622882857, 140722001810512, 6946539, 7946256, 8589934592, 140722001810480, 5483931, 0}}}}}<br> bctx = 0x0<br> ret = -1<br> tvb = {tv_sec = 140722001810144, tv_usec = 140164674297872}<br> tve = {tv_sec = 4297232, tv_usec = 6982157}<br> tz = {tz_minuteswest = 0, tz_dsttime = -127}<br> diff = 0<br> inb = {<br> s = 0xa954a0 <buf> "INVITE sip:+918123088839@172.27.6.5:5060;transport=udp SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZH0UFZ5Xj2\r\nMax-Forwards: 69\r\nFrom: sip:10.201.8.5;tag=9s2l"..., len = 778}<br> netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0}<br> keng = 0x0<br> evp = {data = 0x7ffc64ee1270, rcv = 0x7ffc64ee16f0, dst = 0x0}<br> cidlockidx = 0<br> cidlockset = 0<br> errsipmsg = 0<br> exectime = 0<br> __FUNCTION__ = "receive_msg"<br>#18 0x0000000000498ca7 in udp_rcv_loop () at core/udp_server.c:548<br> len = 778<br> buf = "INVITE sip:+918123088839@172.27.6.5:5060;transport=udp SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZH0UFZ5Xj2\r\nMax-Forwards: 69\r\nFrom: sip:10.201.8.5;tag=9s2l"...<br> tmp = 0x0<br> from = 0x7f7aa1b0fda8<br> fromlen = 16<br> ri = {src_ip = {af = 2, len = 4, u = {addrl = {84461834, 140722001811280}, addr32 = {84461834, 0, 1693325136, 32764}, addr16 = {51466, 1288, 0, 0, 5968,<br> 25838, 32764, 0}, addr = "\n\311\b\005\000\000\000\000P\027\356d\374\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {84286380, 0}, addr32 = {<br> 84286380, 0, 0, 0}, addr16 = {7084, 1286, 0, 0, 0, 0, 0, 0}, addr = "\254\033\006\005", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,<br> proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\n\311\b\005\000\000\000\000\000\000\000"}, sin = {<br>---Type <return> to continue, or q <return> to quit---<br> sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 84461834}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2,<br> sin6_port = 50195, sin6_flowinfo = 84461834, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},<br> __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f7aa1a83510, proto = 1 '\001', proto_pad0 = 24 '\030', proto_pad1 = 25838}<br> evp = {data = 0x0, rcv = 0x0, dst = 0x0}<br> printbuf = "\020\222A\000\000\000\000\000`\036\356d\374\177", '\000' <repeats 18 times>, "\260\025\356d\374\177\000\000\233\255S", '\000' <repeats 21 times>, "\320\025\356d\374\177\000\000\233\255S\000\000\000\000\000\023\225\200\000\000\000\000\000\bz}\000\000\000\000\000 ", '\000' <repeats 13 times>, "\200\300\000\000\000\000@~J\377\037\060L\000\017\000\000\000\000\026\356d\035\000\000\061\240\025\356d\374\177\000\000 \030\356d\002\000\000\000`\241\251\241z\177\000\000\000\325w\000\000\000\000\000\b\027\356d\374\177\000\000\177\000\000\000\000\000\000\000\060\nm[z\177\000\000\020\222A\000\000\000\000\000"...<br> i = -1<br> j = 49280<br> l = 1194642238<br> __FUNCTION__ = "udp_rcv_loop"<br>#19 0x0000000000425f31 in main_loop () at main.c:1673<br> i = 1<br> pid = 0<br> si = 0x7f7aa1a83510<br> si_desc = "udp receiver child=1 sock=<a href="http://172.27.6.5:5060" target="_blank">172.27.6.5:5060</a> (<a href="http://172.27.6.5:5060" target="_blank">172.27.6.5:5060</a>)\000\000\000\000\000\020\222A\000\000\000\000\000\310&E[z\177", '\000' <repeats 14 times>, "\001\000\000\000\320\030\356d\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\352\260\241z\177\000"<br> nrprocs = 8<br> woneinit = 1<br> __FUNCTION__ = "main_loop"<br>#20 0x000000000042e63a in main (argc=13, argv=0x7ffc64ee1e68) at main.c:2802<br> cfg_stream = 0x2a68010<br> c = -1<br> r = 0<br> tmp = 0x7ffc64ee2ed8 ""<br> tmp_len = 2496<br> port = 2496<br> proto = 2496<br> ahost = 0x0<br> aport = 0<br> options = 0x77db90 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"<br> ret = -1<br>---Type <return> to continue, or q <return> to quit---<br> seed = 2887616840<br> rfd = 4<br> debug_save = 0<br> debug_flag = 0<br> dont_fork_cnt = 0<br> n_lst = 0x4000000100<br> p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff><br> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode = 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096,<br> st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec = 629166543}, st_mtim = {tv_sec = 1586435764, tv_nsec = 286216393}, st_ctim = {tv_sec = 1586435764,<br> tv_nsec = 286216393}, __unused = {0, 0, 0}}<br> tbuf = '\000' <repeats 376 times>...<br> option_index = 0<br> long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val = 118}, {<br> name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {<br> name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {<br> name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}<br> __FUNCTION__ = "main"<br>(gdb)<br><br>==========================================================================================================================================<br>CRASH-2:<br>-------<br>Logs:<br><br>Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: CRITICAL: <core> [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(78, abcdefed)[0x7f7a5bd1e5d8:0x7f7a5bd1e610]! Memory allocator was called from dialog: dlg_hash.c:392. Fragment marked by dialog: dlg_handlers.c:308. Exec from core/mem/q_malloc.c:566.<br>Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: CRITICAL: <core> [core/mem/q_malloc.c:155]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten [0x7f7a5bd1e568:0x7f7a5bd1e5a0] - fragment marked by dialog: dlg_handlers.c:308<br>Apr 9 18:57:06 localhost systemd: kamailio.service: main process exited, code=dumped, status=6/ABRT<br><br><br>Full BackTrace:<br><br>[root@localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb /usr/local/sbin/kamailio core.kamailio.995.1586438824.867<br>GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7<br>Copyright (C) 2017 Free Software Foundation, Inc.<br>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>This is free software: you are free to change and redistribute it.<br>There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>and "show warranty" for details.<br>This GDB was configured as "x86_64-redhat-linux-gnu".<br>Type "show configuration" for configuration details.<br>For bug reporting instructions, please see:<br><<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>.<br>Find the GDB manual and other documentation resources online at:<br><<a href="http://www.gnu.org/software/gdb/documentation/" target="_blank">http://www.gnu.org/software/gdb/documentation/</a>>.<br>For help, type "help".<br>Type "apropos word" to search for commands related to "word"...<br>Reading symbols from /usr/local/sbin/kamailio...done.<br>[New LWP 867]<br>[Thread debugging using libthread_db enabled]<br>Using host libthread_db library "/lib64/libthread_db.so.1".<br>Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.<br>Program terminated with signal SIGABRT, Aborted.<br>#0 0x00007f7aa3a29337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55<br>55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);<br>(gdb) bt full<br>#0 0x00007f7aa3a29337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55<br> resultvar = 0<br> pid = 867<br> selftid = 867<br>#1 0x00007f7aa3a2aa28 in __GI_abort () at abort.c:90<br> save_stage = 2<br> act = {__sigaction_handler = {sa_handler = 0x7cafe4, sa_sigaction = 0x7cafe4}, sa_mask = {__val = {8424376, 8442432, 140161962737664, 0, 0, 0, 0, 0,<br> 21474836483, 140163493761024, 9015136354304, 64355789963264, 140163493996064, 0, 140164708874416, 8171492}}, sa_flags = -1685147642,<br> sa_restorer = 0x7f7aa3aeb8b0 <__syslog>}<br> sigs = {__val = {32, 0 <repeats 15 times>}}<br>#2 0x000000000069deac in qm_debug_check_frag (qm=0x7f7a5b419000, f=0x7f7a5bd1e5d8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", line=392,<br> efile=0x8094ff "core/mem/q_malloc.c", eline=566) at core/mem/q_malloc.c:158<br> p = 0x7f7a5bd1e568<br> __FUNCTION__ = "qm_debug_check_frag"<br>#3 0x00000000006a2119 in qm_free (qmp=0x7f7a5b419000, p=0x7f7a5bd1e688, file=0x7f7a9b8e2187 "dialog: dlg_hash.c",<br> func=0x7f7a9b8e4ef9 <__FUNCTION__.13276> "destroy_dlg", line=392, mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:566<br> qm = 0x7f7a5b419000<br> f = 0x7f7a5bd1e650<br> size = 24<br> next = 0x7f7a5bd1e6d0<br> prev = 0x7f7a5bd1e5d8<br> __FUNCTION__ = "qm_free"<br>#4 0x00000000006aacba in qm_shm_free (qmp=0x7f7a5b419000, p=0x7f7a5bd1e688, file=0x7f7a9b8e2187 "dialog: dlg_hash.c",<br> func=0x7f7a9b8e4ef9 <__FUNCTION__.13276> "destroy_dlg", line=392, mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:1275<br>No locals.<br>#5 0x00007f7a9b84fe73 in destroy_dlg (dlg=0x7f7a5b9c3f80) at dlg_hash.c:392<br> ret = 0<br> var = 0x7f7aa1b2a4f8<br> __FUNCTION__ = "destroy_dlg"<br>#6 0x00007f7a9b850162 in destroy_dlg_table () at dlg_hash.c:438<br> dlg = 0x7f7a5b747498<br> l_dlg = 0x7f7a5b9c3f80<br> i = 3854<br>---Type <return> to continue, or q <return> to quit---<br> __FUNCTION__ = "destroy_dlg_table"<br>#7 0x00007f7a9b8c4f17 in mod_destroy () at dialog.c:806<br>No locals.<br>#8 0x000000000053a94a in destroy_modules () at core/sr_module.c:746<br> t = 0x7f7aa1a9a160<br> foo = 0x7f7aa1a99c40<br> __FUNCTION__ = "destroy_modules"<br>#9 0x000000000041a0c7 in cleanup (show_status=1) at main.c:555<br> memlog = 0<br> __FUNCTION__ = "cleanup"<br>#10 0x000000000041b7f6 in shutdown_children (sig=15, show_status=1) at main.c:696<br> __FUNCTION__ = "shutdown_children"<br>#11 0x000000000041e422 in handle_sigs () at main.c:796<br> chld = 0<br> chld_status = 134<br> any_chld_stopped = 1<br> memlog = 1533928056<br> __FUNCTION__ = "handle_sigs"<br>#12 0x0000000000427814 in main_loop () at main.c:1806<br> i = 8<br> pid = 901<br> si = 0x0<br> si_desc = "udp receiver child=7 sock=<a href="http://172.27.6.5:5090" target="_blank">172.27.6.5:5090</a>\000(<a href="http://172.27.6.5:5060" target="_blank">172.27.6.5:5060</a>)\000\000\000\000\000\020\222A\000\000\000\000\000\310&E[z\177", '\000' <repeats 14 times>, "\001\000\000\000\320\030\356d\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\352\260\241z\177\000"<br> nrprocs = 8<br> woneinit = 1<br> __FUNCTION__ = "main_loop"<br>#13 0x000000000042e63a in main (argc=13, argv=0x7ffc64ee1e68) at main.c:2802<br> cfg_stream = 0x2a68010<br> c = -1<br> r = 0<br> tmp = 0x7ffc64ee2ed8 ""<br> tmp_len = 2496<br>---Type <return> to continue, or q <return> to quit---<br> port = 2496<br> proto = 2496<br> ahost = 0x0<br> aport = 0<br> options = 0x77db90 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"<br> ret = -1<br> seed = 2887616840<br> rfd = 4<br> debug_save = 0<br> debug_flag = 0<br> dont_fork_cnt = 0<br> n_lst = 0x4000000100<br> p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff><br> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode = 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096,<br> st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec = 629166543}, st_mtim = {tv_sec = 1586435764, tv_nsec = 286216393}, st_ctim = {tv_sec = 1586435764,<br> tv_nsec = 286216393}, __unused = {0, 0, 0}}<br> tbuf = '\000' <repeats 376 times>...<br> option_index = 0<br> long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val = 118}, {<br> name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {<br> name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {<br> name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}<br> __FUNCTION__ = "main"<br>(gdb)<br><br>===========================================================================================================================================<br>CRASH-3:<br>---------<br>Secnario:<br><br>This crash is seen after the previous crashes happened and kamailio restarted quickly and was handling new calls and the in-dialog messages from prior to the restart.<br><br>Logs:<br><br>Apr 9 18:59:51 localhost kernel: kamailio[3775]: segfault at 7f4500000078 ip 00007f4500000078 sp 00007ffcd4255688 error 14 in libbz2.so.1.0.6[7f452f812000+f000]<br>Apr 9 18:59:53 localhost mysqld: 2020-04-09 18:59:53 308 [Warning] Aborted connection 308 to db: 'kamailio' user: 'kamailio' host: 'localhost' (Got an error reading communication packets)<br>Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3774]: ALERT: <core> [main.c:767]: handle_sigs(): child process 3775 exited by a signal 11<br>Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3774]: ALERT: <core> [main.c:770]: handle_sigs(): core was generated<br>Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3808]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 13<br><br>Full BackTrace:<br><br>[root@localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb /usr/local/sbin/kamailio core.kamailio.995.1586438991.3775<br>GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7<br>Copyright (C) 2017 Free Software Foundation, Inc.<br>License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" target="_blank">http://gnu.org/licenses/gpl.html</a>><br>This is free software: you are free to change and redistribute it.<br>There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>and "show warranty" for details.<br>This GDB was configured as "x86_64-redhat-linux-gnu".<br>Type "show configuration" for configuration details.<br>For bug reporting instructions, please see:<br><<a href="http://www.gnu.org/software/gdb/bugs/" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>.<br>Find the GDB manual and other documentation resources online at:<br><<a href="http://www.gnu.org/software/gdb/documentation/" target="_blank">http://www.gnu.org/software/gdb/documentation/</a>>.<br>For help, type "help".<br>Type "apropos word" to search for commands related to "word"...<br>Reading symbols from /usr/local/sbin/kamailio...done.<br>[New LWP 3775]<br>[Thread debugging using libthread_db enabled]<br>Using host libthread_db library "/lib64/libthread_db.so.1".<br>Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.<br>Program terminated with signal SIGSEGV, Segmentation fault.<br>#0 0x00007f4500000078 in ?? ()<br>(gdb) bt full<br>#0 0x00007f4500000078 in ?? ()<br>No symbol table info available.<br>#1 0x00007f4573f11d7b in run_trans_callbacks_internal (cb_lst=0x7f45329421e0, type=512, trans=0x7f4532942168, params=0x7ffcd4255790) at t_hooks.c:254<br> cbp = 0x7f45322f9b40<br> backup_from = 0xaf1930 <def_list+16><br> backup_to = 0xaf1938 <def_list+24><br> backup_dom_from = 0xaf1940 <def_list+32><br> backup_dom_to = 0xaf1948 <def_list+40><br> backup_uri_from = 0xaf1920 <def_list><br> backup_uri_to = 0xaf1928 <def_list+8><br> backup_xavps = 0xaf1370 <_xavp_list_head><br> __FUNCTION__ = "run_trans_callbacks_internal"<br>#2 0x00007f4573f11f7b in run_trans_callbacks_with_buf (type=512, rbuf=0x7f4532942238, req=0x7f4532a03d88, repl=0x7f4577407f10, flags=0) at t_hooks.c:297<br> params = {req = 0x7f4532a03d88, rpl = 0x7f4577407f10, param = 0x7f45322f9b50, code = 200, flags = 0, branch = 0, t_rbuf = 0x7f4532942238,<br> dst = 0x7f4532942288, send_buf = {<br> s = 0x7f45318d3f00 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91hCISeVu.MVjk\r\nFrom: sip:10.201.8.5;tag=9CgNCj5xs3ih1R-o4eMnHS3RcqY2k0P-\r\nTo: sip:+91"..., len = 376}}<br> trans = 0x7f4532942168<br>#3 0x00007f4573eaabe1 in relay_reply (t=0x7f4532942168, p_msg=0x7f4577407f10, branch=0, msg_status=200, cancel_data=0x7ffcd4255b60, do_put_on_wait=1)<br> at t_reply.c:2021<br> relay = 0<br> save_clone = 0<br> buf = 0x7f4577409540 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91hCISeVu.MVjk\r\nFrom: sip:10.201.8.5;tag=9CgNCj5xs3ih1R-o4eMnHS3RcqY2k0P-\r\nTo: sip:+91"...<br> res_len = 376<br> relayed_code = 200<br> relayed_msg = 0x7f4577407f10<br> reply_bak = 0x1ec2dd8<br> bm = {to_tag_val = {s = 0x7ffcd4255910 "\220Y%\324\374\177", len = 4959780}}<br> totag_retr = 0<br> reply_status = RPS_COMPLETED<br> uas_rb = 0x7f4532942238<br> to_tag = 0x4b77854c<br>---Type <return> to continue, or q <return> to quit---<br> reason = {s = 0x1d4255910 <error: Cannot access memory at address 0x1d4255910>, len = 819120656}<br> onsend_params = {req = 0x69def9 <qm_insert_free+59>, rpl = 0x0, param = 0x2000000, code = 32255512, flags = 0, branch = 0, t_rbuf = 0x1, dst = 0x7f4530d2ca10,<br> send_buf = {s = 0x13eac0 <error: Cannot access memory at address 0x13eac0>, len = 8}}<br> ip = {af = 0, len = 4128684032, u = {addrl = {139936328892432, 140723867703520}, addr32 = {1999421456, 32581, 3559217376, 32764}, addr16 = {49168, 30508,<br> 32581, 0, 22752, 54309, 32764, 0}, addr = "\020\300,wE\177\000\000\340X%\324\374\177\000"}}<br> __FUNCTION__ = "relay_reply"<br>#4 0x00007f4573eaf0c0 in reply_received (p_msg=0x7f4577407f10) at t_reply.c:2540<br> msg_status = 200<br> last_uac_status = 0<br> ack = 0x0<br> ack_len = 3559218416<br> branch = 0<br> reply_status = 0<br> onreply_route = 0<br> cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text = {s = 0x0, len = 0}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 0}}}}<br> uac = 0x7f4532942380<br> t = 0x7f4532942168<br> lack_dst = {send_sock = 0xf618000000000000, to = {s = {sa_family = 34124, sa_data = "wK\000\000\000\000\020\222A\000\000\000\000"}, sin = {sin_family = 34124,<br> sin_port = 19319, sin_addr = {s_addr = 0}, sin_zero = "\020\222A\000\000\000\000"}, sin6 = {sin6_family = 34124, sin6_port = 19319, sin6_flowinfo = 0,<br> sin6_addr = {__in6_u = {__u6_addr8 = "\020\222A\000\000\000\000\000\360\\%\324\374\177\000", __u6_addr16 = {37392, 65, 0, 0, 23792, 54309, 32764, 0},<br> __u6_addr32 = {4297232, 0, 3559218416, 32764}}}, sin6_scope_id = 2000513248}}, id = 32581, send_flags = {f = 34124, blst_imask = 19319},<br> proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}<br> backup_user_from = 0x0<br> backup_user_to = 0x7f4577408f38<br> backup_domain_from = 0x7ffcd4255c00<br> backup_domain_to = 0x7f4577407f10<br> backup_uri_from = 0x419210 <_start><br> backup_uri_to = 0x4b77854c<br> backup_xavps = 0x7f4577389c38<br> replies_locked = 1<br> branch_ret = 0<br> prev_branch = 0<br> blst_503_timeout = 0<br>---Type <return> to continue, or q <return> to quit---<br> hf = 0x0<br> onsend_params = {req = 0x0, rpl = 0x0, param = 0x7ffcd4255c00, code = 1935113626, flags = 32581, branch = 0, t_rbuf = 0x7f45773d71c8, dst = 0x0, send_buf = {<br> s = 0xa9566b <buf+459> "", len = 11097626}}<br> ctx = {rec_lev = 4297232, run_flags = 0, last_retcode = -735745408, jmp_env = {{__jmpbuf = {0, 0, 140723867704064, 7325262, 1305280, 8, 1, 139936330190760},<br> __mask_was_saved = -735749488, __saved_mask = {__val = {4993055, 18446744073709551615, 139936330186512, 32255952, 1266124108, 140723867704196,<br> 140723867704200, 139936329984224, 139936330186512, 139936330191304, 11097707, 0, 11097705, 4297232, 1266124108, 4297232}}}}}<br> bctx = 0x7ffcd4255cf0<br> keng = 0x0<br> ret = 32764<br> evname = {s = 0x7f4573f46ee2 "on_sl_reply", len = 11}<br> __FUNCTION__ = "reply_received"<br>#5 0x000000000052edf7 in do_forward_reply (msg=0x7f4577407f10, mode=0) at core/forward.c:745<br> new_buf = 0x0<br> dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0},<br> sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {<br> __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0,<br> send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}<br> new_len = 0<br> r = 0<br> ip = {af = 3559218448, len = 32764, u = {addrl = {139936279475385, 4306042945}, addr32 = {1950004409, 32581, 11075649, 1}, addr16 = {46265, 29754, 32581, 0,<br> 65, 169, 1, 0}, addr = "\271\264:tE\177\000\000A\000\251\000\001\000\000"}}<br> s = 0x419210 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w"<br> len = 0<br> __FUNCTION__ = "do_forward_reply"<br>#6 0x000000000053096f in forward_reply (msg=0x7f4577407f10) at core/forward.c:846<br>No locals.<br>#7 0x000000000059444d in receive_msg (<br> buf=0xa954a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0, SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"..., len=459, rcv_info=0x7ffcd4256310) at core/receive.c:510<br> msg = 0x7f4577407f10<br> ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {1266124108, 3199900600670044426, 4297232, 140723867708032, 0, 0,<br> 3199900600563089674, -3201500360552660726}, __mask_was_saved = 0, __saved_mask = {__val = {8, 2, 139936330063368, 140723867705200, 4993055, 0,<br> 21483258624, 760, 1266124108, 140723867705456, 6946539, 7946256, 4294967296, 140723867705424, 5483931, 0}}}}}<br>---Type <return> to continue, or q <return> to quit---<br> bctx = 0x0<br> ret = 1<br> tvb = {tv_sec = 140723867705088, tv_usec = 139936328892432}<br> tve = {tv_sec = 4297232, tv_usec = 6982157}<br> tz = {tz_minuteswest = 0, tz_dsttime = -127}<br> diff = 0<br> inb = {<br> s = 0xa954a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0, SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"..., len = 459}<br> netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0}<br> keng = 0x0<br> evp = {data = 0x7ffcd4255e90, rcv = 0x7ffcd4256310, dst = 0x0}<br> cidlockidx = 0<br> cidlockset = 0<br> errsipmsg = 0<br> exectime = 0<br> __FUNCTION__ = "receive_msg"<br>#8 0x0000000000498ca7 in udp_rcv_loop () at core/udp_server.c:548<br> len = 459<br> buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0, SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"...<br> tmp = 0x0<br> from = 0x7f45773e9e40<br> fromlen = 16<br> ri = {src_ip = {af = 2, len = 4, u = {addrl = {2032933804, 140723867706224}, addr32 = {2032933804, 0, 3559220080, 32764}, addr16 = {7084, 31020, 0, 0, 25456,<br> 54309, 32764, 0}, addr = "\254\033,y\000\000\000\000pc%\324\374\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {84286380, 0}, addr32 = {84286380,<br> 0, 0, 0}, addr16 = {7084, 1286, 0, 0, 0, 0, 0, 0}, addr = "\254\033\006\005", '\000' <repeats 11 times>}}, src_port = 5090, dst_port = 5060,<br> proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\342\254\033,y\000\000\000\000\000\000\000"}, sin = {sin_family = 2,<br> sin_port = 57875, sin_addr = {s_addr = 2032933804}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 57875,<br> sin6_flowinfo = 2032933804, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0,<br> 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f457735d510, proto = 1 '\001', proto_pad0 = 100 'd', proto_pad1 = -11227}<br> evp = {data = 0x0, rcv = 0x0, dst = 0x0}<br> printbuf = "\020\222A\000\000\000\000\000\200j%\324\374\177\000\000\060\004~\000\000\000\000\000\000\000\000\000|\000\000\000Dg~\000\000\000\000\000\275\006~\00---Type <return> to continue, or q <return> to quit---<br>0\000\000\000\000\060\000\000\000\000\000\000\000\020\300,wE\177\000\000\360a%\324\024\000\000\000\060\262x\000\000\000\000\000\023\225\200\000\000\000\000\000D\262x\000\000\000\000\000L\205wK\000\000\000\000\020\222A\000\000\000\000\000\220a%\324\374\177\000\000\032\204_\000\000\000\000\000\250\235>wE\177\000\000\060z\265", '\000' <repeats 17 times>, "\004\000\000\000\060b%\324\374\177\000\000~\214_\000\000\000\000\000\177\000\000\000\000\000\000\000\234\251\372\060E\177\000\000\020\222A\000\000\000\000\000"...<br> i = 6<br> j = 49280<br> l = 1242268245<br> __FUNCTION__ = "udp_rcv_loop"<br>#9 0x0000000000425f31 in main_loop () at main.c:1673<br> i = 0<br> pid = 0<br> si = 0x7f457735d510<br> si_desc = "udp receiver child=0 sock=<a href="http://172.27.6.5:5060" target="_blank">172.27.6.5:5060</a> (<a href="http://172.27.6.5:5060" target="_blank">172.27.6.5:5060</a>)\000\000\000\000\000\020\222A\000\000\000\000\000\310\306\322\060E\177", '\000' <repeats 14 times>, "\001\000\000\000\360d%\324\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\212>wE\177\000"<br> nrprocs = 8<br> woneinit = 0<br> __FUNCTION__ = "main_loop"<br>#10 0x000000000042e63a in main (argc=13, argv=0x7ffcd4256a88) at main.c:2802<br> cfg_stream = 0x16dd010<br> c = -1<br> r = 0<br> tmp = 0x7ffcd4257ed8 ""<br> tmp_len = 2496<br> port = 2496<br> proto = 2496<br> ahost = 0x0<br> aport = 0<br> options = 0x77db90 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"<br> ret = -1<br> seed = 637115892<br> rfd = 4<br> debug_save = 0<br> debug_flag = 0<br>---Type <return> to continue, or q <return> to quit---<br> dont_fork_cnt = 0<br> n_lst = 0x4000000100<br> p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff><br> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode = 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 100, st_blksize = 4096,<br> st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec = 629166543}, st_mtim = {tv_sec = 1586438826, tv_nsec = 779661371}, st_ctim = {tv_sec = 1586438826,<br> tv_nsec = 779661371}, __unused = {0, 0, 0}}<br> tbuf = '\000' <repeats 376 times>...<br> option_index = 0<br> long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val = 118}, {<br> name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {<br> name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {<br> name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}<br> __FUNCTION__ = "main"<br>(gdb)<br><br>=================================================================================================================================================<br></div><div><br></div><div>Regards,</div><div>Harneet</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 3, 2020 at 11:06 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>If it is something that it repeats, try to get the core file, the
log message says core was not generated. <br>
</p>
<p>You have to start kamailio as user root in order to have all
privileges, then before starting kamailio, in the terminal, run:</p>
<p>ulimit -c unlimited</p>
<p>Now start kamailio and then reproduce the case. From the corefile
using gdb get the output of "bt full". The corefile can be located
in / folder or where you set the working directory via kamailio's
-w command line options. It would be good to enable core file per
process as well.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 03.04.20 16:42, harneet singh wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi All,
<div><br>
</div>
<div>As part of our very low cps load at 50 calls per second
through Kamailio Version 5.3.2 (Dispatcher Module used with
Call Load based Routing - Algorithm 10. Machine is a Centos:
7.7 with kernel version 3.10), we are seeing the kamailio
continuously crashing. Please see the data below for 3 types
of segfaults seen:</div>
<div><br>
</div>
<div>Crash-1:<br>
--------<br>
Apr 3 19:32:25 FE-A07-34-VM6 kernel: kamailio[10382]:
segfault at 7ff300000078 ip 00007ff300000078 sp
00007ffe5f2eafc8 error 14 in
libbz2.so.1.0.6[7ff331582000+f000]<br>
Apr 3 19:32:25 FE-A07-34-VM6 mysqld: 2020-04-03 19:32:25 6131
[Warning] Aborted connection 6131 to db: 'kamailio' user:
'kamailio' host: 'localhost' (Got an error reading
communication packets)<br>
Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10423]:
CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF
on 13<br>
Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10381]:
ALERT: <core> [main.c:767]: handle_sigs(): child process
10382 exited by a signal 11<br>
Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10381]:
ALERT: <core> [main.c:770]: handle_sigs(): core was not
generated<br>
===============================<br>
Crash-2: <br>
-------<br>
Apr 3 19:36:11 FE-A07-34-VM6 kernel: kamailio[12838]:
segfault at 7efe00000078 ip 00007efe00000078 sp
00007ffdc876b868 error 15 in zero
(deleted)[7efdc407f000+40000000]<br>
Apr 3 19:36:11 FE-A07-34-VM6 mysqld: 2020-04-03 19:36:11 6175
[Warning] Aborted connection 6175 to db: 'kamailio' user:
'kamailio' host: 'localhost' (Got an error reading
communication packets)<br>
Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12874]:
CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF
on 18<br>
Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12826]:
ALERT: <core> [main.c:767]: handle_sigs(): child process
12838 exited by a signal 11<br>
Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12826]:
ALERT: <core> [main.c:770]: handle_sigs(): core was not
generated<br>
============================<br>
Crash-3:<br>
--------<br>
Apr 3 19:40:53 FE-A07-34-VM6 kernel: kamailio[13542]:
segfault at 80 ip 000000000065e193 sp 00007ffdae10c1f0 error 4
in kamailio[400000+476000]<br>
Apr 3 19:40:53 FE-A07-34-VM6 mysqld: 2020-04-03 19:40:53 6222
[Warning] Aborted connection 6222 to db: 'kamailio' user:
'kamailio' host: 'localhost' (Got an error reading
communication packets)<br>
Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13516]:
ALERT: <core> [main.c:767]: handle_sigs(): child process
13542 exited by a signal 11<br>
Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13558]:
CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF
on 38<br>
Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13516]:
ALERT: <core> [main.c:770]: handle_sigs(): core was not
generated</div>
<div><br clear="all">
<div>Any pointers for resolution are most welcome as we need
to quickly resolve this. </div>
<div><br>
</div>
<div>Regards,</div>
<div>Harneet Singh </div>
-- <br>
<div dir="ltr">"Once you eliminate the
impossible, whatever remains, no matter how improbable, must
be the truth" - Sir Arthur Conan Doyle<br>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">"Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth" - Sir Arthur Conan Doyle<br></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">"Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth" - Sir Arthur Conan Doyle<br></div>