[SR-Users] Kamailio 5.3.2 crashing continuously
harneet singh
hbilling at gmail.com
Mon Apr 13 12:28:32 CEST 2020
Hi,
Looks like the Crash-1 is happening when the previous fragment's end check1
pattern seems to have been overwritten and does not match with the expected
'0xc0c0c0c0'. I am not sure though how to go about debugging further on
this issue.
I have verified that the previous fragment's check and the previous
fragment's end check 2 are both fine.
Can someone help out here on how to go about troubleshooting this?
Regards,
Harneet Singh
On Fri, Apr 10, 2020 at 1:49 PM harneet singh <hbilling at gmail.com> wrote:
> Hi Daniel,
>
> Please see the requested details below:
>
> SETUP: Kamailio Version 5.3.3 being used. Similar issues were seen in
> Kamailio 5.3.2 as well so we upgraded anyhow.
> Kamailio is acting as a Call Stateful Proxy with Dispatcher module
> enabled with Algorith-10 being used(Call Load based Routing) though there
> is only one UAS in our case for now.
> The load is a very simple one.
>
> UAC(Sipp) --------- Kamailio ----------- UAS(Sipp)
>
> Initial INVITE-- 200OK -- ACK happens properly. Then UAS sends an UPDATE
> after 1 minute which is passed properly to UAC which sends a 200OK and that
> reaches UAS. After another minute, the UAS sends another refresh UPDATE to
> Kamailio and at the same time UAC sends BYE for the call. Please note that
> this is a very time sensitive issue and happens only when such a glare
> happens at Kamailio. Crash is not seen with a single call. However, at 50
> cps, the crashes appears at a regular interval of 4-5 minutes.
>
> Crash-1 and Crash-2 are seen almost at the same timestamp whereas the
> Crash-3 is seen on the new restarted kamailio instance that is handling
> calls and the in-dialog messages of the prevoious kamailio instance.
>
> [root at localhost tmp]# ll
> total 3161744
> -rw------- 1 kamailio kamailio 1111052288 Apr 9 18:57
> core.kamailio.995.1586438821.869 ----> CRASH-1
> -rw------- 1 kamailio kamailio 1110773760 Apr 9 18:57
> core.kamailio.995.1586438824.867 ----> CRASH-2
> -rw------- 1 kamailio kamailio 1111052288 Apr 9 18:59
> core.kamailio.995.1586438991.3775 ----> CRASH-3
>
> CRASH-1:
> ---------
>
> Logs:
>
> Apr 9 18:57:01 localhost /usr/local/sbin/kamailio[869]: CRITICAL: {1
> 10648 INVITE NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND} <core>
> [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev. fragm.
> tail overwritten(78, abcdefed)[0x7f7a5bd1e5d8:0x7f7a5bd1e610]! Memory
> allocator was called from dialog: dlg_hash.c:544. Fragment marked by
> dialog: dlg_handlers.c:308. Exec from core/mem/q_malloc.c:391.
> Apr 9 18:57:01 localhost /usr/local/sbin/kamailio[869]: CRITICAL: {1
> 10648 INVITE NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND} <core>
> [core/mem/q_malloc.c:155]: qm_debug_check_frag(): BUG: qm: prev. fragm.
> tail overwritten [0x7f7a5bd1e568:0x7f7a5bd1e5a0] - fragment marked by
> dialog: dlg_handlers.c:308
> Apr 9 18:57:03 localhost mysqld: 2020-04-09 18:57:03 276 [Warning]
> Aborted connection 276 to db: 'kamailio' user: 'kamailio' host: 'localhost'
> (Got an error reading communication packets)
> Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: ALERT: <core>
> [main.c:767]: handle_sigs(): child process 869 exited by a signal 6
> Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: ALERT: <core>
> [main.c:770]: handle_sigs(): core was generated
>
> Full BackTrace:
>
> [root at localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb
> /usr/local/sbin/kamailio core.kamailio.995.1586438821.869
> GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/local/sbin/kamailio...done.
> [New LWP 869]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/local/sbin/kamailio -P
> /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007f7aa3a29337 in __GI_raise (sig=sig at entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:55
> 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> (gdb) bt full
> #0 0x00007f7aa3a29337 in __GI_raise (sig=sig at entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:55
> resultvar = 0
> pid = 869
> selftid = 869
> #1 0x00007f7aa3a2aa28 in __GI_abort () at abort.c:90
> save_stage = 2
> act = {__sigaction_handler = {sa_handler = 0x7cafe4, sa_sigaction
> = 0x7cafe4}, sa_mask = {__val = {50, 11474112, 0, 851352, 1301680, 1304944,
> 8, 2,
> 21474836483, 140163493761024, 9015141347359, 98784247808,
> 140163493996064, 32252848, 140164708874416, 8171492}}, sa_flags = 50,
> sa_restorer = 0x7f7aa3aeb8b0 <__syslog>}
> sigs = {__val = {32, 0 <repeats 15 times>}}
> #2 0x000000000069deac in qm_debug_check_frag (qm=0x7f7a5b419000,
> f=0x7f7a5bd1e5d8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", line=544,
> efile=0x8094ff "core/mem/q_malloc.c", eline=391) at
> core/mem/q_malloc.c:158
> p = 0x7f7a5bd1e568
> __FUNCTION__ = "qm_debug_check_frag"
> #3 0x000000000069f9b9 in qm_malloc (qmp=0x7f7a5b419000, size=8,
> file=0x7f7a9b8e2187 "dialog: dlg_hash.c",
> func=0x7f7a9b8e4f30 <__FUNCTION__.13318> "dlg_set_leg_info", line=544,
> mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:391
> qm = 0x7f7a5b419000
> f = 0x7f7a5bd1e5d8
> hash = 2
> list_cntr = 1
> __FUNCTION__ = "qm_malloc"
> #4 0x00000000006aaa6d in qm_shm_malloc (qmp=0x7f7a5b419000, size=5,
> file=0x7f7a9b8e2187 "dialog: dlg_hash.c",
> func=0x7f7a9b8e4f30 <__FUNCTION__.13318> "dlg_set_leg_info", line=544,
> mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:1226
> r = 0x7f7a5b9b7050
> #5 0x00007f7a9b850fcd in dlg_set_leg_info (dlg=0x7f7a5b9ba288,
> tag=0x7f7aa1b2f210, rr=0x7ffc64edf810, contact=0x7ffc64edf820,
> cseq=0x7ffc64edf830, leg=0)
> at dlg_hash.c:544
> cs = {
> s = 0xa955fc <buf+348> "10648 INVITE\r\nAllow: PRACK, INVITE,
> ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE,
> OPTIONS\r\nSupported: replaces, 100rel, timer,
> norefersub\r\nSession-Expires: 120\r\nMin-SE: 120\r\nUser-"..., len = 5}
> __FUNCTION__ = "dlg_set_leg_info"
> #6 0x00007f7a9b87bd78 in populate_leg_info (dlg=0x7f7a5b9ba288,
> msg=0x7f7aa1b2df10, t=0x0, leg=0, tag=0x7f7aa1b2f210) at dlg_handlers.c:266
> skip_recs = 0
> own_rr = 0
> ---Type <return> to continue, or q <return> to quit---
> cseq = {
> s = 0xa955fc <buf+348> "10648 INVITE\r\nAllow: PRACK, INVITE,
> ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE,
> OPTIONS\r\nSupported: replaces, 100rel, timer,
> norefersub\r\nSession-Expires: 120\r\nMin-SE: 120\r\nUser-"..., len = 5}
> contact = {
> s = 0xa955b2 <buf+274> "sip:10.201.8.5:5060;ob>\r\nCall-ID:
> NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK,
> INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE,
> OPTIONS\r\nSupported: rep"..., len = 22}
> rr_set = {s = 0x0, len = 0}
> __FUNCTION__ = "populate_leg_info"
> #7 0x00007f7a9b883f38 in dlg_new_dialog (req=0x7f7aa1b2df10, t=0x0,
> run_initial_cbs=1) at dlg_handlers.c:949
> dlg = 0x7f7a5b9ba288
> s = {s = 0xf614f80000000000 <error: Cannot access memory at
> address 0xf614f80000000000>, len = -1582559920}
> callid = {
> s = 0xa955d4 <buf+308>
> "NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK,
> INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE,
> OPTIONS\r\nSupported: replaces, 100rel, timer, norefersub\r\n"..., len = 32}
> ftag = {
> s = 0xa95564 <buf+196> "9s2l4B7KJz0pm8tKXwR39PGRVnJm8sK3\r\nTo:
> sip:+918123088839 at 172.27.6.5\r\nContact: <sip:10.201.8.5:5060;ob>\r\nCall-ID:
> NbfWyDfdCt6TCoZbZw.78R.f7gh-HCND\r\nCSeq: 10648 INVITE\r\nAllow: PRACK,
> INVITE, ACK, BYE, CA"..., len = 32}
> ttag = {s = 0x0, len = 0}
> req_uri = {s = 0x7f7aa1a5d910 "sip:+918123088839 at 172.27.6.5:5060;transport=udp",
> len = 47}
> dir = 0
> __FUNCTION__ = "dlg_new_dialog"
> #8 0x00007f7a9b890325 in dlg_manage (msg=0x7f7aa1b2df10) at
> dlg_handlers.c:1864
> tag = {s = 0x0, len = 0}
> backup_mode = 0
> dlg = 0x0
> t = 0x0
> __FUNCTION__ = "dlg_manage"
> #9 0x00007f7a9b8c869c in w_dlg_manage (msg=0x7f7aa1b2df10, s1=0x0,
> s2=0x0) at dialog.c:1094
> No locals.
> #10 0x0000000000458b36 in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ac3fb0,
> msg=0x7f7aa1b2df10) at core/action.c:1071
> ret = -5
> v = 1
> dst = {send_sock = 0x419210 <_start>, to = {s = {sa_family = 7776,
> sa_data = "\356d\374\177\000\000\000\000\000\000\000\000\000"}, sin =
> {sin_family = 7776,
> ---Type <return> to continue, or q <return> to quit---
> sin_port = 25838, sin_addr = {s_addr = 32764}, sin_zero =
> "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 7776, sin6_port =
> 25838,
> sin6_flowinfo = 32764, sin6_addr = {__in6_u = {__u6_addr8 =
> '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
> __u6_addr32 = {0, 0, 0,
> 0}}}, sin6_scope_id = 1693319264}}, id = 32764,
> send_flags = {f = 56144, blst_imask = 102}, proto = 0 '\000', proto_pad0 =
> 0 '\000',
> proto_pad1 = 0}
> tmp = 0x0
> new_uri = 0x0
> end = 0x7f7a00000000 <error: Cannot access memory at address
> 0x7f7a00000000>
> crt = 0x0
> cmd = 0x7f7aa1a9a230
> len = 32634
> user = -1638375131
> uri = {user = {s = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
> 1693326944}, passwd = {s = 0x0, len = 0}, host = {
> s = 0x7ffc64edfee0 "`", len = 6739658}, port = {s = 0x2
> <error: Cannot access memory at address 0x2>, len = 1300448}, params = {
> s = 0x7ffc64edfdb0 "\310\375\253\241z\177", len = 4993055},
> sip_params = {s = 0x0, len = -1583406912}, headers = {
> s = 0x100000000 <error: Cannot access memory at address
> 0x100000000>, len = -1582113008}, port_no = 64968, proto = 41387, type =
> 32634,
> flags = (unknown: 1693318768), transport = {s = 0x7d4950 "core",
> len = 1}, ttl = {s = 0x7ffc64edfee0 "`", len = 6721818}, user_param = {
> s = 0x7f7aa1ac2938 "\001", len = 1693319192}, maddr = {s =
> 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, method = {
> s = 0x1000cf960 <error: Cannot access memory at address
> 0x1000cf960>, len = 0}, lr = {s = 0x13e970 <error: Cannot access memory at
> address 0x13e970>,
> len = 0}, r2 = {s = 0x7f7aa1ac1fb8 "\004", len = 1693319576},
> gr = {s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440}, transport_val = {
> s = 0x7ffc64edff50 "`", len = 4611518}, ttl_val = {s =
> 0x7ffc64edfed0 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1abfdc8
> "\316\002",
> len = 1693323440}, maddr_val = {s = 0x0, len = 1}, method_val
> = {s = 0x60bb3a29 <error: Cannot access memory at address 0x60bb3a29>, len
> = -1251003067},
> lr_val = {s = 0x400000000 <error: Cannot access memory at
> address 0x400000000>, len = 1693319572}, r2_val = {s = 0x0, len = 0},
> gr_val = {
> s = 0x46dc132b38f3545 <error: Cannot access memory at address
> 0x46dc132b38f3545>, len = 1622882857}}
> next_hop = {user = {s = 0x0, len = 1693319040}, passwd = {s =
> 0x7ffc64ee0000 "`", len = 6729767}, host = {
> s = 0x1 <error: Cannot access memory at address 0x1>, len =
> 45273232}, port = {s = 0x7ffc64edfdd0 "\340\376\355d\374\177", len = 4},
> params = {
> s = 0x7f7aa1a934e0 "\364.7\235z\177", len = -1548974973},
> sip_params = {s = 0x7f7aa1ac5320 "\002", len = 1}, headers = {
> s = 0x7f7a00000000 <error: Cannot access memory at address
> 0x7f7a00000000>, len = -1582113008}, port_no = 6760, proto = 41388, type =
> 32634,
> flags = (unknown: 1693318384), transport = {s = 0x7f7aa1b2df10
> ">\330\001", len = -1}, ttl = {s = 0x7ffc64edfd60 "\340\376\355d\374\177",
> len = 6721818},
> user_param = {s = 0x0, len = 0}, maddr = {s = 0x7f7aa1aa62b8 "",
> len = -1582113008}, method = {
> s = 0x1ffffffff <error: Cannot access memory at address
> 0x1ffffffff>, len = 0}, lr = {
> s = 0xf612d00000000000 <error: Cannot access memory at address
> 0xf612d00000000000>, len = 0}, r2 = {s = 0x7f7aa1ac2940 "\004", len =
> 1693319192}, gr = {
> s = 0x7f7aa1b2df10 ">\330\001", len = 1693323440},
> transport_val = {s = 0x7ffc64edfdd0 "\340\376\355d\374\177", len =
> 4611518}, ttl_val = {
> s = 0x7f7aa1a2a520 "", len = -1582113008}, user_param_val = {s
> = 0x7f7aa1ac1a68 "\333\002", len = 1693323440}, maddr_val = {s = 0x0, len =
> 1},
> ---Type <return> to continue, or q <return> to quit---
> method_val = {s = 0x60bb3a29 <error: Cannot access memory at
> address 0x60bb3a29>, len = -1334889147}, lr_val = {
> s = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
> 1693326944}, r2_val = {s = 0x0, len = 0}, gr_val = {
> s = 0x46dc132be8f3545 <error: Cannot access memory at address
> 0x46dc132be8f3545>, len = 1622882857}}
> u = 0x7f7aa1b2f4b0
> port = 0
> dst_host = 0x0
> i = 6741152
> flags = -74119067
> avp = 0x7ffc64ee01e0
> st = {flags = 4297232, id = 0, name = {n = 1693326944, s = {s =
> 0x7ffc64ee1e60 "\r", len = 0}, re = 0x7ffc64ee1e60}, avp = 0x0}
> sct = 0x0
> sjt = 0x7f7aa1b2df10
> rve = 0x0
> mct = 0x4000000000
> rv = 0x46dc132bf6f3545
> rv1 = 0x0
> c1 = {cache_type = 1693319040, val_type = 32764, c = {avp_val = {n
> = -1582113008, s = {s = 0x7f7aa1b2df10 ">\330\001", len = -1582553112},
> re = 0x7f7aa1b2df10}, pval = {rs = {s = 0x7f7aa1b2df10
> ">\330\001", len = -1582553112}, ri = 1693323440, flags = 32764}},
> i2s =
> "\000\000\000\000\000\000\000\000\377\377\377\377\000\000\000\000):\273`\000"}
> s = {s = 0x7ffc64edfc50 "`\375\355d\374\177", len = 4611518}
> srevp = {0x0, 0x0}
> evp = {data = 0x0, rcv = 0x0, dst = 0x0}
> mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
> str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
> NOSUBTYPE, u = {
> number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
> 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string
> = 0x0, str = {
> s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
> {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
> data = 0x0,
> attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
> 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
> 0x0}}, {
> type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
> 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
> = {number = 0,
> string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
> 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
> = {s = 0x0,
> len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
> __FUNCTION__ = "do_action"
> #11 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1ac3fb0,
> msg=0x7f7aa1b2df10) at core/action.c:1576
> t = 0x7f7aa1ac3fb0
> ret = -1
> ---Type <return> to continue, or q <return> to quit---
> tvb = {tv_sec = 0, tv_usec = 0}
> tve = {tv_sec = 0, tv_usec = 0}
> tz = {tz_minuteswest = 0, tz_dsttime = 4}
> tdiff = 1
> __FUNCTION__ = "run_actions"
> #12 0x0000000000458aa5 in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ac4100,
> msg=0x7f7aa1b2df10) at core/action.c:1062
> ret = 1
> v = 1
> dst = {send_sock = 0x0, to = {s = {sa_family = 1856, sa_data =
> "\356d\374\177\000\000\360\026\253\241z\177\000"}, sin = {sin_family =
> 1856, sin_port = 25838,
> sin_addr = {s_addr = 32764}, sin_zero =
> "\360\026\253\241z\177\000"}, sin6 = {sin6_family = 1856, sin6_port =
> 25838, sin6_flowinfo = 32764, sin6_addr = {
> __in6_u = {__u6_addr8 =
> "\360\026\253\241z\177\000\000\020\337\262\241z\177\000", __u6_addr16 =
> {5872, 41387, 32634, 0, 57104, 41394, 32634, 0},
> __u6_addr32 = {2712344304, 32634, 2712854288, 32634}}},
> sin6_scope_id = 1693321136}}, id = 32764, send_flags = {f = 54986,
> blst_imask = 102},
> proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}
> tmp = 0x7f7a9c5ecc83 <acc_log_request+8047> "\270\001"
> new_uri = 0x7f7aa1b2df10 ">\330\001"
> end = 0x7f7aa1ab16d0 "\a"
> crt = 0x7ffc64ee08e0 "\300\t\356d\374\177"
> cmd = 0x7f7aa1a949c8
> len = 0
> user = 0
> uri = {user = {s = 0x0, len = 1559864800}, passwd = {s =
> 0x2f00000000 <error: Cannot access memory at address 0x2f00000000>, len =
> 1536887583}, host = {
> s = 0x0, len = 0}, port = {s = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
> 1693326944}, params = {s = 0x0,
> len = 0}, sip_params = {s = 0x7ffc64ee0550
> "\360\026\253\241z\177", len = 7393852}, headers = {s = 0x7f7a5b41bb90 "",
> len = -1582113008}, port_no = 43232,
> proto = 41386, type = 32634, flags = (unknown: 1693320640),
> transport = {s = 0x100000000 <error: Cannot access memory at address
> 0x100000000>, len = -1},
> ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr
> = {s = 0x7f7aa1b2e1b0 "\253T\251", len = 1693321024}, method = {s =
> 0x7f7aa1ab16f0 "",
> len = -1582113008}, lr = {s = 0x7ffc64ee05f0
> "p\006\356d\374\177", len = -1647784851}, r2 = {
> s = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
> 1693326944}, gr = {s = 0x0, len = 0}, transport_val = {
> s = 0x7ffc64ee0740 "", len = -1582622992}, ttl_val = {s =
> 0x7f7aa1b2e1b0 "\253T\251", len = -1582113008}, user_param_val = {s =
> 0x7f7aa1aaa8e0 "q\002",
> len = 1693323440}, maddr_val = {s = 0x0, len = 1622882857},
> method_val = {
> s = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
> 1693326944}, lr_val = {s = 0x0, len = 0}, r2_val = {
> s = 0x7ffc64ee0670 "\360\006\356d\374\177", len =
> -1647782719}, gr_val = {
> s = 0x46dc13540ef3545 <error: Cannot access memory at address
> 0x46dc13540ef3545>, len = -282512059}}
> next_hop = {user = {s = 0xa8428197 <error: Cannot access memory at
> address 0xa8428197>, len = -1541505685}, passwd = {
> ---Type <return> to continue, or q <return> to quit---
> s = 0x19 <error: Cannot access memory at address 0x19>, len =
> 1693320032}, host = {
> s = 0xf611b40000000000 <error: Cannot access memory at address
> 0xf611b40000000000>, len = 6982157}, port = {s = 0x7ffc64ee0380
> "\310i\252\241z\177",
> len = -1583407088}, params = {s = 0x7f7aa1a934e0
> "\364.7\235z\177", len = -1583407088}, sip_params = {s = 0x7ffc64ee03c0
> "\360\003\356d\374\177",
> len = 6995517}, headers = {s = 0x7f7aa1a2a520 "", len =
> -1582113008}, port_no = 27080, proto = 41386, type = 32634, flags =
> (unknown: 1693320256),
> transport = {s = 0xcfff0 <error: Cannot access memory at address
> 0xcfff0>, len = 1301656}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0,
> len = 0},
> maddr = {s = 0x7ffc64ee03f0 "\020\222A", len = 4993055}, method
> = {s = 0xffffffff00000000 <error: Cannot access memory at address
> 0xffffffff00000000>,
> len = 0}, lr = {s = 0x1ec23d0 <error: Cannot access memory at
> address 0x1ec23d0>, len = 1622882857}, r2 = {
> s = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
> 1693326944}, gr = {s = 0x0, len = 0}, transport_val = {
> s = 0x7ffc64ee0520 "", len = 4611518}, ttl_val = {s =
> 0x7ffc64ee0520 "", len = -1582113008}, user_param_val = {s = 0x7f7aa1aa69c8
> "X\002",
> len = 1693323440}, maddr_val = {s = 0x0, len = 1622882857},
> method_val = {
> s = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w", len =
> 1693326944}, lr_val = {s = 0x0, len = 0}, r2_val = {
> s = 0x7ffc64ee0590 "@\a\356d\374\177", len = 4609789}, gr_val
> = {s = 0x46dc1354fef3545 <error: Cannot access memory at address
> 0x46dc1354fef3545>,
> len = -282512059}}
> u = 0x0
> port = 0
> dst_host = 0x7ffc64ee07b0
> i = 6744853
> flags = 0
> avp = 0x7ffc64ee0930
> st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len =
> 1693320208}, re = 0x0}, avp = 0x4656fd <run_actions+1860>}
> sct = 0x7f7a9c616b90 <__FUNCTION__.11588>
> sjt = 0x419c61f7fc
> rve = 0x7f7aa1ac3798
> mct = 0x7f7a9c615c60
> rv = 0x7ffc64ee1e60
> rv1 = 0x7ffc64ee10b0
> c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c =
> {avp_val = {n = 0, s = {s = 0x0, len = 1}, re = 0x0}, pval = {rs = {s =
> 0x0, len = 1}, ri = 0,
> flags = 0}}, i2s =
> "\260Q\260\241z\177\000\000):\273`\000\000\000\000\020\222A\000\000"}
> s = {s = 0x0, len = 0}
> srevp = {0x7ffc64ee0860, 0x7ffc64ee0418}
> evp = {data = 0x0, rcv = 0x0, dst = 0x0}
> mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
> str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
> NOSUBTYPE, u = {
> number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
> 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string
> = 0x0, str = {
> ---Type <return> to continue, or q <return> to quit---
> s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
> {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
> data = 0x0,
> attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
> 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
> 0x0}}, {
> type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
> 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
> = {number = 0,
> string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
> 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
> = {s = 0x0,
> len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
> __FUNCTION__ = "do_action"
> #13 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1abd308,
> msg=0x7f7aa1b2df10) at core/action.c:1576
> t = 0x7f7aa1ac4100
> ret = 1
> tvb = {tv_sec = 0, tv_usec = 0}
> tve = {tv_sec = 0, tv_usec = 0}
> tz = {tz_minuteswest = -1671324464, tz_dsttime = 32634}
> tdiff = 2712291872
> __FUNCTION__ = "run_actions"
> #14 0x000000000045555c in do_action (h=0x7ffc64ee10b0, a=0x7f7aa1ab2d40,
> msg=0x7f7aa1b2df10) at core/action.c:695
> ret = -5
> v = 0
> dst = {send_sock = 0x823590, to = {s = {sa_family = 33536, sa_data
> = "\200\000V\001\000\000\260d\202\000\000\000\000"}, sin = {sin_family =
> 33536,
> sin_port = 128, sin_addr = {s_addr = 342}, sin_zero =
> "\260d\202\000\000\000\000"}, sin6 = {sin6_family = 33536, sin6_port = 128,
> sin6_flowinfo = 342,
> sin6_addr = {__in6_u = {__u6_addr8 =
> "\260d\202\000\000\000\000\000\030:\202\000\000\000\000", __u6_addr16 =
> {25776, 130, 0, 0, 14872, 130, 0, 0},
> __u6_addr32 = {8545456, 0, 8534552, 0}}}, sin6_scope_id
> = 2712859096}}, id = 32634, send_flags = {f = 8208, blst_imask = 41375},
> proto = 122 'z',
> proto_pad0 = 127 '\177', proto_pad1 = 0}
> tmp = 0x7ffc64ee10d0 "\020\222A"
> new_uri = 0x7ffc64ee0f64 "\374\177"
> end = 0x1ec3308 <error: Cannot access memory at address 0x1ec3308>
> crt = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w"
> cmd = 0x7f7aa1a931d0
> len = 0
> user = 1622882857
> uri = {user = {s = 0x8163a0 "core", len = 0}, passwd = {s = 0x0,
> len = 0}, host = {s = 0xc300000000 <error: Cannot access memory at address
> 0xc300000000>,
> len = 11097831}, port = {s = 0x164ee0c50 <error: Cannot access
> memory at address 0x164ee0c50>, len = 11098026}, params = {s =
> 0x7ffc64ee0cb0 "\001",
> len = 7299836}, sip_params = {s = 0x7ffc64ee0d42 "\251", len =
> 6982157}, headers = {s = 0x7ffc64ee0c80 "", len = -1583407088}, port_no =
> 0, proto = 0,
> type = 4128681984, flags = (unknown: 2711560208), transport = {s
> = 0x7ffc64ee0cc0 "\360\f\356d\374\177", len = -1632797901}, ttl = {
> ---Type <return> to continue, or q <return> to quit---
> s = 0x69def9 <qm_insert_free+59> "\005\363\a", len = 0},
> user_param = {s = 0x2000000 <error: Cannot access memory at address
> 0x2000000>, len = 32255496},
> maddr = {s = 0xcf960 <error: Cannot access memory at address
> 0xcf960>, len = 1298936}, method = {
> s = 0x13e970 <error: Cannot access memory at address
> 0x13e970>, len = 8}, lr = {s = 0x1 <error: Cannot access memory at address
> 0x1>, len = 6938361},
> r2 = {s = 0x7ffc64ee0cf0 "\020\016\356d\374\177", len =
> 4993055}, gr = {s = 0x0, len = -1583176416}, transport_val = {
> s = 0x80ba1a2a520 <error: Cannot access memory at address
> 0x80ba1a2a520>, len = -1583176416}, ttl_val = {s = 0x7ffc64ee0e10
> "\340\016\356d\374\177",
> len = 6955649}, user_param_val = {s = 0x1ec2dc8 <error: Cannot
> access memory at address 0x1ec2dc8>, len = 1622882857}, maddr_val = {
> s = 0x7ffc64ee0e10 "\340\016\356d\374\177", len = 6946539},
> method_val = {s = 0x823590 "core", len = 11097317}, lr_val = {
> s = 0x13e970 <error: Cannot access memory at address
> 0x13e970>, len = -1582108200}, r2_val = {s = 0xa957aa <buf+778> "", len =
> 11097873}, gr_val = {
> s = 0x7ffc64ee0d80 "\330\361\262\241z\177", len = 4993055}}
> next_hop = {user = {s = 0x2 <error: Cannot access memory at
> address 0x2>, len = 4959780}, passwd = {s = 0x7ffc64ee0aa0
> "\300\v\356d\374\177", len = 4993055},
> host = {s = 0x0, len = -1583397840}, port = {s = 0x5200000000
> <error: Cannot access memory at address 0x5200000000>, len = -1583397840},
> params = {
> s = 0x7ffc64ee0bc0 "H", len = 6955649}, sip_params = {s =
> 0x2009e6676b0 <error: Cannot access memory at address 0x2009e6676b0>, len =
> 1548742352},
> headers = {s = 0x7f7a9e66e830 <__FUNCTION__.12289>
> "update_totag_set", len = -1637445885}, port_no = 4976, proto = 175, type =
> ERROR_URI_T,
> flags = (unknown: 11475272), transport = {s = 0xaf1940
> <def_list+32> "", len = 11475256}, ttl = {
> s = 0x5000af1930 <error: Cannot access memory at address
> 0x5000af1930>, len = 1548742368}, user_param = {s = 0xaf1920 <def_list>
> "@t\233[z\177", len = 0},
> maddr = {s = 0x7f7a9e6676b0 "tm", len = 6982157}, method = {s =
> 0x7ffc64ee0b60 "", len = -1583407088}, lr = {s = 0x7f7aa1b2f780 "\300
> \237\241z\177",
> len = -1583407088}, r2 = {s = 0x7ffc64ee0ba0
> "\320\f\356d\374\177", len = -1632797901}, gr = {s = 0x7f7aa19f27c0 "", len
> = 0}, transport_val = {
> s = 0x2000000 <error: Cannot access memory at address
> 0x2000000>, len = 32258408}, ttl_val = {
> s = 0xcf418 <error: Cannot access memory at address 0xcf418>,
> len = 1296024}, user_param_val = {
> s = 0x40013e970 <error: Cannot access memory at address
> 0x40013e970>, len = 8479648}, maddr_val = {s = 0x1 <error: Cannot access
> memory at address 0x1>,
> len = -1582235496}, method_val = {s = 0x7ffc64ee0cd0 "", len =
> 7110832}, lr_val = {s = 0x0, len = 8422144}, r2_val = {
> s = 0x48 <error: Cannot access memory at address 0x48>, len =
> 1622882857}, gr_val = {s = 0x7ffc64ee0cd0 "", len = 6946539}}
> u = 0xf6199c0000000000
> port = 0
> dst_host = 0x0
> i = 2
> flags = 0
> avp = 0xa957aa <buf+778>
> st = {flags = 33554432, id = 0, name = {n = 32253432, s = {s =
> 0x1ec25f8 <error: Cannot access memory at address 0x1ec25f8>, len =
> 851296}, re = 0x1ec25f8},
> avp = 0x13da08}
> sct = 0x7ffc64ee0e40
> sjt = 0x7f7aa1b2f1a0
> rve = 0x7f7aa1ab2528
> ---Type <return> to continue, or q <return> to quit---
> mct = 0x1ec2e08
> rv = 0x0
> rv1 = 0x0
> c1 = {cache_type = 1693321776, val_type = 32764, c = {avp_val = {n
> = -1583407088, s = {s = 0x7f7aa19f2010 "\001", len = 391}, re =
> 0x7f7aa19f2010}, pval = {
> rs = {s = 0x7f7aa19f2010 "\001", len = 391}, ri =
> -1583407088, flags = 32634}}, i2s =
> "p\n\356d\374\177\000\000\063{\255\236z\177\000\000HF\267\\z\177"}
> s = {s = 0x300000002 <error: Cannot access memory at address
> 0x300000002>, len = 6982157}
> srevp = {0x0, 0x8094ff}
> evp = {data = 0x0, rcv = 0x0, dst = 0x0}
> mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0,
> str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type =
> NOSUBTYPE, u = {
> number = 0, string = 0x0, str = {s = 0x0, len = 0}, data =
> 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string
> = 0x0, str = {
> s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}},
> {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0},
> data = 0x0,
> attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number =
> 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select =
> 0x0}}, {
> type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s =
> 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u
> = {number = 0,
> string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
> 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str
> = {s = 0x0,
> len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
> __FUNCTION__ = "do_action"
> #15 0x00000000004656fd in run_actions (h=0x7ffc64ee10b0, a=0x7f7aa1aa48d0,
> msg=0x7f7aa1b2df10) at core/action.c:1576
> t = 0x7f7aa1ab2d40
> ret = 1
> tvb = {tv_sec = 0, tv_usec = 0}
> tve = {tv_sec = 0, tv_usec = 0}
> tz = {tz_minuteswest = 11098026, tz_dsttime = 0}
> tdiff = 1622882857
> __FUNCTION__ = "run_actions"
> #16 0x0000000000465e86 in run_top_route (a=0x7f7aa1aa48d0,
> msg=0x7f7aa1b2df10, c=0x0) at core/action.c:1661
> ctx = {rec_lev = 3, run_flags = 0, last_retcode = -1, jmp_env =
> {{__jmpbuf = {1622882857, 319123583751107909, 4297232, 140722001813088, 0,
> 0,
> 319123583451215173, -318338964762577595}, __mask_was_saved
> = 0, __saved_mask = {__val = {0, 4611686022722355200, 140164675591952,
> 140722001809824,
> 5326593, 4297232, 140722001813088, 0, 0, 5988290944,
> 140164675591952, 1073741825, 0, 140164675389112, 1622882857, 4297232}}}}}
> p = 0x7ffc64ee10b0
> ret = 0
> sfbk = 0
> #17 0x000000000059344c in receive_msg (
> buf=0xa954a0 <buf> "INVITE sip:+918123088839 at 172.27.6.5:5060;transport=udp
> SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZ---Type
> <return> to continue, or q <return> to quit---
> H0UFZ5Xj2\r\nMax-Forwards: 69\r\nFrom: sip:10.201.8.5;tag=9s2l"...,
> len=778, rcv_info=0x7ffc64ee16f0) at core/receive.c:424
> msg = 0x7f7aa1b2df10
> ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env =
> {{__jmpbuf = {1622882857, 319123583751107909, 4297232, 140722001813088, 0,
> 0,
> 319123583451215173, -318338964762577595}, __mask_was_saved
> = 0, __saved_mask = {__val = {8, 2, 140164675468656, 140722001810256,
> 4993055, 0,
> 21483258624, 912, 1622882857, 140722001810512, 6946539,
> 7946256, 8589934592, 140722001810480, 5483931, 0}}}}}
> bctx = 0x0
> ret = -1
> tvb = {tv_sec = 140722001810144, tv_usec = 140164674297872}
> tve = {tv_sec = 4297232, tv_usec = 6982157}
> tz = {tz_minuteswest = 0, tz_dsttime = -127}
> diff = 0
> inb = {
> s = 0xa954a0 <buf> "INVITE sip:+918123088839 at 172.27.6.5:5060;transport=udp
> SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZH0UFZ5Xj2\r\nMax-Forwards:
> 69\r\nFrom: sip:10.201.8.5;tag=9s2l"..., len = 778}
> netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0}
> keng = 0x0
> evp = {data = 0x7ffc64ee1270, rcv = 0x7ffc64ee16f0, dst = 0x0}
> cidlockidx = 0
> cidlockset = 0
> errsipmsg = 0
> exectime = 0
> __FUNCTION__ = "receive_msg"
> #18 0x0000000000498ca7 in udp_rcv_loop () at core/udp_server.c:548
> len = 778
> buf = "INVITE sip:+918123088839 at 172.27.6.5:5060;transport=udp
> SIP/2.0\r\nVia: SIP/2.0/UDP 10.201.8.5:5060;rport;branch=z9hG4bKPjWG-1KGYBTODST0R7xuJoWbZH0UFZ5Xj2\r\nMax-Forwards:
> 69\r\nFrom: sip:10.201.8.5;tag=9s2l"...
> tmp = 0x0
> from = 0x7f7aa1b0fda8
> fromlen = 16
> ri = {src_ip = {af = 2, len = 4, u = {addrl = {84461834,
> 140722001811280}, addr32 = {84461834, 0, 1693325136, 32764}, addr16 =
> {51466, 1288, 0, 0, 5968,
> 25838, 32764, 0}, addr =
> "\n\311\b\005\000\000\000\000P\027\356d\374\177\000"}}, dst_ip = {af = 2,
> len = 4, u = {addrl = {84286380, 0}, addr32 = {
> 84286380, 0, 0, 0}, addr16 = {7084, 1286, 0, 0, 0, 0, 0,
> 0}, addr = "\254\033\006\005", '\000' <repeats 11 times>}}, src_port =
> 5060, dst_port = 5060,
> proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s =
> {sa_family = 2, sa_data =
> "\023\304\n\311\b\005\000\000\000\000\000\000\000"}, sin = {
> ---Type <return> to continue, or q <return> to quit---
> sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
> 84461834}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family
> = 2,
> sin6_port = 50195, sin6_flowinfo = 84461834, sin6_addr =
> {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0,
> 0, 0, 0, 0, 0},
> __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
> bind_address = 0x7f7aa1a83510, proto = 1 '\001', proto_pad0 = 24 '\030',
> proto_pad1 = 25838}
> evp = {data = 0x0, rcv = 0x0, dst = 0x0}
> printbuf = "\020\222A\000\000\000\000\000`\036\356d\374\177",
> '\000' <repeats 18 times>, "\260\025\356d\374\177\000\000\233\255S", '\000'
> <repeats 21 times>,
> "\320\025\356d\374\177\000\000\233\255S\000\000\000\000\000\023\225\200\000\000\000\000\000\bz}\000\000\000\000\000
> ", '\000' <repeats 13 times>, "\200\300\000\000\000\000@~J\377\037\060L\000\017\000\000\000\000\026\356d\035\000\000\061\240\025\356d\374\177\000\000
> \030\356d\002\000\000\000`\241\251\241z\177\000\000\000\325w\000\000\000\000\000\b\027\356d\374\177\000\000\177\000\000\000\000\000\000\000\060\nm[z\177\000\000\020\222A\000\000\000\000\000"...
> i = -1
> j = 49280
> l = 1194642238
> __FUNCTION__ = "udp_rcv_loop"
> #19 0x0000000000425f31 in main_loop () at main.c:1673
> i = 1
> pid = 0
> si = 0x7f7aa1a83510
> si_desc = "udp receiver child=1 sock=172.27.6.5:5060 (
> 172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310&E[z\177",
> '\000' <repeats 14 times>,
> "\001\000\000\000\320\030\356d\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\352\260\241z\177\000"
> nrprocs = 8
> woneinit = 1
> __FUNCTION__ = "main_loop"
> #20 0x000000000042e63a in main (argc=13, argv=0x7ffc64ee1e68) at
> main.c:2802
> cfg_stream = 0x2a68010
> c = -1
> r = 0
> tmp = 0x7ffc64ee2ed8 ""
> tmp_len = 2496
> port = 2496
> proto = 2496
> ahost = 0x0
> aport = 0
> options = 0x77db90
> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
> ret = -1
> ---Type <return> to continue, or q <return> to quit---
> seed = 2887616840
> rfd = 4
> debug_save = 0
> debug_flag = 0
> dont_fork_cnt = 0
> n_lst = 0x4000000100
> p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff>
> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode =
> 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 40,
> st_blksize = 4096,
> st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec =
> 629166543}, st_mtim = {tv_sec = 1586435764, tv_nsec = 286216393}, st_ctim =
> {tv_sec = 1586435764,
> tv_nsec = 286216393}, __unused = {0, 0, 0}}
> tbuf = '\000' <repeats 376 times>...
> option_index = 0
> long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0,
> val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val =
> 118}, {
> name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024},
> {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {
> name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val =
> 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {
> name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val =
> 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
> __FUNCTION__ = "main"
> (gdb)
>
>
> ==========================================================================================================================================
> CRASH-2:
> -------
> Logs:
>
> Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: CRITICAL: <core>
> [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev. fragm.
> tail overwritten(78, abcdefed)[0x7f7a5bd1e5d8:0x7f7a5bd1e610]! Memory
> allocator was called from dialog: dlg_hash.c:392. Fragment marked by
> dialog: dlg_handlers.c:308. Exec from core/mem/q_malloc.c:566.
> Apr 9 18:57:03 localhost /usr/local/sbin/kamailio[867]: CRITICAL: <core>
> [core/mem/q_malloc.c:155]: qm_debug_check_frag(): BUG: qm: prev. fragm.
> tail overwritten [0x7f7a5bd1e568:0x7f7a5bd1e5a0] - fragment marked by
> dialog: dlg_handlers.c:308
> Apr 9 18:57:06 localhost systemd: kamailio.service: main process exited,
> code=dumped, status=6/ABRT
>
>
> Full BackTrace:
>
> [root at localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb
> /usr/local/sbin/kamailio core.kamailio.995.1586438824.867
> GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/local/sbin/kamailio...done.
> [New LWP 867]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/local/sbin/kamailio -P
> /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x00007f7aa3a29337 in __GI_raise (sig=sig at entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:55
> 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
> (gdb) bt full
> #0 0x00007f7aa3a29337 in __GI_raise (sig=sig at entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:55
> resultvar = 0
> pid = 867
> selftid = 867
> #1 0x00007f7aa3a2aa28 in __GI_abort () at abort.c:90
> save_stage = 2
> act = {__sigaction_handler = {sa_handler = 0x7cafe4, sa_sigaction
> = 0x7cafe4}, sa_mask = {__val = {8424376, 8442432, 140161962737664, 0, 0,
> 0, 0, 0,
> 21474836483, 140163493761024, 9015136354304, 64355789963264,
> 140163493996064, 0, 140164708874416, 8171492}}, sa_flags = -1685147642,
> sa_restorer = 0x7f7aa3aeb8b0 <__syslog>}
> sigs = {__val = {32, 0 <repeats 15 times>}}
> #2 0x000000000069deac in qm_debug_check_frag (qm=0x7f7a5b419000,
> f=0x7f7a5bd1e5d8, file=0x7f7a9b8e2187 "dialog: dlg_hash.c", line=392,
> efile=0x8094ff "core/mem/q_malloc.c", eline=566) at
> core/mem/q_malloc.c:158
> p = 0x7f7a5bd1e568
> __FUNCTION__ = "qm_debug_check_frag"
> #3 0x00000000006a2119 in qm_free (qmp=0x7f7a5b419000, p=0x7f7a5bd1e688,
> file=0x7f7a9b8e2187 "dialog: dlg_hash.c",
> func=0x7f7a9b8e4ef9 <__FUNCTION__.13276> "destroy_dlg", line=392,
> mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:566
> qm = 0x7f7a5b419000
> f = 0x7f7a5bd1e650
> size = 24
> next = 0x7f7a5bd1e6d0
> prev = 0x7f7a5bd1e5d8
> __FUNCTION__ = "qm_free"
> #4 0x00000000006aacba in qm_shm_free (qmp=0x7f7a5b419000,
> p=0x7f7a5bd1e688, file=0x7f7a9b8e2187 "dialog: dlg_hash.c",
> func=0x7f7a9b8e4ef9 <__FUNCTION__.13276> "destroy_dlg", line=392,
> mname=0x7f7a9b8e2180 "dialog") at core/mem/q_malloc.c:1275
> No locals.
> #5 0x00007f7a9b84fe73 in destroy_dlg (dlg=0x7f7a5b9c3f80) at
> dlg_hash.c:392
> ret = 0
> var = 0x7f7aa1b2a4f8
> __FUNCTION__ = "destroy_dlg"
> #6 0x00007f7a9b850162 in destroy_dlg_table () at dlg_hash.c:438
> dlg = 0x7f7a5b747498
> l_dlg = 0x7f7a5b9c3f80
> i = 3854
> ---Type <return> to continue, or q <return> to quit---
> __FUNCTION__ = "destroy_dlg_table"
> #7 0x00007f7a9b8c4f17 in mod_destroy () at dialog.c:806
> No locals.
> #8 0x000000000053a94a in destroy_modules () at core/sr_module.c:746
> t = 0x7f7aa1a9a160
> foo = 0x7f7aa1a99c40
> __FUNCTION__ = "destroy_modules"
> #9 0x000000000041a0c7 in cleanup (show_status=1) at main.c:555
> memlog = 0
> __FUNCTION__ = "cleanup"
> #10 0x000000000041b7f6 in shutdown_children (sig=15, show_status=1) at
> main.c:696
> __FUNCTION__ = "shutdown_children"
> #11 0x000000000041e422 in handle_sigs () at main.c:796
> chld = 0
> chld_status = 134
> any_chld_stopped = 1
> memlog = 1533928056
> __FUNCTION__ = "handle_sigs"
> #12 0x0000000000427814 in main_loop () at main.c:1806
> i = 8
> pid = 901
> si = 0x0
> si_desc = "udp receiver child=7 sock=172.27.6.5:5090\000(
> 172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310&E[z\177",
> '\000' <repeats 14 times>,
> "\001\000\000\000\320\030\356d\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\352\260\241z\177\000"
> nrprocs = 8
> woneinit = 1
> __FUNCTION__ = "main_loop"
> #13 0x000000000042e63a in main (argc=13, argv=0x7ffc64ee1e68) at
> main.c:2802
> cfg_stream = 0x2a68010
> c = -1
> r = 0
> tmp = 0x7ffc64ee2ed8 ""
> tmp_len = 2496
> ---Type <return> to continue, or q <return> to quit---
> port = 2496
> proto = 2496
> ahost = 0x0
> aport = 0
> options = 0x77db90
> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
> ret = -1
> seed = 2887616840
> rfd = 4
> debug_save = 0
> debug_flag = 0
> dont_fork_cnt = 0
> n_lst = 0x4000000100
> p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff>
> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode =
> 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 40,
> st_blksize = 4096,
> st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec =
> 629166543}, st_mtim = {tv_sec = 1586435764, tv_nsec = 286216393}, st_ctim =
> {tv_sec = 1586435764,
> tv_nsec = 286216393}, __unused = {0, 0, 0}}
> tbuf = '\000' <repeats 376 times>...
> option_index = 0
> long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0,
> val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val =
> 118}, {
> name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024},
> {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {
> name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val =
> 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {
> name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val =
> 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
> __FUNCTION__ = "main"
> (gdb)
>
>
> ===========================================================================================================================================
> CRASH-3:
> ---------
> Secnario:
>
> This crash is seen after the previous crashes happened and kamailio
> restarted quickly and was handling new calls and the in-dialog messages
> from prior to the restart.
>
> Logs:
>
> Apr 9 18:59:51 localhost kernel: kamailio[3775]: segfault at 7f4500000078
> ip 00007f4500000078 sp 00007ffcd4255688 error 14 in
> libbz2.so.1.0.6[7f452f812000+f000]
> Apr 9 18:59:53 localhost mysqld: 2020-04-09 18:59:53 308 [Warning]
> Aborted connection 308 to db: 'kamailio' user: 'kamailio' host: 'localhost'
> (Got an error reading communication packets)
> Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3774]: ALERT: <core>
> [main.c:767]: handle_sigs(): child process 3775 exited by a signal 11
> Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3774]: ALERT: <core>
> [main.c:770]: handle_sigs(): core was generated
> Apr 9 18:59:53 localhost /usr/local/sbin/kamailio[3808]: CRITICAL: <core>
> [core/pass_fd.c:277]: receive_fd(): EOF on 13
>
> Full BackTrace:
>
> [root at localhost tmp]# /opt/rh/devtoolset-7/root/bin/gdb
> /usr/local/sbin/kamailio core.kamailio.995.1586438991.3775
> GNU gdb (GDB) Red Hat Enterprise Linux 8.0.1-36.el7
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/local/sbin/kamailio...done.
> [New LWP 3775]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/usr/local/sbin/kamailio -P
> /var/run/kamailio/kamailio.pid -f /usr/local/etc/ka'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007f4500000078 in ?? ()
> (gdb) bt full
> #0 0x00007f4500000078 in ?? ()
> No symbol table info available.
> #1 0x00007f4573f11d7b in run_trans_callbacks_internal
> (cb_lst=0x7f45329421e0, type=512, trans=0x7f4532942168,
> params=0x7ffcd4255790) at t_hooks.c:254
> cbp = 0x7f45322f9b40
> backup_from = 0xaf1930 <def_list+16>
> backup_to = 0xaf1938 <def_list+24>
> backup_dom_from = 0xaf1940 <def_list+32>
> backup_dom_to = 0xaf1948 <def_list+40>
> backup_uri_from = 0xaf1920 <def_list>
> backup_uri_to = 0xaf1928 <def_list+8>
> backup_xavps = 0xaf1370 <_xavp_list_head>
> __FUNCTION__ = "run_trans_callbacks_internal"
> #2 0x00007f4573f11f7b in run_trans_callbacks_with_buf (type=512,
> rbuf=0x7f4532942238, req=0x7f4532a03d88, repl=0x7f4577407f10, flags=0) at
> t_hooks.c:297
> params = {req = 0x7f4532a03d88, rpl = 0x7f4577407f10, param =
> 0x7f45322f9b50, code = 200, flags = 0, branch = 0, t_rbuf = 0x7f4532942238,
> dst = 0x7f4532942288, send_buf = {
> s = 0x7f45318d3f00 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
> 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91hCISeVu.MVjk\r\nFrom:
> sip:10.201.8.5;tag=9CgNCj5xs3ih1R-o4eMnHS3RcqY2k0P-\r\nTo: sip:+91"..., len
> = 376}}
> trans = 0x7f4532942168
> #3 0x00007f4573eaabe1 in relay_reply (t=0x7f4532942168,
> p_msg=0x7f4577407f10, branch=0, msg_status=200, cancel_data=0x7ffcd4255b60,
> do_put_on_wait=1)
> at t_reply.c:2021
> relay = 0
> save_clone = 0
> buf = 0x7f4577409540 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
> 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91hCISeVu.MVjk\r\nFrom:
> sip:10.201.8.5;tag=9CgNCj5xs3ih1R-o4eMnHS3RcqY2k0P-\r\nTo: sip:+91"...
> res_len = 376
> relayed_code = 200
> relayed_msg = 0x7f4577407f10
> reply_bak = 0x1ec2dd8
> bm = {to_tag_val = {s = 0x7ffcd4255910 "\220Y%\324\374\177", len =
> 4959780}}
> totag_retr = 0
> reply_status = RPS_COMPLETED
> uas_rb = 0x7f4532942238
> to_tag = 0x4b77854c
> ---Type <return> to continue, or q <return> to quit---
> reason = {s = 0x1d4255910 <error: Cannot access memory at address
> 0x1d4255910>, len = 819120656}
> onsend_params = {req = 0x69def9 <qm_insert_free+59>, rpl = 0x0,
> param = 0x2000000, code = 32255512, flags = 0, branch = 0, t_rbuf = 0x1,
> dst = 0x7f4530d2ca10,
> send_buf = {s = 0x13eac0 <error: Cannot access memory at address
> 0x13eac0>, len = 8}}
> ip = {af = 0, len = 4128684032, u = {addrl = {139936328892432,
> 140723867703520}, addr32 = {1999421456, 32581, 3559217376, 32764}, addr16 =
> {49168, 30508,
> 32581, 0, 22752, 54309, 32764, 0}, addr =
> "\020\300,wE\177\000\000\340X%\324\374\177\000"}}
> __FUNCTION__ = "relay_reply"
> #4 0x00007f4573eaf0c0 in reply_received (p_msg=0x7f4577407f10) at
> t_reply.c:2540
> msg_status = 200
> last_uac_status = 0
> ack = 0x0
> ack_len = 3559218416
> branch = 0
> reply_status = 0
> onreply_route = 0
> cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text
> = {s = 0x0, len = 0}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 0}}}}
> uac = 0x7f4532942380
> t = 0x7f4532942168
> lack_dst = {send_sock = 0xf618000000000000, to = {s = {sa_family =
> 34124, sa_data = "wK\000\000\000\000\020\222A\000\000\000\000"}, sin =
> {sin_family = 34124,
> sin_port = 19319, sin_addr = {s_addr = 0}, sin_zero =
> "\020\222A\000\000\000\000"}, sin6 = {sin6_family = 34124, sin6_port =
> 19319, sin6_flowinfo = 0,
> sin6_addr = {__in6_u = {__u6_addr8 =
> "\020\222A\000\000\000\000\000\360\\%\324\374\177\000", __u6_addr16 =
> {37392, 65, 0, 0, 23792, 54309, 32764, 0},
> __u6_addr32 = {4297232, 0, 3559218416, 32764}}},
> sin6_scope_id = 2000513248}}, id = 32581, send_flags = {f = 34124,
> blst_imask = 19319},
> proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}
> backup_user_from = 0x0
> backup_user_to = 0x7f4577408f38
> backup_domain_from = 0x7ffcd4255c00
> backup_domain_to = 0x7f4577407f10
> backup_uri_from = 0x419210 <_start>
> backup_uri_to = 0x4b77854c
> backup_xavps = 0x7f4577389c38
> replies_locked = 1
> branch_ret = 0
> prev_branch = 0
> blst_503_timeout = 0
> ---Type <return> to continue, or q <return> to quit---
> hf = 0x0
> onsend_params = {req = 0x0, rpl = 0x0, param = 0x7ffcd4255c00,
> code = 1935113626, flags = 32581, branch = 0, t_rbuf = 0x7f45773d71c8, dst
> = 0x0, send_buf = {
> s = 0xa9566b <buf+459> "", len = 11097626}}
> ctx = {rec_lev = 4297232, run_flags = 0, last_retcode =
> -735745408, jmp_env = {{__jmpbuf = {0, 0, 140723867704064, 7325262,
> 1305280, 8, 1, 139936330190760},
> __mask_was_saved = -735749488, __saved_mask = {__val =
> {4993055, 18446744073709551615, 139936330186512, 32255952, 1266124108,
> 140723867704196,
> 140723867704200, 139936329984224, 139936330186512,
> 139936330191304, 11097707, 0, 11097705, 4297232, 1266124108, 4297232}}}}}
> bctx = 0x7ffcd4255cf0
> keng = 0x0
> ret = 32764
> evname = {s = 0x7f4573f46ee2 "on_sl_reply", len = 11}
> __FUNCTION__ = "reply_received"
> #5 0x000000000052edf7 in do_forward_reply (msg=0x7f4577407f10, mode=0) at
> core/forward.c:745
> new_buf = 0x0
> dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000'
> <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
> {s_addr = 0},
> sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
> {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {
> __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 =
> {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id =
> 0}}, id = 0,
> send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000',
> proto_pad0 = 0 '\000', proto_pad1 = 0}
> new_len = 0
> r = 0
> ip = {af = 3559218448, len = 32764, u = {addrl = {139936279475385,
> 4306042945}, addr32 = {1950004409, 32581, 11075649, 1}, addr16 = {46265,
> 29754, 32581, 0,
> 65, 169, 1, 0}, addr =
> "\271\264:tE\177\000\000A\000\251\000\001\000\000"}}
> s = 0x419210 <_start>
> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300\200\237w"
> len = 0
> __FUNCTION__ = "do_forward_reply"
> #6 0x000000000053096f in forward_reply (msg=0x7f4577407f10) at
> core/forward.c:846
> No locals.
> #7 0x000000000059444d in receive_msg (
> buf=0xa954a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0,
> SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"...,
> len=459, rcv_info=0x7ffcd4256310) at core/receive.c:510
> msg = 0x7f4577407f10
> ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env =
> {{__jmpbuf = {1266124108, 3199900600670044426, 4297232, 140723867708032, 0,
> 0,
> 3199900600563089674, -3201500360552660726},
> __mask_was_saved = 0, __saved_mask = {__val = {8, 2, 139936330063368,
> 140723867705200, 4993055, 0,
> 21483258624, 760, 1266124108, 140723867705456, 6946539,
> 7946256, 4294967296, 140723867705424, 5483931, 0}}}}}
> ---Type <return> to continue, or q <return> to quit---
> bctx = 0x0
> ret = 1
> tvb = {tv_sec = 140723867705088, tv_usec = 139936328892432}
> tve = {tv_sec = 4297232, tv_usec = 6982157}
> tz = {tz_minuteswest = 0, tz_dsttime = -127}
> diff = 0
> inb = {
> s = 0xa954a0 <buf> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
> 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0,
> SIP/2.0/UDP 10.201.8.5:5060;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"...,
> len = 459}
> netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0}
> keng = 0x0
> evp = {data = 0x7ffcd4255e90, rcv = 0x7ffcd4256310, dst = 0x0}
> cidlockidx = 0
> cidlockset = 0
> errsipmsg = 0
> exectime = 0
> __FUNCTION__ = "receive_msg"
> #8 0x0000000000498ca7 in udp_rcv_loop () at core/udp_server.c:548
> len = 459
> buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 172.27.6.5:5060;branch=z9hG4bK8eec.864091e3a052fee70fca6fca9405f4bb.0,
> SIP/2.0/UDP 10.201.8.5:5060
> ;received=10.201.8.5;rport=5060;branch=z9hG4bKPj6P--gitumZddIspOER91h"...
> tmp = 0x0
> from = 0x7f45773e9e40
> fromlen = 16
> ri = {src_ip = {af = 2, len = 4, u = {addrl = {2032933804,
> 140723867706224}, addr32 = {2032933804, 0, 3559220080, 32764}, addr16 =
> {7084, 31020, 0, 0, 25456,
> 54309, 32764, 0}, addr =
> "\254\033,y\000\000\000\000pc%\324\374\177\000"}}, dst_ip = {af = 2, len =
> 4, u = {addrl = {84286380, 0}, addr32 = {84286380,
> 0, 0, 0}, addr16 = {7084, 1286, 0, 0, 0, 0, 0, 0}, addr =
> "\254\033\006\005", '\000' <repeats 11 times>}}, src_port = 5090, dst_port
> = 5060,
> proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s =
> {sa_family = 2, sa_data =
> "\023\342\254\033,y\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
> sin_port = 57875, sin_addr = {s_addr = 2032933804}, sin_zero
> = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
> 57875,
> sin6_flowinfo = 2032933804, sin6_addr = {__in6_u =
> {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
> 0, 0}, __u6_addr32 = {0, 0,
> 0, 0}}}, sin6_scope_id = 0}}, bind_address =
> 0x7f457735d510, proto = 1 '\001', proto_pad0 = 100 'd', proto_pad1 = -11227}
> evp = {data = 0x0, rcv = 0x0, dst = 0x0}
> printbuf =
> "\020\222A\000\000\000\000\000\200j%\324\374\177\000\000\060\004~\000\000\000\000\000\000\000\000\000|\000\000\000Dg~\000\000\000\000\000\275\006~\00---Type
> <return> to continue, or q <return> to quit---
> 0\000\000\000\000\060\000\000\000\000\000\000\000\020\300,wE\177\000\000\360a%\324\024\000\000\000\060\262x\000\000\000\000\000\023\225\200\000\000\000\000\000D\262x\000\000\000\000\000L\205wK\000\000\000\000\020\222A\000\000\000\000\000\220a%\324\374\177\000\000\032\204_\000\000\000\000\000\250\235>wE\177\000\000\060z\265",
> '\000' <repeats 17 times>,
> "\004\000\000\000\060b%\324\374\177\000\000~\214_\000\000\000\000\000\177\000\000\000\000\000\000\000\234\251\372\060E\177\000\000\020\222A\000\000\000\000\000"...
> i = 6
> j = 49280
> l = 1242268245
> __FUNCTION__ = "udp_rcv_loop"
> #9 0x0000000000425f31 in main_loop () at main.c:1673
> i = 0
> pid = 0
> si = 0x7f457735d510
> si_desc = "udp receiver child=0 sock=172.27.6.5:5060 (
> 172.27.6.5:5060)\000\000\000\000\000\020\222A\000\000\000\000\000\310\306\322\060E\177",
> '\000' <repeats 14 times>,
> "\001\000\000\000\360d%\324\374\177\000\000\200\252j\000\000\000\000\000`\342y\000\000\000\000\000X\212>wE\177\000"
> nrprocs = 8
> woneinit = 0
> __FUNCTION__ = "main_loop"
> #10 0x000000000042e63a in main (argc=13, argv=0x7ffcd4256a88) at
> main.c:2802
> cfg_stream = 0x16dd010
> c = -1
> r = 0
> tmp = 0x7ffcd4257ed8 ""
> tmp_len = 2496
> port = 2496
> proto = 2496
> ahost = 0x0
> aport = 0
> options = 0x77db90
> ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
> ret = -1
> seed = 637115892
> rfd = 4
> debug_save = 0
> debug_flag = 0
> ---Type <return> to continue, or q <return> to quit---
> dont_fork_cnt = 0
> n_lst = 0x4000000100
> p = 0xf0b5ff <error: Cannot access memory at address 0xf0b5ff>
> st = {st_dev = 20, st_ino = 16135899, st_nlink = 2, st_mode =
> 16832, st_uid = 995, st_gid = 992, __pad0 = 0, st_rdev = 0, st_size = 100,
> st_blksize = 4096,
> st_blocks = 0, st_atim = {tv_sec = 1586172064, tv_nsec =
> 629166543}, st_mtim = {tv_sec = 1586438826, tv_nsec = 779661371}, st_ctim =
> {tv_sec = 1586438826,
> tv_nsec = 779661371}, __unused = {0, 0, 0}}
> tbuf = '\000' <repeats 376 times>...
> option_index = 0
> long_options = {{name = 0x78070a "help", has_arg = 0, flag = 0x0,
> val = 104}, {name = 0x77a42c "version", has_arg = 0, flag = 0x0, val =
> 118}, {
> name = 0x78070f "alias", has_arg = 1, flag = 0x0, val = 1024},
> {name = 0x780715 "subst", has_arg = 1, flag = 0x0, val = 1025}, {
> name = 0x78071b "substdef", has_arg = 1, flag = 0x0, val =
> 1026}, {name = 0x780724 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {
> name = 0x78072e "server-id", has_arg = 1, flag = 0x0, val =
> 1028}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
> __FUNCTION__ = "main"
> (gdb)
>
>
> =================================================================================================================================================
>
> Regards,
> Harneet
>
> On Fri, Apr 3, 2020 at 11:06 PM Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>> Hello,
>>
>> If it is something that it repeats, try to get the core file, the log
>> message says core was not generated.
>>
>> You have to start kamailio as user root in order to have all privileges,
>> then before starting kamailio, in the terminal, run:
>>
>> ulimit -c unlimited
>>
>> Now start kamailio and then reproduce the case. From the corefile using
>> gdb get the output of "bt full". The corefile can be located in / folder or
>> where you set the working directory via kamailio's -w command line options.
>> It would be good to enable core file per process as well.
>>
>> Cheers,
>> Daniel
>> On 03.04.20 16:42, harneet singh wrote:
>>
>> Hi All,
>>
>> As part of our very low cps load at 50 calls per second through Kamailio
>> Version 5.3.2 (Dispatcher Module used with Call Load based Routing -
>> Algorithm 10. Machine is a Centos: 7.7 with kernel version 3.10), we are
>> seeing the kamailio continuously crashing. Please see the data below for 3
>> types of segfaults seen:
>>
>> Crash-1:
>> --------
>> Apr 3 19:32:25 FE-A07-34-VM6 kernel: kamailio[10382]: segfault at
>> 7ff300000078 ip 00007ff300000078 sp 00007ffe5f2eafc8 error 14 in
>> libbz2.so.1.0.6[7ff331582000+f000]
>> Apr 3 19:32:25 FE-A07-34-VM6 mysqld: 2020-04-03 19:32:25 6131 [Warning]
>> Aborted connection 6131 to db: 'kamailio' user: 'kamailio' host:
>> 'localhost' (Got an error reading communication packets)
>> Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10423]: CRITICAL: <core>
>> [core/pass_fd.c:277]: receive_fd(): EOF on 13
>> Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10381]: ALERT: <core>
>> [main.c:767]: handle_sigs(): child process 10382 exited by a signal 11
>> Apr 3 19:32:25 FE-A07-34-VM6 /usr/sbin/kamailio[10381]: ALERT: <core>
>> [main.c:770]: handle_sigs(): core was not generated
>> ===============================
>> Crash-2:
>> -------
>> Apr 3 19:36:11 FE-A07-34-VM6 kernel: kamailio[12838]: segfault at
>> 7efe00000078 ip 00007efe00000078 sp 00007ffdc876b868 error 15 in zero
>> (deleted)[7efdc407f000+40000000]
>> Apr 3 19:36:11 FE-A07-34-VM6 mysqld: 2020-04-03 19:36:11 6175 [Warning]
>> Aborted connection 6175 to db: 'kamailio' user: 'kamailio' host:
>> 'localhost' (Got an error reading communication packets)
>> Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12874]: CRITICAL: <core>
>> [core/pass_fd.c:277]: receive_fd(): EOF on 18
>> Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12826]: ALERT: <core>
>> [main.c:767]: handle_sigs(): child process 12838 exited by a signal 11
>> Apr 3 19:36:11 FE-A07-34-VM6 /usr/sbin/kamailio[12826]: ALERT: <core>
>> [main.c:770]: handle_sigs(): core was not generated
>> ============================
>> Crash-3:
>> --------
>> Apr 3 19:40:53 FE-A07-34-VM6 kernel: kamailio[13542]: segfault at 80 ip
>> 000000000065e193 sp 00007ffdae10c1f0 error 4 in kamailio[400000+476000]
>> Apr 3 19:40:53 FE-A07-34-VM6 mysqld: 2020-04-03 19:40:53 6222 [Warning]
>> Aborted connection 6222 to db: 'kamailio' user: 'kamailio' host:
>> 'localhost' (Got an error reading communication packets)
>> Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13516]: ALERT: <core>
>> [main.c:767]: handle_sigs(): child process 13542 exited by a signal 11
>> Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13558]: CRITICAL: <core>
>> [core/pass_fd.c:277]: receive_fd(): EOF on 38
>> Apr 3 19:40:53 FE-A07-34-VM6 /usr/sbin/kamailio[13516]: ALERT: <core>
>> [main.c:770]: handle_sigs(): core was not generated
>>
>> Any pointers for resolution are most welcome as we need to quickly
>> resolve this.
>>
>> Regards,
>> Harneet Singh
>> --
>> "Once you eliminate the impossible, whatever remains, no matter how
>> improbable, must be the truth" - Sir Arthur Conan Doyle
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>
>>
>
> --
> "Once you eliminate the impossible, whatever remains, no matter how
> improbable, must be the truth" - Sir Arthur Conan Doyle
>
--
"Once you eliminate the impossible, whatever remains, no matter how
improbable, must be the truth" - Sir Arthur Conan Doyle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200413/3248667f/attachment.html>
More information about the sr-users
mailing list