[SR-Users] Re: Kamailio stops processing the calls - restart fixes it.

Hello, Just to sort out some easier reasons: do you increased the shared memory pool and private memory pool in your configuration? The defaults are pretty low. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com -----Original Message----- From: Muhammad Danish Moosa &lt;danishmoosa(a)gmail.com&gt; Sent: Mittwoch, 8. März 2023 02:21 To: Henning Westerholt &lt;hw(a)gilawa.com&gt; Cc: Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Re: [SR-Users] Kamailio stops processing the calls - restart fixes it. Hi Guys, While I am still testing v 5.6.4 on my lab, Production server (v 5.5.3) became unresponsive again. I took "bt full" to all processes and have been analysing that. It looks like a memory allocation issue. I see frequent occurrences of "No symbol table info available". This is a purpose specific server with plenty of resources available including 8G RAM and hardly a fraction of that is used. Only main services are redis and Kamailio. I am not using mysql as well. It has only 30-40 concurrent calls. . I have seen selinux is permissive and uimits has liberated values. Do I need to do any other special OS level settings? Or is it really a version upgrade required? Should I get rid of KEMI or do config directly in the cfg file? I had seen another user reported that issue https://github.com/kamailio/kamailio/issues/2380 and was advised to use v 5.3.5, This is pretty strange because I had experience with opensips before , even with the very old version it used to handle 3k calls and literally several years of non-interruptive function without a single restart or fine tuning etc. Not sure I can attach the logs here , please see the excerpts below. Excerpts from bt full: Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f5ec46fc980 in __pause_nocancel () from /lib64/libc.so.6 #0 0x00007f5ec46fc980 in __pause_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x000000000042dec6 in main_loop () at main.c:1904 i = 8 pid = 22833 si = 0x0 si_desc = "udp receiver child=7 sock=10.10.16.240:5060 \000:\000\000\000\001\000\000\000\376\177\000\000\260\232\000\000\000\000\000\000\300\305E\001\000\000\000\000\300\351\202\000k\000\000\000[̀", '\000' <repeats 13 times>, "\340\311\337\333\376\177\000\000\030\342}\000\000\000\000\000\027\000\000\000\000\000\000\000@ \300A\000\000\000\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #2 0x000000000043684b in main (argc=5, argv=0x7ffedbdfcac8) at main.c:3053 cfg_stream = 0x1456040 c = -1 r = 0 tmp = 0x0 tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x7e1208 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2616789248 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffedbdfc980 p = 0x0 st = {st_dev = 19, st_ino = 21232, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1675551785, tv_nsec = 549443593}, st_mtim = {tv_sec = 1677547400, tv_nsec = 664002717}, st_ctim = {tv_sec = 1677547400, tv_nsec = 664002717}, __unused = {0, 0, 0}} tbuf = "\377\377\377\377", '\000' <repeats 12 times>, "\350sd\304^\177\000\000\310d3\305^\177", '\000' <repeats 90 times>, "\200\022\254\000\000\000\000\000\200\304A\000\000\000\000\000\300\312\337\333\376\177", '\000' <repeats 26 times>, "\356=\023\305^\177\000\000\001", '\000' <repeats 23 times>... option_index = 0 long_options = {{name = 0x7e361f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7de694 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7e3624 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7e362a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7e3630 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7e3639 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7e3643 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7e364d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7e3658 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7e3661 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7e366c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7e3672 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x7e367c "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" [Inferior 1 (process 22811) detached] x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 #0 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5ec46bfba2 in _L_lock_16654 () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5ec46bc7e3 in malloc () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f5ec46aea6a in open_memstream () from /lib64/libc.so.6 No symbol table info available. #4 0x00007f5ec472f65a in __vsyslog_chk () from /lib64/libc.so.6 No symbol table info available. #5 0x00007f5ec472fbbf in syslog () from /lib64/libc.so.6 No symbol table info available. #6 0x0000000000424070 in sig_usr (signo=17) at main.c:920 __llevel = 3 memlog = 0 __FUNCTION__ = "sig_usr" #7 <signal handler called> No symbol table info available. #8 0x00007f5ec46b9094 in _int_malloc () from /lib64/libc.so.6 No symbol table info available. #9 0x00007f5ec46bc78c in malloc () from /lib64/libc.so.6 No symbol table info available. #10 0x00007f5ec241fd0e in luaM_realloc_ () from /lib64/liblua-5.1.so No symbol table info available. #11 0x00007f5ec24239b8 in luaS_newlstr () from /lib64/liblua-5.1.so No symbol table info available. #12 0x00007f5ec2417b5a in lua_pushlstring () from /lib64/liblua-5.1.so No symbol table info available. #13 0x00007f5ec2427ad3 in emptybuffer () from /lib64/liblua-5.1.so No symbol table info available. #14 0x00007f5ec24288b9 in luaL_pushresult () from /lib64/liblua-5.1.so No symbol table info available. #15 0x00007f5ec242bb0b in read_chars () from /lib64/liblua-5.1.so No symbol table info available. #16 0x00007f5ec242bd32 in g_read () from /lib64/liblua-5.1.so No symbol table info available. #17 0x00007f5ec241c324 in luaD_precall () from /lib64/liblua-5.1.so No symbol table info available. #18 0x00007f5ec2426e57 in luaV_execute () from /lib64/liblua-5.1.so No symbol table info available. #19 0x00007f5ec241c74d in luaD_call () from /lib64/liblua-5.1.so No symbol table info available. #20 0x00007f5ec241ba6e in luaD_rawrunprotected () from /lib64/ liblua-5.1.so No symbol table info available. #21 0x00007f5ec241c8da in luaD_pcall () from /lib64/liblua-5.1.so No symbol table info available. #22 0x00007f5ec241844d in lua_pcall () from /lib64/liblua-5.1.so No symbol table info available. #23 0x00007f5ec2660cbc in app_lua_run_ex (msg=0x7f5ec3f20a78, func=0x7f5ec26804ae "ksr_request_route", p1=0x0, p2=0x0, p3=0x0, emode=1) at app_lua_api.c:713 n = 0 ret = 0 txt = {s = 0x7ffedbdfbb38 "Z\336P", len = 22679472} bmsg = 0x0 ltop = 17 __FUNCTION__ = "app_lua_run_ex" #24 0x00007f5ec264a6c9 in sr_kemi_config_engine_lua (msg=0x7f5ec3f20a78, rtype=1, rname=0x0, rparam=0x0) at app_lua_mod.c:119 ret = -1 __FUNCTION__ = "sr_kemi_config_engine_lua" #25 0x00000000004a2592 in sr_kemi_route (keng=0xae6060 <_sr_kemi_eng_list>, msg=0x7f5ec3f20a78, rtype=1, ename=0x0, edata=0x0) at core/kemi.c:3471 sfbk = 0 ret = 7 #26 0x00000000005dc9aa in receive_msg (buf=0xad0020 <buf.7142> "REFER sip: 192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK04B64e74bc7ae4d17c8\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bK2280db05\r\nFrom: \"Danish Queue\""..., len=1366, rcv_info=0x7ffedbdfc200) at core/receive.c:488 msg = 0x7f5ec3f20a78 Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 #0 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5ec46bfba2 in _L_lock_16654 () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5ec46bc7e3 in malloc () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f5ec46aea6a in open_memstream () from /lib64/libc.so.6 No symbol table info available. #4 0x00007f5ec472f65a in __vsyslog_chk () from /lib64/libc.so.6 No symbol table info available. #5 0x00007f5ec472fbbf in syslog () from /lib64/libc.so.6 No symbol table info available. #6 0x0000000000424070 in sig_usr (signo=17) at main.c:920 __llevel = 3 memlog = 0 __FUNCTION__ = "sig_usr" #7 <signal handler called> No symbol table info available. #8 0x00007f5ec46b908a in _int_malloc () from /lib64/libc.so.6 No symbol table info available. #9 0x00007f5ec46bc78c in malloc () from /lib64/libc.so.6 No symbol table info available. #10 0x00007f5ec241fd0e in luaM_realloc_ () from /lib64/liblua-5.1.so No symbol table info available. #11 0x00007f5ec24239b8 in luaS_newlstr () from /lib64/liblua-5.1.so No symbol table info available. #12 0x00007f5ec2417b5a in lua_pushlstring () from /lib64/liblua-5.1.so No symbol table info available. #13 0x00007f5ec2427ad3 in emptybuffer () from /lib64/liblua-5.1.so No symbol table info available. #14 0x00007f5ec24288b9 in luaL_pushresult () from /lib64/liblua-5.1.so No symbol table info available. #15 0x00007f5ec242bb0b in read_chars () from /lib64/liblua-5.1.so No symbol table info available. #16 0x00007f5ec242bd32 in g_read () from /lib64/liblua-5.1.so No symbol table info available. #17 0x00007f5ec241c324 in luaD_precall () from /lib64/liblua-5.1.so No symbol table info available. #18 0x00007f5ec2426e57 in luaV_execute () from /lib64/liblua-5.1.so No symbol table info available. #19 0x00007f5ec241c74d in luaD_call () from /lib64/liblua-5.1.so No symbol table info available. #20 0x00007f5ec241ba6e in luaD_rawrunprotected () from /lib64/ liblua-5.1.so No symbol table info available. #21 0x00007f5ec241c8da in luaD_pcall () from /lib64/liblua-5.1.so No symbol table info available. #22 0x00007f5ec241844d in lua_pcall () from /lib64/liblua-5.1.so No symbol table info available. #23 0x00007f5ec2660cbc in app_lua_run_ex (msg=0x7f5ec3f26158, func=0x7f5ec26804ae "ksr_request_route", p1=0x0, p2=0x0, p3=0x0, emode=1) at app_lua_api.c:713 n = 0 ret = 0 txt = {s = 0x7ffedbdfbb38 "Z\336P", len = 22689984} bmsg = 0x0 ltop = 17 __FUNCTION__ = "app_lua_run_ex" #24 0x00007f5ec264a6c9 in sr_kemi_config_engine_lua (msg=0x7f5ec3f26158, rtype=1, rname=0x0, rparam=0x0) at app_lua_mod.c:119 ret = -1 __FUNCTION__ = "sr_kemi_config_engine_lua" #25 0x00000000004a2592 in sr_kemi_route (keng=0xae6060 <_sr_kemi_eng_list>, msg=0x7f5ec3f26158, rtype=1, ename=0x0, edata=0x0) at core/kemi.c:3471 sfbk = 0 ret = 7 #26 0x00000000005dc9aa in receive_msg (buf=0xad0020 <buf.7142> "REFER sip: 192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""..., len=1372, rcv_info=0x7ffedbdfc200) at core/receive.c:488 msg = 0x7f5ec3f26158 ctx = {rec_lev = 259, run_flags = 0, last_retcode = 1677719528, jmp_env = {{__jmpbuf = {8717649, 0, 2, 0, 22808048, 0, 0, 140045002453120}, __mask_was_saved = 8, __saved_mask = {__val = {8589934604, 528280977410, 257698037764, 1, 39600, 21349824, 0, 0, 0, 140732587295568, 8311117, 4309056, 8573144, 1372, 140044999523263, 0}}}}} bctx = 0x0 ret = 0 tvb = {tv_sec = 1677719528, tv_usec = 404766} tve = {tv_sec = 0, tv_usec = 0} diff = 0 inb = {s = 0xad0020 <buf.7142> "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""..., len = 1372} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0} keng = 0xae6060 <_sr_kemi_eng_list> evp = {data = 0x7ffedbdfbd30, obuf = {s = 0x0, len = 0}, rcv = 0x7ffedbdfc200, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 exectime = 1 __FUNCTION__ = "receive_msg" #27 0x000000000047e32e in udp_rcv_loop () at core/udp_server.c:543 len = 1372 buf = "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""... tmp = 0x7f5ebd6c48f0 "" fromaddr = 0x7f5ec3eec738 fromaddrlen = 16 rcvi = {src_ip = {af = 2, len = 4, u = {addrl = {3088058890, 0}, addr32 = {3088058890, 0, 0, 0}, addr16 = {2570, 47120, 0, 0, 0, 0, 0, 0}, addr = "\n\n\020\270", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {4027582986, 0}, addr32 = {4027582986, 0, 0, 0}, addr16 = {2570, 61456, 0, 0, 0, 0, 0, 0}, addr = "\n\n\020\360", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\n\n\020\270\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 3088058890}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 3088058890, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, __ss_padding = "\023\304\n\n\020\270", '\000' <repeats 111 times>, __ss_align = 0}}, bind_address = 0x7f5ec3ec6308, rflags = (unknown: 0), proto = 1 '\001', proto_pad0 = 0 '\000', proto_pad1 = 0} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} printbuf = "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0 0D 0A Via: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK0D \000A \000\300\337\333\376\177\000\000\351\305h\301^\177\000\000\330Ђ\000\000\000\000\000\a\000\000\000\004\000\000\000P\301\337\333\376\177\000\000\267\343T\000\000\000\000\000\360\016\201\000\000\000\000\000\360׀\000\000\000\000\000\004\000\000\000\000\000\000\000\004\000\000\000\004\000\000\000\230\337h\301^\177\000\000\334\001"... i = 100 j = 106 l = 4 __FUNCTION__ = "udp_rcv_loop" #28 0x000000000042b3a5 in main_loop () at main.c:1730 i = 3 pid = 0 si = 0x7f5ec3ec6308 si_desc = "udp receiver child=3 sock=192.168.10.240:5060 \000:\000\000\000\001\000\000\000\376\177\000\000\260\232\000\000\000\000\000\000\300\305E\001\000\000\000\000\300\351\202\000k\000\000\000[̀", '\000' <repeats 13 times>, "\340\311\337\333\376\177\000\000\030\342}\000\000\000\000\000\027\000\000\000\000\000\000\000@ \300A\000\000\000\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #29 0x000000000043684b in main (argc=5, argv=0x7ffedbdfcac8) at main.c:3053 cfg_stream = 0x1456040 c = -1 r = 0 tmp = 0x0 tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x7e1208 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2616789248 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffedbdfc980 p = 0x0 st = {st_dev = 19, st_ino = 21232, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1675551785, tv_nsec = 549443593}, st_mtim = {tv_sec = 1677547400, tv_nsec = 664002717}, st_ctim = {tv_sec = 1677547400, tv_nsec = 664002717}, __unused = {0, 0, 0}} tbuf = "\377\377\377\377", '\000' <repeats 12 times>, "\350sd\304^\177\000\000\310d3\305^\177", '\000' <repeats 90 times>, "\200\022\254\000\000\000\000\000\200\304A\000\000\000\000\000\300\312\337\333\376\177", '\000' <repeats 26 times>, "\356=\023\305^\177\000\000\001", '\000' <repeats 23 times>... option_index = 0 On Fri, Mar 3, 2023 at 12:50 AM Henning Westerholt &lt;hw(a)gilawa.com&gt; wrote: https://rpm.kamailio.org/centos/ - maybe this works for you. fixes it. wrote: commented in the linked issue? compilers can report different warnings, so it's sometimes happening. If they are just warnings, they should not create an issue. from deb.kamailio.org - you don't need to build by yourself. fixes it. 00:05:39 lab-danish /usr/local/sbin/kamailio[24642]: ERROR: find bind_acc Mar 3 00:05:39 lab-danish /usr/local/sbin/kamailio[24642]: ERROR: API Mar 3 00:05:39 lab-danish /usr/local/sbin/kamailio[24642]: ERROR: apparently it's not. the most tested and supported version ? [-Wmissing-braces] [-Wmissing-braces] wrote: releases only contains bugfixes, documentation enhancements and similar. Only rarely regressions happen. But you should of course test it. restart fixes it. wrote: first step, I would suggest you to update the system to one of the supported releases, e.g. the latest 5.6.x or 5.5.x. fixes it. proxy between 2 endpoints. Everything works fine for weeks and suddenly I see kamailio stops responding. From pcap I can see kamailio is not proxying the session progress and bombarding invites to one endpoint without any reason. Even that invite was stripped on the Body part. not take bt full yet. based on KEMI). only to the sender! -- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless