Hi Kamailio crushes with lots of such CRITICAL errors:
Aug 16 15:09:48 kam-fe1 kamailio[8485]: CRITICAL: <core> [tcp_read.c:1654]: handle_io(): io_watch_del failed for 0x7f3e610a3a70 id 105 fd -1, state -1, flags 4028, main fd -1, refcnt -2117 ([178.209.103.226]:40855 -> [178 .209.103.226]:5060) Aug 16 15:09:48 kam-fe1 kamailio[8479]: CRITICAL: <core> [io_wait.h:594]: io_watch_del(): invalid fd -1, not in [0, 23) Aug 16 15:09:48 kam-fe1 kamailio[8481]: WARNING: <core> [tcp_read.c:1629]: handle_io(): F_TCPCONN connection marked as bad: 0x7f3e608454e8 id 78 refcnt -257 Aug 16 15:09:48 kam-fe1 kamailio[8379]: INFO: <script>: 52373459-303f6c6b@192.168.2.2|log|source 128.204.55.172:5160 Aug 16 15:09:48 kam-fe1 kamailio[8485]: WARNING: <core> [tcp_read.c:1629]: handle_io(): F_TCPCONN connection marked as bad: 0x7f3e610a3a70 id 105 refcnt -2117 Aug 16 15:09:48 kam-fe1 kamailio[8479]: CRITICAL: <core> [tcp_read.c:1654]: handle_io(): io_watch_del failed for 0x7f3e63c73d90 id 984 fd -1, state -1, flags 5022, main fd -1, refcnt -126097 ([81.211.54.218]:51642 -> [81. 211.54.218]:5060) Aug 16 15:09:48 kam-fe1 kamailio[8481]: CRITICAL: <core> [io_wait.h:594]: io_watch_del(): invalid fd -1, not in [0, 2) Aug 16 15:09:48 kam-fe1 kamailio[8379]: INFO: <script>: 52373459-303f6c6b@192.168.2.2|log|from sip:Menedzher1_UYHPQ@29161.ztpbx.ru Aug 16 15:09:48 kam-fe1 kamailio[8485]: CRITICAL: <core> [io_wait.h:594]: io_watch_del(): invalid fd -1, not in [0, 2) Aug 16 15:09:48 kam-fe1 kamailio[8377]: ALERT: <core> [main.c:728]: handle_sigs(): child process 8504 exited by a signal 11
+++++++++++++++++++++++++++++++++ kamailio -v version: kamailio 4.3.5 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:51:55 Mar 8 2016 with gcc 4.4.7
What do you mean by crashes? Does it stop running or just printing error messages? If stops, provide the logs at shutdown.
it stops.
Aug 16 15:09:48 kam-fe1 kamailio[8481]: CRITICAL: <core> [io_wait.h:594]: io_watch_del(): invalid fd -1, not in [0, 2) Aug 16 15:09:48 kam-fe1 kamailio[8401]: INFO: <script>: 60BF-319A-48029472E1C3E56C1892-064@SipHost|log|originated from external sources Aug 16 15:09:48 kam-fe1 kamailio[8485]: CRITICAL: <core> [io_wait.h:594]: io_watch_del(): invalid fd -1, not in [0, 2) Aug 16 15:09:48 kam-fe1 kamailio[8479]: WARNING: <core> [tcp_read.c:1629]: handle_io(): F_TCPCONN connection marked as bad: 0x7f3e63c73d90 id 984 refcnt -126097 Aug 16 15:09:48 kam-fe1 kamailio[8481]: CRITICAL: <core> [tcp_read.c:1654]: handle_io(): io_watch_del failed for 0x7f3e608454e8 id 78 fd -1, state -1, flags 4028, main fd -1, refcnt -257 ([31.28.8.48]:1208 -> [31.28.8.48] :5060) Aug 16 15:09:48 kam-fe1 kamailio[8401]: INFO: <script>: 60BF-319A-48029472E1C3E56C1892-064@SipHost|end|issued auth challenge to new registration for sip:00001@4955055132.vats.gobaza.ru 79.174.32.137:5060 Aug 16 15:09:48 kam-fe1 kamailio[8379]: INFO: <script>: 52373459-303f6c6b@192.168.2.2|start|recieved udp request NOTIFY sip:29161.ztpbx.ru Aug 16 15:09:48 kam-fe1 kamailio[8485]: CRITICAL: <core> [tcp_read.c:1654]: handle_io(): io_watch_del failed for 0x7f3e610a3a70 id 105 fd -1, state -1, flags 4028, main fd -1, refcnt -2117 ([178.209.103.226]:40855 -> [178 .209.103.226]:5060) Aug 16 15:09:48 kam-fe1 kamailio[8479]: CRITICAL: <core> [io_wait.h:594]: io_watch_del(): invalid fd -1, not in [0, 23) Aug 16 15:09:48 kam-fe1 kamailio[8481]: WARNING: <core> [tcp_read.c:1629]: handle_io(): F_TCPCONN connection marked as bad: 0x7f3e608454e8 id 78 refcnt -257 Aug 16 15:09:48 kam-fe1 kamailio[8379]: INFO: <script>: 52373459-303f6c6b@192.168.2.2|log|source 128.204.55.172:5160 Aug 16 15:09:48 kam-fe1 kamailio[8485]: WARNING: <core> [tcp_read.c:1629]: handle_io(): F_TCPCONN connection marked as bad: 0x7f3e610a3a70 id 105 refcnt -2117 Aug 16 15:09:48 kam-fe1 kamailio[8479]: CRITICAL: <core> [tcp_read.c:1654]: handle_io(): io_watch_del failed for 0x7f3e63c73d90 id 984 fd -1, state -1, flags 5022, main fd -1, refcnt -126097 ([81.211.54.218]:51642 -> [81. 211.54.218]:5060) Aug 16 15:09:48 kam-fe1 kamailio[8481]: CRITICAL: <core> [io_wait.h:594]: io_watch_del(): invalid fd -1, not in [0, 2) Aug 16 15:09:48 kam-fe1 kamailio[8379]: INFO: <script>: 52373459-303f6c6b@192.168.2.2|log|from sip:Menedzher1_UYHPQ@29161.ztpbx.ru Aug 16 15:09:48 kam-fe1 kamailio[8485]: CRITICAL: <core> [io_wait.h:594]: io_watch_del(): invalid fd -1, not in [0, 2) Aug 16 15:09:48 kam-fe1 kamailio[8377]: ALERT: <core> [main.c:728]: handle_sigs(): child process 8504 exited by a signal 11
As soon as I get a back trace I will provide it.
[kamailio_backtrace.txt](https://github.com/kamailio/kamailio/files/426350/kamailio_backtrace.txt)
Hello I got a backtrace
the Private memory was 4 mb. I increased the Private memory to 32 mb. Probably it will help to resolve the issue.
Private memory increase did not help. Kamailio crushed yestearday
hi almost all "gdb/bt full" contain: #13 0x00000000005f916c in parse_disposition (s=0xa75468 <tcp_reader_ltimer+228904>, disp=<optimized out>) at parser/parse_disposition.c:60 disp_p = <optimized out> new_p = <optimized out> state = 2 saved_state = 2 tmp = 0x7f7eb893b810 "" end = 0x126c9b6d00000000 <error: Cannot access memory at address 0x126c9b6d00000000> __func__ = "parse_disposition"
may be it helps to solve the issue?
Did you install from source code or from packages? Because the symbols are missing in most of the frames in backtraces, looking invalid and being not useful to track down the issue.
Core.dumps were generated in production server. I do not know how to define if kamailio-dbg rpm was install in that production server.
I made gdb at another server with kamailio-dbg installed.
The production kamailio was installed from Kazoo repository(2600hz rpm)Core.dumps were generated in production server. I do not know how to define if kamailio-dbg rpm was install in that production server.Is it safe to install this rpm on production server (it has a high load) so probably it may be done at night?I made gdb at another server with kamailio-dbg installed.
On Friday, August 26, 2016 12:20 PM, Daniel-Constantin Mierla notifications@github.com wrote:
Did you install from source code or from packages? Because the symbols are missing in most of the frames in backtraces, looking invalid and being not useful to track down the issue.— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
Package kazoo-kamailio-debuginfo-4.3.4-8.el6.x86_64 already installed and latest version
Can you try with master branch or backport the patch referenced above into your clone? If all ok, then I will backport to stable branches inside kamailio repo.
Use also the second patch referenced above, because your version was older than I first looked at.
Closed #748.
Reopen if the latest patches don't fix it.
-
Daniel-Constantin Mierla -
mbike2000ru