[kamailio/kamailio] Kamailio became unresponsive during runtime (#2380)
<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports. If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list: * http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list: * http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue. If there is no content to be filled in a section, the entire section can be removed. You can delete the comments from the template sections when filling. You can delete next line and everything above before submitting (it is a comment). --> ### Description Hi, I ran into an issue where the Kamailio service seemingly froze (didn't crash, but failed to receive and deliver calls) while running, I went through the logs a little bit, and saw the same log pattern (pasted below) recouring every few months or so, on an older installation (5.2.0). Usually, it ended up automatically restarting the service. But not this time, I had to reload it manually to make the service work again. Is this a known issue? How can I make sure that the service will be able to restart the next time it happens? Edward ### Troubleshooting #### Debugging Data <!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location. gdb /path/to/kamailio /path/to/ bt full info locals list If you are familiar with gdb, feel free to attach more of what you consider to be relevant. --> ``` GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/kamailio...done. warning: exec file is newer than core file. [New LWP 11413] Core was generated by `kamailio'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f4e4a52e428 in ?? () (gdb) Quit (gdb) bt full #0 0x00007f4e4a52e428 in ?? () No symbol table info available. #1 0x00007f4e4a53002a in ?? () No symbol table info available. #2 0x0000000000000020 in ?? () No symbol table info available. #3 0x0000000000000000 in ?? () No symbol table info available. (gdb) info locals No symbol table info available. (gdb) list 1871 & now if we don't need it */ 1872 #ifdef USE_SLOW_TIMER 1873 + 1 /* slow timer process */ 1874 #endif 1875 #ifdef USE_TCP 1876 +((!tcp_disable)?( 1/* tcp main */ + tcp_listeners ):0) 1877 #endif 1878 #ifdef USE_SCTP 1879 +((!sctp_disable)?sctp_listeners:0) 1880 #endif (gdb) ``` #### Log Messages <!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> ``` Jun 28 16:35:22 kamprodegres kamailio: INFO: <core> [core/sctp_core.c:74]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module Jun 28 16:35:22 kamprodegres kamailio: WARNING: <core> [core/socket_info.c:1394]: fix_hostname(): could not rev. resolve 70.36.25.87 Jun 28 16:35:22 kamprodegres kamailio: WARNING: <core> [core/socket_info.c:1394]: fix_hostname(): could not rev. resolve 70.36.25.87 Jun 28 16:35:22 kamprodegres kamailio: INFO: <core> [core/tcp_main.c:5042]: init_tcp(): using epoll_lt as the io watch method (auto detected) Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: jsonrpcs [jsonrpcs_sock.c:197]: jsonrpc_dgram_mod_init(): the socket /var/run/kamailio/kamailio_rpc.sock already exists, trying to delete it... Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: rr [rr_mod.c:177]: mod_init(): outbound module not available Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [main.c:2779]: main(): processes (at least): 40 - shm size: 67108864 - pkg size: 8388608 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 Jun 28 16:35:22 kamprodegres kamailio[8518]: INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/8518 Jun 28 16:35:22 kamprodegres kamailio[8519]: INFO: ctl [io_listener.c:214]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) Jun 29 18:13:37 kamprodegres kamailio[8513]: INFO: {1 581461 CANCEL 402954192_49936786@67.231.13.146} tm [t_reply.c:478]: _reply_light(): can't generate 487 reply when a final 404 was sent out Jun 29 18:27:00 kamprodegres kamailio[8508]: INFO: {1 211236 CANCEL 405019408_127896329@67.231.13.146} tm [t_reply.c:478]: _reply_light(): can't generate 487 reply when a final 404 was sent out Jun 29 18:51:10 kamprodegres /usr/local/sbin/kamailio[2269]: NOTICE: <core> [main.c:725]: handle_sigs(): Thank you for flying kamailio!!! Jun 29 18:51:10 kamprodegres /usr/local/sbin/kamailio[2326]: INFO: <core> [main.c:847]: sig_usr(): signal 15 received ``` ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.3.2 (x86_64/linux) 7ba545 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 7ba545 compiled on 16:37:32 Jun 14 2020 with gcc 5.4.0 ``` * **Operating System**: <!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) --> ``` Ubuntu 16.04 ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380
Hello, you need to install kamailio debugging symbols to get meaningful information from the backtrace. From the log message it looks ok. You should first try to update to the latest 5.3 version (e.g. 5.3.5 released recently), as it might be already fixed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-651328024
Hello, you need to install kamailio debugging symbols to get meaningful information from the backtrace. From the log message it looks ok. You should first try to update to the latest 5.3 version (e.g. 5.3.5 released recently), as it might be already fixed.
Can I do that after the coredump has already been generated? If so, can you point me to how can I do that? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-651329820
Yes, look for a packaged called kamailio-dbg, kamailio-debug or similar and install it. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-651330503
Hmm. I did install it, but I get the same output. `root@kamprodegres:/# gdb /usr/local/sbin/kamailio core GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/kamailio...done. warning: exec file is newer than core file. [New LWP 11413] Core was generated by `kamailio'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f4e4a52e428 in ?? () (gdb) bt full #0 0x00007f4e4a52e428 in ?? () No symbol table info available. #1 0x00007f4e4a53002a in ?? () No symbol table info available. #2 0x0000000000000020 in ?? () No symbol table info available. #3 0x0000000000000000 in ?? () No symbol table info available. (gdb) info locals No symbol table info available. (gdb) Quit (gdb) quit root@kamprodegres:/# apt-get install kamailio-dbg Reading package lists... Done Building dependency tree Reading state information... Done kamailio-dbg is already the newest version (4.3.4-1.1ubuntu2.1). The following packages were automatically installed and are no longer required: gyp javascript-common libjs-inherits libjs-jquery libjs-node-uuid libjs-underscore libuv1 libuv1-dev node-abbrev node-ansi node-ansi-color-table node-archy node-async node-block-stream node-combined-stream node-cookie-jar node-delayed-stream node-forever-agent node-form-data node-fstream node-fstream-ignore node-github-url-from-git node-glob node-graceful-fs node-gyp node-inherits node-ini node-json-stringify-safe node-lockfile node-lru-cache node-mime node-minimatch node-mkdirp node-mute-stream node-node-uuid node-nopt node-normalize-package-data node-npmlog node-once node-osenv node-qs node-read node-read-package-json node-request node-retry node-rimraf node-semver node-sha node-sigmund node-slide node-tar node-tunnel-agent node-underscore node-which python-pkg-resources Use 'sudo apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 442 not upgraded. root@kamprodegres:/# ` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-651339931
Ok, in this case it the backtrace information might be from somewhere else. But I would suggest to update to the lastest kamailio version first. It looks a bit like a crash during shutdown, which might be already fixed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-651341362
Are you aware of similar bug that has been fixed? Edward On Mon, Jun 29, 2020, 23:24 Henning Westerholt <notifications@github.com> wrote:
Ok, in this case it the backtrace information might be from somewhere else. But I would suggest to update to the lastest kamailio version first. It looks a bit like a crash during shutdown, which might be already fixed.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/kamailio/kamailio/issues/2380#issuecomment-651341362>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACM76S472YM6GDETFOBEX4LRZD2A5ANCNFSM4OLN6KAQ> .
-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-651547173
This is hard to say without more details. You'll find all fixes in the Changelog: https://www.kamailio.org/pub/kamailio/5.3.5/ChangeLog Close this one, please re-open if the bug appears again after you updated to 5.3.5. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-651560460
Closed #2380. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#event-3495969512
I have the same issue in Kamailio 5.3.5, right now. Here is backtrace information. warning: exec file is newer than core file. [New LWP 56381] Core was generated by `/usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamail'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f7837e917bb in ?? () (gdb) bt full #0 0x00007f7837e917bb in ?? () No symbol table info available. #1 0x0000000000002000 in ?? () No symbol table info available. #2 0x0000000000000000 in ?? () No symbol table info available. (gdb) info locals No symbol table info available. (gdb) No symbol table info available. (gdb) No symbol table info available. (gdb) No symbol table info available. (gdb) No symbol table info available. (gdb) No symbol table info available. (gdb) list 1883 +((!sctp_disable)?sctp_listeners:0) 1884 #endif 1885 ; 1886 } 1887 1888 int main(int argc, char** argv) 1889 { 1890 1891 FILE* cfg_stream; 1892 int c,r; (gdb) I know i am using old version of Kamailio (5.3.5), But i want to know what is the problem? -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-2193897312 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/2380/2193897312@github.com>
As already discussed, its unfortunately not possible to support this old end of life versions in the open source project. Please upgrade to a supported version (5.7.x or 5.8.x) of Kamailio. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380#issuecomment-2193900892 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/2380/2193900892@github.com>
participants (3)
-
Henning Westerholt -
Mojtaba Esfandiari.S
-
ObakeFilter