May 3 20:18:09 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:488]: connect_servers(): Connecting to servers for priority 2: May 3 20:18:09 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:494]: connect_servers(): Connected to host 219.216.65.195 on port 9999 May 3 20:18:09 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46726]: INFO: ctl [io_listener.c:214]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46708]: INFO: jsonrpc-c [jsonrpc_request.c:66]: jsonrpc_request(): jsonrp_requestctmb address is 0x7f741674e440 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46708]: ERROR: jsonrpc-c [jsonrpc_request.c:94]: jsonrpc_request(): START TM MODULE May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46708]: ERROR: jsonrpc-c [jsonrpc_request.c:95]: jsonrpc_request(): hash is 4976 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46708]: ERROR: jsonrpc-c [jsonrpc_request.c:96]: jsonrpc_request(): label is 181128518 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:189]: cmd_pipe_cb(): SEND STRING IS87:{ "id": 2, "jsonrpc": "2.0", "method": "start", "params": { "sdp": "aa", "id": 1000 } }, May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: ERROR: jsonrpc-c [netstring.c:73]: netstring_read_fd(): bytes is 99 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: ERROR: jsonrpc-c [netstring.c:74]: netstring_read_fd(): read_len is 99 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:281]: socket_cb(): NETSTRING IS{"jsonrpc":"2.0","id":2,"result":"This is result!","params":{"sdpAnswer":"This is sdpAnswer!"}} May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc.c:96]: handle_jsonrpc_response(): id is 2 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc.c:105]: handle_jsonrpc_response(): GET THE RESULT step one May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc.c:107]: handle_jsonrpc_response(): PROCESS THE STEP TWO May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:130]: result_cb(): 2 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:134]: result_cb(): action s address 0x7f741b3b73c0 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:135]: result_cb(): action ss adress 0x7ffe40907dd0 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:136]: result_cb(): t_hash4976 May 3 20:18:19 lwm-PowerEdge-R720 kernel: [4074069.476634] kamailio[46727]: segfault at 38 ip 00007f7418b148a0 sp 00007ffe40907580 error 4 in tm.so[7f7418a50000+11e000] May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:137]: result_cb(): t_label181128518 May 3 20:18:19 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46727]: INFO: jsonrpc-c [jsonrpc_io.c:140]: result_cb(): tmb address is 0x7f741674e440 May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46745]: CRITICAL: <core> [pass_fd.c:277]: receive_fd(): EOF on 5 May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46694]: ALERT: <core> [main.c:727]: handle_sigs(): child process 46727 exited by a signal 11 May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46694]: ALERT: <core> [main.c:730]: handle_sigs(): core was generated May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46694]: INFO: <core> [main.c:743]: handle_sigs(): terminating due to SIGCHLD May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46745]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46743]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46741]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46726]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46700]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46699]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46697]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46701]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46712]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46703]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46708]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46716]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46702]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46695]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46704]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46698]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46710]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46731]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46733]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46696]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received
I pringt some of the main opeartion to find it mainly caused by the call tmb.t_continue(cmd->t_hash, cmd->t_label, a) in jsonrpc_io.c at the line of 137. Please help me !
The log messages indicate that a core file was generated:
``` May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46694]: ALERT: [main.c:727]: handle_sigs(): child process 46727 exited by a signal 11 May 3 20:18:20 lwm-PowerEdge-R720 /usr/local/sbin/kamailio[46694]: ALERT: [main.c:730]: handle_sigs(): core was generated ```
Can you locate it and get the output of `bt full` from `gdb` investigating the core? The corefile may be either in `/` (root folder), the runtime directory for kamailio or a special folder configured in the OS for dumping the core files.
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.23.so...done. done. Reading symbols from /lib/x86_64-linux-gnu/libgcc_s.so.1...(no debugging symbols found)...done. Reading symbols from /usr/local/lib64/kamailio/modules/mi_fifo.so...done. Reading symbols from /usr/local/lib64/kamailio/libkmi.so.1...done. Reading symbols from /usr/local/lib64/kamailio/modules/kex.so...done. Reading symbols from /usr/local/lib64/kamailio/libkcore.so.1...done. Reading symbols from /usr/local/lib64/kamailio/libsrutils.so.1...done. Reading symbols from /usr/local/lib64/kamailio/modules/corex.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/tm.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/tmx.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/sl.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/rr.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/pv.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/maxfwd.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/usrloc.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/registrar.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/textops.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/siputils.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/xlog.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/sanity.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/ctl.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/cfg_rpc.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/mi_rpc.so...done. Reading symbols from /usr/local/lib64/kamailio/libbinrpc.so.0...done. Reading symbols from /usr/local/lib64/kamailio/modules/acc.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/jsonrpc-c.so...done. Reading symbols from /lib/x86_64-linux-gnu/libjson-c.so.2...(no debugging symbols found)...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5...(no debugging symbols found)...done. Reading symbols from /usr/local/lib64/kamailio/modules/sdpops.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/auth.so...done. Reading symbols from /usr/local/lib64/kamailio/modules/auth_db.so...done. Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.23.so...done. done. Reading symbols from /lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2...(no debugging symbols found)...done. Reading symbols from /lib/x86_64-linux-gnu/libnss_dns.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_dns-2.23.so...done. done. 0x00007f2286cc7290 in __pause_nocancel () at ../sysdeps/unix/syscall-template.S:84 84 ../sysdeps/unix/syscall-template.S: No such file or directory. (gdb) bt #0 0x00007f2286cc7290 in __pause_nocancel () at ../sysdeps/unix/syscall-template.S:84 #1 0x000000000053abed in main_loop () at main.c:1706 #2 0x00000000005402f1 in main (argc=5, argv=0x7ffc5151b8a8) at main.c:2553 (gdb) bt #0 0x00007f2286cc7290 in __pause_nocancel () at ../sysdeps/unix/syscall-template.S:84 #1 0x000000000053abed in main_loop () at main.c:1706
I just wonder if i hvae to change it into debug mode.Thanks.
It seems that the core of the process that crashed was overwritten at shutdown. Can you enable core file per pid with the next command:
``` echo "1" > /proc/sys/kernel/core_uses_pid ``` Then reproduce the case. You should get more core files, each with the pid number in the name. Take the bt for every core file and paste them here.
gdb /usr/local/sbin/kamailio core.1778 GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/kamailio...done. [New LWP 1778] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/local/sbin/kamailio -P /var/run/kamailio.pid -f /usr/local/etc/kamailio//k'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f9adcd708a0 in cfg_update_local (no_cbs=0) at ../../cfg/cfg_struct.h:358 358 cfg_child_cb = cfg_child_cb->next; (gdb) bt #0 0x00007f9adcd708a0 in cfg_update_local (no_cbs=0) at ../../cfg/cfg_struct.h:358 #1 0x00007f9adcd731e0 in t_continue (hash_index=3009, label=1421635639, route=0x7f9adf6133c0) at t_suspend.c:181 #2 0x00007f9ada79a0e9 in result_cb (result=0x293c6c0, data=0x7f9ad5f80ee0 "8\020\370՚\177", error=0) at jsonrpc_io.c:142 #3 0x00007f9ada790cc7 in handle_jsonrpc_response (response=0x293c2e0) at jsonrpc.c:108 #4 0x00007f9ada79d378 in socket_cb (fd=12, event=2, arg=0x7f9adf652668) at jsonrpc_io.c:291 #5 0x00007f9ada3494c9 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 #6 0x00007f9ada7987bc in jsonrpc_io_child_process (cmd_pipe=7, _servers=0x7f9adf601f90 "219.216.65.195:9999,2") at jsonrpc_io.c:92 #7 0x00007f9ada792a54 in child_init (rank=0) at jsonrpc_mod.c:137 #8 0x0000000000548cc7 in init_mod_child (m=0x7f9adf600678, rank=0) at sr_module.c:927 #9 0x0000000000548a04 in init_mod_child (m=0x7f9adf600a50, rank=0) at sr_module.c:924 #10 0x0000000000548a04 in init_mod_child (m=0x7f9adf601288, rank=0) at sr_module.c:924 #11 0x0000000000548a04 in init_mod_child (m=0x7f9adf6018c8, rank=0) at sr_module.c:924 #12 0x0000000000548fcc in init_child (rank=0) at sr_module.c:953 #13 0x000000000053a04d in main_loop () at main.c:1654 #14 0x00000000005402f1 in main (argc=5, argv=0x7fffe49f1518) at main.c:2553
I've run kamailio again and got only one core.
Somehow this went on the side, but I just pushed a patch that hopefully takes care of it.
On the other hand, I think janssonrpcc might be a better alternative -- iirc, it was added by the same developer as jsonrpcc, targeting to be a better replacement for jsonrpcc.
Closing -- if still an issue after the commit referenced above, reopen.
Closed #1520.
-
Daniel-Constantin Mierla -
theblackworld