Description

I upgraded kamailio from 4.4.5 to 5.0.0 using the Debian repositories for Debian Jessie.
After upgrading the packages using aptitude the command kamcmd ws.dump does not generate any output but a coredump.

Reproduction

I can reproduce it anytime by running kamcmd ws.dump

Debugging Data

[0][root@webrtc:~]# gdb /usr/sbin/kamailio /tmp/core
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/fe/4d6c322f76df685bbec9adafde99fc43c0bc6a.debug...done.
done.
[New LWP 9079]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f684d974a3b in rpc_struct_add (s=0x14bf340, fmt=0x7f684ca4f4ae "[{") at binrpc_run.c:1101
1101	binrpc_run.c: Datei oder Verzeichnis nicht gefunden.
(gdb) backtrace
#0  0x00007f684d974a3b in rpc_struct_add (s=0x14bf340, fmt=0x7f684ca4f4ae "[{") at binrpc_run.c:1101
#1  0x00007f684ca3a02d in ws_rpc_dump (rpc=0x7f684dba2740 <binrpc_callbacks>, ctx=0x7ffd8a1ce500) at ws_conn.c:625
#2  0x00007f684d97172d in process_rpc_req (buf=0x14a72a4 "\241\003\nY\302\tq\221\bws.dump", size=17, bytes_needed=0x7ffd8a1ce608, sh=0x7ffd8a1ce630, 
    saved_state=0x14b72a8) at binrpc_run.c:675
#3  0x00007f684d9677bd in handle_stream_read (s_c=0x14a7270, idx=-1) at io_listener.c:511
#4  0x00007f684d969212 in handle_io (fm=0x7f684fd6b1d8, events=1, idx=-1) at io_listener.c:706
#5  0x00007f684d960abf in io_wait_loop_epoll (h=0x7f684db92400 <io_h>, t=10, repeat=0) at ../../core/io_wait.h:1065
#6  0x00007f684d96495c in io_listen_loop (fd_no=1, cs_lst=0x14a1900) at io_listener.c:281
#7  0x00007f684d98048d in mod_child (rank=0) at ctl.c:335
#8  0x000000000053d8c2 in init_mod_child (m=0x7f684fcc2908, rank=0) at core/sr_module.c:921
#9  0x000000000053d5e0 in init_mod_child (m=0x7f684fcc2f70, rank=0) at core/sr_module.c:918
#10 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc35e0, rank=0) at core/sr_module.c:918
#11 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc3de8, rank=0) at core/sr_module.c:918
#12 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc4550, rank=0) at core/sr_module.c:918
#13 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc51d8, rank=0) at core/sr_module.c:918
#14 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc5c70, rank=0) at core/sr_module.c:918
#15 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6138, rank=0) at core/sr_module.c:918
#16 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6670, rank=0) at core/sr_module.c:918
#17 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6e28, rank=0) at core/sr_module.c:918
#18 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc7268, rank=0) at core/sr_module.c:918
#19 0x000000000053dbf0 in init_child (rank=0) at core/sr_module.c:947
#20 0x000000000042357c in main_loop () at main.c:1700
#21 0x0000000000429f71 in main (argc=15, argv=0x7ffd8a1cf208) at main.c:2639
(gdb) 
#0  0x00007f684d974a3b in rpc_struct_add (s=0x14bf340, fmt=0x7f684ca4f4ae "[{") at binrpc_run.c:1101
#1  0x00007f684ca3a02d in ws_rpc_dump (rpc=0x7f684dba2740 <binrpc_callbacks>, ctx=0x7ffd8a1ce500) at ws_conn.c:625
#2  0x00007f684d97172d in process_rpc_req (buf=0x14a72a4 "\241\003\nY\302\tq\221\bws.dump", size=17, bytes_needed=0x7ffd8a1ce608, sh=0x7ffd8a1ce630, 
    saved_state=0x14b72a8) at binrpc_run.c:675
#3  0x00007f684d9677bd in handle_stream_read (s_c=0x14a7270, idx=-1) at io_listener.c:511
#4  0x00007f684d969212 in handle_io (fm=0x7f684fd6b1d8, events=1, idx=-1) at io_listener.c:706
#5  0x00007f684d960abf in io_wait_loop_epoll (h=0x7f684db92400 <io_h>, t=10, repeat=0) at ../../core/io_wait.h:1065
#6  0x00007f684d96495c in io_listen_loop (fd_no=1, cs_lst=0x14a1900) at io_listener.c:281
#7  0x00007f684d98048d in mod_child (rank=0) at ctl.c:335
#8  0x000000000053d8c2 in init_mod_child (m=0x7f684fcc2908, rank=0) at core/sr_module.c:921
#9  0x000000000053d5e0 in init_mod_child (m=0x7f684fcc2f70, rank=0) at core/sr_module.c:918
#10 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc35e0, rank=0) at core/sr_module.c:918
#11 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc3de8, rank=0) at core/sr_module.c:918
#12 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc4550, rank=0) at core/sr_module.c:918
#13 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc51d8, rank=0) at core/sr_module.c:918
#14 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc5c70, rank=0) at core/sr_module.c:918
#15 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6138, rank=0) at core/sr_module.c:918
#16 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6670, rank=0) at core/sr_module.c:918
#17 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6e28, rank=0) at core/sr_module.c:918
#18 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc7268, rank=0) at core/sr_module.c:918
#19 0x000000000053dbf0 in init_child (rank=0) at core/sr_module.c:947
#20 0x000000000042357c in main_loop () at main.c:1700
#21 0x0000000000429f71 in main (argc=15, argv=0x7ffd8a1cf208) at main.c:2639
(gdb) bt full
#0  0x00007f684d974a3b in rpc_struct_add (s=0x14bf340, fmt=0x7f684ca4f4ae "[{") at binrpc_run.c:1101
        ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7ffd8a1ce3d0, reg_save_area = 0x7ffd8a1ce2e0}}
        err = 0
        avp = {name = {s = 0x7f684ca4f4a2 "connections", len = 11}, type = 3, u = {strval = {s = 0x0, len = 0}, fval = 0, intval = 0, end = 0}}
        rs = 0x14c1390
        __FUNCTION__ = "rpc_struct_add"
#1  0x00007f684ca3a02d in ws_rpc_dump (rpc=0x7f684dba2740 <binrpc_callbacks>, ctx=0x7ffd8a1ce500) at ws_conn.c:625
        h = 6587863
        connections = 0
        truncated = 0
        order = 0
        found = 0
        wsc = 0x7f684fd634f8
        sorder = {s = 0x7f684d98a158 "if you get this string, you don'tcheck rpc_scan return code !!! (very bad)", len = 74}
        th = 0x14bf340
        ih = 0xad9180 <rpc_hash_table>
        dh = 0x38a1ce4b0
        __FUNCTION__ = "ws_rpc_dump"
#2  0x00007f684d97172d in process_rpc_req (buf=0x14a72a4 "\241\003\nY\302\tq\221\bws.dump", size=17, bytes_needed=0x7ffd8a1ce608, sh=0x7ffd8a1ce630, 
    saved_state=0x14b72a8) at binrpc_run.c:675
        err = 0
        val = {name = {s = 0x7ffd8a1ce5e0 "\360\346\034\212\375\177", len = 1301657797}, type = 1, u = {strval = {s = 0x14a72ad "ws.dump", len = 7}, 
            fval = 1.06996027198961e-316, intval = 21656237, end = 21656237}}
        rpc_e = 0x7f684fd63528
        f_ctx = {in = {ctx = {tlen = 10, cookie = 1505888625, type = 0, flags = 1, offset = 10, in_struct = 0, in_array = 0}, s = 0x14a72b5 "", end = 0x14a72b5 "", 
            record_no = 0, in_struct = 0}, out = {pkt = {body = 0x14b7330 "\003\203", end = 0x14bf330 "", crt = 0x14b7332 ""}, structs = {next = 0x14bf340, 
              prev = 0x14bf340}}, send_h = 0x7ffd8a1ce630, method = 0x14a72ad "ws.dump", gc = 0x0, replied = 0, err_code = 0, err_phrase = {s = 0x0, len = 0}}
        ctx = 0x7ffd8a1ce500
        __FUNCTION__ = "process_rpc_req"
#3  0x00007f684d9677bd in handle_stream_read (s_c=0x14a7270, idx=-1) at io_listener.c:511
        bytes_free = 65535
        bytes_read = 17
---Type <return> to continue, or q <return> to quit---
        bytes_needed = 21633280
        bytes_processed = 0
        r = 0x14a7290
        sh = {fd = 8, type = 0, from = {sa_in = {s = {sa_family = 29296, sa_data = "J\001\000\000\000\000\001\000\311Oh\177\000"}, sin = {sin_family = 29296, 
                sin_port = 330, sin_addr = {s_addr = 0}, sin_zero = "\001\000\311Oh\177\000"}, sin6 = {sin6_family = 29296, sin6_port = 330, sin6_flowinfo = 0, 
                sin6_addr = {__in6_u = {__u6_addr8 = "\001\000\311Oh\177\000\000\370\224\242\000\000\000\000", __u6_addr16 = {1, 20425, 32616, 0, 38136, 162, 0, 
                      0}, __u6_addr32 = {1338572801, 32616, 10654968, 0}}}, sin6_scope_id = 1}}, sa_un = {sun_family = 29296, 
              sun_path = "J\001\000\000\000\000\001\000\311Oh\177\000\000\370\224\242\000\000\000\000\000\001\000\000\000\000\000\000\000\330H\036Gh\177", '\000' <repeats 18 times>, "\260\346\034\212\001\000\000\000\330H\036Gh\177\000\000\260\346\034\212\375\177\000\000\255k\225Mh\177\000\000\300\346\034\212\375\177\000\000\330H\036Gh\177\000\000\002\375\255?\000"}}, from_len = 0}
        __FUNCTION__ = "handle_stream_read"
#4  0x00007f684d969212 in handle_io (fm=0x7f684fd6b1d8, events=1, idx=-1) at io_listener.c:706
        ret = 1
        __FUNCTION__ = "handle_io"
#5  0x00007f684d960abf in io_wait_loop_epoll (h=0x7f684db92400 <io_h>, t=10, repeat=0) at ../../core/io_wait.h:1065
        n = 1
        r = 0
        fm = 0x7f684fd6b1d8
        revents = 1
        __FUNCTION__ = "io_wait_loop_epoll"
#6  0x00007f684d96495c in io_listen_loop (fd_no=1, cs_lst=0x14a1900) at io_listener.c:281
        max_fd_no = 195
        poll_err = 0x0
        poll_method = 2
        cs = 0x0
        type = 2
        __FUNCTION__ = "io_listen_loop"
#7  0x00007f684d98048d in mod_child (rank=0) at ctl.c:335
        pid = 0
        cs = 0x4178f0 <_start>
        rpc_handler = 1
        __FUNCTION__ = "mod_child"
---Type <return> to continue, or q <return> to quit---
#8  0x000000000053d8c2 in init_mod_child (m=0x7f684fcc2908, rank=0) at core/sr_module.c:921
        __FUNCTION__ = "init_mod_child"
#9  0x000000000053d5e0 in init_mod_child (m=0x7f684fcc2f70, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#10 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc35e0, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#11 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc3de8, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#12 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc4550, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#13 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc51d8, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#14 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc5c70, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#15 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6138, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#16 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6670, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#17 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6e28, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#18 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc7268, rank=0) at core/sr_module.c:918
        __FUNCTION__ = "init_mod_child"
#19 0x000000000053dbf0 in init_child (rank=0) at core/sr_module.c:947
No locals.
#20 0x000000000042357c in main_loop () at main.c:1700
        i = 4
        pid = 9077
        si = 0x0
        si_desc = "udp receiver child=3 sock=10.3.66.231:5060\000\363o\000\000\000\361\240u\000\000\000\000\000\000\366\374\363\340\341\232\n\004\000\000\000\000\000\000\000\002\375\255?\000\000\000\000\360xA\000\000\000\000\000\000\362\034\212\375\177", '\000' <repeats 18 times>, "\060\357\034\212\375\177\000\000\035\322^\000\000\000\000"
        nrprocs = 4
---Type <return> to continue, or q <return> to quit---
        woneinit = 1
        __FUNCTION__ = "main_loop"
#21 0x0000000000429f71 in main (argc=15, argv=0x7ffd8a1cf208) at main.c:2639
        cfg_stream = 0x140d010
        c = -1
        r = 0
        tmp = 0x7ffd8a1d0ef2 ""
        tmp_len = 1346830760
        port = 32616
        proto = -1977814848
        options = 0x737490 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
        ret = -1
        seed = 27692109
        rfd = 4
        debug_save = 0
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = 0x0
        p = 0x1 <error: Cannot access memory at address 0x1>
        st = {st_dev = 15, st_ino = 14760, st_nlink = 2, st_mode = 16832, st_uid = 109, st_gid = 114, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, 
          st_blocks = 0, st_atim = {tv_sec = 1488887721, tv_nsec = 984216540}, st_mtim = {tv_sec = 1488888110, tv_nsec = 550708698}, st_ctim = {
            tv_sec = 1488888110, tv_nsec = 550708698}, __glibc_reserved = {0, 0, 0}}
        __FUNCTION__ = "main"
(gdb) 
(gdb) info locals
ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7ffd8a1ce3d0, reg_save_area = 0x7ffd8a1ce2e0}}
err = 0
avp = {name = {s = 0x7f684ca4f4a2 "connections", len = 11}, type = 3, u = {strval = {s = 0x0, len = 0}, fval = 0, intval = 0, end = 0}}
rs = 0x14c1390
__FUNCTION__ = "rpc_struct_add"
(gdb) list
1096	in binrpc_run.c
(gdb) quit

Log Messages

Mar  7 13:15:45 webrtc kernel: [9074388.189612] kamailio[9526]: segfault at 7fab895ad49d ip 00007fab8a4d2a3b sp 00007fff0712d3d0 error 7 in ctl.so[7fab8a4a1000+4e000]
Mar  7 13:15:45 webrtc /usr/sbin/kamailio[9518]: ALERT: <core> [main.c:744]: handle_sigs(): child process 9526 exited by a signal 11
Mar  7 13:15:45 webrtc /usr/sbin/kamailio[9518]: ALERT: <core> [main.c:747]: handle_sigs(): core was generated
Mar  7 13:15:45 webrtc systemd[1]: kamailio.service: main process exited, code=exited, status=1/FAILURE
Mar  7 13:15:45 webrtc systemd[1]: Unit kamailio.service entered failed state.

Additional Information

[0][root@webrtc:~]# dpkg -l|grep kamailio
ii  kamailio                            5.0.0+jessie                      amd64        very fast, dynamic and configurable SIP server
ii  kamailio-dbg:amd64                  5.0.0+jessie                      amd64        very fast and configurable SIP server [debug symbols]
ii  kamailio-extra-modules:amd64        5.0.0+jessie                      amd64        Extra modules for the Kamailio SIP Server
ii  kamailio-outbound-modules:amd64     5.0.0+jessie                      amd64        SIP Outbound module for the Kamailio SIP server
ii  kamailio-utils-modules:amd64        5.0.0+jessie                      amd64        Utility functions for the Kamailio SIP server
ii  kamailio-websocket-modules:amd64    5.0.0+jessie                      amd64        WebSocket module for the Kamailio SIP server
[0][root@webrtc:~]#

[130][root@webrtc:~]# kamailio -v
version: kamailio 5.0.0 (x86_64/linux) 
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled with gcc 4.9.2
[0][root@webrtc:~]#

Freshly updated Debian Jessie

[0][root@webrtc:~]# cat /etc/issue
Debian GNU/Linux 8 \n \l

[0][root@webrtc:~]# uname -a
Linux webrtc 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux
[0][root@webrtc:~]#


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.