[sr-dev] [kamailio/kamailio] kamcmd ws.dump generates coredump (#1022)

### Description

I upgraded kamailio from 4.4.5 to 5.0.0 using the Debian repositories for Debian Jessie. After upgrading the packages using aptitude the command `kamcmd ws.dump` does not generate any output but a coredump.

#### Reproduction

I can reproduce it anytime by running `kamcmd ws.dump`

#### Debugging Data

``` [0][root@webrtc:~]# gdb /usr/sbin/kamailio /tmp/core GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/fe/4d6c322f76df685bbec9adafde99fc43c0bc6a.debug...done. done. [New LWP 9079] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f684d974a3b in rpc_struct_add (s=0x14bf340, fmt=0x7f684ca4f4ae "[{") at binrpc_run.c:1101 1101 binrpc_run.c: Datei oder Verzeichnis nicht gefunden. (gdb) backtrace #0 0x00007f684d974a3b in rpc_struct_add (s=0x14bf340, fmt=0x7f684ca4f4ae "[{") at binrpc_run.c:1101 #1 0x00007f684ca3a02d in ws_rpc_dump (rpc=0x7f684dba2740 <binrpc_callbacks>, ctx=0x7ffd8a1ce500) at ws_conn.c:625 #2 0x00007f684d97172d in process_rpc_req (buf=0x14a72a4 "\241\003\nY\302\tq\221\bws.dump", size=17, bytes_needed=0x7ffd8a1ce608, sh=0x7ffd8a1ce630, saved_state=0x14b72a8) at binrpc_run.c:675 #3 0x00007f684d9677bd in handle_stream_read (s_c=0x14a7270, idx=-1) at io_listener.c:511 #4 0x00007f684d969212 in handle_io (fm=0x7f684fd6b1d8, events=1, idx=-1) at io_listener.c:706 #5 0x00007f684d960abf in io_wait_loop_epoll (h=0x7f684db92400 <io_h>, t=10, repeat=0) at ../../core/io_wait.h:1065 #6 0x00007f684d96495c in io_listen_loop (fd_no=1, cs_lst=0x14a1900) at io_listener.c:281 #7 0x00007f684d98048d in mod_child (rank=0) at ctl.c:335 #8 0x000000000053d8c2 in init_mod_child (m=0x7f684fcc2908, rank=0) at core/sr_module.c:921 #9 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc2f70, rank=0) at core/sr_module.c:918 #10 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc35e0, rank=0) at core/sr_module.c:918 #11 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc3de8, rank=0) at core/sr_module.c:918 #12 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc4550, rank=0) at core/sr_module.c:918 #13 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc51d8, rank=0) at core/sr_module.c:918 #14 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc5c70, rank=0) at core/sr_module.c:918 #15 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6138, rank=0) at core/sr_module.c:918 #16 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6670, rank=0) at core/sr_module.c:918 #17 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6e28, rank=0) at core/sr_module.c:918 #18 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc7268, rank=0) at core/sr_module.c:918 #19 0x000000000053dbf0 in init_child (rank=0) at core/sr_module.c:947 #20 0x000000000042357c in main_loop () at main.c:1700 #21 0x0000000000429f71 in main (argc=15, argv=0x7ffd8a1cf208) at main.c:2639 (gdb) #0 0x00007f684d974a3b in rpc_struct_add (s=0x14bf340, fmt=0x7f684ca4f4ae "[{") at binrpc_run.c:1101 #1 0x00007f684ca3a02d in ws_rpc_dump (rpc=0x7f684dba2740 <binrpc_callbacks>, ctx=0x7ffd8a1ce500) at ws_conn.c:625 #2 0x00007f684d97172d in process_rpc_req (buf=0x14a72a4 "\241\003\nY\302\tq\221\bws.dump", size=17, bytes_needed=0x7ffd8a1ce608, sh=0x7ffd8a1ce630, saved_state=0x14b72a8) at binrpc_run.c:675 #3 0x00007f684d9677bd in handle_stream_read (s_c=0x14a7270, idx=-1) at io_listener.c:511 #4 0x00007f684d969212 in handle_io (fm=0x7f684fd6b1d8, events=1, idx=-1) at io_listener.c:706 #5 0x00007f684d960abf in io_wait_loop_epoll (h=0x7f684db92400 <io_h>, t=10, repeat=0) at ../../core/io_wait.h:1065 #6 0x00007f684d96495c in io_listen_loop (fd_no=1, cs_lst=0x14a1900) at io_listener.c:281 #7 0x00007f684d98048d in mod_child (rank=0) at ctl.c:335 #8 0x000000000053d8c2 in init_mod_child (m=0x7f684fcc2908, rank=0) at core/sr_module.c:921 #9 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc2f70, rank=0) at core/sr_module.c:918 #10 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc35e0, rank=0) at core/sr_module.c:918 #11 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc3de8, rank=0) at core/sr_module.c:918 #12 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc4550, rank=0) at core/sr_module.c:918 #13 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc51d8, rank=0) at core/sr_module.c:918 #14 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc5c70, rank=0) at core/sr_module.c:918 #15 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6138, rank=0) at core/sr_module.c:918 #16 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6670, rank=0) at core/sr_module.c:918 #17 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6e28, rank=0) at core/sr_module.c:918 #18 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc7268, rank=0) at core/sr_module.c:918 #19 0x000000000053dbf0 in init_child (rank=0) at core/sr_module.c:947 #20 0x000000000042357c in main_loop () at main.c:1700 #21 0x0000000000429f71 in main (argc=15, argv=0x7ffd8a1cf208) at main.c:2639 (gdb) bt full #0 0x00007f684d974a3b in rpc_struct_add (s=0x14bf340, fmt=0x7f684ca4f4ae "[{") at binrpc_run.c:1101 ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7ffd8a1ce3d0, reg_save_area = 0x7ffd8a1ce2e0}} err = 0 avp = {name = {s = 0x7f684ca4f4a2 "connections", len = 11}, type = 3, u = {strval = {s = 0x0, len = 0}, fval = 0, intval = 0, end = 0}} rs = 0x14c1390 __FUNCTION__ = "rpc_struct_add" #1 0x00007f684ca3a02d in ws_rpc_dump (rpc=0x7f684dba2740 <binrpc_callbacks>, ctx=0x7ffd8a1ce500) at ws_conn.c:625 h = 6587863 connections = 0 truncated = 0 order = 0 found = 0 wsc = 0x7f684fd634f8 sorder = {s = 0x7f684d98a158 "if you get this string, you don'tcheck rpc_scan return code !!! (very bad)", len = 74} th = 0x14bf340 ih = 0xad9180 <rpc_hash_table> dh = 0x38a1ce4b0 __FUNCTION__ = "ws_rpc_dump" #2 0x00007f684d97172d in process_rpc_req (buf=0x14a72a4 "\241\003\nY\302\tq\221\bws.dump", size=17, bytes_needed=0x7ffd8a1ce608, sh=0x7ffd8a1ce630, saved_state=0x14b72a8) at binrpc_run.c:675 err = 0 val = {name = {s = 0x7ffd8a1ce5e0 "\360\346\034\212\375\177", len = 1301657797}, type = 1, u = {strval = {s = 0x14a72ad "ws.dump", len = 7}, fval = 1.06996027198961e-316, intval = 21656237, end = 21656237}} rpc_e = 0x7f684fd63528 f_ctx = {in = {ctx = {tlen = 10, cookie = 1505888625, type = 0, flags = 1, offset = 10, in_struct = 0, in_array = 0}, s = 0x14a72b5 "", end = 0x14a72b5 "", record_no = 0, in_struct = 0}, out = {pkt = {body = 0x14b7330 "\003\203", end = 0x14bf330 "", crt = 0x14b7332 ""}, structs = {next = 0x14bf340, prev = 0x14bf340}}, send_h = 0x7ffd8a1ce630, method = 0x14a72ad "ws.dump", gc = 0x0, replied = 0, err_code = 0, err_phrase = {s = 0x0, len = 0}} ctx = 0x7ffd8a1ce500 __FUNCTION__ = "process_rpc_req" #3 0x00007f684d9677bd in handle_stream_read (s_c=0x14a7270, idx=-1) at io_listener.c:511 bytes_free = 65535 bytes_read = 17 ---Type <return> to continue, or q <return> to quit--- bytes_needed = 21633280 bytes_processed = 0 r = 0x14a7290 sh = {fd = 8, type = 0, from = {sa_in = {s = {sa_family = 29296, sa_data = "J\001\000\000\000\000\001\000\311Oh\177\000"}, sin = {sin_family = 29296, sin_port = 330, sin_addr = {s_addr = 0}, sin_zero = "\001\000\311Oh\177\000"}, sin6 = {sin6_family = 29296, sin6_port = 330, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\001\000\311Oh\177\000\000\370\224\242\000\000\000\000", __u6_addr16 = {1, 20425, 32616, 0, 38136, 162, 0, 0}, __u6_addr32 = {1338572801, 32616, 10654968, 0}}}, sin6_scope_id = 1}}, sa_un = {sun_family = 29296, sun_path = "J\001\000\000\000\000\001\000\311Oh\177\000\000\370\224\242\000\000\000\000\000\001\000\000\000\000\000\000\000\330H\036Gh\177", '\000' <repeats 18 times>, "\260\346\034\212\001\000\000\000\330H\036Gh\177\000\000\260\346\034\212\375\177\000\000\255k\225Mh\177\000\000\300\346\034\212\375\177\000\000\330H\036Gh\177\000\000\002\375\255?\000"}}, from_len = 0} __FUNCTION__ = "handle_stream_read" #4 0x00007f684d969212 in handle_io (fm=0x7f684fd6b1d8, events=1, idx=-1) at io_listener.c:706 ret = 1 __FUNCTION__ = "handle_io" #5 0x00007f684d960abf in io_wait_loop_epoll (h=0x7f684db92400 <io_h>, t=10, repeat=0) at ../../core/io_wait.h:1065 n = 1 r = 0 fm = 0x7f684fd6b1d8 revents = 1 __FUNCTION__ = "io_wait_loop_epoll" #6 0x00007f684d96495c in io_listen_loop (fd_no=1, cs_lst=0x14a1900) at io_listener.c:281 max_fd_no = 195 poll_err = 0x0 poll_method = 2 cs = 0x0 type = 2 __FUNCTION__ = "io_listen_loop" #7 0x00007f684d98048d in mod_child (rank=0) at ctl.c:335 pid = 0 cs = 0x4178f0 <_start> rpc_handler = 1 __FUNCTION__ = "mod_child" ---Type <return> to continue, or q <return> to quit--- #8 0x000000000053d8c2 in init_mod_child (m=0x7f684fcc2908, rank=0) at core/sr_module.c:921 __FUNCTION__ = "init_mod_child" #9 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc2f70, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #10 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc35e0, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #11 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc3de8, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #12 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc4550, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #13 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc51d8, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #14 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc5c70, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #15 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6138, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #16 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6670, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #17 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc6e28, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #18 0x000000000053d5e0 in init_mod_child (m=0x7f684fcc7268, rank=0) at core/sr_module.c:918 __FUNCTION__ = "init_mod_child" #19 0x000000000053dbf0 in init_child (rank=0) at core/sr_module.c:947 No locals. #20 0x000000000042357c in main_loop () at main.c:1700 i = 4 pid = 9077 si = 0x0 si_desc = "udp receiver child=3 sock=10.3.66.231:5060\000\363o\000\000\000\361\240u\000\000\000\000\000\000\366\374\363\340\341\232\n\004\000\000\000\000\000\000\000\002\375\255?\000\000\000\000\360xA\000\000\000\000\000\000\362\034\212\375\177", '\000' <repeats 18 times>, "\060\357\034\212\375\177\000\000\035\322^\000\000\000\000" nrprocs = 4 ---Type <return> to continue, or q <return> to quit--- woneinit = 1 __FUNCTION__ = "main_loop" #21 0x0000000000429f71 in main (argc=15, argv=0x7ffd8a1cf208) at main.c:2639 cfg_stream = 0x140d010 c = -1 r = 0 tmp = 0x7ffd8a1d0ef2 "" tmp_len = 1346830760 port = 32616 proto = -1977814848 options = 0x737490 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 27692109 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0x1 <error: Cannot access memory at address 0x1> st = {st_dev = 15, st_ino = 14760, st_nlink = 2, st_mode = 16832, st_uid = 109, st_gid = 114, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1488887721, tv_nsec = 984216540}, st_mtim = {tv_sec = 1488888110, tv_nsec = 550708698}, st_ctim = { tv_sec = 1488888110, tv_nsec = 550708698}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" (gdb) (gdb) info locals ap = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7ffd8a1ce3d0, reg_save_area = 0x7ffd8a1ce2e0}} err = 0 avp = {name = {s = 0x7f684ca4f4a2 "connections", len = 11}, type = 3, u = {strval = {s = 0x0, len = 0}, fval = 0, intval = 0, end = 0}} rs = 0x14c1390 __FUNCTION__ = "rpc_struct_add" (gdb) list 1096 in binrpc_run.c (gdb) quit ```

#### Log Messages

``` Mar 7 13:15:45 webrtc kernel: [9074388.189612] kamailio[9526]: segfault at 7fab895ad49d ip 00007fab8a4d2a3b sp 00007fff0712d3d0 error 7 in ctl.so[7fab8a4a1000+4e000] Mar 7 13:15:45 webrtc /usr/sbin/kamailio[9518]: ALERT: <core> [main.c:744]: handle_sigs(): child process 9526 exited by a signal 11 Mar 7 13:15:45 webrtc /usr/sbin/kamailio[9518]: ALERT: <core> [main.c:747]: handle_sigs(): core was generated Mar 7 13:15:45 webrtc systemd[1]: kamailio.service: main process exited, code=exited, status=1/FAILURE Mar 7 13:15:45 webrtc systemd[1]: Unit kamailio.service entered failed state. ```

### Additional Information

``` [0][root@webrtc:~]# dpkg -l|grep kamailio ii kamailio 5.0.0+jessie amd64 very fast, dynamic and configurable SIP server ii kamailio-dbg:amd64 5.0.0+jessie amd64 very fast and configurable SIP server [debug symbols] ii kamailio-extra-modules:amd64 5.0.0+jessie amd64 Extra modules for the Kamailio SIP Server ii kamailio-outbound-modules:amd64 5.0.0+jessie amd64 SIP Outbound module for the Kamailio SIP server ii kamailio-utils-modules:amd64 5.0.0+jessie amd64 Utility functions for the Kamailio SIP server ii kamailio-websocket-modules:amd64 5.0.0+jessie amd64 WebSocket module for the Kamailio SIP server [0][root@webrtc:~]#

[130][root@webrtc:~]# kamailio -v version: kamailio 5.0.0 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 4.9.2 [0][root@webrtc:~]# ```

* **Operating System**:

Freshly updated Debian Jessie

``` [0][root@webrtc:~]# cat /etc/issue Debian GNU/Linux 8 \n \l

[0][root@webrtc:~]# uname -a Linux webrtc 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux [0][root@webrtc:~]# ```