[SR-Users] Diameter (CDP) crashes on startup. ERROR: cdp [peerstatemachine.c:634]: I_Snd_CER(): I_Snd_CER(): Error on finding local host address > Socket operation on non-socket

Hello! Please help me to find the issue with CDP module as it blocks Kamailio to start. [root@10-10-10-10 kamailio]# cat diameter.xml <?xml version="1.0" encoding="UTF-8"?<DiameterPeer FQDN="10-10-10-10.ams.proxy.dev" Realm="proxy.dev" Product_Name="Diameter Credit Control" Vendor_Id="10415" AcceptUnknownPeers="1" DropUnknownOnDisconnect="1" Tc="30" Workers="4" QueueLength="32" <Peer FQDN="dev-diameter-1.ams.proxy.dev" Realm="proxy.dev" port="3868"/ <Acceptor port="3868" bind="10.10.10.10"/ <Auth id="16777216" vendor="10415"/ <Acct id="16777216" vendor="0" / <Auth id="16777216" vendor="10415"/ <Auth id="16777216" vendor="0" / <SupportedVendor vendor="10415"/ <Realm name="proxy.dev" <Route FQDN="dev-diameter-1.ams.proxy.dev" metric="1"/ </Realm <DefaultRoute FQDN="dev-diameter-1.ams.proxy.dev" metric="10"/</DiameterPeer 38(7825) INFO: cdp [worker.c:332]: worker_process(): [1] Worker process started... 37(7824) INFO: cdp [worker.c:332]: worker_process(): [0] Worker process started... 39(7826) INFO: cdp [worker.c:332]: worker_process(): [2] Worker process started... 40(7827) INFO: cdp [worker.c:332]: worker_process(): [3] Worker process started... 41(7828) INFO: cdp [receiver.c:450]: receiver_process(): receiver_process(): [] Receiver process doing init on new process... 41(7828) INFO: cdp [receiver.c:455]: receiver_process(): receiver_process(): [] Receiver process starting up... 42(7829) INFO: cdp [receiver.c:450]: receiver_process(): receiver_process(): [dev-diameter-1.ams.proxy.dev] Receiver process doing init on new process... 42(7829) INFO: cdp [receiver.c:184]: add_serviced_peer(): add_serviced_peer(): Adding serviced_peer_t to receiver for peer [dev-diameter-1.ams.proxy.dev] 42(7829) INFO: cdp [receiver.c:455]: receiver_process(): receiver_process(): [dev-diameter-1.ams.proxy.dev] Receiver process starting up... 43(7830) INFO: cdp [acceptor.c:81]: acceptor_process(): Acceptor process starting up... 0(7787) INFO: cdp [cdp_mod.c:244]: cdp_child_init(): ... CDiameterPeer child started 43(7830) WARNING: cdp [tcp_accept.c:121]: create_socket(): create_socket(): Trying to open/bind/listen on 10.10.10.10 port 3868 43(7830) WARNING: cdp [tcp_accept.c:146]: create_socket(): create_socket(): Successful socket open/bind/listen on 10.10.10.10 port 3868 43(7830) INFO: cdp [acceptor.c:95]: acceptor_process(): Acceptor opened sockets. Entering accept loop ... 44(7831) INFO: cdp [timer.c:205]: timer_process(): Timer process starting up... [root@10-10-10-10 kamailio]# 44(7831) INFO: cdp [peerstatemachine.c:525]: I_Snd_Conn_Req(): I_Snd_Conn_Req(): Peer dev-diameter-1.ams.proxy.dev 44(7831) INFO: cdp [receiver.c:869]: peer_connect(): peer_connect(): Trying to connect to 10.10.10.20 port 3868 44(7831) INFO: cdp [receiver.c:937]: peer_connect(): peer_connect(): Peer dev-diameter-1.ams.proxy.dev:3868 connected 42(7829) ERROR: cdp [peerstatemachine.c:634]: I_Snd_CER(): I_Snd_CER(): Error on finding local host address > Socket operation on non-socket 42(7829) CRITICAL: <core> [core/mem/q_malloc.c:149]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(0, abcdefed)[0x7f3ecb7ef018:0x7f3ecb7ef050]! Memory allocator was called from cdp: diameter_avp.c:365. Fragment marked by cdp: diameter_avp.c:142. Exec from core/mem/q_malloc.c:504. 45(7832) CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 52 0(7787) ALERT: <core> [main.c:739]: handle_sigs(): child process 7829 exited by a signal 6 0(7787) ALERT: <core> [main.c:742]: handle_sigs(): core was generated 0(7787) INFO: <core> [main.c:764]: handle_sigs(): terminating due to SIGCHLD DNS record dev-diameter-1.ams.proxy.dev is pointed to 10.10.10.20 NU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/usr/sbin/kamailio.debug...done. done. [New LWP 7829] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `kamailio -f kamailio.cfg -DDDD -E'. Program terminated with signal 6, Aborted. #0 0x00007f3ed7dc0207 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-34.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libgcc-4.8.5-36.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libstdc++-4.8.5-36.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 mariadb-libs-5.5.60-1.el7_5.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 pcre-8.32-17.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64 (gdb) bt #0 0x00007f3ed7dc0207 in raise () from /lib64/libc.so.6 #1 0x00007f3ed7dc18f8 in abort () from /lib64/libc.so.6 #2 0x000000000067d5f7 in qm_debug_check_frag (qm=0x7f3ecb4d0000, f=0x7f3ecb7ef018, file=0x7f3ed068d932 "cdp: diameter_avp.c", line=365, efile=0x7df137 "core/mem/q_malloc.c", eline=504) at core/mem/q_malloc.c:151 #3 0x000000000068077a in qm_free (qmp=0x7f3ecb4d0000, p=0x7f3ecb7ef050, file=0x7f3ed068d932 "cdp: diameter_avp.c", func=0x7f3ed068f418 <__FUNCTION__.7016> "AAAFreeAVP", line=365, mname=0x7f3ed068d760 "cdp") at core/mem/q_malloc.c:504 #4 0x000000000068a2d6 in qm_shm_free (qmp=0x7f3ecb4d0000, p=0x7f3ecb7ef050, file=0x7f3ed068d932 "cdp: diameter_avp.c", func=0x7f3ed068f418 <__FUNCTION__.7016> "AAAFreeAVP", line=365, mname=0x7f3ed068d760 "cdp") at core/mem/q_malloc.c:1268 #5 0x00007f3ed06691c8 in AAAFreeAVP (avp=0x7ffd86bdd300) at diameter_avp.c:365 #6 0x00007f3ed0636d66 in AAAFreeAVPList (avpList=0x7f3ecb7eea40) at diameter_msg.c:396 #7 0x00007f3ed0637123 in AAAFreeMessage (msg=0x7ffd86bdd3b0) at diameter_msg.c:416 #8 0x00007f3ed06088ec in Process_CEA (p=0x7f3ecb7a00d8, cea=0x7f3ecb7ee9f0) at peerstatemachine.c:804 #9 0x00007f3ed0601ab1 in sm_process (p=0x7f3ecb7a00d8, event=I_Rcv_CEA, msg=0x7f3ecb7ee9f0, peer_locked=0, sock=8) at peerstatemachine.c:166 #10 0x00007f3ed0659825 in receive_message (msg=0x7f3ecb7ee9f0, sp=0x7f3ed76feec8) at receiver.c:1128 #11 0x00007f3ed064f295 in do_receive (sp=0x7f3ed76feec8) at receiver.c:593 #12 0x00007f3ed0653068 in receive_loop (original_peer=0x7f3ecb7a00d8) at receiver.c:800 #13 0x00007f3ed064c7c6 in receiver_process (p=0x7f3ecb7a00d8) at receiver.c:459 #14 0x00007f3ed05fd7de in diameter_peer_start (blocking=0) at diameter_peer.c:289 #15 0x00007f3ed05ef9b1 in cdp_child_init (rank=0) at cdp_mod.c:243 #16 0x0000000000544e31 in init_mod_child (m=0x7f3ed7630170, rank=0) at core/sr_module.c:943 #17 0x0000000000544ad3 in init_mod_child (m=0x7f3ed7630e90, rank=0) at core/sr_module.c:939 #18 0x0000000000544ad3 in init_mod_child (m=0x7f3ed7631290, rank=0) at core/sr_module.c:939 #19 0x0000000000544ad3 in init_mod_child (m=0x7f3ed7631700, rank=0) at core/sr_module.c:939 #20 0x0000000000545205 in init_child (rank=0) at core/sr_module.c:970 #21 0x0000000000424f85 in main_loop () at main.c:1701 #22 0x000000000042b87e in main (argc=5, argv=0x7ffd86bde218) at main.c:2638 (gdb) bt full #0 0x00007f3ed7dc0207 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f3ed7dc18f8 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x000000000067d5f7 in qm_debug_check_frag (qm=0x7f3ecb4d0000, f=0x7f3ecb7ef018, file=0x7f3ed068d932 "cdp: diameter_avp.c", line=365, efile=0x7df137 "core/mem/q_malloc.c", eline=504) at core/mem/q_malloc.c:151 __FUNCTION__ = "qm_debug_check_frag" #3 0x000000000068077a in qm_free (qmp=0x7f3ecb4d0000, p=0x7f3ecb7ef050, file=0x7f3ed068d932 "cdp: diameter_avp.c", func=0x7f3ed068f418 <__FUNCTION__.7016> "AAAFreeAVP", line=365, mname=0x7f3ed068d760 "cdp") at core/mem/q_malloc.c:504 qm = 0x7f3ecb4d0000 f = 0x7f3ecb7ef018 size = 140726864040496 next = 0x7df14b prev = 0x7f3ed067f329 __FUNCTION__ = "qm_free" #4 0x000000000068a2d6 in qm_shm_free (qmp=0x7f3ecb4d0000, p=0x7f3ecb7ef050, file=0x7f3ed068d932 "cdp: diameter_avp.c", func=0x7f3ed068f418 <__FUNCTION__.7016> "AAAFreeAVP", line=365, mname=0x7f3ed068d760 "cdp") at core/mem/q_malloc.c:1268 No locals. #5 0x00007f3ed06691c8 in AAAFreeAVP (avp=0x7ffd86bdd300) at diameter_avp.c:365 __FUNCTION__ = "AAAFreeAVP" #6 0x00007f3ed0636d66 in AAAFreeAVPList (avpList=0x7f3ecb7eea40) at diameter_msg.c:396 avp_t = 0x7f3ecb7ef050 avp = 0x7f3ecb7eef40 #7 0x00007f3ed0637123 in AAAFreeMessage (msg=0x7ffd86bdd3b0) at diameter_msg.c:416 __FUNCTION__ = "AAAFreeMessage" #8 0x00007f3ed06088ec in Process_CEA (p=0x7f3ecb7a00d8, cea=0x7f3ecb7ee9f0) at peerstatemachine.c:804 avp = 0x7f3ecb7ef440 #9 0x00007f3ed0601ab1 in sm_process (p=0x7f3ecb7a00d8, event=I_Rcv_CEA, msg=0x7f3ecb7ee9f0, peer_locked=0, sock=8) at peerstatemachine.c:166 result_code = -798486160 next_event = 32574 msg_received = 0 __FUNCTION__ = "sm_process" #10 0x00007f3ed0659825 in receive_message (msg=0x7f3ecb7ee9f0, sp=0x7f3ed76feec8) at receiver.c:1128 avp1 = 0xd0684fc0 avp2 = 0x7f3ecb7eed10 __FUNCTION__ = "receive_message" #11 0x00007f3ed064f295 in do_receive (sp=0x7f3ed76feec8) at receiver.c:593 cnt = 184 n = 184 version = 1 dst = 0x7f3ecb7ef4f4 "" dmsg = 0x7f3ecb7ee9f0 __FUNCTION__ = "do_receive" #12 0x00007f3ed0653068 in receive_loop (original_peer=0x7f3ecb7a00d8) at receiver.c:800 rfds = {__fds_bits = {256, 0 <repeats 15 times>}} efds = {__fds_bits = {0 <repeats 16 times>}} tv = {tv_sec = 0, tv_usec = 999998} n = 1 max = 49 cnt = 1 msg = 0x0 sp = 0x7f3ed76feec8 sp2 = 0x7f3ed76feec8 p = 0x7f3ecb7a00d8 fd = 8 fd_exchange_pipe_local = 49 __FUNCTION__ = "receive_loop" #13 0x00007f3ed064c7c6 in receiver_process (p=0x7f3ecb7a00d8) at receiver.c:459 ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "receiver_process" #14 0x00007f3ed05fd7de in diameter_peer_start (blocking=0) at diameter_peer.c:289 pid = 0 k = -1 p = 0x7f3ecb7a00d8 __FUNCTION__ = "diameter_peer_start" #15 0x00007f3ed05ef9b1 in cdp_child_init (rank=0) at cdp_mod.c:243 __FUNCTION__ = "cdp_child_init" #16 0x0000000000544e31 in init_mod_child (m=0x7f3ed7630170, rank=0) at core/sr_module.c:943 __FUNCTION__ = "init_mod_child" #17 0x0000000000544ad3 in init_mod_child (m=0x7f3ed7630e90, rank=0) at core/sr_module.c:939 __FUNCTION__ = "init_mod_child" #18 0x0000000000544ad3 in init_mod_child (m=0x7f3ed7631290, rank=0) at core/sr_module.c:939 __FUNCTION__ = "init_mod_child" #19 0x0000000000544ad3 in init_mod_child (m=0x7f3ed7631700, rank=0) at core/sr_module.c:939 __FUNCTION__ = "init_mod_child" #20 0x0000000000545205 in init_child (rank=0) at core/sr_module.c:970 No locals. #21 0x0000000000424f85 in main_loop () at main.c:1701 i = 32 pid = 7821 si = 0x0 si_desc = "udp receiver child=31 sock=10.10.10.10:5060\000\177\000\000`\336\275\206\375\177\000\000A\307g\000\000\000\000\000`\202A\000\000\000\000\000ȖP\313>\177", '\000' <repeats 14 times>, "\001\000\000\000\260\336\275\206\375\177\000\000\234\240h\000\000\000\000\000 \243w\000\000\000\000\000H\361m\327>\177\000" nrprocs = 32 woneinit = 1 __FUNCTION__ = "main_loop" #22 0x000000000042b87e in main (argc=5, argv=0x7ffd86bde218) at main.c:2638 cfg_stream = 0x2b13020 c = -1 r = 0 tmp = 0x0 tmp_len = 0 port = 0 proto = 0 options = 0x758490 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 4201223303 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 4 n_lst = 0x0 p = 0x0 st = {st_dev = 19, st_ino = 82198, st_nlink = 2, st_mode = 16832, st_uid = 995, st_gid = 2, __pad0 = 0, st_rdev = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1547739314, tv_nsec = 582000000}, st_mtim = {tv_sec = 1547743303, tv_nsec = 194148453}, st_ctim = {tv_sec = 1547743303, tv_nsec = 194148453}, __unused = {0, 0, 0}} __FUNCTION__ = "main" (gdb) -- BR, Denys Pozniak