Hello,

before you investigate more the cause of this segfault, I strongly suggest that you update to a recent 5.4.x or 5.5.x version. There have been extensions and bugfixes all over the place, also for the DMQ module and for better container support.

[..]

version: kamailio 5.2.5 (x86_64/linux)

[..]

This version reached its end of life.

If you can reproduce this with a newer version, please open an issue on our github tracker about it.

Cheers,

Henning

--

Henning Westerholt – https://skalatan.de/blog/

Kamailio services – https://gilawa.com

From: sr-dev <sr-dev-bounces@lists.kamailio.org> On Behalf Of Sergio Carvalho
Sent: Friday, September 10, 2021 5:41 PM
To: sr-dev@lists.kamailio.org
Subject: [sr-dev] SIGSEGV on DMQ

Hi everyone,

I'm trying to use the DMQ module to share the user location info across our Kamailio's cluster.  But when I start the service under docker I get a segmentation fault violation.  After analysing the core dump with the LLDB I get (if you want I can share the core dump with you using Google Drive):

(lldb) bt
* thread #1, name = 'kamailio', stop reason = signal SIGSEGV
  * frame #0: 0x000000000071c63c kamailio`get_hdr_field(buf="", end="", hdr=0x0000000000000000) at msg_parser.c:144:5
    frame #1: 0x0000000000575b02 kamailio`resolve_select(s=0x00007ffd470c6da0) at select.c:384:3
    frame #2: 0x00007fbbd7637fb9
    frame #3: 0x00000000005ad785 kamailio`clen_builder(msg=0x00007fbbd7f6e008, clen_len=0x0000000000000000, diff=0, body_only=0) at msg_translator.c:451:3
    frame #4: 0x0000000000439aa7 kamailio`_dns_hash_find(name=0x00007fbbd7f6e498, type=32765, h=0x0000000000000000, err=0x0000000000000000) at dns_cache.c:551:4
    frame #5: 0x0000000000440cb2 kamailio`dns_cache_do_request(name=0x0000000000000000, type=0) at dns_cache.c:1767:3
    frame #6: 0x000000000043f8f6 kamailio`dns_cache_mk_rd_entry(name=0x0000000000000000, type=0, rd_lst=0x0000000000000000) at dns_cache.c:1226:7
    frame #7: 0x000000000046501e kamailio`print_action(t=0x0000000000000000) at route_struct.c:467:4
    frame #8: 0x0000000000471f2f kamailio`xavp_print_list_content(head=0x00007ffd470c31a0, level=32699) at xavp.c:584:2
    frame #9: 0x0000000000462047 kamailio`print_action(t=0x0000000000000000) at route_struct.c:392:4
    frame #10: 0x0000000000471f2f kamailio`xavp_print_list_content(head=0x00007ffd470c31a0, level=32699) at xavp.c:584:2
    frame #11: 0x00000000004654dc kamailio`print_action(t=0x0000000000000000) at route_struct.c:470:4
    frame #12: 0x0000000000471f2f kamailio`xavp_print_list_content(head=0x00007ffd470c31a0, level=32699) at xavp.c:584:2
    frame #13: 0x0000000000465532 kamailio`print_action(t=0x0000000003863810) at route_struct.c:470:4
    frame #14: 0x0000000000471f2f kamailio`xavp_print_list_content(head=0x00007ffd470c31a0, level=32699) at xavp.c:584:2
    frame #15: 0x0000000000462047 kamailio`print_action(t=0x00000000006da9d3) at route_struct.c:392:4
    frame #16: 0x0000000000471f2f kamailio`xavp_print_list_content(head=0x00007ffd470c31a0, level=32699) at xavp.c:584:2
    frame #17: 0x0000000000472680 kamailio`xavp_get_list_key_names(xavp=0x00007ffd470c6da0) at xavp.c:622:3
    frame #18: 0x00007fbbb357da54
    frame #19: 0x00007fbbb3580370
    frame #20: 0x00007fbbb358317b
    frame #21: 0x00007fbbb3583147
    frame #22: 0x00007fbbb3585e8d
    frame #23: 0x00007fbbaec3ef23
    frame #24: 0x00007fbbaec3d5da
    frame #25: 0x00007fbbaec4e158
    frame #26: 0x00007fbbaec4aadd
    frame #27: 0x00007fbbaec53f05
    frame #28: 0x0000000000541563 kamailio`handle_tcp_child(tcp_c=0x00000004b4481868, fd_i=136) at tcp_main.c:3477:1
    frame #29: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x0000000000000000, fd_i=32699) at tcp_main.c:3470:5
    frame #30: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00000000470c6da0, fd_i=0) at tcp_main.c:3470:5
    frame #31: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00000000b4481868, fd_i=1863578990) at tcp_main.c:3470:5
    frame #32: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00000000b4481868, fd_i=1863578990) at tcp_main.c:3470:5
    frame #33: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00007fbbb4481868, fd_i=1863578990) at tcp_main.c:3470:5
    frame #34: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00007ffd470c64d0, fd_i=32699) at tcp_main.c:3470:5
    frame #35: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00007ffd470c658c, fd_i=32765) at tcp_main.c:3470:5
    frame #36: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00000000b4481868, fd_i=1863578990) at tcp_main.c:3470:5
    frame #37: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00007fbbb4481868, fd_i=1863578990) at tcp_main.c:3470:5
    frame #38: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00007fbbd8059e38, fd_i=32699) at tcp_main.c:3470:5
    frame #39: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00007ffd470c6770, fd_i=32765) at tcp_main.c:3470:5
    frame #40: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x0000000000000000, fd_i=32765) at tcp_main.c:3470:5
    frame #41: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00007fbbb5179a80, fd_i=0) at tcp_main.c:3470:5
    frame #42: 0x0000000000541223 kamailio`handle_tcp_child(tcp_c=0x00000065dbb52b93, fd_i=32765) at tcp_main.c:3470:5
    frame #43: 0x00000000005418fa kamailio`handle_ser_child(p=0x00000065dbb52b93, fd_i=32765) at tcp_main.c:3513:3
    frame #44: 0x0000000000425d6e kamailio`main_loop at main.c:1620:4
    frame #45: 0x000000000042c6af kamailio`main(argc=13, argv=0x00007ffd470c6da8) at main.c:2568:3

Using GDB I get the following info:

#22 0x00007efcca978e8d in request () from /usr/lib64/kamailio/modules/tm.so
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#23 0x00007efcc6031f23 in dmq_send_message (peer=0x7efccc5545c0, body=0x7efcef106ef8, node=0x7efccc5ded10, resp_cback=0x7efcc625d370 <notification_callback>, max_forwards=2,
    content_type=0x7efcc625d360 <notification_content_type>) at dmq_funcs.c:251
        uac_r = {method = 0x7efcc625d3b0 <dmq_request_method>, headers = 0x7ffe13cb34b0, body = 0x7efcef106ef8, ssock = 0x7efcc625da20 <dmq_server_socket>, dialog = 0x7efccc5e0918,
          cb_flags = 1024, cb = 0x7efcc602eb7b <dmq_tm_callback>, cbp = 0x7efccc5def58, callid = 0x0}
        str_hdr = {s = 0x7efcef1004e8 "Max-Forwards: 2\r\nContent-Type: text/plain\r\n", len = 43}
        from = {s = 0x7efcef107248 "sip:notification_peer@10.12.4.238:5090", len = 38}
        to = {s = 0x7efcef11d100 "sip:notification_peer@10.12.4.170:5090", len = 38}
        cb_param = 0x7efccc5def58
        result = 0
        len = 43
        __FUNCTION__ = "dmq_send_message"
#24 0x00007efcc60305da in bcast_dmq_message1 (peer=0x7efccc5545c0, body=0x7efcef106ef8, except=0x0, resp_cback=0x7efcc625d370 <notification_callback>, max_forwards=2,
    content_type=0x7efcc625d360 <notification_content_type>, incl_inactive=1) at dmq_funcs.c:170
        node = 0x7efccc5ded10
        __FUNCTION__ = "bcast_dmq_message1"
#25 0x00007efcc6041158 in request_nodelist (node=0x7efccc5deac8, forward=2) at notification_peer.c:573
        body = 0x7efcef106ef8
        ret = 0
        __FUNCTION__ = "request_nodelist"
#26 0x00007efcc603dadd in add_server_and_notify (paddr=0x7efcc625da30 <dmq_notification_address>) at notification_peer.c:340
        puri_data = "sip:10.12.5.29:5090\000\376\177\000\000\250I+\002\000\000\000\000\230L+\002\000\000\000\000\260;\313\023\376\177\000\000\250c+\002\000\000\000\000`;\313\023\376\177", '\000' <repeats 17 times>, "\200`?\313\023\376\177\000\000\325\222\255\307\374~\000\000\024=\313\023\376\177\000\000\030=\313\023\376\177\000\000\034=\313\023\376\177\000\000\260:\313\023\376\177\000\000*\000\000\000\000\000\000\000]s\f\000\000\000\000\000\001\000\000\000\000\000\000\000\244\201\000\000\204\352", '\000' <repeats 19 times>, "\340\f\000\000\000\000\000\000\020\000\000\000\000\000\000p\006\000\000\000\000\000\000"...
        puri_list = {0x7ffe13cb3950 "sip:10.12.5.29:5090", 0x7ffe13cb3a50 "sip:10.12.4.170:5090", 0x7ffe13cb3b50 "x\321-\002", 0x7ffe13cb3c50 "", 0x7ffe13cb3d50 " @\313\023\376\177",
          0x7ffe13cb3e50 "", 0x7ffe13cb3f50 "", 0x7ffe13cb4050 "", 0x7ffe13cb4150 "2\247h", 0x7ffe13cb4250 "pB\313\023\376\177", 0x7ffe13cb4350 "\200S\237\312\374~",
          0x7ffe13cb4450 "8\341\006\357\374~", 0x7ffe13cb4550 "\320E\313\023\376\177", 0x7ffe13cb4650 "\210*,\002", 0x7ffe13cb4750 "\330N\313\023\376\177", 0x7ffe13cb4850 "",
          0x7ffe13cb4950 "", 0x7ffe13cb4a50 "\200J\313\023\376\177", 0x7ffe13cb4b50 "\035\362\067\037\003", 0x7ffe13cb4c50 "\320L\313\023\376\177", 0x7ffe13cb4d50 "\177", 0x7ffe13cb4e50 "n",
          0x7ffe13cb4f50 "1\325\\\002", 0x7ffe13cb5050 "\260P\313\023\376\177", 0x7ffe13cb5150 "", 0x7ffe13cb5250 "\320\343\270\307\374~", 0x7ffe13cb5350 "", 0x7ffe13cb5450 "xV,\002",
          0x7ffe13cb5550 "2\247h", 0x7ffe13cb5650 "\320\235\005\306\374~"}
        pfirst = 0x7efccc5deac8
        pnode = 0x7efccc5ded10
        host_cnt = 2
        index = 2
        puri = {{user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x7efceefdde8c "kamailio-kazoo.svc.cpaas.eu-west-1.aws.prv.talkdeskstg.com:5090", len = 58}, port = {
              s = 0x7efceefddec7 "5090", len = 4}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 5090, proto = 0, type = SIP_URI_T,
            flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {
              s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0},
            maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}}}
        pstr = {{s = 0x7ffe13cb3a50 "sip:10.12.4.170:5090", len = 20}}
        __FUNCTION__ = "add_server_and_notify"
#27 0x00007efcc6046f05 in child_init (rank=0) at dmq.c:302
        i = 4
        newpid = 160
---Type <return> to continue, or q <return> to quit---
        __FUNCTION__ = "child_init"
#28 0x0000000000541563 in init_mod_child ()
No symbol table info available.
#29 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#30 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#31 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#32 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#33 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#34 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#35 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#36 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#37 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#38 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#39 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#40 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#41 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#42 0x0000000000541223 in init_mod_child ()
No symbol table info available.
#43 0x00000000005418fa in init_child ()
No symbol table info available.
#44 0x0000000000425d6e in main_loop ()
No symbol table info available.
#45 0x000000000042c6af in main ()
No symbol table info available.

DMQ module parameters are:

#!ifdef DMQ_ROLE

loadmodule "dmq.so"

loadmodule "dmq_usrloc.so"

# ---- dmq params ----

modparam("dmq", "server_address", "sip:DMQ_SERVER_ADDRESS")

modparam("dmq", "notification_address", "sip:DMQ_NOTIFY_ADDRESS")

modparam("dmq", "ping_interval", 5)

modparam("dmq", "multi_notify", 1)

modparam("dmq", "num_workers", 4)

modparam("dmq", "worker_usleep", 0)

modparam("dmq_usrloc", "enable", 1)

#!endif

Kamailio version and flags are:

version: kamailio 5.2.5 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144 MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 01:14:59 Nov 19 2019 with gcc 7.3.1

Do you have any idea what might be causing this SEGV?

Best regards,

Sergio Carvalho