Greetings, all, I have a redirect server doing CNAM lookup (via http_async) and STIR/SHAKEN validation with secsipidx that is crashing every few hours. Kamailio 6.0.1 running in a Debian 12 container on an Amazon Linux 2023 host with ARM CPU.
root@0689d4fe5fed:/# kamailio -v version: kamailio 6.0.1 (aarch64/linux) fce50d flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: fce50d compiled on 15:50:40 Apr 1 2025 with gcc 12.2.0
Back trace is here, so I think it's a secsipid related problem? Slightly sanitized this output (Removed SIP message specifics in step #42). What should be my next course of action?
#0 __pthread_kill_implementation (threadid=281473196830752, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44 #1 0x0000ffff95c90ab4 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78 #2 0x0000ffff95c4a72c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x0000ffff95c3747c in __GI_abort () at ./stdlib/abort.c:79 #4 0x0000ffff95c84aac in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffff95d66d28 "%s\n") at ../sysdeps/posix/libc_fatal.c:156 #5 0x0000ffff95c9aebc in malloc_printerr (str=str@entry=0xffff95d62098 "double free or corruption (!prev)") at ./malloc/malloc.c:5660 #6 0x0000ffff95c9cee8 in _int_free (av=0xffff95db0af0 <main_arena>, p=p@entry=0xaaab05dd41b0, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4587 #7 0x0000ffff95c9f77c in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385 #8 0x0000ffff90eb0a0c in w_secsipid_get_url (msg=0xffff92f53030, purl=0xffff92e84e10 "\220M\350\222\377\377", povar=0xffff92e84fd0 "\004") at secsipid_mod.c:961 #9 0x0000aaaaea4fd710 in do_action (h=0xffffff40a610, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1133 #10 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40a610, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1620 #11 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff40bc50, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1683 #12 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff40bc50, msg=0xffff92f53030, i=0xffffff40ab48, rv=0xffff92e8fc78, cache=0x0) at core/rvalue.c:973 #13 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff40bc50, msg=0xffff92f53030, res=0xffffff40ab48, rve=0xffff92e8fc70) at core/rvalue.c:1852 #14 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff40bc50, msg=0xffff92f53030, res=0xffffff40b080, rve=0xffff92e903b0) at core/rvalue.c:1862 #15 0x0000aaaaea4fd090 in do_action (h=0xffffff40bc50, a=0xffff92e91810, msg=0xffff92f53030) at core/action.c:1099 #16 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40bc50, a=0xffff92e8d690, msg=0xffff92f53030) at core/action.c:1620 #17 0x0000aaaaea4f9400 in do_action (h=0xffffff40bc50, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:711 #18 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40bc50, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:1620 #19 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff40daf0, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:1683 #20 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff40daf0, msg=0xffff92f53030, i=0xffffff40c188, rv=0xffff92e82378, cache=0x0) at core/rvalue.c:973 #21 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff40daf0, msg=0xffff92f53030, res=0xffffff40c188, rve=0xffff92e82370) at core/rvalue.c:1852 #22 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff40daf0, msg=0xffff92f53030, res=0xffffff40c6c0, rve=0xffff92e82ab0) at core/rvalue.c:1862 #23 0x0000aaaaea4fd090 in do_action (h=0xffffff40daf0, a=0xffff92e83a90, msg=0xffff92f53030) at core/action.c:1099 #24 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e83a90, msg=0xffff92f53030) at core/action.c:1620 #25 0x0000aaaaea4fd5e8 in do_action (h=0xffffff40daf0, a=0xffff92e83bf0, msg=0xffff92f53030) at core/action.c:1119 #26 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e79450, msg=0xffff92f53030) at core/action.c:1620 #27 0x0000aaaaea4f9400 in do_action (h=0xffffff40daf0, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:711 #28 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:1620 #29 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff4101f8, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:1683 #30 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff4101f8, msg=0xffff92f53030, i=0xffffff40e028, rv=0xffff92e75878, cache=0x0) at core/rvalue.c:973 #31 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff4101f8, msg=0xffff92f53030, res=0xffffff40e028, rve=0xffff92e75870) at core/rvalue.c:1852 #32 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff4101f8, msg=0xffff92f53030, res=0xffffff40e560, rve=0xffff92e75fb0) at core/rvalue.c:1862 #33 0x0000aaaaea4fd090 in do_action (h=0xffffff4101f8, a=0xffff92e76e10, msg=0xffff92f53030) at core/action.c:1099 #34 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92e74500, msg=0xffff92f53030) at core/action.c:1620 #35 0x0000aaaaea5095fc in do_action (h=0xffffff4101f8, a=0xffff92e79110, msg=0xffff92f53030) at core/action.c:1401 #36 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92e71a40, msg=0xffff92f53030) at core/action.c:1620 #37 0x0000aaaaea4f9400 in do_action (h=0xffffff4101f8, a=0xffff92dd8520, msg=0xffff92f53030) at core/action.c:711 #38 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92dd7e30, msg=0xffff92f53030) at core/action.c:1620 #39 0x0000aaaaea4fd5a8 in do_action (h=0xffffff4101f8, a=0xffff92dd8680, msg=0xffff92f53030) at core/action.c:1115 #40 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92dce4c0, msg=0xffff92f53030) at core/action.c:1620 #41 0x0000aaaaea50dbf4 in run_top_route (a=0xffff92dce4c0, msg=0xffff92f53030, c=0x0) at core/action.c:1703 #42 0x0000aaaaea6bfff8 in receive_msg ( buf=0xaaaaeac284b8 <buf> "INVITE sip:2345678901@1.2.3.4 SIP/2.0\r\nRecord-Route: sip:1.2.3.5;lr=on;ftag=gK045643d5\r\nRecord-Route: sip:1.2.3.6;lr=on;ftag=gK045643d5\r\nVia: SIP/2.0/UDP 1.2.3.5:5060;branch"..., len=2492, rcv_info=0xffffff410a10) at core/receive.c:520 #43 0x0000aaaaea85abe0 in udp_rcv_loop () at core/udp_server.c:770 #44 0x0000aaaaea4e26a8 in main_loop () at main.c:1895 #45 0x0000aaaaea4f1e88 in main (argc=16, argv=0xffffff411178) at main.c:3406
Kaufman Senior Voice Engineer
E: bkaufman@bcmone.com
SIP.US Client Support: 800.566.9810 | SIPTRUNK Client Support: 800.250.6510 | Flowroute Client Support: 855.356.9768
[img]https://www.sip.us [img]https://www.siptrunk.com [img]https://www.flowroute.com
NOTE: This e-mail and any attachments are from BCM One, Inc. and are intended solely for the use and review of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail (and any attachments) from your computer and do not copy or disclose it to anyone else.
I'd go to frame 8 and print the value of _secsipid_get_url_val.s / _secsipid_get_url_val.len. It's obviously not NULL, but I wonder what it's pointing to.
On May 9, 2025, at 11:47 AM, Ben Kaufman via sr-users sr-users@lists.kamailio.org wrote:
Greetings, all, I have a redirect server doing CNAM lookup (via http_async) and STIR/SHAKEN validation with secsipidx that is crashing every few hours. Kamailio 6.0.1 running in a Debian 12 container on an Amazon Linux 2023 host with ARM CPU.
root@0689d4fe5fed:/# kamailio -v version: kamailio 6.0.1 (aarch64/linux) fce50d flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: fce50d compiled on 15:50:40 Apr 1 2025 with gcc 12.2.0
Back trace is here, so I think it's a secsipid related problem? Slightly sanitized this output (Removed SIP message specifics in step #42). What should be my next course of action?
#0 __pthread_kill_implementation (threadid=281473196830752, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44 #1 0x0000ffff95c90ab4 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78 #2 0x0000ffff95c4a72c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x0000ffff95c3747c in __GI_abort () at ./stdlib/abort.c:79 #4 0x0000ffff95c84aac in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffff95d66d28 "%s\n") at ../sysdeps/posix/libc_fatal.c:156 #5 0x0000ffff95c9aebc in malloc_printerr (str=str@entry=0xffff95d62098 "double free or corruption (!prev)") at ./malloc/malloc.c:5660 #6 0x0000ffff95c9cee8 in _int_free (av=0xffff95db0af0 <main_arena>, p=p@entry=0xaaab05dd41b0, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4587 #7 0x0000ffff95c9f77c in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385 #8 0x0000ffff90eb0a0c in w_secsipid_get_url (msg=0xffff92f53030, purl=0xffff92e84e10 "\220M\350\222\377\377", povar=0xffff92e84fd0 "\004") at secsipid_mod.c:961 #9 0x0000aaaaea4fd710 in do_action (h=0xffffff40a610, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1133 #10 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40a610, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1620 #11 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff40bc50, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1683 #12 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff40bc50, msg=0xffff92f53030, i=0xffffff40ab48, rv=0xffff92e8fc78, cache=0x0) at core/rvalue.c:973 #13 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff40bc50, msg=0xffff92f53030, res=0xffffff40ab48, rve=0xffff92e8fc70) at core/rvalue.c:1852 #14 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff40bc50, msg=0xffff92f53030, res=0xffffff40b080, rve=0xffff92e903b0) at core/rvalue.c:1862 #15 0x0000aaaaea4fd090 in do_action (h=0xffffff40bc50, a=0xffff92e91810, msg=0xffff92f53030) at core/action.c:1099 #16 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40bc50, a=0xffff92e8d690, msg=0xffff92f53030) at core/action.c:1620 #17 0x0000aaaaea4f9400 in do_action (h=0xffffff40bc50, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:711 #18 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40bc50, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:1620 #19 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff40daf0, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:1683 #20 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff40daf0, msg=0xffff92f53030, i=0xffffff40c188, rv=0xffff92e82378, cache=0x0) at core/rvalue.c:973 #21 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff40daf0, msg=0xffff92f53030, res=0xffffff40c188, rve=0xffff92e82370) at core/rvalue.c:1852 #22 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff40daf0, msg=0xffff92f53030, res=0xffffff40c6c0, rve=0xffff92e82ab0) at core/rvalue.c:1862 #23 0x0000aaaaea4fd090 in do_action (h=0xffffff40daf0, a=0xffff92e83a90, msg=0xffff92f53030) at core/action.c:1099 #24 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e83a90, msg=0xffff92f53030) at core/action.c:1620 #25 0x0000aaaaea4fd5e8 in do_action (h=0xffffff40daf0, a=0xffff92e83bf0, msg=0xffff92f53030) at core/action.c:1119 #26 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e79450, msg=0xffff92f53030) at core/action.c:1620 #27 0x0000aaaaea4f9400 in do_action (h=0xffffff40daf0, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:711 #28 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:1620 #29 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff4101f8, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:1683 #30 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff4101f8, msg=0xffff92f53030, i=0xffffff40e028, rv=0xffff92e75878, cache=0x0) at core/rvalue.c:973 #31 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff4101f8, msg=0xffff92f53030, res=0xffffff40e028, rve=0xffff92e75870) at core/rvalue.c:1852 #32 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff4101f8, msg=0xffff92f53030, res=0xffffff40e560, rve=0xffff92e75fb0) at core/rvalue.c:1862 #33 0x0000aaaaea4fd090 in do_action (h=0xffffff4101f8, a=0xffff92e76e10, msg=0xffff92f53030) at core/action.c:1099 #34 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92e74500, msg=0xffff92f53030) at core/action.c:1620 #35 0x0000aaaaea5095fc in do_action (h=0xffffff4101f8, a=0xffff92e79110, msg=0xffff92f53030) at core/action.c:1401 #36 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92e71a40, msg=0xffff92f53030) at core/action.c:1620 #37 0x0000aaaaea4f9400 in do_action (h=0xffffff4101f8, a=0xffff92dd8520, msg=0xffff92f53030) at core/action.c:711 #38 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92dd7e30, msg=0xffff92f53030) at core/action.c:1620 #39 0x0000aaaaea4fd5a8 in do_action (h=0xffffff4101f8, a=0xffff92dd8680, msg=0xffff92f53030) at core/action.c:1115 #40 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92dce4c0, msg=0xffff92f53030) at core/action.c:1620 #41 0x0000aaaaea50dbf4 in run_top_route (a=0xffff92dce4c0, msg=0xffff92f53030, c=0x0) at core/action.c:1703 #42 0x0000aaaaea6bfff8 in receive_msg ( buf=0xaaaaeac284b8 <buf> "INVITE sip:2345678901@1.2.3.4 SIP/2.0\r\nRecord-Route: sip:1.2.3.5;lr=on;ftag=gK045643d5\r\nRecord-Route: sip:1.2.3.6;lr=on;ftag=gK045643d5\r\nVia: SIP/2.0/UDP 1.2.3.5:5060;branch"..., len=2492, rcv_info=0xffffff410a10) at core/receive.c:520 #43 0x0000aaaaea85abe0 in udp_rcv_loop () at core/udp_server.c:770 #44 0x0000aaaaea4e26a8 in main_loop () at main.c:1895 #45 0x0000aaaaea4f1e88 in main (argc=16, argv=0xffffff411178) at main.c:3406
Kaufman Senior Voice Engineer
E: bkaufman@bcmone.com
SIP.US Client Support: 800.566.9810 | SIPTRUNK Client Support: 800.250.6510 | Flowroute Client Support: 855.356.9768 NOTE: This e-mail and any attachments are from BCM One, Inc. and are intended solely for the use and review of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail (and any attachments) from your computer and do not copy or disclose it to anyone else. __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
That was my first thought, but I didn't want to jump to conclusions here. It's a valid URL and I'm able to download the certificate. I believe that the signing certificates should be cached, but it could be the first attempt to grab this specific certificate. I just figured out how to get the core dumps on this system today, so it's the only example I have but I will be checking to see if it looks to be consistent to this cert/vendor.
#8 0x0000ffff90eb0a0c in w_secsipid_get_url (msg=0xffff92f53030, purl=0xffff92e84e10 "\220M\350\222\377\377", povar=0xffff92e84fd0 "\004") at secsipid_mod.c:961 ret = 0 ovar = 0x21f val = {rs = {s = 0x0, len = -12542544}, ri = 187651052223396, flags = -12542408} surl = {s = 0xffff92d52280 "https://sticr.stir.comcast.com/9d3a9bd1-f4f9-43ef-920a-bbcd0b02cb18.pem", len = 71} __func__ = "w_secsipid_get_url"
Kaufman Senior Voice Engineer
E: bkaufman@bcmone.com
SIP.US Client Support: 800.566.9810 | SIPTRUNK Client Support: 800.250.6510 | Flowroute Client Support: 855.356.9768
[img]https://www.sip.us [img]https://www.siptrunk.com [img]https://www.flowroute.com
________________________________ From: Alex Balashov via sr-users Sent: Friday, May 9, 2025 11:09 AM To: Kamailio (SER) - Users Mailing List Cc: Alex Balashov Subject: [SR-Users] Re: Kamailio Crash
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'd go to frame 8 and print the value of _secsipid_get_url_val.s / _secsipid_get_url_val.len. It's obviously not NULL, but I wonder what it's pointing to.
On May 9, 2025, at 11:47 AM, Ben Kaufman via sr-users sr-users@lists.kamailio.org wrote:
Greetings, all, I have a redirect server doing CNAM lookup (via http_async) and STIR/SHAKEN validation with secsipidx that is crashing every few hours. Kamailio 6.0.1 running in a Debian 12 container on an Amazon Linux 2023 host with ARM CPU.
root@0689d4fe5fed:/# kamailio -v version: kamailio 6.0.1 (aarch64/linux) fce50d flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: fce50d compiled on 15:50:40 Apr 1 2025 with gcc 12.2.0
Back trace is here, so I think it's a secsipid related problem? Slightly sanitized this output (Removed SIP message specifics in step #42). What should be my next course of action?
#0 __pthread_kill_implementation (threadid=281473196830752, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44 #1 0x0000ffff95c90ab4 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78 #2 0x0000ffff95c4a72c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x0000ffff95c3747c in __GI_abort () at ./stdlib/abort.c:79 #4 0x0000ffff95c84aac in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffff95d66d28 "%s\n") at ../sysdeps/posix/libc_fatal.c:156 #5 0x0000ffff95c9aebc in malloc_printerr (str=str@entry=0xffff95d62098 "double free or corruption (!prev)") at ./malloc/malloc.c:5660 #6 0x0000ffff95c9cee8 in _int_free (av=0xffff95db0af0 <main_arena>, p=p@entry=0xaaab05dd41b0, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4587 #7 0x0000ffff95c9f77c in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385 #8 0x0000ffff90eb0a0c in w_secsipid_get_url (msg=0xffff92f53030, purl=0xffff92e84e10 "\220M\350\222\377\377", povar=0xffff92e84fd0 "\004") at secsipid_mod.c:961 #9 0x0000aaaaea4fd710 in do_action (h=0xffffff40a610, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1133 #10 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40a610, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1620 #11 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff40bc50, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1683 #12 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff40bc50, msg=0xffff92f53030, i=0xffffff40ab48, rv=0xffff92e8fc78, cache=0x0) at core/rvalue.c:973 #13 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff40bc50, msg=0xffff92f53030, res=0xffffff40ab48, rve=0xffff92e8fc70) at core/rvalue.c:1852 #14 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff40bc50, msg=0xffff92f53030, res=0xffffff40b080, rve=0xffff92e903b0) at core/rvalue.c:1862 #15 0x0000aaaaea4fd090 in do_action (h=0xffffff40bc50, a=0xffff92e91810, msg=0xffff92f53030) at core/action.c:1099 #16 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40bc50, a=0xffff92e8d690, msg=0xffff92f53030) at core/action.c:1620 #17 0x0000aaaaea4f9400 in do_action (h=0xffffff40bc50, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:711 #18 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40bc50, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:1620 #19 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff40daf0, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:1683 #20 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff40daf0, msg=0xffff92f53030, i=0xffffff40c188, rv=0xffff92e82378, cache=0x0) at core/rvalue.c:973 #21 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff40daf0, msg=0xffff92f53030, res=0xffffff40c188, rve=0xffff92e82370) at core/rvalue.c:1852 #22 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff40daf0, msg=0xffff92f53030, res=0xffffff40c6c0, rve=0xffff92e82ab0) at core/rvalue.c:1862 #23 0x0000aaaaea4fd090 in do_action (h=0xffffff40daf0, a=0xffff92e83a90, msg=0xffff92f53030) at core/action.c:1099 #24 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e83a90, msg=0xffff92f53030) at core/action.c:1620 #25 0x0000aaaaea4fd5e8 in do_action (h=0xffffff40daf0, a=0xffff92e83bf0, msg=0xffff92f53030) at core/action.c:1119 #26 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e79450, msg=0xffff92f53030) at core/action.c:1620 #27 0x0000aaaaea4f9400 in do_action (h=0xffffff40daf0, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:711 #28 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:1620 #29 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff4101f8, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:1683 #30 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff4101f8, msg=0xffff92f53030, i=0xffffff40e028, rv=0xffff92e75878, cache=0x0) at core/rvalue.c:973 #31 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff4101f8, msg=0xffff92f53030, res=0xffffff40e028, rve=0xffff92e75870) at core/rvalue.c:1852 #32 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff4101f8, msg=0xffff92f53030, res=0xffffff40e560, rve=0xffff92e75fb0) at core/rvalue.c:1862 #33 0x0000aaaaea4fd090 in do_action (h=0xffffff4101f8, a=0xffff92e76e10, msg=0xffff92f53030) at core/action.c:1099 #34 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92e74500, msg=0xffff92f53030) at core/action.c:1620 #35 0x0000aaaaea5095fc in do_action (h=0xffffff4101f8, a=0xffff92e79110, msg=0xffff92f53030) at core/action.c:1401 #36 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92e71a40, msg=0xffff92f53030) at core/action.c:1620 #37 0x0000aaaaea4f9400 in do_action (h=0xffffff4101f8, a=0xffff92dd8520, msg=0xffff92f53030) at core/action.c:711 #38 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92dd7e30, msg=0xffff92f53030) at core/action.c:1620 #39 0x0000aaaaea4fd5a8 in do_action (h=0xffffff4101f8, a=0xffff92dd8680, msg=0xffff92f53030) at core/action.c:1115 #40 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92dce4c0, msg=0xffff92f53030) at core/action.c:1620 #41 0x0000aaaaea50dbf4 in run_top_route (a=0xffff92dce4c0, msg=0xffff92f53030, c=0x0) at core/action.c:1703 #42 0x0000aaaaea6bfff8 in receive_msg ( buf=0xaaaaeac284b8 <buf> "INVITE sip:2345678901@1.2.3.4 SIP/2.0\r\nRecord-Route: sip:1.2.3.5;lr=on;ftag=gK045643d5\r\nRecord-Route: sip:1.2.3.6;lr=on;ftag=gK045643d5\r\nVia: SIP/2.0/UDP 1.2.3.5:5060;branch"..., len=2492, rcv_info=0xffffff410a10) at core/receive.c:520 #43 0x0000aaaaea85abe0 in udp_rcv_loop () at core/udp_server.c:770 #44 0x0000aaaaea4e26a8 in main_loop () at main.c:1895 #45 0x0000aaaaea4f1e88 in main (argc=16, argv=0xffffff411178) at main.c:3406
Kaufman Senior Voice Engineer
E: bkaufman@bcmone.com
SIP.US Client Support: 800.566.9810 | SIPTRUNK Client Support: 800.250.6510 | Flowroute Client Support: 855.356.9768 NOTE: This e-mail and any attachments are from BCM One, Inc. and are intended solely for the use and review of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail (and any attachments) from your computer and do not copy or disclose it to anyone else. __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
-- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://urldefense.com/v3/__https://evaristesys.com__;!!KWzduNI!e3spbT2au32M... Tel: +1-706-510-6800
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
I've finally managed to capture one of the messages responsible for the issue, and it looks like the Identity header is malformed - it is missing the alg and ppt parameters. I'll try to mock up and replicate the behavior.
Kaufman Senior Voice Engineer
E: bkaufman@bcmone.com
SIP.US Client Support: 800.566.9810 | SIPTRUNK Client Support: 800.250.6510 | Flowroute Client Support: 855.356.9768
[img]https://www.sip.us [img]https://www.siptrunk.com [img]https://www.flowroute.com
________________________________ From: Ben Kaufman via sr-users Sent: Friday, May 9, 2025 11:29 AM To: Kamailio (SER) - Users Mailing List Cc: Ben Kaufman Subject: [SR-Users] Re: Kamailio Crash
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
That was my first thought, but I didn't want to jump to conclusions here. It's a valid URL and I'm able to download the certificate. I believe that the signing certificates should be cached, but it could be the first attempt to grab this specific certificate. I just figured out how to get the core dumps on this system today, so it's the only example I have but I will be checking to see if it looks to be consistent to this cert/vendor.
#8 0x0000ffff90eb0a0c in w_secsipid_get_url (msg=0xffff92f53030, purl=0xffff92e84e10 "\220M\350\222\377\377", povar=0xffff92e84fd0 "\004") at secsipid_mod.c:961 ret = 0 ovar = 0x21f val = {rs = {s = 0x0, len = -12542544}, ri = 187651052223396, flags = -12542408} surl = {s = 0xffff92d52280 "https://sticr.stir.comcast.com/9d3a9bd1-f4f9-43ef-920a-bbcd0b02cb18.pemhttps://urldefense.com/v3/__https://sticr.stir.comcast.com/9d3a9bd1-f4f9-43ef-920a-bbcd0b02cb18.pem__;!!KWzduNI!a6MjQ2Dl6ax2lsjR7BTYV_-DMZNwdewupTw5DUC0i3x5XyWsRoB0zxq-BDRhPKVgvFJY3WTyfKwEYcnAgeYUbjc$", len = 71} __func__ = "w_secsipid_get_url"
Kaufman Senior Voice Engineer
E: bkaufman@bcmone.com
SIP.US Client Support: 800.566.9810 | SIPTRUNK Client Support: 800.250.6510 | Flowroute Client Support: 855.356.9768
[img]https://urldefense.com/v3/__https://www.sip.us__;!!KWzduNI!a6MjQ2Dl6ax2lsjR7BTYV_-DMZNwdewupTw5DUC0i3x5XyWsRoB0zxq-BDRhPKVgvFJY3WTyfKwEYcnAgKpEyaQ$ [img]https://urldefense.com/v3/__https://www.siptrunk.com__;!!KWzduNI!a6MjQ2Dl6ax2lsjR7BTYV_-DMZNwdewupTw5DUC0i3x5XyWsRoB0zxq-BDRhPKVgvFJY3WTyfKwEYcnAml7EXX8$ [img]https://www.flowroute.com
________________________________ From: Alex Balashov via sr-users Sent: Friday, May 9, 2025 11:09 AM To: Kamailio (SER) - Users Mailing List Cc: Alex Balashov Subject: [SR-Users] Re: Kamailio Crash
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
I'd go to frame 8 and print the value of _secsipid_get_url_val.s / _secsipid_get_url_val.len. It's obviously not NULL, but I wonder what it's pointing to.
On May 9, 2025, at 11:47 AM, Ben Kaufman via sr-users sr-users@lists.kamailio.org wrote:
Greetings, all, I have a redirect server doing CNAM lookup (via http_async) and STIR/SHAKEN validation with secsipidx that is crashing every few hours. Kamailio 6.0.1 running in a Debian 12 container on an Amazon Linux 2023 host with ARM CPU.
root@0689d4fe5fed:/# kamailio -v version: kamailio 6.0.1 (aarch64/linux) fce50d flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: fce50d compiled on 15:50:40 Apr 1 2025 with gcc 12.2.0
Back trace is here, so I think it's a secsipid related problem? Slightly sanitized this output (Removed SIP message specifics in step #42). What should be my next course of action?
#0 __pthread_kill_implementation (threadid=281473196830752, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44 #1 0x0000ffff95c90ab4 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78 #2 0x0000ffff95c4a72c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #3 0x0000ffff95c3747c in __GI_abort () at ./stdlib/abort.c:79 #4 0x0000ffff95c84aac in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffff95d66d28 "%s\n") at ../sysdeps/posix/libc_fatal.c:156 #5 0x0000ffff95c9aebc in malloc_printerr (str=str@entry=0xffff95d62098 "double free or corruption (!prev)") at ./malloc/malloc.c:5660 #6 0x0000ffff95c9cee8 in _int_free (av=0xffff95db0af0 <main_arena>, p=p@entry=0xaaab05dd41b0, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4587 #7 0x0000ffff95c9f77c in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385 #8 0x0000ffff90eb0a0c in w_secsipid_get_url (msg=0xffff92f53030, purl=0xffff92e84e10 "\220M\350\222\377\377", povar=0xffff92e84fd0 "\004") at secsipid_mod.c:961 #9 0x0000aaaaea4fd710 in do_action (h=0xffffff40a610, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1133 #10 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40a610, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1620 #11 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff40bc50, a=0xffff92e8e9b0, msg=0xffff92f53030) at core/action.c:1683 #12 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff40bc50, msg=0xffff92f53030, i=0xffffff40ab48, rv=0xffff92e8fc78, cache=0x0) at core/rvalue.c:973 #13 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff40bc50, msg=0xffff92f53030, res=0xffffff40ab48, rve=0xffff92e8fc70) at core/rvalue.c:1852 #14 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff40bc50, msg=0xffff92f53030, res=0xffffff40b080, rve=0xffff92e903b0) at core/rvalue.c:1862 #15 0x0000aaaaea4fd090 in do_action (h=0xffffff40bc50, a=0xffff92e91810, msg=0xffff92f53030) at core/action.c:1099 #16 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40bc50, a=0xffff92e8d690, msg=0xffff92f53030) at core/action.c:1620 #17 0x0000aaaaea4f9400 in do_action (h=0xffffff40bc50, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:711 #18 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40bc50, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:1620 #19 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff40daf0, a=0xffff92e82210, msg=0xffff92f53030) at core/action.c:1683 #20 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff40daf0, msg=0xffff92f53030, i=0xffffff40c188, rv=0xffff92e82378, cache=0x0) at core/rvalue.c:973 #21 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff40daf0, msg=0xffff92f53030, res=0xffffff40c188, rve=0xffff92e82370) at core/rvalue.c:1852 #22 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff40daf0, msg=0xffff92f53030, res=0xffffff40c6c0, rve=0xffff92e82ab0) at core/rvalue.c:1862 #23 0x0000aaaaea4fd090 in do_action (h=0xffffff40daf0, a=0xffff92e83a90, msg=0xffff92f53030) at core/action.c:1099 #24 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e83a90, msg=0xffff92f53030) at core/action.c:1620 #25 0x0000aaaaea4fd5e8 in do_action (h=0xffffff40daf0, a=0xffff92e83bf0, msg=0xffff92f53030) at core/action.c:1119 #26 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e79450, msg=0xffff92f53030) at core/action.c:1620 #27 0x0000aaaaea4f9400 in do_action (h=0xffffff40daf0, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:711 #28 0x0000aaaaea50d3c4 in run_actions (h=0xffffff40daf0, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:1620 #29 0x0000aaaaea50db44 in run_actions_safe (h=0xffffff4101f8, a=0xffff92e75710, msg=0xffff92f53030) at core/action.c:1683 #30 0x0000aaaaea71a7e0 in rval_get_long (h=0xffffff4101f8, msg=0xffff92f53030, i=0xffffff40e028, rv=0xffff92e75878, cache=0x0) at core/rvalue.c:973 #31 0x0000aaaaea7200f0 in rval_expr_eval_long (h=0xffffff4101f8, msg=0xffff92f53030, res=0xffffff40e028, rve=0xffff92e75870) at core/rvalue.c:1852 #32 0x0000aaaaea720118 in rval_expr_eval_long (h=0xffffff4101f8, msg=0xffff92f53030, res=0xffffff40e560, rve=0xffff92e75fb0) at core/rvalue.c:1862 #33 0x0000aaaaea4fd090 in do_action (h=0xffffff4101f8, a=0xffff92e76e10, msg=0xffff92f53030) at core/action.c:1099 #34 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92e74500, msg=0xffff92f53030) at core/action.c:1620 #35 0x0000aaaaea5095fc in do_action (h=0xffffff4101f8, a=0xffff92e79110, msg=0xffff92f53030) at core/action.c:1401 #36 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92e71a40, msg=0xffff92f53030) at core/action.c:1620 #37 0x0000aaaaea4f9400 in do_action (h=0xffffff4101f8, a=0xffff92dd8520, msg=0xffff92f53030) at core/action.c:711 #38 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92dd7e30, msg=0xffff92f53030) at core/action.c:1620 #39 0x0000aaaaea4fd5a8 in do_action (h=0xffffff4101f8, a=0xffff92dd8680, msg=0xffff92f53030) at core/action.c:1115 #40 0x0000aaaaea50d3c4 in run_actions (h=0xffffff4101f8, a=0xffff92dce4c0, msg=0xffff92f53030) at core/action.c:1620 #41 0x0000aaaaea50dbf4 in run_top_route (a=0xffff92dce4c0, msg=0xffff92f53030, c=0x0) at core/action.c:1703 #42 0x0000aaaaea6bfff8 in receive_msg ( buf=0xaaaaeac284b8 <buf> "INVITE sip:2345678901@1.2.3.4 SIP/2.0\r\nRecord-Route: sip:1.2.3.5;lr=on;ftag=gK045643d5\r\nRecord-Route: sip:1.2.3.6;lr=on;ftag=gK045643d5\r\nVia: SIP/2.0/UDP 1.2.3.5:5060;branch"..., len=2492, rcv_info=0xffffff410a10) at core/receive.c:520 #43 0x0000aaaaea85abe0 in udp_rcv_loop () at core/udp_server.c:770 #44 0x0000aaaaea4e26a8 in main_loop () at main.c:1895 #45 0x0000aaaaea4f1e88 in main (argc=16, argv=0xffffff411178) at main.c:3406
Kaufman Senior Voice Engineer
E: bkaufman@bcmone.com
SIP.US Client Support: 800.566.9810 | SIPTRUNK Client Support: 800.250.6510 | Flowroute Client Support: 855.356.9768 NOTE: This e-mail and any attachments are from BCM One, Inc. and are intended solely for the use and review of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail (and any attachments) from your computer and do not copy or disclose it to anyone else. __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
-- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://urldefense.com/v3/__https://evaristesys.com__;!!KWzduNI!e3spbT2au32M... Tel: +1-706-510-6800
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
-
Alex Balashov -
Ben Kaufman