February 2024 - sr-dev - kamailio.org

[kamailio/kamailio] t_uac_send parser does not like exclamation or other non alphabetic characters in Contact header (Issue #3682)
by xaled1 24 Jul '24

24 Jul '24

Hi there is an issue with the SIP parser for the t_uac_send function https://www.kamailio.org/docs/modules/devel/modules/tm.html#tm.f.t_uac_send If I call this function for example: t_uac_send("INVITE", "sip:test@123.123.123.123:5060", "", "" ,"Content-Type: text/plain\r\nContact: <sip:85951b3d-096c-95d6-5f05-4f38095aca9f@172.17.64.120:5060;transport=tcp>;+u.sip!devicename.ccm.cisco.com=\"SEP5486BC7F2BBC\"", "Testbody") Then the resulting request is broken - it's probably due to a character (exclamation mark?) in the Contact header.... I tested it with a call from kemi, but it probably also occurs when you run it with a normal kamailio -script tests -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3682 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3682(a)github.com>

3 4

[kamailio/kamailio] Dialog DMQ sync: Timer needs fixing, firing on wrong node! (Issue #3656)
by Benoît Panizzon 17 Jul '24

17 Jul '24

Situation: Two Kamailio nodes syncing dialog profiles via DMQ Observation: When a dialog timeout is encountered, about half of the time, the timeout is triggered on the peer node not handling the dialog preventing the call to be correctly terminated and also leaving entries in the dialog database which never get deleted. https://kamailio.org/docs/modules/5.7.x/modules/dialog.html#dialog.p.enable… Makes clear, that only the node which is handling the dialog in question, can make changes not related to the dialog profiles. So when a dialog times out, it is this node which has to trigger the timeout, not any other one. When looking at the source code, it is clear, that the 'lifetime' is transmitted via DMQ dialog message to the peer nodes, which in turn arm a timer. So it is obvious, this time will trigger, sometimes before the instance handling the dialog itself triggers that timer. With some assistance of @oej I found a way to alter the JSON payload to extend the lifetime on the peer nodes: ``` route[DMQ_CAPTURE] { if(is_method("KDMQ")) { if(has_body("application/json") && $fU == 'dialog') { if (jansson_get("lifetime", $rb, "$var(lifetime)")) { $var(new_lifetime) = $var(lifetime) + 60; # Add 60 seconds on DMQ peer to make sure it expires AFTER main node. $var(newrb) = $rb; jansson_set("integer", "lifetime", $var(new_lifetime), "$var(newrb)"); set_body("$var(newrb)","application/json"); msg_apply_changes(); } } dmq_handle_message(); exit; } } ``` Testing with this config fixed the issue of the timeout firing on a peer node instead of the node handling the dialog. Issue is present in 5.5, 5.6 and after looking at the code, I assume also on 5.7 -Benoît -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3656 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3656(a)github.com>

6 15

[kamailio/kamailio] Debian package build fails for Kamailio 5.7.4 (Issue #3722)
by Alexander Bakker 15 Jul '24

15 Jul '24

### Description The Debian package build fails for Kamailio 5.7.4 because it can't find static libssl. This happens because ``LIBSSL_STATIC_SRCPATH`` is set to the wrong location in the rules file. ### Troubleshooting #### Reproduction 1. Run ``ln -s pkg/kamailio/deb/bookworm debian`` 2. Run ``make deb`` #### Output ``` gcc -shared -g -m64 -Wl,-O2 -Wl,-E -Wl,-z,relro -pthread -rdynamic "-ldl" -Wl,-Bsymbolic-functions tls_bio.o tls_cfg.o tls_config.o tls_ct_wrq.o tls_domain.o tls_dump_vf.o tls_init.o tls_locking.o tls_map.o tls_rand.o tls_rpc.o tls_select.o tls_server.o tls_util.o tls_verify.o tlsa_mod.o -lm /usr/lib64/libssl.a /usr/lib64/libcrypto.a -o tlsa.so /usr/bin/ld: cannot find /usr/lib64/libssl.a: No such file or directory /usr/bin/ld: cannot find /usr/lib64/libcrypto.a: No such file or directory collect2: error: ld returned 1 exit status make[5]: *** [../../Makefile.rules:191: tlsa.so] Error 1 ``` ### Possible Solutions An extremely hacky workaround is to patch the rules file to point ``LIBSSL_STATIC_SRCPATH`` to the right location: ``` sed -i -e "/LIBSSL_STATIC_SRCPATH/a export\\ LIBSSL_STATIC_SRCPATH=\\/usr\\/lib\\/$(uname -m)-linux-gnu" pkg/kamailio/deb/bookworm/rules ``` ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.7.4 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 12.2.0 ``` * **Operating System**: ``` Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm ``` ``` Linux 6.5.0-14-generic #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 x86_64 GNU/Linux ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3722 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3722(a)github.com>

3 10

[kamailio/kamailio] No CDR is written on failover (Issue #3254)
by Earn330 15 Jul '24

15 Jul '24

Hi, I think there is an issue with writing CDRs on failover scenarios. I have two servers with Kamailio. Both server running Debian 11 (bulleye) and latest Kamailio 5.6.1 from git. I'm using DMQ to sync dialogs and htable between these servers, so both servers have the same knowledge of dialog state. Both Kamailio uses nobind option, so I can switch a VIP from one server to the other one. This is managed with keepalived. Switching the VIP from one server the other one works fine an while a call is running I can switch the VIP. I can see that the BYE Message is handled OK after I made a switch. I can see that acc is triggered but what is missing is acc_cdr in this case. I'm using htable to fill all necessary variables to complete the CDR an I can see that all values are synced correctly. The acc_cdr is created fine if there is no failover so i think it can't be an configuration issue. Both server holds the same Kamailio configuration except the IP. `version: kamailio 5.6.1 (x86_64/linux) bfc5c2-dirty flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: bfc5c2 -dirty compiled with gcc 10.2.1` `root@voip-lab-proxy01:~# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye` `root@voip-lab-proxy01:~# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 38 bits physical, 48 bits virtual CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 23 Model name: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz Stepping: 10 CPU MHz: 2500.088 BogoMIPS: 5000.17 Virtualization: VT-x Hypervisor vendor: KVM Virtualization type: full L1d cache: 128 KiB L1i cache: 128 KiB L2 cache: 16 MiB L3 cache: 16 MiB NUMA node0 CPU(s): 0-3 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Mitigation; PTE Inversion; VMX EPT disabled Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Unknown: No mitigations Vulnerability Retbleed: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni vmx ssse3 cx16 pdcm sse4_1 x2apic tsc_deadline_timer xsave hypervisor lahf_lm cpuid_fault pti tpr_shadow vnmi flexpriority vpid tsc_adjust arat arch_capabilit ies` If you need more information, please let me know. As I'm running this on a test system I can reproduce the issue at any time. -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3254 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3254(a)github.com>

5 9

[kamailio/kamailio] extented haproxy protocol support (PR #3731)
by sergey-safarov 12 Jul '24

12 Jul '24

#### Pre-Submission Checklist - [x] Commit message has the format required by CONTRIBUTING guide - [x] Commits are split per component (core, individual modules, libs, utils, ...) - [x] Each component has a single commit (if not, squash them into one commit) - [x] No commits to README files for modules (changes must be done to docbook files in `doc/` subfolder, the README file is autogenerated) #### Type Of Change - [ ] Small bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds new functionality) - [ ] Breaking change (fix or feature that would change existing functionality) #### Checklist: - [ ] PR should be backported to stable branches - [x] Tested changes locally - [x] Related to issue #3667 #### Description Allow creating Record-Route and Via header using destination address and port in the haproxy protocol header. PR created behalf @ivanuschak You can view, comment on, or merge this pull request online at: https://github.com/kamailio/kamailio/pull/3731 -- Commit Summary -- * core: extented haproxy protocol parser * rr: extented haproxy protocol suppport * path: extented haproxy protocol suppport * websocket: extented haproxy protocol suppport * siptrace: extented haproxy protocol suppport -- File Changes -- M src/core/forward.h (4) M src/core/msg_translator.c (137) M src/core/parser/msg_parser.h (1) M src/core/receive.c (6) M src/core/receive.h (3) M src/core/tcp_conn.h (1) M src/core/tcp_main.c (22) M src/core/tcp_read.c (20) M src/modules/path/path.c (2) M src/modules/rr/loose.c (4) M src/modules/rr/record.c (6) M src/modules/siptrace/siptrace.c (89) M src/modules/websocket/ws_frame.c (9) M src/modules/websocket/ws_handshake.c (3) -- Patch Links -- https://github.com/kamailio/kamailio/pull/3731.patch https://github.com/kamailio/kamailio/pull/3731.diff -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/pull/3731 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/pull/3731(a)github.com>

6 22

[kamailio/kamailio] WIP pkg: RPM packaging add kamailio-tls_wolfssl subpackage (PR #3592)
by space88man 09 Jul '24

09 Jul '24

#### Pre-Submission Checklist    - [X] Commit message has the format required by CONTRIBUTING guide - [X] Commits are split per component (core, individual modules, libs, utils, ...) - [X] Each component has a single commit (if not, squash them into one commit) - [X] No commits to README files for modules (changes must be done to docbook files in `doc/` subfolder, the README file is autogenerated) #### Type Of Change - [ ] Small bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds new functionality) - [ ] Breaking change (fix or feature that would change existing functionality) #### Checklist:  - [ ] PR should be backported to stable branches - [X] Tested changes locally - [ ] Related to issue #XXXX (replace XXXX with an open issue number) #### Description Not for merging yet. Package tls_wolfssl as an alternative to OpenSSL 3.x. Depends on PR https://github.com/kamailio/kamailio/pull/3590 You can view, comment on, or merge this pull request online at: https://github.com/kamailio/kamailio/pull/3592 -- Commit Summary -- * pkg: RPM packaging add kamailio-tls_wolfssl subpackage -- File Changes -- M pkg/kamailio/obs/kamailio.spec (18) -- Patch Links -- https://github.com/kamailio/kamailio/pull/3592.patch https://github.com/kamailio/kamailio/pull/3592.diff -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/pull/3592 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/pull/3592(a)github.com>

5 27

[kamailio/kamailio] performance regression related to new TCP connection tracking feature with a high number of connections (Issue #3759)
by Henning Westerholt 05 Jul '24

05 Jul '24

### Description On systems with a high number of TCP sessions there can be a significant performance regression observed, probably related to the newly added TCP connection tracking feature. ### Troubleshooting #### Reproduction No special configuration is necessary, just install the latest 5.7.x release, e.g. 5.7.3 on a production system with a lot of clients connected over TCP or TLS. You need to have a large number of clients connected to be able to observe the regression. For a high number of connections (e.g. more then 20.000 up to 30.000 connections) the Kamailio servers uses about 30% to 50% more CPU as with the old version. #### Debugging Data Two graphs were attached to this issue. The first shows the CPU load before (less load) and after the upgrade (increased load). The second is a flamegraph that shows that over 80% of the CPU time is spent in the newly added function tcp_connection_limit_srcip(). <img width="372" alt="cpu-load-before-after" src="https://github.com/kamailio/kamailio/assets/6481937/ec92c41b-25e2-4847-a4bc…"> ![flamegraph](https://github.com/kamailio/kamailio/assets/6481937/5c2e3982-2d5b-4acd-a774-bad8fa64bd0a) Most of the CPU time is spend in the TCP main process, as expected. #### Log Messages No special log messages could be observed. #### SIP Traffic ### Possible Solutions The TCP limit feature should probably be optimized to not cause such a large performance regression. It should be also possible to deactivate it completly and therefore getting a comparable performance as before the feature addition. ### Additional Information Kamailio 5.7.3 and probably also git master version. * **Operating System**: Debian 11, Debian 12 -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3759 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3759(a)github.com>

2 6

[kamailio/kamailio] tcpops module event_route[tcp:closed] route block $conid is null (Issue #3767)
by liangbaolin 05 Jul '24

05 Jul '24

### Description During the process of using version 5.7.4 of Kamailio, I found that the $conid cannot be correctly obtained in the TCP: closed routing block of the TCP module event_route. I am certain that it is normal in version 5.4.8. ### Troubleshooting ``` pv [pv_core.c:2725]: pv_get_tcpconn_id() ``` #### Reproduction The test scenario initiated a register registration process for me, and after completing the disconnection of the TCP connection, triggered this event. #### Log Messages  ``` 2024-02-26T01:35:55.902002512Z 52(98) DEBUG: <core> [core/receive.c:635]: receive_msg(): cleaning up 2024-02-26T01:35:55.902034482Z 52(98) DEBUG: <core> [core/tcp_read.c:280]: tcp_read_data(): EOF on connection 0x7f99a1546a80 (state: 0, flags: c018) - FD 24, bytes 0, rd-flags 10000 ([12.0.1.6]:15821 -> [12.0.1.6]:5060)52(98) DEBUG: <core> [core/tcp_read.c:343]: tcp_read(): read 0 bytes: 2024-02-26T01:35:55.902051542Z 2024-02-26T01:35:55.902058282Z 52(98) DEBUG: <core> [core/tcp_read.c:1532]: tcp_read_req(): read= 0 bytes, parsed=0, state=0, error=1 2024-02-26T01:35:55.902063282Z 52(98) DEBUG: <core> [core/tcp_read.c:1534]: tcp_read_req(): last char=0x00, parsed msg= 2024-02-26T01:35:55.902069402Z 2024-02-26T01:35:55.902074032Z 52(98) DEBUG: <core> [core/tcp_read.c:1544]: tcp_read_req(): EOF 2024-02-26T01:35:55.902101372Z 52(98) DEBUG: <core> [core/io_wait.h:598]: io_watch_del(): DBG: io_watch_del (0x559e8a029740, 24, -1, 0x10) fd_no=2 called 2024-02-26T01:35:55.902153162Z 52(98) DEBUG: <core> [core/tcp_read.c:1927]: handle_io(): removing from list 0x7f99a1546a80 id 2 fd 24, state 2, flags c018, main fd 134, refcnt 2 ([12.0.1.6]:15821 -> [12.0.1.6]:5060) 2024-02-26T01:35:55.902168112Z 52(98) DEBUG: <core> [core/tcp_read.c:1702]: release_tcpconn(): releasing con 0x7f99a1546a80, state -1, fd=24, id=2 ([12.0.1.6]:15821 -> [12.0.1.6]:5060) 2024-02-26T01:35:55.902173372Z 52(98) DEBUG: <core> [core/tcp_read.c:1705]: release_tcpconn(): extra_data (nil) 2024-02-26T01:35:55.902208402Z 83(129) DEBUG: <core> [core/tcp_main.c:3747]: handle_tcp_child(): reader response= 7f99a1546a80, -1 from 1 2024-02-26T01:35:55.902224742Z 83(129) DEBUG: <core> [core/tcp_main.c:3904]: handle_tcp_child(): case CONN_EOF tcp_emit_closed_event before 5 2024-02-26T01:35:55.902229802Z 83(129) DEBUG: <core> [core/tcp_main.c:3671]: tcp_emit_closed_event(): TCP closed event creation triggered (reason: 0) 2024-02-26T01:35:55.902318572Z 83(129) DEBUG: tcpops [tcpops.c:311]: tcpops_handle_tcp_closed(): received TCP closed event 2024-02-26T01:35:55.902331352Z 83(129) DEBUG: tcpops [tcpops.c:261]: tcpops_tcp_closed_run_route(): event reason id: 0 rt: 1 2024-02-26T01:35:55.902335132Z 83(129) ERROR: tcpops [tcpops.c:278]: tcpops_tcp_closed_run_route(): run_top_route() before tev->id: 2 2024-02-26T01:35:55.902480972Z 83(129) DEBUG: <core> [core/tcp_main.c:1713]: _tcpconn_find(): 2 port 0 2024-02-26T01:35:55.902493872Z 83(129) DEBUG: pv [pv_core.c:2725]: pv_get_tcpconn_id(): no conid 2024-02-26T01:35:55.902498762Z 83(129) DEBUG: pv [pv_core.c:2726]: pv_get_tcpconn_id(): msg->rcv.proto_reserved1: 2 2024-02-26T01:35:55.902503592Z 83(129) DEBUG: <core> [core/tcp_main.c:1713]: _tcpconn_find(): 2 port 0 2024-02-26T01:35:55.902508262Z 83(129) DEBUG: <core> [core/pvapi.c:1418]: pv_printf_mode(): final buffer length 111 2024-02-26T01:35:55.902513422Z 83(129) ERROR: <script>: tcp connection closed conid: (<null>), tcp(conid): <null>, Ri: 12.0.1.185, Rp: 5060 , si: 12.0.1.6 , sp: 15821 2024-02-26T01:35:55.902607692Z 83(129) DEBUG: <core> [core/tcp_main.c:1713]: _tcpconn_find(): 2 port 0 2024-02-26T01:35:55.902625622Z 83(129) DEBUG: pv [pv_core.c:2725]: pv_get_tcpconn_id(): no conid 2024-02-26T01:35:55.902630672Z 83(129) DEBUG: pv [pv_core.c:2726]: pv_get_tcpconn_id(): msg->rcv.proto_reserved1: 2 2024-02-26T01:35:55.902635822Z 83(129) ERROR: tcpops [tcpops.c:280]: tcpops_tcp_closed_run_route(): run_top_route() after tev->id: 2 2024-02-26T01:35:58.777642759Z 41(87) DEBUG: ims_dialog [dlg_handlers.c:2026]: print_all_dlgs(): ********************41(87) DEBUG: ims_dialog [dlg_handlers.c:2027]: print_all_dlgs(): printing 65536 dialogs ``` #### SIP Traffic  ``` my test script block event_route[tcp:closed] { xlog("L_ERR", "$proto connection closed conid: ($conid), tcp(conid): $tcp(conid), Ri: $Ri, Rp: $Rp , si: $si , sp: $sp\n"); } file: src/modules/pv/pv_core.c int pv_get_tcpconn_id(struct sip_msg *msg, pv_param_t *param, pv_value_t *res) { ... if((con = tcpconn_get(msg->rcv.proto_reserved1, 0, 0, 0, 0)) == NULL) // return false { LM_DBG("no conid\n"); //additional custom log LM_DBG("msg->rcv.proto_reserved1: %d\n",msg->rcv.proto_reserved1); return pv_get_null(msg, param, res); } .... } file: src/core/tcp_main.c struct tcp_connection *tcpconn_lookup(int id, struct ip_addr *ip, int port, union sockaddr_union *local_addr, int try_local_port, ticks_t timeout) { ... if(unlikely(c == NULL)) { c = _tcpconn_find(id, ip, port, &local_ip, local_port); // retrun c == null } ... } struct tcp_connection *_tcpconn_find( int id, struct ip_addr *ip, int port, struct ip_addr *l_ip, int l_port) { ... if(likely(id)) { hash = tcp_id_hash(id); for(c = tcpconn_id_hash[hash]; c; c = c->id_next) { //tcpconn_id_hash find c is null #ifdef EXTRA_DEBUG LM_DBG("c=%p, c->id=%d, port=%d\n", c, c->id, c->rcv.src_port); print_ip("ip=", &c->rcv.src_ip, "\n"); #endif .... } inline static int tcpconn_try_unhash(struct tcp_connection *tcpconn) { ... _tcpconn_detach(tcpconn); //executed ... lock_release(&tcpconn->write_lock); ... } static inline void _tcpconn_detach(struct tcp_connection *c) { ... tcpconn_listrm(tcpconn_id_hash[c->id_hash], c, id_next, id_prev); //remove c ... } ``` ### Possible Solutions  ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.7.4 (x86_64/linux) 9920d9-dirty flags: , EXTRA_DEBUGUSE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 9920d9 -dirty compiled on 09:59:52 Feb 26 2024 with gcc 7.5.0 ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3767 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3767(a)github.com>

5 8

[kamailio/kamailio] Kamailio became unresponsive during runtime (#2380)
by ObakeFilter 27 Jun '24

27 Jun '24

### Description Hi, I ran into an issue where the Kamailio service seemingly froze (didn't crash, but failed to receive and deliver calls) while running, I went through the logs a little bit, and saw the same log pattern (pasted below) recouring every few months or so, on an older installation (5.2.0). Usually, it ended up automatically restarting the service. But not this time, I had to reload it manually to make the service work again. Is this a known issue? How can I make sure that the service will be able to restart the next time it happens? Edward ### Troubleshooting #### Debugging Data  ``` GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/kamailio...done. warning: exec file is newer than core file. [New LWP 11413] Core was generated by `kamailio'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f4e4a52e428 in ?? () (gdb) Quit (gdb) bt full #0 0x00007f4e4a52e428 in ?? () No symbol table info available. #1 0x00007f4e4a53002a in ?? () No symbol table info available. #2 0x0000000000000020 in ?? () No symbol table info available. #3 0x0000000000000000 in ?? () No symbol table info available. (gdb) info locals No symbol table info available. (gdb) list 1871 & now if we don't need it */ 1872 #ifdef USE_SLOW_TIMER 1873 + 1 /* slow timer process */ 1874 #endif 1875 #ifdef USE_TCP 1876 +((!tcp_disable)?( 1/* tcp main */ + tcp_listeners ):0) 1877 #endif 1878 #ifdef USE_SCTP 1879 +((!sctp_disable)?sctp_listeners:0) 1880 #endif (gdb) ``` #### Log Messages  ``` Jun 28 16:35:22 kamprodegres kamailio: INFO: <core> [core/sctp_core.c:74]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module Jun 28 16:35:22 kamprodegres kamailio: WARNING: <core> [core/socket_info.c:1394]: fix_hostname(): could not rev. resolve 70.36.25.87 Jun 28 16:35:22 kamprodegres kamailio: WARNING: <core> [core/socket_info.c:1394]: fix_hostname(): could not rev. resolve 70.36.25.87 Jun 28 16:35:22 kamprodegres kamailio: INFO: <core> [core/tcp_main.c:5042]: init_tcp(): using epoll_lt as the io watch method (auto detected) Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: jsonrpcs [jsonrpcs_sock.c:197]: jsonrpc_dgram_mod_init(): the socket /var/run/kamailio/kamailio_rpc.sock already exists, trying to delete it... Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: rr [rr_mod.c:177]: mod_init(): outbound module not available Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [main.c:2779]: main(): processes (at least): 40 - shm size: 67108864 - pkg size: 8388608 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 Jun 28 16:35:22 kamprodegres kamailio[8518]: INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/8518 Jun 28 16:35:22 kamprodegres kamailio[8519]: INFO: ctl [io_listener.c:214]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) Jun 29 18:13:37 kamprodegres kamailio[8513]: INFO: {1 581461 CANCEL 402954192_49936786(a)67.231.13.146} tm [t_reply.c:478]: _reply_light(): can't generate 487 reply when a final 404 was sent out Jun 29 18:27:00 kamprodegres kamailio[8508]: INFO: {1 211236 CANCEL 405019408_127896329(a)67.231.13.146} tm [t_reply.c:478]: _reply_light(): can't generate 487 reply when a final 404 was sent out Jun 29 18:51:10 kamprodegres /usr/local/sbin/kamailio[2269]: NOTICE: <core> [main.c:725]: handle_sigs(): Thank you for flying kamailio!!! Jun 29 18:51:10 kamprodegres /usr/local/sbin/kamailio[2326]: INFO: <core> [main.c:847]: sig_usr(): signal 15 received ``` ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.3.2 (x86_64/linux) 7ba545 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 7ba545 compiled on 16:37:32 Jun 14 2020 with gcc 5.4.0 ``` * **Operating System**:  ``` Ubuntu 16.04 ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2380

3 10

[kamailio/kamailio] Kamailio 5.6 (and 5.7) core dumps. (Issue #3725)
by Mattis Lind 19 Jun '24

19 Jun '24

### Description We have a quite basic scenario which sends a SIP MESSAGE and then receives SIP MESSAGE back. It also sets up two voice calls using SIP INVITE. It is only this specific scenario that is causing the core dump. Some time the core dump occur just a few seconds after the call/message scenario, but occasionally it takes up to 50 minutes. We have been using the uac module for sending SIP MESSAGE in several other scenarios without experiencing a crash. We always see the same lines in the log just prior to the core dump: CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 49 We are using Kamailio 5.6 retrieved from the kamailio repository: http://deb.kamailio.org/kamailio56. We are running Kamailio in a Docker container which runs on "5.10.0-25-cloud-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64 GNU/Linux" We have also tried Kamailio 5.7 from the same repo which crashes in exactly the same way. ### Troubleshooting #### Reproduction We have troubles to consistently reproduce it and it only happen from time to time when running a specific scenario sending SIP MESSAGE using the uac module. It does not happen for every call of this call scenario. At one instance Kamailo crashed with just 15 seconds in-between during the call scenario. #### Debugging Data ``` # gdb /usr/sbin/kamailio /core GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio... (No debugging symbols found in /usr/sbin/kamailio) warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing [New LWP 4777] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -DD -M 18 -m 192 -A serverId=17173 -A sendTraceLocal="sip:10'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:77 77 ../sysdeps/x86_64/multiarch/strlen-evex.S: No such file or directory. (gdb) bt full #0 __strlen_evex () at ../sysdeps/x86_64/multiarch/strlen-evex.S:77 No locals. #1 0x00007f5425e34b78 in __vfprintf_internal (s=s@entry=0x55a06e969a60, format=format@entry=0x55a06e558020 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n", ap=ap@entry=0x7ffd6bebfb50, mode_flags=mode_flags@entry=0) at vfprintf-internal.c:1647 len = <optimized out> step0_jumps = {0, 1717, 1621, 3413, 3317, 3997, 2677, 2837, 3613, 1773, 4309, 4445, 3517, 4437, 4389, 2789, 4197, 3917, 3221, 2997, 1141, 1365, 1997, 1925, 1885, 733, 3709, 533, 533, 4101} space = <optimized out> is_short = <optimized out> use_outdigits = 0 outc = <optimized out> step1_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 1773, 4309, 4445, 3517, 4437, 4389, 2789, 4197, 3917, 3221, 2997, 1141, 1365, 1997, 1925, 1885, 733, 3709, 533, 533, 0} group = 0 prec = -1 step2_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4309, 4445, 3517, 4437, 4389, 2789, 4197, 3917, 3221, 2997, 1141, 1365, 1997, 1925, 1885, 733, 3709, 533, 533, 0} string = 0x756d6f7266222c22 <error: Cannot access memory at address 0x756d6f7266222c22> left = 0 is_long_double = <optimized out> width = 0 signed_number = <optimized out> step3a_jumps = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4213, 0, 0, 0, 4389, 2789, 4197, 3917, 3221, 0, 0, 0, 0, 1925, 0, 0, 0, 0, 0, 0} alt = <optimized out> showsign = 0 is_long = 0 is_char = <optimized out> pad = <optimized out> step3b_jumps = {0 <repeats 11 times>, 3517, 0, 0, 4389, 2789, 4197, 3917, 3221, 2997, 1141, 1365, 1997, 1925, 1885, 733, 3709, 0, 0, 0} step4_jumps = {0 <repeats 14 times>, 4389, 2789, 4197, 3917, 3221, 2997, 1141, 1365, 1997, 1925, 1885, 733, 3709, 0, 0, 0} args_value = <optimized out> is_negative = <optimized out> number = {longlong = <optimized out>, word = <optimized out>} base = <optimized out> the_arg = {pa_wchar = 4777 L'\x12a9', pa_int = 4777, pa_long_int = 4777, pa_long_long_int = 4777, pa_u_int = 4777, pa_u_long_int = 4777, pa_u_long_long_int = 4777, pa_double = 2.3601515901836347e-320, pa_long_double = 1.74131181638025811763e-4947, pa_float128 = 3.09319115455554459548860449034534676e-4962, pa_string = 0x12a9 <error: Cannot access memory at address 0x12a9>, pa_wstring = 0x12a9 <error: Cannot access memory at address 0x12a9>, pa_pointer = 0x12a9, pa_user = 0x12a9} spec = 115 's' _buffer = {__routine = 0x4, __arg = 0xd, __canceltype = 1855363680, __prev = 0xe0} _avail = <optimized out> thousands_sep = 0x0 grouping = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff> done = 238 f = 0x55a06e5580a7 "s:%lu. Exec from %s:%u.\n" lead_str_end = 0x55a06e558020 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n" end_of_spec = <optimized out> work_buffer = "h\r\000\000\000\000\000\000\000\000\000\000\060\000\000\000\000\000\000\000\375\177\000\000 \372\353k\375\177\000\000\237MSn\n\000\000\000\000\000\000\000\240U", '\000' <repeats 18 times>, "P\225Un\240U\000\000\000\000\000\000[\214A\323\f\000\000\000\000\000\000\000\377\377\377\377\377\377\377\377\206\002", '\000' <repeats 14 times>, "\004\000\000\000\000\000\000\000 \367\353k\375\177\000\000\301\225Un\240U\000\000\323\262\vn\240U\000\000\000\000\000\000\000\000\000\000s\374On\240U\000\000W\225Un\240U\000\000\350>\212\027T\177\000\000 \323\365$T\177\000\000\060\a\354k\375\177\000\00--Type <RET> for more, q to quit, c to continue without paging-- 0\220\371\227n\240U\000\000"... workend = 0x7ffd6bebf9f8 "" ap_save = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7ffd6bebfc30, reg_save_area = 0x7ffd6bebfb70}} nspecs_done = 10 save_errno = 4 readonly_format = 0 do_longlong_number = <optimized out> __result = <optimized out> #2 0x00007f5425ec079f in __vsyslog_internal (pri=<optimized out>, fmt=0x55a06e558020 "%s: %.*s%s%s%sBUG: qm: fragm. %p (address %p) beginning overwritten (%lx)! Memory allocator was called from %s:%u. Fragment marked by %s:%lu. Exec from %s:%u.\n", ap=0x7ffd6bebfb50, mode_flags=0) at ../misc/syslog.c:233 now_tm = {tm_sec = 8, tm_min = 18, tm_hour = 12, tm_mday = 23, tm_mon = 0, tm_year = 124, tm_wday = 2, tm_yday = 22, tm_isdst = 0, tm_gmtoff = 0, tm_zone = 0x55a06e94c5e0 "UTC"} now = 1706012288 fd = <optimized out> f = 0x55a06e969a60 buf = 0x0 bufsize = 0 msgoff = 21 saved_errno = <optimized out> failbuf = "`\232\226n\240U\000\000\000\204\201\247[\214A\323`\374\353k\375\177\000\000\300p\371%T" clarg = {buf = <optimized out>, oldaction = <optimized out>} #3 0x00007f5425ec0c46 in __syslog (pri=<optimized out>, fmt=<optimized out>) at ../misc/syslog.c:117 ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7ffd6bebfc70, reg_save_area = 0x7ffd6bebfb70}} #4 0x000055a06e3b7839 in ?? () No symbol table info available. #5 0x000055a06e3bc039 in qm_free () No symbol table info available. #6 0x000055a06e3c7c28 in qm_shm_free () No symbol table info available. #7 0x00007f542325fb8e in uac_send_tm_callback () from /usr/lib/x86_64-linux-gnu/kamailio/modules/uac.so No symbol table info available. #8 0x00007f5424a2f002 in run_trans_callbacks_internal () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. #9 0x00007f5424a2f179 in run_trans_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. #10 0x00007f54249d5e8c in free_cell_helper () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. #11 0x00007f5424aa8f82 in wait_handler () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. #12 0x000055a06e37a263 in ?? () No symbol table info available. #13 0x000055a06e37a79d in ?? () No symbol table info available. #14 0x000055a06e37acc6 in timer_main () No symbol table info available. #15 0x000055a06e0a5f62 in main_loop () No symbol table info available. --Type <RET> for more, q to quit, c to continue without paging-- #16 0x000055a06e0b120c in main () No symbol table info available. (gdb) info locals No locals. (gdb) list 72 in ../sysdeps/x86_64/multiarch/strlen-evex.S ``` #### Log Messages ``` Log file: 2024-01-23T13:18:08.828+01:00 Jan 23 12:18:08 /usr/sbin/kamailio[4789]: INFO: <script>: Incoming SIP TCP request conid 21 call-id un0rihsRLJLvP-grn6LO-A 2024-01-23T13:18:08.835+01:00 Jan 23 12:18:08 /usr/sbin/kamailio[4789]: INFO: <script>: Incoming SIP TCP request conid 21 call-id WQjhldpRJbxjZRYe7fWgbw 2024-01-23T13:18:08.860+01:00 Jan 23 12:18:08 /usr/sbin/kamailio[4795]: CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 49 2024-01-23T13:18:09.486+01:00 Jan 23 12:18:09 /usr/sbin/kamailio[4751]: ALERT: <core> [main.c:783]: handle_sigs(): child process 4777 exited by a signal 11 2024-01-23T13:18:09.486+01:00 Jan 23 12:18:09 /usr/sbin/kamailio[4751]: ALERT: <core> [main.c:787]: handle_sigs(): core was generated 2024-01-23T13:18:09.516+01:00 Jan 23 12:18:09 /usr/sbin/kamailio[4751]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized 2024-01-23T13:18:09.570+01:00 Started /root/sipconfig/startkamailio.sh 2024-01-23T13:18:09.570+01:00 info: :-) Starting Kamailio ``` #### SIP Traffic ### Possible Solutions ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` # /usr/sbin/kamailio -v version: kamailio 5.6.5 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 10.2.1 ``` * **Operating System**: ``` # lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye Linux ip-nn-nn-nn-nn 5.10.0-25-cloud-amd64 #1 SMP Debian 5.10.191-1 (2023-08-16) x86_64 GNU/Linux ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3725 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/issues/3725(a)github.com>

4 8