sr-users October 2023

sr-users@lists.kamailio.org

47 participants
47 discussions

503(service unavailable)
by masoud96＠mail.ru 16 Oct '23

16 Oct '23

Hello! It often happens to me that the service begins to refuse service (503) and so far I have not been able to understand the exact reason. Typically, before clients start receiving 503, many before them receive a 408 code. As soon as the service starts responding with 503, it does not recover on its own, you have to restart. There is no way to understand from the logs what led to this (nothing unusual). My server is powerful enough for the number of users of my service (what I mean is that there is no problem with resources). For example, 64 cores, 125GB of DDR4 RAM, 2TB of disk, of which 1TB SSD for the database, 10Gigabit channel, this is only a server for Kamailio for Rtpengines separate servers. Please help me figure out what causes a service failure!? OS: Debian 11, 64 cores, 125GB of DDR4 RAM; DB: MariaDB 10.5.15; kamailio -v version: kamailio 5.5.3 (x86_64/linux) 473cef flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. udp_children: 30; tcp_children: 34; TLS: YES; async_workers: 64. Incoming calls are sent via push notifications( Federico Cabiddu method: https://www.voztovoice.org/sites/default/files/KamilioWorld2015%20-Federico…). NetBridging(for SIP and RTPEngine). ims_charging for billing (integration with our billing system using the Diameter protocol). In the logs exactly at the moment when the service began to refuse, the recording disappears completely for 7 minutes, absolutely no entries, after which the recording of the TCP queue full begins. Oct 3 18:38:07 sip1-life3 kamailio[3380553]: CRITICAL: {2 3691 INVITE 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:38:09 sip1-life3 kamailio[3380539]: CRITICAL: {2 3691 INVITE 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:38:12 sip1-life3 kamailio[3380534]: CRITICAL: {2 30106 INVITE 032f8650-1d90-49ed-82f1-c50ccbef191d} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:45:14 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 285 requests queued (total handled 2899418) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 286 requests queued (total handled 2964063) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 286 requests queued (total handled 2923484) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 286 requests queued (total handled 2911633) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 286 requests queued (total handled 2907132) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 286 requests queued (total handled 2903281) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 286 requests queued (total handled 2903438) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 286 requests queued (total handled 2899419) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 286 requests queued (total handled 2897842) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 286 requests queued (total handled 2895429) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 286 requests queued (total handled 2895303) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 22, socket 246: queue full, 286 requests queued (total handled 2891873) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 26, socket 254: queue full, 286 requests queued (total handled 2892619) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 27, socket 256: queue full, 286 requests queued (total handled 2891769) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry 0, retries 98, connection 0x7f0b6d974348, tcp so Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 30, socket 262: queue full, 286 requests queued (total handled 2891614) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 31, socket 264: queue full, 286 requests queued (total handled 2890778) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 33, socket 268: queue full, 286 requests queued (total handled 2890852) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 0, socket 202: queue full, 287 requests queued (total handled 3071438) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 287 requests queued (total handled 2964064) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 2, socket 206: queue full, 287 requests queued (total handled 2944928) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 287 requests queued (total handled 2923485) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 5, socket 212: queue full, 287 requests queued (total handled 2915778) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 287 requests queued (total handled 2911634) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 287 requests queued (total handled 2907133) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 287 requests queued (total handled 2903282) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 287 requests queued (total handled 2903439) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 10, socket 222: queue full, 287 requests queued (total handled 2900227) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 287 requests queued (total handled 2899420) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 287 requests queued (total handled 2897843) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 204 , queue entry 0, retries 88, connection 0x7f0b742a1968, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 13, socket 228: queue full, 287 requests queued (total handled 2896210) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 210 , queue entry 0, retries 86, connection 0x7f0b74ffaa08, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 287 requests queued (total handled 2895430) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 214 , queue entry 0, retries 88, connection 0x7f0b75003218, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 216 , queue entry 1, retries 86, connection 0x7f0b75007620, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 218 , queue entry 2, retries 84, connection 0x7f0b74772f70, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 287 requests queued (total handled 2895304) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 220 , queue entry 0, retries 87, connection 0x7f0b74777378, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry 1, retries 83, connection 0x7f0b7477fb88, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 21, socket 244: queue full, 287 requests queued (total handled 2893042)

4 7

183 Session Progress Acknowledgment
by Ali Taher 14 Oct '23

14 Oct '23

Hello, Is there a way to send PRACK as reply to a 183 Session Progress (with Require: 100rel) sent to Kamailio sip proxy? Regards,

2 3

Kamailio as SIP proxy
by Ali Taher 12 Oct '23

12 Oct '23

Hello, I'm using Kamailio as SIP proxy between Session Manager and SBC So here is the scenario : Session Manager sends a call (Invite) to Kamailio, kamailio needs to relay it to SBC after doing some modification. (-----Session Manager-----) ----------- > (----Kamailio----) ----------- > (-----SBC-----) My routing logic is similar to the below: route{ if ((method==OPTIONS) && (! uri=~"sip:.*[@]+.*")) { options_reply(); } if (is_method("INVITE")) { sql_query("cb", "select number from pool1 order by random() limit 1", "ra"); $var(rand)=$dbr(ra=>[0,0]); uac_replace_from($var(rand),"sip:$var(rand)@192.168.1.1"); xlog("L_INFO","Random: $var(rand)"); $td = "192.168.1.10"; $rd = "192.168.1.10"; t_relay(); } } Where: 192.168.1.1 is Kamailio IP 192.168.1.10 is SBC IP The issue is that I'm not sure to handle the other SIP packets like 183,CANCEL, BYE etc that can be received from both SBC and session Manager? What to add to my configuration to handle it properly? Regards,

2 1

Using DMQ to sync the TM module.
by Michel Pelletier 11 Oct '23

11 Oct '23

Hi, I have 2 kamailio instances behind a load balancer. The problem I have is that the load balancer can only track TCP connections, but not UDP. So one Kamailio instance might send a request using UDP, while the corresponding UDP reply arrives on the other. This doesn't play well with the (stateful) TM module. Is there a way to synchronize the TM module accross Kamailio instances using DMQ? Cheers, Michel Pelletier

4 8

rtpproxy -- timeout_socket
by James Lipski 11 Oct '23

11 Oct '23

Hello, I'm a bit confused about a piece of the documentation. I'm using rtpproxy 3.0.1 + rtpproxy module on kamailio 5.6.4. On the rtpproxy module, I'm trying to get the "timeout_socket" param working. From the documentation, I have modparam("rtpproxy", "timeout_socket", "xmlrpc:http://192.168.155.64:8888/XMLRPC") modparam("xmlrpc", "route", "XMLRPC_ROUTE") modparam("xmlrpc", "mode", 1) modparam("xmlrpc", "url_match", "^/XMLRPC") /usr/bin/rtpproxy -i -u rtpproxy -n tcp:192.168.155.64:8888 -l 192.168.155.64 -s udp:127.0.0.1:7333 -m 29152 -M 49150 -d DBUG However when purposely trying to timeout RTP, I'm seeing the following in my rtpproxy logs: Oct 10 00:31:22 localhost rtpproxy[3802129]: DBUG:GLOBAL:rtpp_command_split:401: received command "3801518_4 USc9,0,8,18,101 219b2465decc-n7al36xva4lw 192.168.155.168 2266 C60D788E-8E14A9FA;1 w6p2ke9ttk;1 xmlrpc:http://192.168.155.64:8888/XMLHTTP" Oct 10 00:31:22 localhost rtpproxy[3802129]: ERR:219b2465decc-n7al36xva4lw:rtpp_command_ul_handle:603: invalid socket name w6p2ke9ttk;1 I'm just wondering if I'm missing anything, do I need to define "listen=tcp:192.168.155.64:8888" in my configs? I know that rtpengine is recommended/ I would like to experiement with rtpproxy. Do I need a specific version of rtpproxy in order for this to work -- I did try it with 2.1.1 however that didn't work. Thanks.

2 2

Kamailio Developers Meeting, Nov 7-8, 2023, in Dusseldorf
by Daniel-Constantin Mierla 11 Oct '23

11 Oct '23

Hello, Kamailio SIP Server project is organizing another meeting of its developers and community members during November 7-8, 2023, hosted again by sipgate.de in Dusseldorf, Germany. The event is intended to facilitate the interaction between Kamailio developers and contributors in order to offer a convenient environment for working together on several topics of high interest for the project, including writing code for Kamailio and its tools, improving documentation, or discuss about future development. Everyone from the community is welcome to join, developer or user interested in helping the project. Please note we have a limited capacity of seats in the meeting room, the main policy for accepting participants being first come first server. Also, very important to be aware that this is not an event to learn how to use Kamailio. More details about the event, the venue, how to register, are available at: * https://www.kamailio.org/w/developers-meeting/ Looking forward to those two intensive hacking Kamailio days in Dusseldorf! Cheers, Daniel -- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy and Development Services Kamailio Advanced Training - Online - Nov 14-16, 2023 -- asipto.com

1 0

Error reporting when developing a custom module to link to a third-party library
by bikq＠163.com 11 Oct '23

11 Oct '23

Dear friends, I have a question: When adding a custom module in kamailio, I referenced a third-party library. There was no problem with compilation, but when I ran it, it reported that the third-party library could not be found. What is the reason? The current Makefile is as follows： ``` # WARNING: do not run this directly, it should be run by the main Makefile include ../../Makefile.defs auto_gen= NAME=rocketmq.so THIRDPARTY=/mnt/vdd/kqbi2/3rdparty ROCKETMQ_ROOT_PATH=${THIRDPARTY}/rocketmq CXX?=g++ LD?=g++ LIBS+=-L$(LOCALBASE)/lib -L$(ROCKETMQ_ROOT_PATH)/lib/linux/x64 crocketmq.o -lstdc++ -lrocketmq DEFS+=-I$(LOCALBASE)/include -I${ROCKETMQ_ROOT_PATH}/include CXXFLAGS=$(CFLAGS:-Wno-deprecated option=) CXXFLAGS+=-Wno-write-strings -Wno-deprecated -Wno-unused-function -Wno-sign-compare -Wno-strict-aliasing include ../../Makefile.modules crocketmq.o: crocketmq.cpp crocketmq.h @echo "Compiling $<" $(CXX) -std=c++11 $(CXXFLAGS) $(CFLAGS) $(C_DEFS) $(DEFS) -c $< -o $@ rocketmq.so: crocketmq.o modclean: -@rm -f crocketmq.o crocketmq.d rocketmq_mod.o rocketmq_mod.d clean clean-modules: modclean proper realclean distclean maintainer-clean: modclean ``` There is no problem in compiling, but when running, the third-party librocketmq.so library cannot be found, and various export LD_LIBRARY_LIBRARY also does not work. ``` 0(13229) DEBUG: <core> [core/cfg.y:2053]: yyparse(): loading module rocketmq.so 0(13229) DEBUG: <core> [core/sr_module.c:516]: ksr_locate_module(): found module to load <lib64/kamailio/modules/rocketmq.so> 0(13229) DEBUG: <core> [core/sr_module.c:573]: ksr_load_module(): trying to load <lib64/kamailio/modules/rocketmq.so> 0(13229) ERROR: <core> [core/sr_module.c:591]: ksr_load_module(): could not open module <lib64/kamailio/modules/rocketmq.so>: librocketmq.so: cannot open shared object file: No such file or directory 0(13229) CRITICAL: <core> [core/cfg.y:3900]: yyerror_at(): parse error in config file /mnt/vdd/kqbi2/install/kamailio/etc/kamailio/kamailio.cfg, line 380, column 12-24: failed to load module ```

1 2

Serial Forking with differing next hop for each branch?
by Barry Flanagan 10 Oct '23

10 Oct '23

Hi I have a scenario where I need to implement serial forking but each destination has also a next hop proxy. I can do the forking using append_branch, but how can I set the destination URI for each branch? In some cases the R-URI does not change, only the next hop proxy. Example: R-URI Next Hop user1(a)domain.com XXX.XXX.XXX.XXX user1(a)domain.com YYY.YYY.YYY.YYY user2(a)domain.com XXX.XXX.XXX.XXX I was thinking I could use a htable to store the next hop using the R-URI as the key, but as can be seen above there are possibly duplicate R-URIs with only the next hop changing. Thanks. -- -Barry

4 18

0.0.0.0 in VIA header
by Lewis Hutchinson 10 Oct '23

10 Oct '23

Morning What have you got your listen / advertise address / port set as? See this link > https://www.kamailio.org/wiki/cookbooks/5.5.x/core#advertised_address I'd probably start by looking at that. Lewis Mission Labs Limited is registered in England, company number 10040088. Trading Office: The Old Milk Depot, Bacup Rd, Rossendale, BB4 7FE. Registered office: The Scalpel, 18th Floor 52 Lime Street, London, EC3M 7AF. Email confidentiality notice: This message is private and confidential. If you have received this message in error, please notify us and remove it from your system. Please consider the environment before you print this email.

1 0

0.0.0.0 in VIA header
by Ali Taher 10 Oct '23

10 Oct '23

Hello, I'm using Kamailio as SIP proxy to receive INVITE packets, randomize A-number and then relay it to a certain gateway. I noticed that a Via header is added to the INVITE sent by Kamailio to the gateway with 0.0.0.0 as address as shown below: Via: SIP/2.0/UDP 0.0.0.0;branch=z9hG4bK2d97.61de75dd58fa7ca7450cc8b9b01db62f.0 As far as I know, the IP that should be used in the new added VIA header must be Kamailio IP, but it's not the case here, knowing that Kamailio is not behind NAT. Regards, Ali Taher

2 1

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

sr-users October 2023