19 Dec
2024
19 Dec
'24
11:17 a.m.
Sergio, we run a very similar scenario in some places with subtle differences, our http
backend is in golang, it’s replicated and load balanced..
We achieve more than 1000 caps with a kind of similar config you’re posting, and also hit
the ’no answering threshold’
2 things/questions.
1. How long it takes for the http backend to answer your query? (GET/POST)?
2. How sure you are the backend is able to handle the rate you’re requesting? Did you
tried by itself to load stress that?
1. In our case, the backend is integrated with external apps (diameter, a web service,
etc) kamailio will wait for the http response, the more it takes to get the answer, the
more it waits the more resources are wasted (and kamailio is not guilty of that),
children start to “occupy” and will not handle invites or any (in our cases, not even
OPTIONS and sip trunks started to go down). We had to cache and modify the backend to
speed it up, we started to play with a children overbook (adding even more cpu’s) to
“gift” more resources to be occupied in the wait and make a false more capacity.
2. I thought the backend was able to handle it “it’s not the problem” but it was at some
point, as our backend is in a K8, increase the replicas helped a lot (after a fine code
tuning in the app first)
In this “scenario” we run 8 x 4 node (16vcpu) clusters and each cluster is around 2500
average caps each with a decent cpu consumption and no loss (very very very tiny loss
during peak hours which are ~4000 caps per cluster.
This is my comment based in our experience in a scenario like yours, doesn’t mean is the
correct or more adequate answer, just an idea where to check first.
Regards
On Dec 19, 2024, at 10:08 AM, Sergio Charrua via
sr-users <sr-users(a)lists.kamailio.org> wrote:
Hello!
for this ST/SH project, we are using Kamailio 5.8.4 in stateless mode, making HTTP
requests to a REST API (Java Spring) that will reply with a JSON object, and then Kamailio
replaces the Contact header and sends a SIP 300 Multiple Choice reply to the SBC that sent
the initial INVITE.
Kamailio is running on a VM with 4vCPUs and 4GB RAM and 2 NICs.
On NIC ens224 there is a Virtual IP managed by Keepalived.
Kamailio is listening on port 5060 on ens224 and set with 32 children process (tried with
8, 16, 24 and also 64, but with this value it was a lot worst).
We are doing load performance tests on our PreProd environment. With the SBC we are
sending 450 to 600 CAPS to Kamailio, and what we noticed is that above 450 CAPS, and after
less than 1 minute, Kamailio only replies SIP 100 to some INVITEs. We also could figure
out that Kamailio is not receiving all the INVITES, despite having proved that ALL invites
are sent to the server.
It seems to me that, for some reason, the OS is somehow not able to deliver *all* SIP
packets to Kamailio, because:
- sngrep does capture all the SIP packets and shows the flows
- though the flow shows SIP invite coming from SBC to Kamailio Server, there is no SIP
100 replied back to SBC, even though this is the 1st thing kamailio does, so, for me,
Kamailio did not received the SIP Invite
- picking a call with reinvites, which after some seconds, SBC cancels, if I copy the
Call-ID value and grep on kamailio's logs, I just do not find any content!
I have read some "network tuning" articles, including Linux Tune Network Stack
(Buffers Size) To Increase Networking Performance - nixCraft
<https://www.cyberciti.biz/faq/linux-tcp-tuning/> and Tuning Kamailio for high
throughput and performance | Evariste Systems Blog
<https://blog.evaristesys.com/2016/02/15/tuning-kamailio-for-high-throughput-and-performance/>
and when running the following command I get:
$ ss -4 -n -l | grep 5060
udp UNCONN 0 0 10.242.17.125:5060 <http://10.242.17.125:5060/>
0.0.0.0:*
udp UNCONN 324224 0 10.242.17.146:5060 <http://10.242.17.146:5060/>
0.0.0.0:*
According to the article, the 3rd column should be as near 0 as possible, ideally 0.
While I am waiting for some sysadmin with root permissions to be available and modify
some of the network parameters on this VM, I wonder if anyone has some tips to share on
how to solve this behaviour. I'm not sure, either, what parameters to change or if
they should be changed! The goal is to have a minimum 750 CAPS.
Lastly, some parts of the code:
Kamailio.cfg:
### LOG Levels: 3=DBG, 2=INFO, 1=NOTICE, 0=WARN,
-1=ERR
debug=1
log_stderror=no
rundir="/tmp"
memdbg=5
memlog=5
log_facility=LOG_LOCAL0
# configure the prefix for all log messages
log_prefix_mode = 1
log_prefix="{$mt $hdr(CSeq) $ci} "
/* number of SIP routing processes */
children=32 # ------------- ALREADY TRIED WITH 8,16,24... no results.....
/* uncomment the next line to disable TCP (default on) */
disable_tcp=no
/* uncomment the next line to disable the auto discovery of local aliases
* based on reverse DNS on IPs (default on) */
# auto_aliases=no
/* add local domain aliases */
# alias="sip.mydomain.com <http://sip.mydomain.com/>"
####### Custom Parameters #########
/* These parameters can be modified runtime via RPC interface
* - see the documentation of 'cfg_rpc' module.
*
* Format: group.id <http://group.id/> = value 'desc' description
* Access: $sel(cfg_get.group.id <http://cfg_get.group.id/>) or @cfg_get.group.id
<http://cfg_get.group.id/> */
####### Modules Section ########
/* set paths to location of modules */
loadmodule "db_mysql.so"
loadmodule "db_cluster.so"
loadmodule "http_client.so"
loadmodule "jsonrpcs.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "sl.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "jansson.so"
loadmodule "snmpstats.so"
loadmodule "file_out.so"
loadmodule "ctl.so"
loadmodule "permissions.so"
loadmodule "xhttp.so"
loadmodule "xhttp_rpc.so"
####### Other Args and Env Vars #########
/* listen addresses */
include_file "listen.cfg"
include_file "db_conn.cfg"
# ----------------- setting module-specific parameters --------------
# --- DB Cluster params ---
# minimum requirement is to have DBURL1 defined
#!ifdef DBURL1
modparam("db_cluster", "connection" , DBURL1)
#!endif
#!ifdef DBURL2
modparam("db_cluster", "connection" , DBURL2)
#!endif
#!ifdef DBURL3
modparam("db_cluster", "connection" , DBURL3)
#!endif
#!ifdef DBURL4
modparam("db_cluster", "connection" , DBURL4)
#!endif
#!ifdef DBURL5
modparam("db_cluster", "connection" , DBURL5)
#!endif
modparam("db_cluster", "cluster", DBCLUSTER)
# --- Permissions params ---
modparam("permissions", "db_url", "cluster://k1")
modparam("permissions", "db_mode", 1)
#modparam("permissions", "reload_delta", 30)
# ----- jsonrpcs params -----
modparam("jsonrpcs", "pretty_format", 1)
/* set the path to RPC fifo control file */
modparam("jsonrpcs", "fifo_name",
"/tmp/kamailio_rpc.fifo")
/* set the path to RPC unix socket control file */
modparam("jsonrpcs", "dgram_socket",
"/tmp/kamailio_rpc.sock")
modparam("jsonrpcs", "transport", 0)
# ----- ctl params -----
/* set the path to RPC unix socket control file */
modparam("ctl", "binrpc", "unix:/tmp/kamailio_ctl")
# ----- http_async_client params -----
modparam("http_client", "query_result", 0)
modparam("http_client", "keep_connections", 0)
modparam("http_client", "connection_timeout",2)
modparam("http_client", "timeout_mode",2)
modparam("http_client", "config_file",
"/usr/local/etc/kamailio/http_client.cfg")
# ---- SNMP Stats params ----
modparam("snmpstats", "sipEntityType", "proxyServer")
modparam("snmpstats", "sipEntityType", "redirectServer")
modparam("snmpstats", "sipEntityType", "other")
modparam("snmpstats", "snmpgetPath", "/usr/bin/")
modparam("snmpstats", "MsgQueueMinorThreshold", 20)
modparam("snmpstats", "MsgQueueMajorThreshold", 100)
modparam("snmpstats", "dlg_minor_threshold", 20)
modparam("snmpstats", "dlg_major_threshold", 100)
modparam("snmpstats", "snmpCommunity", "kamailio")
# ---- File_Out params ----
modparam("file_out", "base_folder",
"/data/sabire002/log/kamailio/")
modparam("file_out", "file",
"name=stsh_requests;interval=1440;extension=.log")
modparam("file_out", "file",
"name=cdr;interval=1440;extension=.log")
modparam("file_out", "file",
"name=http;interval=1440;extension=.log")
####### Routing Logic ########
include_file "includes/reqinit.cfg"
include_file "includes/handle_options.cfg"
include_file "includes/handle_cancel.cfg"
include_file "includes/handle_stir_shaken_stateless.cfg"
include_file "includes/handle_http_rpc.cfg"
request_route {
$avp(START_TIME)=$utimef(%Y-%m-%d %H:%M:%S);
$avp(GROUPID) = allow_address_group($si, $sp);
if ( $avp(GROUPID) == 100 || !allow_address_group($si, $sp) ) {
xlog("L_INFO", "INIT - $si:$sp is not in the allowed ACL Group
ID !\n");
#sl_reply("401", "Address not allowed");
exit;
};
if (is_method("ACK") ) { #&& t_check_trans() ){
exit;
}
if (is_method("INVITE")) {
file_out("cdr","$rm|$ft|$tt|$ci|$rs|$rr|$Ts|$avp(START_TIME)|$fU|$fd|$si|$tU|$rU|$rd|$utimef(%Y-%m-%d
%H:%M:%S)");
send_reply("100","Trying");
}
route(HANDLE_OPTIONS);
route(REQINIT);
xlog("L_INFO"," ********** Route START ***********");
# log the basic info regarding this call
xlog("L_INFO","===================================================
\n");
xlog("L_INFO","New SIP message $rm with call-ID $ci received $pr request
$rm $ou source $si:$sp from $fu to $tu \n");
xlog("L_INFO","===================================================
\n");
route(HANDLE_STIRSHAKEN);
route(HANDLE_CANCEL);
if (method == "INVITE"){
route(RELAY);
}
}
route[RELAY] {
# Sends a 300 Multiple Choices back to the proxy that requested the routing lookup
xlog("L_INFO","RELAY - send reply \n");
file_out("cdr","RELAY|$ft|$tt|$ci|$rs|$rr|$Ts|$avp(START_TIME)|$fU|$fd|$si|$tU|$rU|$rd|$utimef(%Y-%m-%d
%H:%M:%S)");
send_reply("300", "Multiple Choices");
exit;
}
As for the HANDLE_STIRSHAKEN route, where the main process is:
route[HANDLE_STIRSHAKEN]
{
if (!is_method("INVITE")) {
return;
}
$xavp(requestTime) = $utimef(%s);
$var(post) = $null; // resets and makes sure the $var(post) variable is null before
usage
[.... get some header values ...]
$var(res) = http_connect("api1", "/stsh",
"application/json", "$var(post)", "$var(result)");
jansson_xdecode($var(result), "json");
if ( $xavp(json=>sip-response-code) == 300 )
{
remove_hf("Contact");
append_to_reply("Contact: $xavp(json=>contact)\r\n");
}
else
{
# add error message to reply
sl_reply("$xavp(json=>sip-response-code)","$xavp(json=>sip-response-text)");
exit;
}
}
Atenciosamente / Kind Regards / Cordialement / Un saludo,
Sérgio Charrua
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions --
sr-users(a)lists.kamailio.org
To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!