2 Aug
2019
2 Aug
'19
6:22 p.m.
On 02.08.19 01:52, jay binks wrote:
Firstly, thankyou for fixing my missing subject...
oops :(
is the traffic sent over UDP?
Yes the traffic is UDP, but im 100% sure the traffic arrived at the
server ( verified by TCPDump on the server ).
If you see the traffic with ngrep. tcpdump, etc ... it still does not
mean it reaches the application layer, these tools hook into network
traffic callbacks and from there the kernel or firewall can still drop
the traffic and the application (kamailio) won't get it. Might not be
the case here, but that can happen.
Can you look with netstat and see if the receiving queue for the
socket of kamailio is buffering data?
Sure, to avoid confusion, can you point me in the right direction
here, as im not familiar with this metric.
If it is udp traffic, do:
netstat -alupn
Look for the lines of kamailio and the sip port (5060), something like:
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
udp 0 0 127.0.0.1:5060
0.0.0.0:* 4960/kamailio
If the Recv-Q value is high, then kamailio cannot handle the traffic
there in a timely manner, so some of them can get dropped at some point,
when the socket buffer is full.
Cheers,
Daniel
What is the operating system? Do you have contrack
(kernel module)
or selinux enabled in the system?
Debian 9, on a dual Xeon(R) Silver 4114 CPU @ 2.20GHz ( 40 cores ).
32 gig of ram, GigE NIC.
No Selinux
following contract modules are loaded :
nf_conntrack_ipv6 20480 1
nf_defrag_ipv6 20480 1 nf_conntrack_ipv6
nf_conntrack_ipv4 16384 1
nf_defrag_ipv4 16384 1 nf_conntrack_ipv4
xt_conntrack 16384 2
nf_conntrack 114688 3
nf_conntrack_ipv6,nf_conntrack_ipv4,xt_conntrack
x_tables 36864 11
xt_comment,xt_multiport,ipt_REJECT,iptable_mangle,ip_tables,iptable_filter,ip6t_REJECT,ip6table_filter,xt_DSCP,xt_conntrack,ip6_tables
Verify also that you don't have pike module enabled with a low
rate limit.
I have removed pike, and replicated without pike loaded.
Just for clarity sake, adding more workers 100% fixed the issue.
What im really seeking is a way to monitor for this scenario so I can
graph it and not get caught out by surprise.
In the last day or 2, ive found that some of the DB Queries in my kam
config were taking longer than they should ( for a few reasons ), so
this will be a largly contributing factor, however ... it still comes
back to, how do I monitor for packets being dropped because there was
no worker.
Thanks all
Jay
Cheers,
Daniel
On 31.07.19 04:57, jay binks wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Hey all,
looking for a little help.
I've been tracking an issue, where a number of SIP Messages (
typically INVITE ) are sent, but Kamailio only see one of them (
the last one). I have verified that the messages are definitely
received on the host, as verified with TCPDUMP, but the INVITES
never hit the request route block. ( the very first thing is to
log the message )
What i've managed to find is, that if I increase worker children,
the problem goes away.
So I've got a problem, I think I've found the solution, but what
I'm struggling with is how to monitor better, so I can be alerted
to this in the future.
I started by looking at "kamctl stats | grep drop"
"core:drop_replies = 14",
"core:drop_requests = 10577"
however drop_requests seems to include explicit drops in my config.
which I do for many reasons, but mainly bad UA or blocked IP's,
so this dosnt seem to be what im after.
I then went digging in the code ( often the best way to find
things ).
in receive.c receive_msg() we find some conditions, where we drop
the packet ( before calling request route ).
right now im looking around line 251 at
LM_DBG("dropping the received message\n");
goto error00;
Following error00 I see it calls STATS_RX_DROPS which increments
stats->received_drops.
This is where it gets fun, I cant find where received_drops is
ever returned in the stats output.
I can see where -SIGUSR1 will trigger this to be dumped, if
Kamailio is compiled with that option ( dosnt seem the debian
packages are ). however, what im seeking clarification on is,
does this get reported in "kamctl stats"
Am I just going crazy or is something not quite right here?
TLDNR; how do I monitor for Kamailio not having enough worker
threads to process incoming messages.
FYI I'm seeing about 50Mbps of incoming SIP on this box, and I do
perform a decent amount of work on some message types.
--
Sincerely
Jay
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
--
Sincerely
Jay