<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="moz-cite-prefix">On 02.08.19 01:52, jay binks wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAO47HpcteMUuvf6rq+tJ8ORvPSFLQLEfTMWLGUguFKR9pqf9-g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Firstly, thankyou for fixing my missing
subject... oops :(</div>
<div dir="ltr"><br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>is the traffic sent over UDP?</p>
</div>
</blockquote>
<div>Yes the traffic is UDP, but im 100% sure the
traffic arrived at the server ( verified by TCPDump on
the server ).</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>If you see the traffic with ngrep. tcpdump, etc ... it still does
not mean it reaches the application layer, these tools hook into
network traffic callbacks and from there the kernel or firewall
can still drop the traffic and the application (kamailio) won't
get it. Might not be the case here, but that can happen.</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAO47HpcteMUuvf6rq+tJ8ORvPSFLQLEfTMWLGUguFKR9pqf9-g@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Can you look with netstat and see if the
receiving queue for the socket of kamailio is
buffering data?</p>
</div>
</blockquote>
<div> Sure, to avoid confusion, can you point me in the
right direction here, as im not familiar with this
metric.</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>If it is udp traffic, do:</p>
<p>netstat -alupn</p>
<p>Look for the lines of kamailio and the sip port (5060), something
like:</p>
<p>Proto Recv-Q Send-Q Local Address Foreign
Address State PID/Program name<br>
</p>
<p>udp 0 0 127.0.0.1:5060
0.0.0.0:* 4960/kamailio</p>
<p>If the Recv-Q value is high, then kamailio cannot handle the
traffic there in a timely manner, so some of them can get dropped
at some point, when the socket buffer is full.</p>
<p>Cheers,<br>
Daniel</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAO47HpcteMUuvf6rq+tJ8ORvPSFLQLEfTMWLGUguFKR9pqf9-g@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>What is the operating system? Do you have
contrack (kernel module) or selinux enabled in the
system?</p>
</div>
</blockquote>
<div>Debian 9, on a dual Xeon(R) Silver 4114 CPU @
2.20GHz ( 40 cores ).</div>
<div>32 gig of ram, GigE NIC.</div>
<div>No Selinux </div>
<div>following contract modules are loaded : </div>
</div>
</div>
</div>
</div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_conntrack_ipv6 20480 1</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_defrag_ipv6 20480 1
nf_conntrack_ipv6</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_conntrack_ipv4 16384 1</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_defrag_ipv4 16384 1
nf_conntrack_ipv4</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>xt_conntrack 16384 2</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_conntrack 114688 3
nf_conntrack_ipv6,nf_conntrack_ipv4,xt_conntrack</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>x_tables 36864 11
xt_comment,xt_multiport,ipt_REJECT,iptable_mangle,ip_tables,iptable_filter,ip6t_REJECT,ip6table_filter,xt_DSCP,xt_conntrack,ip6_tables</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Verify also that you don't have pike module
enabled with a low rate limit.</p>
</div>
</blockquote>
<div>I have removed pike, and replicated without pike
loaded.</div>
<div><br>
</div>
<div>Just for clarity sake, adding more workers 100%
fixed the issue.</div>
<div>What im really seeking is a way to monitor for this
scenario so I can graph it and not get caught out by
surprise.</div>
<div><br>
</div>
<div>In the last day or 2, ive found that some of the DB
Queries in my kam config were taking longer than they
should ( for a few reasons ), so this will be a largly
contributing factor, however ... it still comes back
to, how do I monitor for packets being dropped because
there was no worker.</div>
<div><br>
</div>
<div>Thanks all</div>
<div><br>
</div>
<div>Jay</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Cheers,<br>
Daniel<br>
</p>
<div
class="gmail-m_-3052619888151593434moz-cite-prefix">On
31.07.19 04:57, jay binks wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hey all,
<div>looking for a little help.</div>
<div><br>
</div>
<div>I've been tracking an issue, where a number
of SIP Messages ( typically INVITE ) are sent,
but Kamailio only see one of them ( the last
one). I have verified that the messages are
definitely received on the host, as verified
with TCPDUMP, but the INVITES never hit the
request route block. ( the very first thing is
to log the message )
<div>
<div><br>
</div>
<div>What i've managed to find is, that if I
increase worker children, the problem goes
away.</div>
<div><br>
</div>
<div>So I've got a problem, I think I've
found the solution, but what I'm
struggling with is how to monitor better,
so I can be alerted to this in the future.</div>
<div><br>
</div>
<div>I started by looking at "kamctl stats |
grep drop"</div>
"core:drop_replies = 14",<br>
"core:drop_requests = 10577"</div>
<div><br>
</div>
<div>however drop_requests seems to include
explicit drops in my config.</div>
<div>which I do for many reasons, but mainly
bad UA or blocked IP's, so this dosnt seem
to be what im after.</div>
<div><br>
</div>
<div>I then went digging in the code ( often
the best way to find things ).</div>
in receive.c receive_msg() we find some
conditions, where we drop the packet ( before
calling request route ).</div>
<div><br>
</div>
<div>right now im looking around line 251 at </div>
<div>LM_DBG("dropping the received message\n"); </div>
<div>goto error00;</div>
<div><br>
</div>
Following error00 I see it calls STATS_RX_DROPS
which increments stats->received_drops.
<div><font face="SFMono-Regular, Consolas,
Liberation Mono, Menlo, monospace"
color="#24292e"><span style="font-size:12px;white-space:pre-wrap">
</span></font></div>
This is where it gets fun, I cant find where
received_drops is ever returned in the stats
output.<br>
<div>
<div><br>
</div>
<div>I can see where -SIGUSR1 will trigger
this to be dumped, if Kamailio is compiled
with that option ( dosnt seem the debian
packages are ). however, what im seeking
clarification on is, does this get reported
in "kamctl stats"</div>
<div><br>
</div>
<div>Am I just going crazy or is something not
quite right here?</div>
<div><br>
</div>
<div>TLDNR; how do I monitor for Kamailio not
having enough worker threads to process
incoming messages.</div>
<div><br>
</div>
<div>FYI I'm seeing about 50Mbps of incoming
SIP on this box, and I do perform a decent
amount of work on some message types.</div>
<div><br>
</div>
<div>-- <br>
<div dir="ltr"
class="gmail-m_-3052619888151593434gmail_signature">Sincerely<br>
<br>
Jay</div>
</div>
</div>
</div>
<br>
<fieldset
class="gmail-m_-3052619888151593434mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-3052619888151593434moz-quote-pre">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="gmail-m_-3052619888151593434moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="gmail-m_-3052619888151593434moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">Sincerely<br>
<br>
Jay</div>
</div>
</div>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a></pre>
</body>
</html>