<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Firstly, thankyou for fixing my missing subject... oops :(</div><div dir="ltr"><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><p>is the traffic sent over UDP?</p></div></blockquote><div>Yes the traffic is UDP, but im 100% sure the traffic arrived at the server ( verified by TCPDump on the server ).</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>Can you look with netstat and see if the receiving queue for the socket of kamailio is buffering data?</p></div></blockquote><div> Sure, to avoid confusion, can you point me in the right direction here, as im not familiar with this metric.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>What is the operating system? Do you have contrack (kernel module) or selinux enabled in the system?</p></div></blockquote><div>Debian 9, on a dual Xeon(R) Silver 4114 CPU @ 2.20GHz ( 40 cores ).</div><div>32 gig of ram, GigE NIC.</div><div>No Selinux </div><div>following contract modules are loaded : </div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div><div class="gmail_quote"><div><div>nf_conntrack_ipv6 20480 1</div></div></div></div></div></div><div><div><div><div class="gmail_quote"><div><div>nf_defrag_ipv6 20480 1 nf_conntrack_ipv6</div></div></div></div></div></div><div><div><div><div class="gmail_quote"><div><div>nf_conntrack_ipv4 16384 1</div></div></div></div></div></div><div><div><div><div class="gmail_quote"><div><div>nf_defrag_ipv4 16384 1 nf_conntrack_ipv4</div></div></div></div></div></div><div><div><div><div class="gmail_quote"><div><div>xt_conntrack 16384 2</div></div></div></div></div></div><div><div><div><div class="gmail_quote"><div><div>nf_conntrack 114688 3 nf_conntrack_ipv6,nf_conntrack_ipv4,xt_conntrack</div></div></div></div></div></div><div><div><div><div class="gmail_quote"><div><div>x_tables 36864 11 xt_comment,xt_multiport,ipt_REJECT,iptable_mangle,ip_tables,iptable_filter,ip6t_REJECT,ip6table_filter,xt_DSCP,xt_conntrack,ip6_tables</div></div></div></div></div></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>Verify also that you don't have pike module enabled with a low rate limit.</p></div></blockquote><div>I have removed pike, and replicated without pike loaded.</div><div><br></div><div>Just for clarity sake, adding more workers 100% fixed the issue.</div><div>What im really seeking is a way to monitor for this scenario so I can graph it and not get caught out by surprise.</div><div><br></div><div>In the last day or 2, ive found that some of the DB Queries in my kam config were taking longer than they should ( for a few reasons ), so this will be a largly contributing factor, however ... it still comes back to, how do I monitor for packets being dropped because there was no worker.</div><div><br></div><div>Thanks all</div><div><br></div><div>Jay</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<p>Cheers,<br>Daniel<br>
</p>
<div class="gmail-m_-3052619888151593434moz-cite-prefix">On 31.07.19 04:57, jay binks wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hey all,<div>looking for a little help.</div>
<div><br>
</div>
<div>I've been tracking an issue, where a number of SIP Messages ( typically INVITE ) are sent, but Kamailio only see one of them ( the last one). I have verified that the messages are definitely received on the host, as verified with TCPDUMP, but the INVITES never hit the request route block. ( the very first thing is to log the message )<div>
<div><br>
</div>
<div>What i've managed to find is, that if I increase worker children, the problem goes away.</div>
<div><br>
</div>
<div>So I've got a problem, I think I've found the solution, but what I'm struggling with is how to monitor better, so I can be alerted to this in the future.</div>
<div><br>
</div>
<div>I started by looking at "kamctl stats | grep drop"</div> "core:drop_replies = 14",<br> "core:drop_requests = 10577"</div>
<div><br>
</div>
<div>however drop_requests seems to include explicit drops in my config.</div>
<div>which I do for many reasons, but mainly bad UA or blocked IP's, so this dosnt seem to be what im after.</div>
<div><br>
</div>
<div>I then went digging in the code ( often the best way to find things ).</div>in receive.c receive_msg() we find some conditions, where we drop the packet ( before calling request route ).</div>
<div><br>
</div>
<div>right now im looking around line 251 at </div>
<div>LM_DBG("dropping the received message\n"); </div>
<div>goto error00;</div>
<div><br>
</div>Following error00 I see it calls STATS_RX_DROPS which increments stats->received_drops.<div><font face="SFMono-Regular, Consolas, Liberation Mono,
Menlo, monospace" color="#24292e"><span style="font-size:12px;white-space:pre-wrap">
</span></font></div>This is where it gets fun, I cant find where received_drops is ever returned in the stats output.<br>
<div>
<div><br>
</div>
<div>I can see where -SIGUSR1 will trigger this to be dumped, if Kamailio is compiled with that option ( dosnt seem the debian packages are ). however, what im seeking clarification on is, does this get reported in "kamctl stats"</div>
<div><br>
</div>
<div>Am I just going crazy or is something not quite right here?</div>
<div><br>
</div>
<div>TLDNR; how do I monitor for Kamailio not having enough worker threads to process incoming messages.</div>
<div><br>
</div>
<div>FYI I'm seeing about 50Mbps of incoming SIP on this box, and I do perform a decent amount of work on some message types.</div>
<div><br>
</div>
<div>-- <br>
<div dir="ltr" class="gmail-m_-3052619888151593434gmail_signature">Sincerely<br>
<br>Jay</div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_-3052619888151593434mimeAttachmentHeader"></fieldset> <pre class="gmail-m_-3052619888151593434moz-quote-pre">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a class="gmail-m_-3052619888151593434moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="gmail-m_-3052619888151593434moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Sincerely<br><br>Jay</div></div></div></div></div>