<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello Jay,</p>
<p>maybe this is interesting for you:</p>
<p><a class="moz-txt-link-freetext" href="https://www.kamailio.org/wiki/cookbooks/5.2.x/core#latency_limit_db">https://www.kamailio.org/wiki/cookbooks/5.2.x/core#latency_limit_db</a></p>
<p>You can log an error message if a DB operation from Kamailio takes a long time.</p>
<p>Cheers,</p>
<p>Henning<br>
</p>
<div class="moz-cite-prefix">Am 02.08.19 um 01:52 schrieb jay binks:<br>
</div>
<blockquote type="cite" cite="mid:CAO47HpcteMUuvf6rq+tJ8ORvPSFLQLEfTMWLGUguFKR9pqf9-g@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Firstly, thankyou for fixing my missing subject... oops :(</div>
<div dir="ltr"><br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>is the traffic sent over UDP?</p>
</div>
</blockquote>
<div>Yes the traffic is UDP, but im 100% sure the traffic arrived at the server ( verified by TCPDump on the server ).</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Can you look with netstat and see if the receiving queue for the socket of kamailio is buffering data?</p>
</div>
</blockquote>
<div> Sure, to avoid confusion, can you point me in the right direction here, as im not familiar with this metric.</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>What is the operating system? Do you have contrack (kernel module) or selinux enabled in the system?</p>
</div>
</blockquote>
<div>Debian 9, on a dual Xeon(R) Silver 4114 CPU @ 2.20GHz ( 40 cores ).</div>
<div>32 gig of ram, GigE NIC.</div>
<div>No Selinux </div>
<div>following contract modules are loaded : </div>
</div>
</div>
</div>
</div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_conntrack_ipv6 20480 1</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_defrag_ipv6 20480 1 nf_conntrack_ipv6</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_conntrack_ipv4 16384 1</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_defrag_ipv4 16384 1 nf_conntrack_ipv4</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>xt_conntrack 16384 2</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>nf_conntrack 114688 3 nf_conntrack_ipv6,nf_conntrack_ipv4,xt_conntrack</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<div class="gmail_quote">
<div>
<div>x_tables 36864 11 xt_comment,xt_multiport,ipt_REJECT,iptable_mangle,ip_tables,iptable_filter,ip6t_REJECT,ip6table_filter,xt_DSCP,xt_conntrack,ip6_tables</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Verify also that you don't have pike module enabled with a low rate limit.</p>
</div>
</blockquote>
<div>I have removed pike, and replicated without pike loaded.</div>
<div><br>
</div>
<div>Just for clarity sake, adding more workers 100% fixed the issue.</div>
<div>What im really seeking is a way to monitor for this scenario so I can graph it and not get caught out by surprise.</div>
<div><br>
</div>
<div>In the last day or 2, ive found that some of the DB Queries in my kam config were taking longer than they should ( for a few reasons ), so this will be a largly contributing factor, however ... it still comes back to, how do I monitor for packets being
dropped because there was no worker.</div>
<div><br>
</div>
<div>Thanks all</div>
<div><br>
</div>
<div>Jay</div>
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px
0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Cheers,<br>
Daniel<br>
</p>
<div class="gmail-m_-3052619888151593434moz-cite-prefix">On 31.07.19 04:57, jay binks wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hey all,
<div>looking for a little help.</div>
<div><br>
</div>
<div>I've been tracking an issue, where a number of SIP Messages ( typically INVITE ) are sent, but Kamailio only see one of them ( the last one). I have verified that the messages are definitely received on the host, as verified with TCPDUMP, but the INVITES
never hit the request route block. ( the very first thing is to log the message )
<div>
<div><br>
</div>
<div>What i've managed to find is, that if I increase worker children, the problem goes away.</div>
<div><br>
</div>
<div>So I've got a problem, I think I've found the solution, but what I'm struggling with is how to monitor better, so I can be alerted to this in the future.</div>
<div><br>
</div>
<div>I started by looking at "kamctl stats | grep drop"</div>
"core:drop_replies = 14",<br>
"core:drop_requests = 10577"</div>
<div><br>
</div>
<div>however drop_requests seems to include explicit drops in my config.</div>
<div>which I do for many reasons, but mainly bad UA or blocked IP's, so this dosnt seem to be what im after.</div>
<div><br>
</div>
<div>I then went digging in the code ( often the best way to find things ).</div>
in receive.c receive_msg() we find some conditions, where we drop the packet ( before calling request route ).</div>
<div><br>
</div>
<div>right now im looking around line 251 at </div>
<div>LM_DBG("dropping the received message\n"); </div>
<div>goto error00;</div>
<div><br>
</div>
Following error00 I see it calls STATS_RX_DROPS which increments stats->received_drops.
<div><font face="SFMono-Regular, Consolas,
Liberation Mono, Menlo, monospace" color="#24292e"><span style="font-size:12px;white-space:pre-wrap"></span></font></div>
This is where it gets fun, I cant find where received_drops is ever returned in the stats output.<br>
<div>
<div><br>
</div>
<div>I can see where -SIGUSR1 will trigger this to be dumped, if Kamailio is compiled with that option ( dosnt seem the debian packages are ). however, what im seeking clarification on is, does this get reported in "kamctl stats"</div>
<div><br>
</div>
<div>Am I just going crazy or is something not quite right here?</div>
<div><br>
</div>
<div>TLDNR; how do I monitor for Kamailio not having enough worker threads to process incoming messages.</div>
<div><br>
</div>
<div>FYI I'm seeing about 50Mbps of incoming SIP on this box, and I do perform a decent amount of work on some message types.</div>
<div><br>
</div>
<div>-- <br>
<div dir="ltr" class="gmail-m_-3052619888151593434gmail_signature">Sincerely<br>
<br>
Jay</div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_-3052619888151593434mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-3052619888151593434moz-quote-pre">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="gmail-m_-3052619888151593434moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="gmail-m_-3052619888151593434moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="gmail-m_-3052619888151593434moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">Sincerely<br>
<br>
Jay</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Henning Westerholt - <a class="moz-txt-link-freetext" href="https://skalatan.de/blog/">https://skalatan.de/blog/</a>
Kamailio services - <a class="moz-txt-link-freetext" href="https://skalatan.de/services">https://skalatan.de/services</a></pre>
</body>
</html>