<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Vorformatiert Zchn";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLVorformatiertZchn
{mso-style-name:"HTML Vorformatiert Zchn";
mso-style-priority:99;
mso-style-link:"HTML Vorformatiert";
font-family:Consolas;}
span.E-MailFormatvorlage21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US">additionally, try to see if you have some general I/O issues, e.g., by observing io-wait CPU, looking to I/O transaction times with monitoring tools etc..<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US">Maybe there are other services also causing excessive I/O.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US">Henning<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US">--
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US">Henning Westerholt –
</span><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://skalatan.de/blog/"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a></span><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US">Kamailio services –
</span><span style="font-size:11.0pt;mso-fareast-language:EN-US"><a href="https://gilawa.com/"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a></span><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:11.0pt;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> sr-users <sr-users-bounces@lists.kamailio.org>
<b>On Behalf Of </b>Daniel-Constantin Mierla<br>
<b>Sent:</b> Wednesday, September 14, 2022 2:25 PM<br>
<b>To:</b> Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>; Amit <amit@brytecall.com><br>
<b>Subject:</b> Re: [SR-Users] Log levels severely impacts performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<p style="margin-left:35.4pt">Hello,<span style="font-size:11.0pt"><o:p></o:p></span></p>
<p style="margin-left:35.4pt">excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.<o:p></o:p></p>
<p style="margin-left:35.4pt">Performance could be also a matter of how many child processes you created, via the children global parameter.<o:p></o:p></p>
<p style="margin-left:35.4pt">Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On 14.09.22 04:18, Amit wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">Hello,</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with
REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt;text-indent:36.0pt"><span style="font-size:11.0pt">Running ‘watch netstat -ulpn’ revealed the buffer issue.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">We usually run the production system at log level INFO (2) and had not ran into any issues in the past.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"><a href="http://www.kamailio.org/wiki/tutorials/3.2.x/syslog">http://www.kamailio.org/wiki/tutorials/3.2.x/syslog</a></span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to
INFO (2).</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">Thank you in advance.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt">Amit</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.0pt"><br>
<br>
<o:p></o:p></span></p>
<pre style="margin-left:35.4pt">__________________________________________________________<o:p></o:p></pre>
<pre style="margin-left:35.4pt">Kamailio - Users Mailing List - Non Commercial Discussions<o:p></o:p></pre>
<pre style="margin-left:35.4pt"> * <a href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a><o:p></o:p></pre>
<pre style="margin-left:35.4pt">Important: keep the mailing list in the recipients, do not reply only to the sender!<o:p></o:p></pre>
<pre style="margin-left:35.4pt">Edit mailing list options or unsubscribe:<o:p></o:p></pre>
<pre style="margin-left:35.4pt"> * <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></pre>
</blockquote>
<pre style="margin-left:35.4pt">-- <o:p></o:p></pre>
<pre style="margin-left:35.4pt">Daniel-Constantin Mierla -- <a href="http://www.asipto.com">www.asipto.com</a><o:p></o:p></pre>
<pre style="margin-left:35.4pt"><a href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
</div>
</body>
</html>