<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
Hi Henning,
<div>That’s just it. We don’t see any IO wait increases during the test case. I’ve only seen it briefly go to %0.13 and then right back to %0.00</div>
<div><br>
</div>
<div>I heard there are ways to time or benchmark different parts of the config. Can you point me in the right direction to perform some logging benchmark tests?</div>
<div>Amit<br>
<br>
<div dir="ltr">
<blockquote type="cite"><br>
On Sep 16, 2022, at 8:32 AM, Henning Westerholt <hw@gilawa.com> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style>@font-face { font-family: "Cambria Math"; }
@font-face { font-family: Calibri; }
p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif; }
a:link, span.MsoHyperlink { color: blue; text-decoration: underline; }
span.E-MailFormatvorlage20 { font-family: Calibri, sans-serif; color: windowtext; }
.MsoChpDefault { font-size: 10pt; }
@page WordSection1 { size: 612pt 792pt; margin: 72pt; }
div.WordSection1 { page: WordSection1; }</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">then have a look to the server stats and IO load during the test case when you observe the particular blocking issues. You should be able to see something then, if its caused from the
Kamailio logging to syslog.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">-- <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning Westerholt –
</span><span style="mso-fareast-language:EN-US"><a href="https://skalatan.de/blog/"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a></span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Kamailio services –
</span><span style="mso-fareast-language:EN-US"><a href="https://gilawa.com/"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a></span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b> sr-users <sr-users-bounces@lists.kamailio.org>
<b>On Behalf Of </b>Amit<br>
<b>Sent:</b> Friday, September 16, 2022 2:14 PM<br>
<b>To:</b> Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org><br>
<b>Subject:</b> Re: [SR-Users] Log levels severely impacts performance<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Hi Alex,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Thanks for your suggestions.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Yes we’ve tested the underlying storage. We specifically upgraded drives on the particular testing server I am using at the moment, and are using a ZFS volume.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">But to answer your specific questions,
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"># hdparm -t /dev/sda2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">/dev/sda2:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Timing buffered disk reads: 4118 MB in 3.00 seconds = 1372.43 MB/sec<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">Also, %iowait remains at 0. I did see it briefly go to %0.13 for one cycle when I sent 2000 simultaneous registrations to Kam.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US">It doesn’t appear to be the disk subsystem.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US" style="font-size:10.5pt;color:black">Amit</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:35.4pt">
<b><span lang="EN-US" style="font-size:12.0pt;color:black">From: </span></b><span lang="EN-US" style="font-size:12.0pt;color:black">sr-users <<a href="mailto:sr-users-bounces@lists.kamailio.org">sr-users-bounces@lists.kamailio.org</a>> on behalf of Alex Balashov
<<a href="mailto:abalashov@evaristesys.com">abalashov@evaristesys.com</a>><br>
<b>Date: </b>Thursday, September 15, 2022 at 11:00 PM<br>
<b>To: </b>Kamailio (SER) - Users Mailing List <<a href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>><br>
<b>Subject: </b>Re: [SR-Users] Log levels severely impacts performance<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-US"><br>
> On Sep 15, 2022, at 10:54 PM, Amit <<a href="mailto:amit@brytecall.com">amit@brytecall.com</a>> wrote:<br>
> <br>
> I’ve already added the “-“ in front of the log file and that didn’t seem to make any recognizable difference.<br>
<br>
That’s the only place that I know.<br>
<br>
Have you tested your storage throughput? Even something as simple as `hdparm -t /dev/sda`[1]?
<br>
<br>
What about monitoring background I/O demand, i.e. `iostat -x 1`? Look at the %util on the drive being written to.<br>
<br>
I know you said your log isn’t being written to the local disk, but other storage activity could still be pushing up your I/O base load.<br>
<br>
— Alex<br>
<br>
[1] Replace `/dev/sda` with your actual storage device.<br>
<br>
-- <br>
Alex Balashov | Principal | Evariste Systems LLC<br>
<br>
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)<br>
Web: <a href="http://www.evaristesys.com/">http://www.evaristesys.com/</a>, <a href="http://www.csrpswitch.com/">
http://www.csrpswitch.com/</a><br>
<br>
<br>
__________________________________________________________<br>
Kamailio - Users Mailing List - Non Commercial Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the recipients, do not reply only to the sender!<br>
Edit mailing list options or unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></span></p>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>