[SR-Users] Log levels severely impacts performance

[Amit]

Hi Henning,
That’s just it. We don’t see any IO wait increases during the test case. I’ve only seen it briefly go to %0.13 and then right back to %0.00

I heard there are ways to time or benchmark different parts of the config. Can you point me in the right direction to perform some logging benchmark tests?
Amit


On Sep 16, 2022, at 8:32 AM, Henning Westerholt <hw at gilawa.com> wrote:


Hello,

then have a look to the server stats and IO load during the test case when you observe the particular blocking issues. You should be able to see something then, if its caused from the Kamailio logging to syslog.

Cheers,

Henning

--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>

From: sr-users <sr-users-bounces at lists.kamailio.org> On Behalf Of Amit
Sent: Friday, September 16, 2022 2:14 PM
To: Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
Subject: Re: [SR-Users] Log levels severely impacts performance

Hi Alex,
Thanks for your suggestions.
Yes we’ve tested the underlying storage. We specifically upgraded drives on the particular testing server I am using at the moment, and are using a ZFS volume.

But to answer your specific questions,
# hdparm -t /dev/sda2
/dev/sda2:
Timing buffered disk reads: 4118 MB in  3.00 seconds = 1372.43 MB/sec

Also, %iowait remains at 0. I did see it briefly go to %0.13 for one cycle when I sent 2000 simultaneous registrations to Kam.

It doesn’t appear to be the disk subsystem.
Amit

From: sr-users <sr-users-bounces at lists.kamailio.org<mailto:sr-users-bounces at lists.kamailio.org>> on behalf of Alex Balashov <abalashov at evaristesys.com<mailto:abalashov at evaristesys.com>>
Date: Thursday, September 15, 2022 at 11:00 PM
To: Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>>
Subject: Re: [SR-Users] Log levels severely impacts performance

> On Sep 15, 2022, at 10:54 PM, Amit <amit at brytecall.com<mailto:amit at brytecall.com>> wrote:
>
> I’ve already added the “-“ in front of the log file and that didn’t seem to make any recognizable difference.

That’s the only place that I know.

Have you tested your storage throughput? Even something as simple as `hdparm -t /dev/sda`[1]?

What about monitoring background I/O demand, i.e. `iostat -x 1`? Look at the %util on the drive being written to.

I know you said your log isn’t being written to the local disk, but other storage activity could still be pushing up your I/O base load.

— Alex

[1] Replace `/dev/sda` with your actual storage device.

--
Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/


__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20220916/e4265894/attachment.htm>