[SR-Users] Log levels severely impacts performance

Thu Sep 15 18:19:18 CEST 2022

Then rework the logs.
Do you have more than one log per registration? If yes, use only one.
If you already have a single log per registration, then try reducing the
size.
Experiment to see where the system breaks. Try new hardware. Decrease the
number of workers to see if it has an impact.

-ovidiu

On Thu, Sep 15, 2022 at 11:30 Amit <amit at brytecall.com> wrote:

> Hi Henning,
>
> No we are using a local graylog server, not cloud.
>
> I suppose if the bottleneck is log generation, it doesn’t matter how or
> where they are handled/sent.
>
> Amit
>
>
>
> *From: *Henning Westerholt <hw at gilawa.com>
> *Date: *Thursday, September 15, 2022 at 9:53 AM
> *To: *Amit <amit at brytecall.com>
> *Cc: *Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
> *Subject: *RE: [SR-Users] Log levels severely impacts performance
>
> Hello Amit,
>
>
>
> this sounds strange. If you send them remotely it should not really affect
> the operation anymore.
>
> Maybe it’s a special limitation of the cloud server, if you use a public
> cloud service provider, but just guessing here.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
> *From:* Amit <amit at brytecall.com>
> *Sent:* Thursday, September 15, 2022 3:16 PM
> *To:* Henning Westerholt <hw at gilawa.com>
> *Cc:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
> *Subject:* Re: [SR-Users] Log levels severely impacts performance
>
>
>
> Hi Henning,
>
>
>
> We are actually sending the logs directly to our local graylog server via
> the $programname property in rsyslog, and we’ve disabled writing Kamailio
> logs to disk (i.e. only in memory by setting Storage=volatile in
> rsyslog.conf) but that didn’t seem to help with the application socket
> receive buffer backlog.
>
>
>
> Keep in mind that for testing purposes, I am blasting Kamailio with a
> fixed 2000 simultaneous registrations, so at log level 2 it really
> struggles and the socket buffer quickly fills up. Reducing the log level to
> 1 allows Kamailio to rip through the requests and the buffer stays at 0.
>
>
>
> ____________________
>
> Amit Nir, MBA
>
> Office: 305.999.0911
>
> www.BryteCall.com
>
>
>
>
>
> *From: *Henning Westerholt <hw at gilawa.com>
> *Date: *Thursday, September 15, 2022 at 9:01 AM
> *To: *Amit <amit at brytecall.com>
> *Cc: *Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
> *Subject: *RE: [SR-Users] Log levels severely impacts performance
>
> Hello,
>
>
>
> how much are you actually logging? As already suggested, maybe just need
> to decrease the amount of log messages.
>
>
>
> If you need this much logging, you can of course forward it to a dedicated
> log server over the network.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
> *From:* Amit <amit at brytecall.com>
> *Sent:* Thursday, September 15, 2022 1:33 PM
> *To:* Henning Westerholt <hw at gilawa.com>
> *Cc:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
> *Subject:* Re: [SR-Users] Log levels severely impacts performance
>
>
>
> Hi Henning,
>
> Thanks for the suggestion.
>
> Disk I/O was our initial assumption on another Kamilio instance we are
> running.
>
> To test this theory we installed Kamailio on separate hardware and
> configurations with faster underlying SSD disks, however we are seeing the
> same crippled performance and congestion in the applications socket receive
> buffer when LOG level is set to INFO (2).
>
>
>
> ____________________
>
> Amit Nir, MBA
>
> Office: 305.999.0911
>
> www.BryteCall.com
>
>
>
>
> On Sep 15, 2022, at 6:14 AM, Henning Westerholt <hw at gilawa.com> wrote:
>
> 
>
> Hello,
>
>
>
> additionally, try to see if you have some general I/O issues, e.g., by
> observing io-wait CPU, looking to I/O transaction times with monitoring
> tools etc..
>
> Maybe there are other services also causing excessive I/O.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
> *From:* sr-users <sr-users-bounces at lists.kamailio.org> *On Behalf Of *Daniel-Constantin
> Mierla
> *Sent:* Wednesday, September 14, 2022 2:25 PM
> *To:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>;
> Amit <amit at brytecall.com>
> *Subject:* Re: [SR-Users] Log levels severely impacts performance
>
>
>
> Hello,
>
> excessive logging is known to impact the performances. If you have many
> log messages printed from config, try to reduce them.
>
> Performance could be also a matter of how many child processes you
> created, via the children global parameter.
>
> Cheers,
> Daniel
>
> On 14.09.22 04:18, Amit wrote:
>
> Hello,
>
>
>
> We’ve been running Kamailio 5.5.0 for a while now and recently ran into an
> issue where the application buffer would get full and Kamailio has a lot of
> difficulty keeping up with REG requests. This severely impacted Kamailio’s
> ability to perform and resulted in many REG timeouts.
>
>
>
> Running ‘watch netstat -ulpn’ revealed the buffer issue.
>
>
>
> We usually run the production system at log level INFO (2) and had not ran
> into any issues in the past.
>
>
>
> When the buffer issue occurred and after some testing, we discovered that
> changing the log level to NOTICE (1) was sufficient to bring the system
> back into operational status.
>
>
>
> For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while
> at log level NOTICE (1) we reach ~500-700 REG/s.
>
>
>
> I did find this article in the wiki and tried using the “-“ in front of
> the log path but this didn’t seem to help.
>
> http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>
>
>
> Even writing the logs directly to our external log server and preventing
> them from being written to the local disk doesn’t seem to alleviate the
> issue when the log level is set to INFO (2).
>
>
>
> Does anyone have any experience with this or have any suggestions? We are
> finding it difficult to scale as a result of this issue.
>
>
>
> Thank you in advance.
>
> Amit
>
>
>
>
>
>
>
> __________________________________________________________
>
> Kamailio - Users Mailing List - Non Commercial Discussions
>
>   * sr-users at lists.kamailio.org
>
> Important: keep the mailing list in the recipients, do not reply only to the sender!
>
> Edit mailing list options or unsubscribe:
>
>   * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
>
> Daniel-Constantin Mierla -- www.asipto.com
>
> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions
>   * sr-users at lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to
> the sender!
> Edit mailing list options or unsubscribe:
>   * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
-- 
VoIP Embedded, Inc.
http://www.voipembedded.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20220915/6e1e9a94/attachment.htm>