15 Sep
2022
15 Sep
'22
5:27 p.m.
Hi Henning,
No we are using a local graylog server, not cloud.
I suppose if the bottleneck is log generation, it doesn’t matter how or where they are
handled/sent.
Amit
From: Henning Westerholt <hw(a)gilawa.com>
Date: Thursday, September 15, 2022 at 9:53 AM
To: Amit <amit(a)brytecall.com>
Cc: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Subject: RE: [SR-Users] Log levels severely impacts performance
Hello Amit,
this sounds strange. If you send them remotely it should not really affect the operation
anymore.
Maybe it’s a special limitation of the cloud server, if you use a public cloud service
provider, but just guessing here.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: Amit <amit(a)brytecall.com>
Sent: Thursday, September 15, 2022 3:16 PM
To: Henning Westerholt <hw(a)gilawa.com>
Cc: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Henning,
We are actually sending the logs directly to our local graylog server via the $programname
property in rsyslog, and we’ve disabled writing Kamailio logs to disk (i.e. only in memory
by setting Storage=volatile in rsyslog.conf) but that didn’t seem to help with the
application socket receive buffer backlog.
Keep in mind that for testing purposes, I am blasting Kamailio with a fixed 2000
simultaneous registrations, so at log level 2 it really struggles and the socket buffer
quickly fills up. Reducing the log level to 1 allows Kamailio to rip through the requests
and the buffer stays at 0.
____________________
Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com<http://www.BryteCall.com>
From: Henning Westerholt <hw@gilawa.com<mailto:hw@gilawa.com>>
Date: Thursday, September 15, 2022 at 9:01 AM
To: Amit <amit@brytecall.com<mailto:amit@brytecall.com>>
Cc: Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>
Subject: RE: [SR-Users] Log levels severely impacts performance
Hello,
how much are you actually logging? As already suggested, maybe just need to decrease the
amount of log messages.
If you need this much logging, you can of course forward it to a dedicated log server over
the network.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: Amit <amit@brytecall.com<mailto:amit@brytecall.com>>
Sent: Thursday, September 15, 2022 1:33 PM
To: Henning Westerholt <hw@gilawa.com<mailto:hw@gilawa.com>>
Cc: Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>
Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Henning,
Thanks for the suggestion.
Disk I/O was our initial assumption on another Kamilio instance we are running.
To test this theory we installed Kamailio on separate hardware and configurations with
faster underlying SSD disks, however we are seeing the same crippled performance and
congestion in the applications socket receive buffer when LOG level is set to INFO (2).
____________________
Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com<http://www.BryteCall.com>
On Sep 15, 2022, at 6:14 AM, Henning Westerholt
<hw@gilawa.com<mailto:hw@gilawa.com>> wrote:
Hello,
additionally, try to see if you have some general I/O issues, e.g., by observing io-wait
CPU, looking to I/O transaction times with monitoring tools etc..
Maybe there are other services also causing excessive I/O.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: sr-users
<sr-users-bounces@lists.kamailio.org<mailto:sr-users-bounces@lists.kamailio.org>>
On Behalf Of Daniel-Constantin Mierla
Sent: Wednesday, September 14, 2022 2:25 PM
To: Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>; Amit
<amit@brytecall.com<mailto:amit@brytecall.com>>
Subject: Re: [SR-Users] Log levels severely impacts performance
Hello,
excessive logging is known to impact the performances. If you have many log messages
printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the
children global parameter.
Cheers,
Daniel
On 14.09.22 04:18, Amit wrote:
Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the
application buffer would get full and Kamailio has a lot of difficulty keeping up with REG
requests. This severely impacted Kamailio’s ability to perform and resulted in many REG
timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues
in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log
level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level
NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but
this didn’t seem to help.
http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being
written to the local disk doesn’t seem to alleviate the issue when the log level is set to
INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it
difficult to scale as a result of this issue.
Thank you in advance.
Amit
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com<http://www.asipto.com>
www.twitter.com/miconda<http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>