Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help. http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance. Amit
Hello,
excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the children global parameter.
Cheers, Daniel
On 14.09.22 04:18, Amit wrote:
Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help.
http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance.
Amit
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Hello,
additionally, try to see if you have some general I/O issues, e.g., by observing io-wait CPU, looking to I/O transaction times with monitoring tools etc.. Maybe there are other services also causing excessive I/O.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Daniel-Constantin Mierla Sent: Wednesday, September 14, 2022 2:25 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; Amit amit@brytecall.com Subject: Re: [SR-Users] Log levels severely impacts performance
Hello,
excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the children global parameter.
Cheers, Daniel On 14.09.22 04:18, Amit wrote: Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help. http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance. Amit
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com
www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
Hi Henning, Thanks for the suggestion. Disk I/O was our initial assumption on another Kamilio instance we are running. To test this theory we installed Kamailio on separate hardware and configurations with faster underlying SSD disks, however we are seeing the same crippled performance and congestion in the applications socket receive buffer when LOG level is set to INFO (2).
____________________ Amit Nir, MBA Office: 305.999.0911 www.BryteCall.com
On Sep 15, 2022, at 6:14 AM, Henning Westerholt hw@gilawa.com wrote:
Hello,
additionally, try to see if you have some general I/O issues, e.g., by observing io-wait CPU, looking to I/O transaction times with monitoring tools etc.. Maybe there are other services also causing excessive I/O.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Daniel-Constantin Mierla Sent: Wednesday, September 14, 2022 2:25 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; Amit amit@brytecall.com Subject: Re: [SR-Users] Log levels severely impacts performance
Hello,
excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the children global parameter.
Cheers, Daniel On 14.09.22 04:18, Amit wrote: Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help. http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance. Amit
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com
www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
Hello,
how much are you actually logging? As already suggested, maybe just need to decrease the amount of log messages.
If you need this much logging, you can of course forward it to a dedicated log server over the network.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Amit amit@brytecall.com Sent: Thursday, September 15, 2022 1:33 PM To: Henning Westerholt hw@gilawa.com Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Henning, Thanks for the suggestion. Disk I/O was our initial assumption on another Kamilio instance we are running. To test this theory we installed Kamailio on separate hardware and configurations with faster underlying SSD disks, however we are seeing the same crippled performance and congestion in the applications socket receive buffer when LOG level is set to INFO (2).
____________________ Amit Nir, MBA Office: 305.999.0911 www.BryteCall.comhttp://www.BryteCall.com
On Sep 15, 2022, at 6:14 AM, Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> wrote: Hello,
additionally, try to see if you have some general I/O issues, e.g., by observing io-wait CPU, looking to I/O transaction times with monitoring tools etc.. Maybe there are other services also causing excessive I/O.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> On Behalf Of Daniel-Constantin Mierla Sent: Wednesday, September 14, 2022 2:25 PM To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org>; Amit <amit@brytecall.commailto:amit@brytecall.com> Subject: Re: [SR-Users] Log levels severely impacts performance
Hello,
excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the children global parameter.
Cheers, Daniel On 14.09.22 04:18, Amit wrote: Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help. http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance. Amit
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com
www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
Hi Henning,
We are actually sending the logs directly to our local graylog server via the $programname property in rsyslog, and we’ve disabled writing Kamailio logs to disk (i.e. only in memory by setting Storage=volatile in rsyslog.conf) but that didn’t seem to help with the application socket receive buffer backlog.
Keep in mind that for testing purposes, I am blasting Kamailio with a fixed 2000 simultaneous registrations, so at log level 2 it really struggles and the socket buffer quickly fills up. Reducing the log level to 1 allows Kamailio to rip through the requests and the buffer stays at 0.
____________________ Amit Nir, MBA Office: 305.999.0911 www.BryteCall.com
From: Henning Westerholt hw@gilawa.com Date: Thursday, September 15, 2022 at 9:01 AM To: Amit amit@brytecall.com Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: RE: [SR-Users] Log levels severely impacts performance Hello,
how much are you actually logging? As already suggested, maybe just need to decrease the amount of log messages.
If you need this much logging, you can of course forward it to a dedicated log server over the network.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Amit amit@brytecall.com Sent: Thursday, September 15, 2022 1:33 PM To: Henning Westerholt hw@gilawa.com Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Henning, Thanks for the suggestion. Disk I/O was our initial assumption on another Kamilio instance we are running. To test this theory we installed Kamailio on separate hardware and configurations with faster underlying SSD disks, however we are seeing the same crippled performance and congestion in the applications socket receive buffer when LOG level is set to INFO (2).
____________________ Amit Nir, MBA Office: 305.999.0911 www.BryteCall.comhttp://www.BryteCall.com
On Sep 15, 2022, at 6:14 AM, Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> wrote: Hello,
additionally, try to see if you have some general I/O issues, e.g., by observing io-wait CPU, looking to I/O transaction times with monitoring tools etc.. Maybe there are other services also causing excessive I/O.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> On Behalf Of Daniel-Constantin Mierla Sent: Wednesday, September 14, 2022 2:25 PM To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org>; Amit <amit@brytecall.commailto:amit@brytecall.com> Subject: Re: [SR-Users] Log levels severely impacts performance
Hello,
excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the children global parameter.
Cheers, Daniel On 14.09.22 04:18, Amit wrote: Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help. http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance. Amit
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com
www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
Hello Amit,
this sounds strange. If you send them remotely it should not really affect the operation anymore. Maybe it’s a special limitation of the cloud server, if you use a public cloud service provider, but just guessing here.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Amit amit@brytecall.com Sent: Thursday, September 15, 2022 3:16 PM To: Henning Westerholt hw@gilawa.com Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Henning,
We are actually sending the logs directly to our local graylog server via the $programname property in rsyslog, and we’ve disabled writing Kamailio logs to disk (i.e. only in memory by setting Storage=volatile in rsyslog.conf) but that didn’t seem to help with the application socket receive buffer backlog.
Keep in mind that for testing purposes, I am blasting Kamailio with a fixed 2000 simultaneous registrations, so at log level 2 it really struggles and the socket buffer quickly fills up. Reducing the log level to 1 allows Kamailio to rip through the requests and the buffer stays at 0.
____________________ Amit Nir, MBA Office: 305.999.0911 www.BryteCall.comhttp://www.BryteCall.com
From: Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> Date: Thursday, September 15, 2022 at 9:01 AM To: Amit <amit@brytecall.commailto:amit@brytecall.com> Cc: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: RE: [SR-Users] Log levels severely impacts performance Hello,
how much are you actually logging? As already suggested, maybe just need to decrease the amount of log messages.
If you need this much logging, you can of course forward it to a dedicated log server over the network.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Amit <amit@brytecall.commailto:amit@brytecall.com> Sent: Thursday, September 15, 2022 1:33 PM To: Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> Cc: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Henning, Thanks for the suggestion. Disk I/O was our initial assumption on another Kamilio instance we are running. To test this theory we installed Kamailio on separate hardware and configurations with faster underlying SSD disks, however we are seeing the same crippled performance and congestion in the applications socket receive buffer when LOG level is set to INFO (2).
____________________ Amit Nir, MBA Office: 305.999.0911 www.BryteCall.comhttp://www.BryteCall.com
On Sep 15, 2022, at 6:14 AM, Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> wrote: Hello,
additionally, try to see if you have some general I/O issues, e.g., by observing io-wait CPU, looking to I/O transaction times with monitoring tools etc.. Maybe there are other services also causing excessive I/O.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> On Behalf Of Daniel-Constantin Mierla Sent: Wednesday, September 14, 2022 2:25 PM To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org>; Amit <amit@brytecall.commailto:amit@brytecall.com> Subject: Re: [SR-Users] Log levels severely impacts performance
Hello,
excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the children global parameter.
Cheers, Daniel On 14.09.22 04:18, Amit wrote: Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help. http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance. Amit
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com
www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
Hi Henning, No we are using a local graylog server, not cloud. I suppose if the bottleneck is log generation, it doesn’t matter how or where they are handled/sent. Amit
From: Henning Westerholt hw@gilawa.com Date: Thursday, September 15, 2022 at 9:53 AM To: Amit amit@brytecall.com Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: RE: [SR-Users] Log levels severely impacts performance Hello Amit,
this sounds strange. If you send them remotely it should not really affect the operation anymore. Maybe it’s a special limitation of the cloud server, if you use a public cloud service provider, but just guessing here.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Amit amit@brytecall.com Sent: Thursday, September 15, 2022 3:16 PM To: Henning Westerholt hw@gilawa.com Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Henning,
We are actually sending the logs directly to our local graylog server via the $programname property in rsyslog, and we’ve disabled writing Kamailio logs to disk (i.e. only in memory by setting Storage=volatile in rsyslog.conf) but that didn’t seem to help with the application socket receive buffer backlog.
Keep in mind that for testing purposes, I am blasting Kamailio with a fixed 2000 simultaneous registrations, so at log level 2 it really struggles and the socket buffer quickly fills up. Reducing the log level to 1 allows Kamailio to rip through the requests and the buffer stays at 0.
____________________ Amit Nir, MBA Office: 305.999.0911 www.BryteCall.comhttp://www.BryteCall.com
From: Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> Date: Thursday, September 15, 2022 at 9:01 AM To: Amit <amit@brytecall.commailto:amit@brytecall.com> Cc: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: RE: [SR-Users] Log levels severely impacts performance Hello,
how much are you actually logging? As already suggested, maybe just need to decrease the amount of log messages.
If you need this much logging, you can of course forward it to a dedicated log server over the network.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Amit <amit@brytecall.commailto:amit@brytecall.com> Sent: Thursday, September 15, 2022 1:33 PM To: Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> Cc: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Henning, Thanks for the suggestion. Disk I/O was our initial assumption on another Kamilio instance we are running. To test this theory we installed Kamailio on separate hardware and configurations with faster underlying SSD disks, however we are seeing the same crippled performance and congestion in the applications socket receive buffer when LOG level is set to INFO (2).
____________________ Amit Nir, MBA Office: 305.999.0911 www.BryteCall.comhttp://www.BryteCall.com
On Sep 15, 2022, at 6:14 AM, Henning Westerholt <hw@gilawa.commailto:hw@gilawa.com> wrote: Hello,
additionally, try to see if you have some general I/O issues, e.g., by observing io-wait CPU, looking to I/O transaction times with monitoring tools etc.. Maybe there are other services also causing excessive I/O.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> On Behalf Of Daniel-Constantin Mierla Sent: Wednesday, September 14, 2022 2:25 PM To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org>; Amit <amit@brytecall.commailto:amit@brytecall.com> Subject: Re: [SR-Users] Log levels severely impacts performance
Hello,
excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the children global parameter.
Cheers, Daniel On 14.09.22 04:18, Amit wrote: Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help. http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance. Amit
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com
www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
Then rework the logs. Do you have more than one log per registration? If yes, use only one. If you already have a single log per registration, then try reducing the size. Experiment to see where the system breaks. Try new hardware. Decrease the number of workers to see if it has an impact.
-ovidiu
On Thu, Sep 15, 2022 at 11:30 Amit amit@brytecall.com wrote:
Hi Henning,
No we are using a local graylog server, not cloud.
I suppose if the bottleneck is log generation, it doesn’t matter how or where they are handled/sent.
Amit
*From: *Henning Westerholt hw@gilawa.com *Date: *Thursday, September 15, 2022 at 9:53 AM *To: *Amit amit@brytecall.com *Cc: *Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Subject: *RE: [SR-Users] Log levels severely impacts performance
Hello Amit,
this sounds strange. If you send them remotely it should not really affect the operation anymore.
Maybe it’s a special limitation of the cloud server, if you use a public cloud service provider, but just guessing here.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com
*From:* Amit amit@brytecall.com *Sent:* Thursday, September 15, 2022 3:16 PM *To:* Henning Westerholt hw@gilawa.com *Cc:* Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Subject:* Re: [SR-Users] Log levels severely impacts performance
Hi Henning,
We are actually sending the logs directly to our local graylog server via the $programname property in rsyslog, and we’ve disabled writing Kamailio logs to disk (i.e. only in memory by setting Storage=volatile in rsyslog.conf) but that didn’t seem to help with the application socket receive buffer backlog.
Keep in mind that for testing purposes, I am blasting Kamailio with a fixed 2000 simultaneous registrations, so at log level 2 it really struggles and the socket buffer quickly fills up. Reducing the log level to 1 allows Kamailio to rip through the requests and the buffer stays at 0.
Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com
*From: *Henning Westerholt hw@gilawa.com *Date: *Thursday, September 15, 2022 at 9:01 AM *To: *Amit amit@brytecall.com *Cc: *Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Subject: *RE: [SR-Users] Log levels severely impacts performance
Hello,
how much are you actually logging? As already suggested, maybe just need to decrease the amount of log messages.
If you need this much logging, you can of course forward it to a dedicated log server over the network.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com
*From:* Amit amit@brytecall.com *Sent:* Thursday, September 15, 2022 1:33 PM *To:* Henning Westerholt hw@gilawa.com *Cc:* Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org *Subject:* Re: [SR-Users] Log levels severely impacts performance
Hi Henning,
Thanks for the suggestion.
Disk I/O was our initial assumption on another Kamilio instance we are running.
To test this theory we installed Kamailio on separate hardware and configurations with faster underlying SSD disks, however we are seeing the same crippled performance and congestion in the applications socket receive buffer when LOG level is set to INFO (2).
Amit Nir, MBA
Office: 305.999.0911
www.BryteCall.com
On Sep 15, 2022, at 6:14 AM, Henning Westerholt hw@gilawa.com wrote:
Hello,
additionally, try to see if you have some general I/O issues, e.g., by observing io-wait CPU, looking to I/O transaction times with monitoring tools etc..
Maybe there are other services also causing excessive I/O.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com
*From:* sr-users sr-users-bounces@lists.kamailio.org *On Behalf Of *Daniel-Constantin Mierla *Sent:* Wednesday, September 14, 2022 2:25 PM *To:* Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; Amit amit@brytecall.com *Subject:* Re: [SR-Users] Log levels severely impacts performance
Hello,
excessive logging is known to impact the performances. If you have many log messages printed from config, try to reduce them.
Performance could be also a matter of how many child processes you created, via the children global parameter.
Cheers, Daniel
On 14.09.22 04:18, Amit wrote:
Hello,
We’ve been running Kamailio 5.5.0 for a while now and recently ran into an issue where the application buffer would get full and Kamailio has a lot of difficulty keeping up with REG requests. This severely impacted Kamailio’s ability to perform and resulted in many REG timeouts.
Running ‘watch netstat -ulpn’ revealed the buffer issue.
We usually run the production system at log level INFO (2) and had not ran into any issues in the past.
When the buffer issue occurred and after some testing, we discovered that changing the log level to NOTICE (1) was sufficient to bring the system back into operational status.
For example, at log level INFO (2) we seem to achieve ~150-200 REG/s while at log level NOTICE (1) we reach ~500-700 REG/s.
I did find this article in the wiki and tried using the “-“ in front of the log path but this didn’t seem to help.
http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
Even writing the logs directly to our external log server and preventing them from being written to the local disk doesn’t seem to alleviate the issue when the log level is set to INFO (2).
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
Thank you in advance.
Amit
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
On 9/13/2022 10:18 PM, Amit wrote:
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
I was just dealing with this behavior. I was logging to a local rsyslog and rsyslog was just writing to a file on a local ssd (unsynced).
It's a ton of log. But meh.
I disabled syslog entirely and enabled log_stderror = yes
Then I just captured stdout+stderr via multilog and write it to a file on the disk. The problem went away.
I think there is some huge overhead in the way kam packages up the log and sends it out to syslog.
Hello,
Kamailio is using the system interface for syslog, this is also a code which is quite stable and has not seen a lot of changes in the last years. People are using Kamailio with a lot of logging in high load situation usually without problems, there has not been a lot of reports about that in the past.
Maybe it something related to a specific distribution version? I checked, the rsyslog is using asynchronous writing as default since a long time. What logging engine do you use?
Cheers,
Henning
Agree, have had some very high-volume systems with prodigious logging output for every call and no problems.
Have also had these types of load problems on other systems with much less logging. The problem was either that asynchronous logging was not turned on, or the I/O subsystem was abnormally slow for other reasons, such as other heavy demand on storage, or just intrinsically slow.
— Alex
On Sep 15, 2022, at 5:18 PM, Henning Westerholt hw@gilawa.com wrote:
Hello,
Kamailio is using the system interface for syslog, this is also a code which is quite stable and has not seen a lot of changes in the last years. People are using Kamailio with a lot of logging in high load situation usually without problems, there has not been a lot of reports about that in the past.
Maybe it something related to a specific distribution version? I checked, the rsyslog is using asynchronous writing as default since a long time. What logging engine do you use?
Cheers,
Henning
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.com
-----Original Message----- From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Jeremy Kister Sent: Thursday, September 15, 2022 9:59 PM To: sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
On 9/13/2022 10:18 PM, Amit wrote:
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
I was just dealing with this behavior. I was logging to a local rsyslog and rsyslog was just writing to a file on a local ssd (unsynced).
It's a ton of log. But meh.
I disabled syslog entirely and enabled log_stderror = yes
Then I just captured stdout+stderr via multilog and write it to a file on the disk. The problem went away.
I think there is some huge overhead in the way kam packages up the log and sends it out to syslog.
--
Jeremy Kister https://jeremy.kister.net./
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Hi Alex, Can you share specifically where in the configs I would be able to confirm that asynchronous logging is enabled? I’ve already added the “-“ in front of the log file and that didn’t seem to make any recognizable difference. Thanks in advance. Amit
From: sr-users sr-users-bounces@lists.kamailio.org on behalf of Alex Balashov abalashov@evaristesys.com Date: Thursday, September 15, 2022 at 5:21 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance Agree, have had some very high-volume systems with prodigious logging output for every call and no problems.
Have also had these types of load problems on other systems with much less logging. The problem was either that asynchronous logging was not turned on, or the I/O subsystem was abnormally slow for other reasons, such as other heavy demand on storage, or just intrinsically slow.
— Alex
On Sep 15, 2022, at 5:18 PM, Henning Westerholt hw@gilawa.com wrote:
Hello,
Kamailio is using the system interface for syslog, this is also a code which is quite stable and has not seen a lot of changes in the last years. People are using Kamailio with a lot of logging in high load situation usually without problems, there has not been a lot of reports about that in the past.
Maybe it something related to a specific distribution version? I checked, the rsyslog is using asynchronous writing as default since a long time. What logging engine do you use?
Cheers,
Henning
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.com
-----Original Message----- From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Jeremy Kister Sent: Thursday, September 15, 2022 9:59 PM To: sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
On 9/13/2022 10:18 PM, Amit wrote:
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
I was just dealing with this behavior. I was logging to a local rsyslog and rsyslog was just writing to a file on a local ssd (unsynced).
It's a ton of log. But meh.
I disabled syslog entirely and enabled log_stderror = yes
Then I just captured stdout+stderr via multilog and write it to a file on the disk. The problem went away.
I think there is some huge overhead in the way kam packages up the log and sends it out to syslog.
--
Jeremy Kister https://jeremy.kister.net./
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Kamailio - Users Mailing List - Non Commercial Discussions
- sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On Sep 15, 2022, at 10:54 PM, Amit amit@brytecall.com wrote:
I’ve already added the “-“ in front of the log file and that didn’t seem to make any recognizable difference.
That’s the only place that I know.
Have you tested your storage throughput? Even something as simple as `hdparm -t /dev/sda`[1]?
What about monitoring background I/O demand, i.e. `iostat -x 1`? Look at the %util on the drive being written to.
I know you said your log isn’t being written to the local disk, but other storage activity could still be pushing up your I/O base load.
— Alex
[1] Replace `/dev/sda` with your actual storage device.
Hi Alex, Thanks for your suggestions. Yes we’ve tested the underlying storage. We specifically upgraded drives on the particular testing server I am using at the moment, and are using a ZFS volume.
But to answer your specific questions, # hdparm -t /dev/sda2 /dev/sda2: Timing buffered disk reads: 4118 MB in 3.00 seconds = 1372.43 MB/sec
Also, %iowait remains at 0. I did see it briefly go to %0.13 for one cycle when I sent 2000 simultaneous registrations to Kam.
It doesn’t appear to be the disk subsystem. Amit
From: sr-users sr-users-bounces@lists.kamailio.org on behalf of Alex Balashov abalashov@evaristesys.com Date: Thursday, September 15, 2022 at 11:00 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
On Sep 15, 2022, at 10:54 PM, Amit amit@brytecall.com wrote:
I’ve already added the “-“ in front of the log file and that didn’t seem to make any recognizable difference.
That’s the only place that I know.
Have you tested your storage throughput? Even something as simple as `hdparm -t /dev/sda`[1]?
What about monitoring background I/O demand, i.e. `iostat -x 1`? Look at the %util on the drive being written to.
I know you said your log isn’t being written to the local disk, but other storage activity could still be pushing up your I/O base load.
— Alex
[1] Replace `/dev/sda` with your actual storage device.
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello,
then have a look to the server stats and IO load during the test case when you observe the particular blocking issues. You should be able to see something then, if its caused from the Kamailio logging to syslog.
Cheers,
Henning
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Amit Sent: Friday, September 16, 2022 2:14 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Alex, Thanks for your suggestions. Yes we've tested the underlying storage. We specifically upgraded drives on the particular testing server I am using at the moment, and are using a ZFS volume.
But to answer your specific questions, # hdparm -t /dev/sda2 /dev/sda2: Timing buffered disk reads: 4118 MB in 3.00 seconds = 1372.43 MB/sec
Also, %iowait remains at 0. I did see it briefly go to %0.13 for one cycle when I sent 2000 simultaneous registrations to Kam.
It doesn't appear to be the disk subsystem. Amit
From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> on behalf of Alex Balashov <abalashov@evaristesys.commailto:abalashov@evaristesys.com> Date: Thursday, September 15, 2022 at 11:00 PM To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: Re: [SR-Users] Log levels severely impacts performance
On Sep 15, 2022, at 10:54 PM, Amit <amit@brytecall.commailto:amit@brytecall.com> wrote:
I've already added the "-" in front of the log file and that didn't seem to make any recognizable difference.
That's the only place that I know.
Have you tested your storage throughput? Even something as simple as `hdparm -t /dev/sda`[1]?
What about monitoring background I/O demand, i.e. `iostat -x 1`? Look at the %util on the drive being written to.
I know you said your log isn't being written to the local disk, but other storage activity could still be pushing up your I/O base load.
- Alex
[1] Replace `/dev/sda` with your actual storage device.
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Henning, That’s just it. We don’t see any IO wait increases during the test case. I’ve only seen it briefly go to %0.13 and then right back to %0.00
I heard there are ways to time or benchmark different parts of the config. Can you point me in the right direction to perform some logging benchmark tests? Amit
On Sep 16, 2022, at 8:32 AM, Henning Westerholt hw@gilawa.com wrote:
Hello,
then have a look to the server stats and IO load during the test case when you observe the particular blocking issues. You should be able to see something then, if its caused from the Kamailio logging to syslog.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Amit Sent: Friday, September 16, 2022 2:14 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
Hi Alex, Thanks for your suggestions. Yes we’ve tested the underlying storage. We specifically upgraded drives on the particular testing server I am using at the moment, and are using a ZFS volume.
But to answer your specific questions, # hdparm -t /dev/sda2 /dev/sda2: Timing buffered disk reads: 4118 MB in 3.00 seconds = 1372.43 MB/sec
Also, %iowait remains at 0. I did see it briefly go to %0.13 for one cycle when I sent 2000 simultaneous registrations to Kam.
It doesn’t appear to be the disk subsystem. Amit
From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> on behalf of Alex Balashov <abalashov@evaristesys.commailto:abalashov@evaristesys.com> Date: Thursday, September 15, 2022 at 11:00 PM To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: Re: [SR-Users] Log levels severely impacts performance
On Sep 15, 2022, at 10:54 PM, Amit <amit@brytecall.commailto:amit@brytecall.com> wrote:
I’ve already added the “-“ in front of the log file and that didn’t seem to make any recognizable difference.
That’s the only place that I know.
Have you tested your storage throughput? Even something as simple as `hdparm -t /dev/sda`[1]?
What about monitoring background I/O demand, i.e. `iostat -x 1`? Look at the %util on the drive being written to.
I know you said your log isn’t being written to the local disk, but other storage activity could still be pushing up your I/O base load.
— Alex
[1] Replace `/dev/sda` with your actual storage device.
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On 15 Sept 2022, at 23:20, Alex Balashov abalashov@evaristesys.com wrote:
The problem was either that asynchronous logging was not turned on,
As standard syslog is very synchronous - how do you turn on asynchronous logging? Curious :-)
/O
Hello,
In the case of rsyslogd the default behaviour is since many years to use asynchronous operations. Refer e.g. to: https://www.rsyslog.com/doc/v8-stable/compatibility/v3compatibility.html
Cheers,
Henning
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Olle E. Johansson Sent: Friday, September 16, 2022 9:47 AM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance
On 15 Sept 2022, at 23:20, Alex Balashov <abalashov@evaristesys.commailto:abalashov@evaristesys.com> wrote:
The problem was either that asynchronous logging was not turned on,
As standard syslog is very synchronous - how do you turn on asynchronous logging? Curious :-)
/O
On 16 Sept 2022, at 10:12, Henning Westerholt hw@gilawa.com wrote:
Hello,
In the case of rsyslogd the default behaviour is since many years to use asynchronous operations. Refer e.g. to: https://www.rsyslog.com/doc/v8-stable/compatibility/v3compatibility.html https://www.rsyslog.com/doc/v8-stable/compatibility/v3compatibility.html
THank you! I will read up on that.
/O
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ https://skalatan.de/blog/ Kamailio services – https://gilawa.com https://gilawa.com/
From: sr-users <sr-users-bounces@lists.kamailio.org mailto:sr-users-bounces@lists.kamailio.org> On Behalf Of Olle E. Johansson Sent: Friday, September 16, 2022 9:47 AM To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org> Subject: Re: [SR-Users] Log levels severely impacts performance
On 15 Sept 2022, at 23:20, Alex Balashov <abalashov@evaristesys.com mailto:abalashov@evaristesys.com> wrote:
The problem was either that asynchronous logging was not turned on,
As standard syslog is very synchronous - how do you turn on asynchronous logging? Curious :-)
/O
Jeremy, Thank you for sharing your experience. I did a little digging on multitool and only found the manpage, but not very much on how to install it. It doesn’t seem to be available via the yum package manager Would you mind sharing some info on installation and how you run it to capture stderr and stdout? Thank you and much appreciated. Amit
From: sr-users sr-users-bounces@lists.kamailio.org on behalf of Jeremy Kister kamailio-01@jeremykister.com Date: Thursday, September 15, 2022 at 4:00 PM To: sr-users@lists.kamailio.org sr-users@lists.kamailio.org Subject: Re: [SR-Users] Log levels severely impacts performance On 9/13/2022 10:18 PM, Amit wrote:
Does anyone have any experience with this or have any suggestions? We are finding it difficult to scale as a result of this issue.
I was just dealing with this behavior. I was logging to a local rsyslog and rsyslog was just writing to a file on a local ssd (unsynced).
It's a ton of log. But meh.
I disabled syslog entirely and enabled log_stderror = yes
Then I just captured stdout+stderr via multilog and write it to a file on the disk. The problem went away.
I think there is some huge overhead in the way kam packages up the log and sends it out to syslog.
--
Jeremy Kister https://jeremy.kister.net./
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
Alex Balashov -
Amit -
Daniel-Constantin Mierla -
Henning Westerholt -
Jeremy Kister -
Olle E. Johansson -
Ovidiu Sas