High load average - sr-users

Daniel-Constantin Mierla

22 May 22 May

12:10 p.m.

Hello, install gdb and, when the load is high, run: kamctl trap It write a file with what kamailio was doing at that moment. Send it over here on mailing list or make it available for download somewhere. We can look at it and guide further about what can be done. Cheers, Daniel On 22.05.20 16:53, Володимир Іванець wrote:

...

Hello everyone! I'm running Kamailio version 5.3.3 on a CentOS 6. I started noticing that "load average" value increases rapidly with the start of Kamailio: # uptime 17:47:52 up 4 days, 17:47, 3 users, load average: 7.02, 7.01, 6.02 It will start to decrease immediately after Kamailio is stopped. Does anyone know what could cause this and how to troubleshoot it? Thank you! _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Reply

Daniel-Constantin Mierla

3 p.m.

Hello, if you can, it would be interesting to get the backtrace and see what was causing the load. Iirc, the Async Task Worker should wait on read on an internal socket, so it should be no CPU used when nothing is transmitted to this type of workers. Cheers, Daniel On 22.05.20 19:20, Володимир Іванець wrote:

...

Hello Daniel, Thank you for your response. I run /kamctl trap/ command but the procedure got stuck. Last line in the generated file contained "---start 12767 -----". 12767 was an Async Task Worker. Since I don't need them I just removed related configuration. It must be left after the testing. This solved the problem. Please let me know if you are still interested in what was going on and if I should restore the configuration and run /kamctl trap/ again. Thank you very much! пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, install gdb and, when the load is high, run: kamctl trap It write a file with what kamailio was doing at that moment. Send it over here on mailing list or make it available for download somewhere. We can look at it and guide further about what can be done. Cheers, Daniel On 22.05.20 16:53, Володимир Іванець wrote:

Hello everyone! I'm running Kamailio version 5.3.3 on a CentOS 6. I started noticing that "load average" value increases rapidly with the start of Kamailio: # uptime 17:47:52 up 4 days, 17:47, 3 users, load average: 7.02, 7.01, 6.02 It will start to decrease immediately after Kamailio is stopped. Does anyone know what could cause this and how to troubleshoot it? Thank you! _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding: https://www.paypal.me/dcmierla

-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Reply

Daniel-Constantin Mierla

9:34 a.m.

Hello, can you install the package with kamailio debugging symbols? Then take again the kamctl trap with two async workers, it should contain more details about what pieces of code run. The package should be named like kamailio-dbg... Besides that, can you also do a 'top' and see what kamailio processes (their PIDs) eat a lot of cpu? Cheers, Daniel On 25.05.20 11:05, Володимир Іванець wrote:

...

Hello, Attached are two files. One for 2 Async Task Workers and one for 8 Workers. The second one was stuck and did not complete. I should point out that the virtual machine has 2 CPU cores. Load average value was stable with 2 workers and was slowly increasing after adding more workers. * workers caused it to increase very fast. Thank you! пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, if you can, it would be interesting to get the backtrace and see what was causing the load. Iirc, the Async Task Worker should wait on read on an internal socket, so it should be no CPU used when nothing is transmitted to this type of workers. Cheers, Daniel On 22.05.20 19:20, Володимир Іванець wrote:

Hello Daniel, Thank you for your response. I run /kamctl trap/ command but the procedure got stuck. Last line in the generated file contained "---start 12767 -----". 12767 was an Async Task Worker. Since I don't need them I just removed related configuration. It must be left after the testing. This solved the problem. Please let me know if you are still interested in what was going on and if I should restore the configuration and run /kamctl trap/ again. Thank you very much! пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, install gdb and, when the load is high, run: kamctl trap It write a file with what kamailio was doing at that moment. Send it over here on mailing list or make it available for download somewhere. We can look at it and guide further about what can be done. Cheers, Daniel On 22.05.20 16:53, Володимир Іванець wrote:

Hello everyone! I'm running Kamailio version 5.3.3 on a CentOS 6. I started noticing that "load average" value increases rapidly with the start of Kamailio: # uptime 17:47:52 up 4 days, 17:47, 3 users, load average: 7.02, 7.01, 6.02 It will start to decrease immediately after Kamailio is stopped. Does anyone know what could cause this and how to troubleshoot it? Thank you! _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding: https://www.paypal.me/dcmierla

-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Reply

Daniel-Constantin Mierla

12:10 p.m.

Hello, the async task workers are in recvfom(), which should not increase any load. Do you have any chance to test on another os/version? Maybe on centos 7 and see if it is the same case? Cheers, Daniel On 25.05.20 17:56, Володимир Іванець wrote:

...

Hello again, I attached a new file. The interesting part is that Kamailio does not load the CPU at all. /Top/ shows it at the bottom. Only the "load average" value gets increased. Thank you! пн, 25 трав. 2020 о 16:34 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, can you install the package with kamailio debugging symbols? Then take again the kamctl trap with two async workers, it should contain more details about what pieces of code run. The package should be named like kamailio-dbg... Besides that, can you also do a 'top' and see what kamailio processes (their PIDs) eat a lot of cpu? Cheers, Daniel On 25.05.20 11:05, Володимир Іванець wrote:

Hello, Attached are two files. One for 2 Async Task Workers and one for 8 Workers. The second one was stuck and did not complete. I should point out that the virtual machine has 2 CPU cores. Load average value was stable with 2 workers and was slowly increasing after adding more workers. * workers caused it to increase very fast. Thank you! пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, if you can, it would be interesting to get the backtrace and see what was causing the load. Iirc, the Async Task Worker should wait on read on an internal socket, so it should be no CPU used when nothing is transmitted to this type of workers. Cheers, Daniel On 22.05.20 19:20, Володимир Іванець wrote:

Hello Daniel, Thank you for your response. I run /kamctl trap/ command but the procedure got stuck. Last line in the generated file contained "---start 12767 -----". 12767 was an Async Task Worker. Since I don't need them I just removed related configuration. It must be left after the testing. This solved the problem. Please let me know if you are still interested in what was going on and if I should restore the configuration and run /kamctl trap/ again. Thank you very much! пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, install gdb and, when the load is high, run: kamctl trap It write a file with what kamailio was doing at that moment. Send it over here on mailing list or make it available for download somewhere. We can look at it and guide further about what can be done. Cheers, Daniel On 22.05.20 16:53, Володимир Іванець wrote:

Hello everyone! I'm running Kamailio version 5.3.3 on a CentOS 6. I started noticing that "load average" value increases rapidly with the start of Kamailio: # uptime 17:47:52 up 4 days, 17:47, 3 users, load average: 7.02, 7.01, 6.02 It will start to decrease immediately after Kamailio is stopped. Does anyone know what could cause this and how to troubleshoot it? Thank you! _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding: https://www.paypal.me/dcmierla

-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Reply

Daniel-Constantin Mierla

26 May 26 May

3:30 a.m.

The load average increase was at runtime, but with the async tasks not actually doing anything. I found a really good article explaining the load average computation -- I haven't read it thoroughly yet, but is very informative: * http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html So it is not only measured the need for CPU, but also the uninterruptible tasks, which in the past was the need for disk I/O, but nowadays can be more than that. The async workers wait on some internal sockets to read the data of the tasks to execute. It uses recvfrom() which is I/O operation and normally should not increase the load. Maybe in a hypervised environment there are some signals waking the readers of internal sockets, or the kernel there counts this operation to be "uninterruptible task". That's why I was curios to see if any other OS/kernel exposes the same situation. On Debian I haven't noticed high load although I have some deployments with async workers doing same rare operations. Cheers, Daniel On 25.05.20 22:37, Sergiu Pojoga wrote:

...

Is Kamailio running in a hypervised environment? If so, I've seen async workers cause high load at runtime, don't recall boot time. On Mon, May 25, 2020 at 12:12 PM Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote: Hello, the async task workers are in recvfom(), which should not increase any load. Do you have any chance to test on another os/version? Maybe on centos 7 and see if it is the same case? Cheers, Daniel On 25.05.20 17:56, Володимир Іванець wrote:

Hello again, I attached a new file. The interesting part is that Kamailio does not load the CPU at all. /Top/ shows it at the bottom. Only the "load average" value gets increased. Thank you! пн, 25 трав. 2020 о 16:34 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, can you install the package with kamailio debugging symbols? Then take again the kamctl trap with two async workers, it should contain more details about what pieces of code run. The package should be named like kamailio-dbg... Besides that, can you also do a 'top' and see what kamailio processes (their PIDs) eat a lot of cpu? Cheers, Daniel On 25.05.20 11:05, Володимир Іванець wrote:

Hello, Attached are two files. One for 2 Async Task Workers and one for 8 Workers. The second one was stuck and did not complete. I should point out that the virtual machine has 2 CPU cores. Load average value was stable with 2 workers and was slowly increasing after adding more workers. * workers caused it to increase very fast. Thank you! пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, if you can, it would be interesting to get the backtrace and see what was causing the load. Iirc, the Async Task Worker should wait on read on an internal socket, so it should be no CPU used when nothing is transmitted to this type of workers. Cheers, Daniel On 22.05.20 19:20, Володимир Іванець wrote:

Hello Daniel, Thank you for your response. I run /kamctl trap/ command but the procedure got stuck. Last line in the generated file contained "---start 12767 -----". 12767 was an Async Task Worker. Since I don't need them I just removed related configuration. It must be left after the testing. This solved the problem. Please let me know if you are still interested in what was going on and if I should restore the configuration and run /kamctl trap/ again. Thank you very much! пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, install gdb and, when the load is high, run: kamctl trap It write a file with what kamailio was doing at that moment. Send it over here on mailing list or make it available for download somewhere. We can look at it and guide further about what can be done. Cheers, Daniel On 22.05.20 16:53, Володимир Іванець wrote: > Hello everyone! > > I'm running Kamailio version 5.3.3 on a CentOS 6. > I started noticing that "load average" value > increases rapidly with the start of Kamailio: > > # uptime > 17:47:52 up 4 days, 17:47, 3 users, load > average: 7.02, 7.01, 6.02 > > It will start to decrease immediately after > Kamailio is stopped. > > Does anyone know what could cause this and how to > troubleshoot it? > > Thank you! > > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding: https://www.paypal.me/dcmierla

-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding: https://www.paypal.me/dcmierla

-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding: https://www.paypal.me/dcmierla _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla

Reply

Alex Balashov

10:05 a.m.

Hi, Some time in late 2014, I reported a similar problem with async workers causing high CPU usage in Xen and KVM environments: https://lists.kamailio.org/pipermail/sr-users/2014-October/085432.html and I think we had a similar discussion. These async workers were just sitting in their normal event loop. This was the main reason why I ended up implementing a lot of async tasks in the more tedious way with mqueue + rtimer; rtimer processes blocking on mqueue did not seem to have this problem. I never really followed up to see if async workers behave better now, but since the vast majority of my installations run in a hypervisor, I didn't want to write a lot of code that uses the conventional convenience wrappers and take the risk. -- Alex On 5/26/20 3:30 AM, Daniel-Constantin Mierla wrote:

...

The load average increase was at runtime, but with the async tasks not actually doing anything. I found a really good article explaining the load average computation -- I haven't read it thoroughly yet, but is very informative: * http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html So it is not only measured the need for CPU, but also the uninterruptible tasks, which in the past was the need for disk I/O, but nowadays can be more than that. The async workers wait on some internal sockets to read the data of the tasks to execute. It uses recvfrom() which is I/O operation and normally should not increase the load. Maybe in a hypervised environment there are some signals waking the readers of internal sockets, or the kernel there counts this operation to be "uninterruptible task". That's why I was curios to see if any other OS/kernel exposes the same situation. On Debian I haven't noticed high load although I have some deployments with async workers doing same rare operations. Cheers, Daniel On 25.05.20 22:37, Sergiu Pojoga wrote:

Is Kamailio running in a hypervised environment? If so, I've seen async workers cause high load at runtime, don't recall boot time. On Mon, May 25, 2020 at 12:12 PM Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote: Hello, the async task workers are in recvfom(), which should not increase any load. Do you have any chance to test on another os/version? Maybe on centos 7 and see if it is the same case? Cheers, Daniel On 25.05.20 17:56, Володимир Іванець wrote:

Hello again, I attached a new file. The interesting part is that Kamailio does not load the CPU at all. /Top/ shows it at the bottom. Only the "load average" value gets increased. Thank you! пн, 25 трав. 2020 о 16:34 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, can you install the package with kamailio debugging symbols? Then take again the kamctl trap with two async workers, it should contain more details about what pieces of code run. The package should be named like kamailio-dbg... Besides that, can you also do a 'top' and see what kamailio processes (their PIDs) eat a lot of cpu? Cheers, Daniel On 25.05.20 11:05, Володимир Іванець wrote:

Hello, Attached are two files. One for 2 Async Task Workers and one for 8 Workers. The second one was stuck and did not complete. I should point out that the virtual machine has 2 CPU cores. Load average value was stable with 2 workers and was slowly increasing after adding more workers. * workers caused it to increase very fast. Thank you! пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: Hello, if you can, it would be interesting to get the backtrace and see what was causing the load. Iirc, the Async Task Worker should wait on read on an internal socket, so it should be no CPU used when nothing is transmitted to this type of workers. Cheers, Daniel On 22.05.20 19:20, Володимир Іванець wrote: > Hello Daniel, > > Thank you for your response. > > I run /kamctl trap/ command but the procedure got > stuck. Last line in the generated file contained > "---start 12767 -----". 12767 was an Async Task > Worker. Since I don't need them I just removed related > configuration. It must be left after the testing. This > solved the problem. > > Please let me know if you are still interested in what > was going on and if I should restore the configuration > and run /kamctl trap/ again. > > Thank you very much! > > пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla > <miconda(a)gmail.com <mailto:miconda@gmail.com>> пише: > > Hello, > > install gdb and, when the load is high, run: > > kamctl trap > > It write a file with what kamailio was doing at > that moment. Send it over here on mailing list or > make it available for download somewhere. We can > look at it and guide further about what can be done. > > Cheers, > Daniel > > On 22.05.20 16:53, Володимир Іванець wrote: >> Hello everyone! >> >> I'm running Kamailio version 5.3.3 on a CentOS 6. >> I started noticing that "load average" value >> increases rapidly with the start of Kamailio: >> >> # uptime >> 17:47:52 up 4 days, 17:47, 3 users, load >> average: 7.02, 7.01, 6.02 >> >> It will start to decrease immediately after >> Kamailio is stopped. >> >> Does anyone know what could cause this and how to >> troubleshoot it? >> >> Thank you! >> >> _______________________________________________ >> Kamailio (SER) - Users Mailing List >> sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org> >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > -- > Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com> > www.twitter.com/miconda <http://www.twitter.com/miconda> --www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> > Funding:https://www.paypal.me/dcmierla > -- Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> --www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding:https://www.paypal.me/dcmierla

-- Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> --www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding:https://www.paypal.me/dcmierla

-- Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> --www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding:https://www.paypal.me/dcmierla _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda Funding:https://www.paypal.me/dcmierla _______________________________________________ Kamailio (SER) - Users Mailing List sr-users(a)lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

Reply