[SR-Users] High load average
Alex Balashov
abalashov at evaristesys.com
Tue May 26 16:05:49 CEST 2020
Hi,
Some time in late 2014, I reported a similar problem with async workers
causing high CPU usage in Xen and KVM environments:
https://lists.kamailio.org/pipermail/sr-users/2014-October/085432.html
and I think we had a similar discussion. These async workers were just
sitting in their normal event loop.
This was the main reason why I ended up implementing a lot of async
tasks in the more tedious way with mqueue + rtimer; rtimer processes
blocking on mqueue did not seem to have this problem.
I never really followed up to see if async workers behave better now,
but since the vast majority of my installations run in a hypervisor, I
didn't want to write a lot of code that uses the conventional
convenience wrappers and take the risk.
-- Alex
On 5/26/20 3:30 AM, Daniel-Constantin Mierla wrote:
> The load average increase was at runtime, but with the async tasks not
> actually doing anything.
>
> I found a really good article explaining the load average computation --
> I haven't read it thoroughly yet, but is very informative:
>
> * http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
>
> So it is not only measured the need for CPU, but also the
> uninterruptible tasks, which in the past was the need for disk I/O, but
> nowadays can be more than that.
>
> The async workers wait on some internal sockets to read the data of the
> tasks to execute. It uses recvfrom() which is I/O operation and normally
> should not increase the load.
>
> Maybe in a hypervised environment there are some signals waking the
> readers of internal sockets, or the kernel there counts this operation
> to be "uninterruptible task". That's why I was curios to see if any
> other OS/kernel exposes the same situation. On Debian I haven't noticed
> high load although I have some deployments with async workers doing same
> rare operations.
>
> Cheers,
> Daniel
>
> On 25.05.20 22:37, Sergiu Pojoga wrote:
>> Is Kamailio running in a hypervised environment? If so, I've seen
>> async workers cause high load at runtime, don't recall boot time.
>>
>> On Mon, May 25, 2020 at 12:12 PM Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>> the async task workers are in recvfom(), which should not increase
>> any load.
>>
>> Do you have any chance to test on another os/version? Maybe on
>> centos 7 and see if it is the same case?
>>
>> Cheers,
>> Daniel
>>
>> On 25.05.20 17:56, Володимир Іванець wrote:
>>> Hello again,
>>>
>>> I attached a new file.
>>>
>>> The interesting part is that Kamailio does not load the CPU at
>>> all. /Top/ shows it at the bottom. Only the "load average" value
>>> gets increased.
>>>
>>> Thank you!
>>>
>>> пн, 25 трав. 2020 о 16:34 Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> пише:
>>>
>>> Hello,
>>>
>>> can you install the package with kamailio debugging symbols?
>>> Then take again the kamctl trap with two async workers, it
>>> should contain more details about what pieces of code run.
>>>
>>> The package should be named like kamailio-dbg...
>>>
>>> Besides that, can you also do a 'top' and see what kamailio
>>> processes (their PIDs) eat a lot of cpu?
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 25.05.20 11:05, Володимир Іванець wrote:
>>>> Hello,
>>>>
>>>> Attached are two files. One for 2 Async Task Workers and one
>>>> for 8 Workers. The second one was stuck and did not complete.
>>>>
>>>> I should point out that the virtual machine has 2 CPU cores.
>>>> Load average value was stable with 2 workers and was slowly
>>>> increasing after adding more workers. * workers caused it to
>>>> increase very fast.
>>>>
>>>> Thank you!
>>>>
>>>> пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla
>>>> <miconda at gmail.com <mailto:miconda at gmail.com>> пише:
>>>>
>>>> Hello,
>>>>
>>>> if you can, it would be interesting to get the backtrace
>>>> and see what was causing the load.
>>>>
>>>> Iirc, the Async Task Worker should wait on read on an
>>>> internal socket, so it should be no CPU used when
>>>> nothing is transmitted to this type of workers.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 22.05.20 19:20, Володимир Іванець wrote:
>>>>> Hello Daniel,
>>>>>
>>>>> Thank you for your response.
>>>>>
>>>>> I run /kamctl trap/ command but the procedure got
>>>>> stuck. Last line in the generated file contained
>>>>> "---start 12767 -----". 12767 was an Async Task
>>>>> Worker. Since I don't need them I just removed related
>>>>> configuration. It must be left after the testing. This
>>>>> solved the problem.
>>>>>
>>>>> Please let me know if you are still interested in what
>>>>> was going on and if I should restore the configuration
>>>>> and run /kamctl trap/ again.
>>>>>
>>>>> Thank you very much!
>>>>>
>>>>> пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla
>>>>> <miconda at gmail.com <mailto:miconda at gmail.com>> пише:
>>>>>
>>>>> Hello,
>>>>>
>>>>> install gdb and, when the load is high, run:
>>>>>
>>>>> kamctl trap
>>>>>
>>>>> It write a file with what kamailio was doing at
>>>>> that moment. Send it over here on mailing list or
>>>>> make it available for download somewhere. We can
>>>>> look at it and guide further about what can be done.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>> On 22.05.20 16:53, Володимир Іванець wrote:
>>>>>> Hello everyone!
>>>>>>
>>>>>> I'm running Kamailio version 5.3.3 on a CentOS 6.
>>>>>> I started noticing that "load average" value
>>>>>> increases rapidly with the start of Kamailio:
>>>>>>
>>>>>> # uptime
>>>>>> 17:47:52 up 4 days, 17:47, 3 users, load
>>>>>> average: 7.02, 7.01, 6.02
>>>>>>
>>>>>> It will start to decrease immediately after
>>>>>> Kamailio is stopped.
>>>>>>
>>>>>> Does anyone know what could cause this and how to
>>>>>> troubleshoot it?
>>>>>>
>>>>>> Thank you!
>>>>>>
>>>>>> _______________________________________________
>>>>>> Kamailio (SER) - Users Mailing List
>>>>>> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com>
>>>>> www.twitter.com/miconda <http://www.twitter.com/miconda> --www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>>>> Funding:https://www.paypal.me/dcmierla
>>>>>
>>>> --
>>>> Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com>
>>>> www.twitter.com/miconda <http://www.twitter.com/miconda> --www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>>> Funding:https://www.paypal.me/dcmierla
>>>>
>>> --
>>> Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com>
>>> www.twitter.com/miconda <http://www.twitter.com/miconda> --www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>> Funding:https://www.paypal.me/dcmierla
>>>
>> --
>> Daniel-Constantin Mierla --www.asipto.com <http://www.asipto.com>
>> www.twitter.com/miconda <http://www.twitter.com/miconda> --www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>> Funding:https://www.paypal.me/dcmierla
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla --www.asipto.com
> www.twitter.com/miconda --www.linkedin.com/in/miconda
> Funding:https://www.paypal.me/dcmierla
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
More information about the sr-users
mailing list