[SR-Users] High load average
Daniel-Constantin Mierla
miconda at gmail.com
Tue May 26 09:30:52 CEST 2020
The load average increase was at runtime, but with the async tasks not
actually doing anything.
I found a really good article explaining the load average computation --
I haven't read it thoroughly yet, but is very informative:
* http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
So it is not only measured the need for CPU, but also the
uninterruptible tasks, which in the past was the need for disk I/O, but
nowadays can be more than that.
The async workers wait on some internal sockets to read the data of the
tasks to execute. It uses recvfrom() which is I/O operation and normally
should not increase the load.
Maybe in a hypervised environment there are some signals waking the
readers of internal sockets, or the kernel there counts this operation
to be "uninterruptible task". That's why I was curios to see if any
other OS/kernel exposes the same situation. On Debian I haven't noticed
high load although I have some deployments with async workers doing same
rare operations.
Cheers,
Daniel
On 25.05.20 22:37, Sergiu Pojoga wrote:
> Is Kamailio running in a hypervised environment? If so, I've seen
> async workers cause high load at runtime, don't recall boot time.
>
> On Mon, May 25, 2020 at 12:12 PM Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
> the async task workers are in recvfom(), which should not increase
> any load.
>
> Do you have any chance to test on another os/version? Maybe on
> centos 7 and see if it is the same case?
>
> Cheers,
> Daniel
>
> On 25.05.20 17:56, Володимир Іванець wrote:
>> Hello again,
>>
>> I attached a new file.
>>
>> The interesting part is that Kamailio does not load the CPU at
>> all. /Top/ shows it at the bottom. Only the "load average" value
>> gets increased.
>>
>> Thank you!
>>
>> пн, 25 трав. 2020 о 16:34 Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> пише:
>>
>> Hello,
>>
>> can you install the package with kamailio debugging symbols?
>> Then take again the kamctl trap with two async workers, it
>> should contain more details about what pieces of code run.
>>
>> The package should be named like kamailio-dbg...
>>
>> Besides that, can you also do a 'top' and see what kamailio
>> processes (their PIDs) eat a lot of cpu?
>>
>> Cheers,
>> Daniel
>>
>> On 25.05.20 11:05, Володимир Іванець wrote:
>>> Hello,
>>>
>>> Attached are two files. One for 2 Async Task Workers and one
>>> for 8 Workers. The second one was stuck and did not complete.
>>>
>>> I should point out that the virtual machine has 2 CPU cores.
>>> Load average value was stable with 2 workers and was slowly
>>> increasing after adding more workers. * workers caused it to
>>> increase very fast.
>>>
>>> Thank you!
>>>
>>> пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> пише:
>>>
>>> Hello,
>>>
>>> if you can, it would be interesting to get the backtrace
>>> and see what was causing the load.
>>>
>>> Iirc, the Async Task Worker should wait on read on an
>>> internal socket, so it should be no CPU used when
>>> nothing is transmitted to this type of workers.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 22.05.20 19:20, Володимир Іванець wrote:
>>>> Hello Daniel,
>>>>
>>>> Thank you for your response.
>>>>
>>>> I run /kamctl trap/ command but the procedure got
>>>> stuck. Last line in the generated file contained
>>>> "---start 12767 -----". 12767 was an Async Task
>>>> Worker. Since I don't need them I just removed related
>>>> configuration. It must be left after the testing. This
>>>> solved the problem.
>>>>
>>>> Please let me know if you are still interested in what
>>>> was going on and if I should restore the configuration
>>>> and run /kamctl trap/ again.
>>>>
>>>> Thank you very much!
>>>>
>>>> пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla
>>>> <miconda at gmail.com <mailto:miconda at gmail.com>> пише:
>>>>
>>>> Hello,
>>>>
>>>> install gdb and, when the load is high, run:
>>>>
>>>> kamctl trap
>>>>
>>>> It write a file with what kamailio was doing at
>>>> that moment. Send it over here on mailing list or
>>>> make it available for download somewhere. We can
>>>> look at it and guide further about what can be done.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 22.05.20 16:53, Володимир Іванець wrote:
>>>>> Hello everyone!
>>>>>
>>>>> I'm running Kamailio version 5.3.3 on a CentOS 6.
>>>>> I started noticing that "load average" value
>>>>> increases rapidly with the start of Kamailio:
>>>>>
>>>>> # uptime
>>>>> 17:47:52 up 4 days, 17:47, 3 users, load
>>>>> average: 7.02, 7.01, 6.02
>>>>>
>>>>> It will start to decrease immediately after
>>>>> Kamailio is stopped.
>>>>>
>>>>> Does anyone know what could cause this and how to
>>>>> troubleshoot it?
>>>>>
>>>>> Thank you!
>>>>>
>>>>> _______________________________________________
>>>>> Kamailio (SER) - Users Mailing List
>>>>> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>>> Funding: https://www.paypal.me/dcmierla
>>>>
>>> --
>>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>> Funding: https://www.paypal.me/dcmierla
>>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>> Funding: https://www.paypal.me/dcmierla
>>
> --
> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
> Funding: https://www.paypal.me/dcmierla
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200526/c984b57b/attachment.html>
More information about the sr-users
mailing list