[SR-Users] High load average

Tue May 26 12:47:38 CEST 2020

Hello,

Thank you for your replies. This happens on ESXi VM and on a bare
metal server. Unfortunately I was not able to start my configuration on the
CentOS 7 to verify if such behavior will repeat there.

Thank you!

вт, 26 трав. 2020 о 10:34 Daniel-Constantin Mierla <miconda at gmail.com> пише:

> The load average increase was at runtime, but with the async tasks not
> actually doing anything.
>
> I found a really good article explaining the load average computation -- I
> haven't read it thoroughly yet, but is very informative:
>
>   * http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html
>
> So it is not only measured the need for CPU, but also the uninterruptible
> tasks, which in the past was the need for disk I/O, but nowadays can be
> more than that.
>
> The async workers wait on some internal sockets to read the data of the
> tasks to execute. It uses recvfrom() which is I/O operation and normally
> should not increase the load.
>
> Maybe in a hypervised environment there are some signals waking the
> readers of internal sockets, or the kernel there counts this operation to
> be "uninterruptible task". That's why I was curios to see if any other
> OS/kernel exposes the same situation. On Debian I haven't noticed high load
> although I have some deployments with async workers doing same rare
> operations.
>
> Cheers,
> Daniel
> On 25.05.20 22:37, Sergiu Pojoga wrote:
>
> Is Kamailio running in a hypervised environment? If so, I've seen async
> workers cause high load at runtime, don't recall boot time.
>
> On Mon, May 25, 2020 at 12:12 PM Daniel-Constantin Mierla <
> miconda at gmail.com> wrote:
>
>> Hello,
>>
>> the async task workers are in recvfom(), which should not increase any
>> load.
>>
>> Do you have any chance to test on another os/version? Maybe on centos 7
>> and see if it is the same case?
>>
>> Cheers,
>> Daniel
>> On 25.05.20 17:56, Володимир Іванець wrote:
>>
>> Hello again,
>>
>> I attached a new file.
>>
>> The interesting part is that Kamailio does not load the CPU at all. *Top* shows
>> it at the bottom. Only the "load average" value gets increased.
>>
>> Thank you!
>>
>> пн, 25 трав. 2020 о 16:34 Daniel-Constantin Mierla <miconda at gmail.com>
>> пише:
>>
>>> Hello,
>>>
>>> can you install the package with kamailio debugging symbols? Then take
>>> again the kamctl trap with two async workers, it should contain more
>>> details about what pieces of code run.
>>>
>>> The package should be named like kamailio-dbg...
>>>
>>> Besides that, can you also do a 'top' and see what kamailio processes
>>> (their PIDs) eat a lot of cpu?
>>>
>>> Cheers,
>>> Daniel
>>> On 25.05.20 11:05, Володимир Іванець wrote:
>>>
>>> Hello,
>>>
>>> Attached are two files. One for 2 Async Task Workers and one for 8
>>> Workers. The second one was stuck and did not complete.
>>>
>>> I should point out that the virtual machine has 2 CPU cores. Load
>>> average value was stable with 2 workers and was slowly increasing after
>>> adding more workers. * workers caused it to increase very fast.
>>>
>>> Thank you!
>>>
>>> пт, 22 трав. 2020 о 22:00 Daniel-Constantin Mierla <miconda at gmail.com>
>>> пише:
>>>
>>>> Hello,
>>>>
>>>> if you can, it would be interesting to get the backtrace and see what
>>>> was causing the load.
>>>>
>>>> Iirc, the Async Task Worker should wait on read on an internal socket,
>>>> so it should be no CPU used when nothing is transmitted to this type of
>>>> workers.
>>>>
>>>> Cheers,
>>>> Daniel
>>>> On 22.05.20 19:20, Володимир Іванець wrote:
>>>>
>>>> Hello Daniel,
>>>>
>>>> Thank you for your response.
>>>>
>>>> I run *kamctl trap* command but the procedure got stuck. Last line in
>>>> the generated file contained "---start 12767 -----".  12767 was an Async
>>>> Task Worker. Since I don't need them I just removed related configuration.
>>>> It must be left after the testing. This solved the problem.
>>>>
>>>> Please let me know if you are still interested in what was going on and
>>>> if I should restore the configuration and run *kamctl trap* again.
>>>>
>>>> Thank you very much!
>>>>
>>>> пт, 22 трав. 2020 о 19:10 Daniel-Constantin Mierla <miconda at gmail.com>
>>>> пише:
>>>>
>>>>> Hello,
>>>>>
>>>>> install gdb and, when the load is high, run:
>>>>>
>>>>> kamctl trap
>>>>>
>>>>> It write a file with what kamailio was doing at that moment. Send it
>>>>> over here on mailing list or make it available for download somewhere. We
>>>>> can look at it and guide further about what can be done.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>> On 22.05.20 16:53, Володимир Іванець wrote:
>>>>>
>>>>> Hello everyone!
>>>>>
>>>>> I'm running Kamailio version 5.3.3 on a CentOS 6. I started noticing
>>>>> that "load average" value increases rapidly with the start of Kamailio:
>>>>>
>>>>> # uptime
>>>>> 17:47:52 up 4 days, 17:47,  3 users,  load average: 7.02, 7.01, 6.02
>>>>>
>>>>> It will start to decrease immediately after Kamailio is stopped.
>>>>>
>>>>> Does anyone know what could cause this and how to troubleshoot it?
>>>>>
>>>>> Thank you!
>>>>>
>>>>> _______________________________________________
>>>>> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>>> Funding: https://www.paypal.me/dcmierla
>>>>>
>>>>> --
>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>> Funding: https://www.paypal.me/dcmierla
>>>>
>>>> --
>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>> Funding: https://www.paypal.me/dcmierla
>>>
>>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Funding: https://www.paypal.me/dcmierla
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Funding: https://www.paypal.me/dcmierla
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200526/bd8363f8/attachment.html>