[SR-Users] async workers causing unexplained high load average

[Daniel-Constantin Mierla]

Can you attach with gdb to one of the async processes and see with the
backtrace command what is the process doing at that time?

Cheers,
Daniel

On 24/10/14 05:30, Alex Balashov wrote:
> I just tested the same thing on a Rackspace VPS (Xen I think) and am
> seeing the same climbing load average with 2 async workers. It seems
> to top out at 1.05.
>
> On 10/23/2014 02:13 PM, Alex Balashov wrote:
>
>> Another thing I have found is that having a certain amount of async
>> workers running, even if they are not doing anything, appears to cause
>> unexplained CPU load, even if the Kamailio instance is completely idle
>> and not processing any calls.
>>
>> Here is the baseline load with no async workers:
>>
>> [root at centosity6 ~]# uptime
>>   14:06:28 up 2 min,  2 users,  load average: 0.01, 0.00, 0.00
>>
>> If I start Kamailio with 1 async worker...
>>
>>     modparam("async", "workers", 1)
>>     async_workers=1
>>
>>   14:07:59 up 3 min,  2 users,  load average: 0.00, 0.00, 0.00
>>
>> Still nothing.
>>
>> Now I try 2:
>>
>>   14:08:54 up 4 min,  2 users,  load average: 0.45, 0.11, 0.04
>>   14:08:59 up 4 min,  2 users,  load average: 0.49, 0.12, 0.04
>>   14:09:04 up 4 min,  2 users,  load average: 0.53, 0.14, 0.04
>>   14:09:09 up 4 min,  2 users,  load average: 0.57, 0.15, 0.05
>>   14:09:14 up 5 min,  2 users,  load average: 0.60, 0.17, 0.05
>>   14:09:19 up 5 min,  2 users,  load average: 0.63, 0.18, 0.06
>>   14:09:31 up 5 min,  2 users,  load average: 0.69, 0.21, 0.07
>>   14:09:37 up 5 min,  2 users,  load average: 0.71, 0.22, 0.07
>>   14:10:35 up 6 min,  2 users,  load average: 0.89, 0.36, 0.13
>>   14:11:07 up 6 min,  2 users,  load average: 0.93, 0.42, 0.16
>>   ...
>>
>> I don't have time right now to see if it will rise to infinity, but it
>> just keeps going up.
>>
>> There are no SIP messages being received at all, and there is absolutely
>> nothing going on.
>>
>> The ascent of the load average is much higher with >= 4 threads. I have
>> 8 "CPUs" available from /proc/cpuinfo (quad-core processor with
>> HyperThreading).
>>
>> When I attach strace to the PID of the Async Worker, there's nothing
>> going on except the normal recvfrom() blocking wait:
>>
>> [root at centosity6 kamailio]# kamctl ps | grep Async
>> which: no greadlink in
>> (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/)
>>
>> Process::  ID=11 PID=1559 Type=Async Task Worker
>> Process::  ID=12 PID=1560 Type=Async Task Worker
>>
>> [root at centosity6 kamailio]# strace -p 1559
>> Process 1559 attached - interrupt to quit
>> recvfrom(9,
>>
>> [root at centosity6 ~]# strace -p 1560
>> Process 1560 attached - interrupt to quit
>> recvfrom(9,
>>
>> Nor do the other Kamailio threads seem to be doing anything unusual. 'ps
>> aux' and 'top' show 0.0% CPU load on the Kamailio processes themselves,
>> and 'iowait' shows the system as 98-99% idle.
>>
>> I suppose it's possible that this is a peculiarity of the testing
>> environment: I am running this on CentOS 6.5 inside VirtualBox, which I
>> think is KVM-based. I have not had a chance to duplicate this experiment
>> in a different kind of execution environment yet. It's possible that the
>> load average is simply being improperly indicated because of the VM
>> environment.
>>
>>
>
>

-- 
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda