[sr-dev] async workers causing unexplained high load average

[Alex Balashov]

Another thing I have found is that having a certain amount of async 
workers running, even if they are not doing anything, appears to cause 
unexplained CPU load, even if the Kamailio instance is completely idle 
and not processing any calls.

Here is the baseline load with no async workers:

[root at centosity6 ~]# uptime
  14:06:28 up 2 min,  2 users,  load average: 0.01, 0.00, 0.00

If I start Kamailio with 1 async worker...

    modparam("async", "workers", 1)
    async_workers=1

  14:07:59 up 3 min,  2 users,  load average: 0.00, 0.00, 0.00

Still nothing.

Now I try 2:

  14:08:54 up 4 min,  2 users,  load average: 0.45, 0.11, 0.04
  14:08:59 up 4 min,  2 users,  load average: 0.49, 0.12, 0.04
  14:09:04 up 4 min,  2 users,  load average: 0.53, 0.14, 0.04
  14:09:09 up 4 min,  2 users,  load average: 0.57, 0.15, 0.05
  14:09:14 up 5 min,  2 users,  load average: 0.60, 0.17, 0.05
  14:09:19 up 5 min,  2 users,  load average: 0.63, 0.18, 0.06
  14:09:31 up 5 min,  2 users,  load average: 0.69, 0.21, 0.07
  14:09:37 up 5 min,  2 users,  load average: 0.71, 0.22, 0.07
  14:10:35 up 6 min,  2 users,  load average: 0.89, 0.36, 0.13
  14:11:07 up 6 min,  2 users,  load average: 0.93, 0.42, 0.16
  ...

I don't have time right now to see if it will rise to infinity, but it 
just keeps going up.

There are no SIP messages being received at all, and there is absolutely 
nothing going on.

The ascent of the load average is much higher with >= 4 threads. I have 
8 "CPUs" available from /proc/cpuinfo (quad-core processor with 
HyperThreading).

When I attach strace to the PID of the Async Worker, there's nothing 
going on except the normal recvfrom() blocking wait:

[root at centosity6 kamailio]# kamctl ps | grep Async
which: no greadlink in 
(/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/)
Process::  ID=11 PID=1559 Type=Async Task Worker
Process::  ID=12 PID=1560 Type=Async Task Worker

[root at centosity6 kamailio]# strace -p 1559
Process 1559 attached - interrupt to quit
recvfrom(9,

[root at centosity6 ~]# strace -p 1560
Process 1560 attached - interrupt to quit
recvfrom(9,

Nor do the other Kamailio threads seem to be doing anything unusual. 'ps 
aux' and 'top' show 0.0% CPU load on the Kamailio processes themselves, 
and 'iowait' shows the system as 98-99% idle.

I suppose it's possible that this is a peculiarity of the testing 
environment: I am running this on CentOS 6.5 inside VirtualBox, which I 
think is KVM-based. I have not had a chance to duplicate this experiment 
in a different kind of execution environment yet. It's possible that the 
load average is simply being improperly indicated because of the VM 
environment.


-- 
Alex Balashov - Principal
Evariste Systems LLC
Tel: +1-678-954-0670
Web: http://www.evaristesys.com/, http://www.alexbalashov.com/

Please be kind to the English language:

http://www.entrepreneur.com/article/232906