<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>The load average increase was at runtime, but with the async
tasks not actually doing anything.</p>
<p>I found a really good article explaining the load average
computation -- I haven't read it thoroughly yet, but is very
informative:<br>
</p>
<p> * <a
href="http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html">http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html</a></p>
<p>So it is not only measured the need for CPU, but also the
uninterruptible tasks, which in the past was the need for disk
I/O, but nowadays can be more than that.</p>
<p>The async workers wait on some internal sockets to read the data
of the tasks to execute. It uses recvfrom() which is I/O operation
and normally should not increase the load.</p>
<p>Maybe in a hypervised environment there are some signals waking
the readers of internal sockets, or the kernel there counts this
operation to be "uninterruptible task". That's why I was curios to
see if any other OS/kernel exposes the same situation. On Debian I
haven't noticed high load although I have some deployments with
async workers doing same rare operations.<br>
</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 25.05.20 22:37, Sergiu Pojoga wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJZsGCmNkM51JXoR3XkrPfdPL3Mk0WpJ2ti737goVXQF2RxWBQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Is Kamailio running in a hypervised environment? If
so, I've seen async workers cause high load at runtime, don't
recall boot time. </div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, May 25, 2020 at 12:12
PM Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com" moz-do-not-send="true">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>the async task workers are in recvfom(), which should not
increase any load.</p>
<p>Do you have any chance to test on another os/version?
Maybe on centos 7 and see if it is the same case?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 25.05.20 17:56, Володимир Іванець wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello again,
<div><br>
</div>
<div>I attached a new file.</div>
<div><br>
</div>
<div>The interesting part is that Kamailio does not load
the CPU at all. <i>Top</i> shows it at the bottom.
Only the "load average" value gets increased.</div>
<div><br>
</div>
<div>Thank you!</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">пн, 25 трав. 2020 о
16:34 Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com" target="_blank"
moz-do-not-send="true">miconda@gmail.com</a>>
пише:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>can you install the package with kamailio
debugging symbols? Then take again the kamctl trap
with two async workers, it should contain more
details about what pieces of code run.<br>
</p>
<p>The package should be named like kamailio-dbg...</p>
<p>Besides that, can you also do a 'top' and see
what kamailio processes (their PIDs) eat a lot of
cpu?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 25.05.20 11:05, Володимир Іванець wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello,
<div><br>
</div>
<div>Attached are two files. One for 2 Async
Task Workers and one for 8 Workers. The second
one was stuck and did not complete.</div>
<div><br>
</div>
<div>I should point out that the virtual machine
has 2 CPU cores. Load average value was stable
with 2 workers and was slowly increasing after
adding more workers. * workers caused it to
increase very fast.</div>
<div><br>
</div>
<div>Thank you!</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">пт, 22 трав.
2020 о 22:00 Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com"
target="_blank" moz-do-not-send="true">miconda@gmail.com</a>>
пише:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>if you can, it would be interesting to
get the backtrace and see what was causing
the load.</p>
<p>Iirc, the Async Task Worker should wait
on read on an internal socket, so it
should be no CPU used when nothing is
transmitted to this type of workers.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 22.05.20 19:20, Володимир Іванець
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello Daniel,
<div><br>
</div>
<div>Thank you for your response.</div>
<div><br>
</div>
<div>I run <i>kamctl trap</i> command
but the procedure got stuck. Last line
in the generated file contained
"---start 12767 -----". 12767 was an
Async Task Worker. Since I don't need
them I just removed related
configuration. It must be left after
the testing. This solved the problem.</div>
<div><br>
</div>
<div>Please let me know if you are still
interested in what was going on and if
I should restore the configuration and
run <i>kamctl trap</i> again.</div>
<div><br>
</div>
<div>Thank you very much!</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">пт, 22
трав. 2020 о 19:10 Daniel-Constantin
Mierla <<a
href="mailto:miconda@gmail.com"
target="_blank"
moz-do-not-send="true">miconda@gmail.com</a>>
пише:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>install gdb and, when the load is
high, run:</p>
<p>kamctl trap</p>
<p>It write a file with what
kamailio was doing at that moment.
Send it over here on mailing list
or make it available for download
somewhere. We can look at it and
guide further about what can be
done.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 22.05.20 16:53, Володимир
Іванець wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello everyone!<br>
<br>
I'm running Kamailio version
5.3.3 on a CentOS 6. I started
noticing that "load average"
value increases rapidly with the
start of Kamailio:
<blockquote style="margin:0px
0px 0px
40px;border:none;padding:0px">
<div># uptime</div>
<div>17:47:52 up 4 days,
17:47, 3 users, load
average: 7.02, 7.01, 6.02</div>
<div><br>
</div>
</blockquote>
It will start to decrease
immediately after Kamailio is
stopped.
<div><br>
</div>
<div>Does anyone know what could
cause this and how to
troubleshoot it?</div>
<div><br>
</div>
<div>Thank you!</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.kamailio.org" target="_blank" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Funding: <a href="https://www.paypal.me/dcmierla" target="_blank" moz-do-not-send="true">https://www.paypal.me/dcmierla</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Funding: <a href="https://www.paypal.me/dcmierla" target="_blank" moz-do-not-send="true">https://www.paypal.me/dcmierla</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Funding: <a href="https://www.paypal.me/dcmierla" target="_blank" moz-do-not-send="true">https://www.paypal.me/dcmierla</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Funding: <a href="https://www.paypal.me/dcmierla" target="_blank" moz-do-not-send="true">https://www.paypal.me/dcmierla</a></pre>
</div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank"
moz-do-not-send="true">sr-users@lists.kamailio.org</a><br>
<a
href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Funding: <a class="moz-txt-link-freetext" href="https://www.paypal.me/dcmierla">https://www.paypal.me/dcmierla</a></pre>
</body>
</html>