<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello Andrew,</p>
<p>I compared the 0 calls and 300 calls dump, indeed there is not that much of a difference. 688 bytes consumed more, they were allocated in pv_cache_add. Can you try another 300 calls and see if there is the same increase?</p>
<p>Cheers,</p>
<p>Henning<br>
</p>
<p><br>
</p>
<p>-root 9302 0.0 5.4 329264 26848 ? Sl 21:18 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32<br>
+root 9302 0.1 6.9 330460 33904 ? Sl 21:18 0:01 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32<br>
<br>
INFO: <core> [core/cfg/cfg_ctx.c:595]: cfg_set_now(): core.mem_dump_pkg has been changed to 9302<br>
ALERT: <core> [core/pt.c:548]: mem_dump_pkg_cb(): Memory status (pkg) of process 9302:<br>
ALERT: qm_status: (0x7ff63029f010):<br>
ALERT: qm_status: heap size= 33554432<br>
-ALERT: qm_status: used= 485216, used+overhead=815112, free=32739320<br>
-ALERT: qm_status: max used (+overhead)= 821904<br>
+ALERT: qm_status: used= 485904, used+overhead=826096, free=32728336<br>
+ALERT: qm_status: max used (+overhead)= 838568<br>
ALERT: qm_status: dumping all alloc'ed. fragments:<br>
ALERT: qm_status: 0. N address=0x7ff6302d86d8 frag=0x7ff6302d86a0 size=1024 used=1<br>
ALERT: qm_status: alloc'd from core: core/str_hash.h: str_hash_alloc(59)<br>
@@ -2599,23 +2599,49 @@<br>
ALERT: qm_status: 872. N address=0x7ff630365280 frag=0x7ff630365248 size=32 used=1<br>
ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_parse_format(1152)<br>
ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed<br>
-ALERT: qm_status: 886. N address=0x7ff6303662c8 frag=0x7ff630366290 size=128 used=1<br>
+ALERT: qm_status: 890. N address=0x7ff630365ff8 frag=0x7ff630365fc0 size=144 used=1<br>
+ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347)<br>
+ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed<br>
+ALERT: qm_status: 894. N address=0x7ff6303662c8 frag=0x7ff630366290 size=128 used=1<br>
+ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347)<br>
+ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed<br>
+ALERT: qm_status: 919. N address=0x7ff630367550 frag=0x7ff630367518 size=136 used=1<br>
+ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347)<br>
+ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed<br>
+ALERT: qm_status: 922. N address=0x7ff630367e40 frag=0x7ff630367e08 size=136 used=1<br>
+ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347)<br>
+ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed<br>
+ALERT: qm_status: 932. N address=0x7ff6303684c8 frag=0x7ff630368490 size=128 used=1<br>
+ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347)<br>
+ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed<br>
+ALERT: qm_status: 975. N address=0x7ff63036a0a0 frag=0x7ff63036a068 size=144 used=1<br>
ALERT: qm_status: alloc'd from core: core/pvapi.c: pv_cache_add(347)<br>
ALERT: qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed<br>
<br>
</p>
<div class="moz-cite-prefix">Am 02.08.19 um 14:16 schrieb Andrew White:<br>
</div>
<blockquote type="cite" cite="mid:17A554B0-5B2B-4A2F-BF37-5973BB9DD180@uconnected.com.au">
Hi Henning!
<div class=""><br class="">
</div>
<div class="">Thanks so much!</div>
<div class=""><br class="">
</div>
<div class="">I spun up a dev machine with the same specs, Kamailio version and config. On each one, I’ve done a ps of the single receiver/worker process, as well as a full dump.</div>
<div class=""><br class="">
</div>
<div class="">0 calls:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/Vdrqkfy9Cm/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/Vdrqkfy9Cm/</a></div>
<div class=""><br class="">
</div>
<div class="">100 calls:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/d765GRqKmx/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/d765GRqKmx/</a></div>
<div class=""><br class="">
</div>
<div class="">200 calls:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/rB2rzgZN7S/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/rB2rzgZN7S/</a></div>
<div class=""><br class="">
</div>
<div class="">300 calls:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/mK4gjJ4tcP/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/mK4gjJ4tcP/</a></div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Interestingly there doesn’t appear to be a major difference from 100-300, which seems odd to me. I feel my dev server isn’t giving me results mirrored with production maybe.</div>
<div class=""><br class="">
</div>
<div class="">Please let me know what you think if you can!</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class="">Andrew<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 2 Aug 2019, at 6:19 am, Henning Westerholt <<a href="mailto:hw@skalatan.de" class="" moz-do-not-send="true">hw@skalatan.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Hi Andrew,</p>
<p class="">answer inline below<br class="">
</p>
<div class="moz-cite-prefix">Am 01.08.19 um 15:34 schrieb Andrew White:<br class="">
</div>
<blockquote type="cite" cite="mid:0D657D17-5035-425C-AD84-7812EA64C536@uconnected.com.au" class="">
Thanks Daniel, you’re fantastic!
<div class=""><br class="">
</div>
<div class="">I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.</div>
<div class=""><br class="">
</div>
<div class="">I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class=""><br class="">
</div>
<div class="">After about 90 calls:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
</div>
<div class=""><br class="">
</div>
<div class="">None of the other 9 Kamailio processes are increasing at all.</div>
<div class=""><br class="">
</div>
<div class="">I ran corex.pkg_summary against one of them and got the following dump:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/SqTF3K5knK/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/SqTF3K5knK/</a></div>
<div class=""><br class="">
</div>
<div class="">I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?</div>
<div class=""><br class="">
</div>
<div class="">Here’s another after another 60 calls:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/9WQXqZtfT2/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/9WQXqZtfT2/</a></div>
<div class=""><br class="">
</div>
</div>
</blockquote>
<p class="">There is not that much difference between the two runs (diff):</p>
<p class="">-journal: Suppressed 19796 messages from /system.slice/kamailio.service<br class="">
+journal: Suppressed 1772 messages from /system.slice/kamailio.service<br class="">
ALERT: <core> [core/pt.c:548]: mem_dump_pkg_cb(): Memory status (pkg) of process 28531:<br class="">
ALERT: qm_status: (0x7f0fc3a2a010):<br class="">
ALERT: qm_status: heap size= 33554432<br class="">
-ALERT: qm_status: used= 538712, used+overhead=883168, free=32671264<br class="">
-ALERT: qm_status: max used (+overhead)= 893040<br class="">
+ALERT: qm_status: used= 590552, used+overhead=939064, free=32615368<br class="">
+ALERT: qm_status: max used (+overhead)= 948112<br class="">
ALERT: qm_status: dumping all alloc'ed. fragments:<br class="">
ALERT: qm_status: 0. N address=0x7f0fc3a636d8 frag=0x7f0fc3a636a0 size=1024 used=1<br class="">
ALERT: qm_status: alloc'd from core: core/str_hash.h: str_hash_alloc(59)<br class="">
</p>
<p class="">The second run shows about 50k more memory usage, but from the dumped allocations they are identical.</p>
<p class="">Two suggestions:</p>
<p class="">- your logging system suppresses actually a lot of debugging output from Kamailio (have a look to the first lines each). Check if its necessary to increase the respective configuration variable in your logging conf to get all log messages.<br class="">
</p>
<p class="">- If you are able to execute this test on a test plattform, configure Kamailio to just use one worker child each, to spot an error easier. Then execute 100 calls, and count the memory increase, another 100 calls and count. Then you can estimate
how much bytes you actually leak per call. Try another memory dump and see if you spot something. Share your results again on the list.<br class="">
</p>
<blockquote type="cite" cite="mid:0D657D17-5035-425C-AD84-7812EA64C536@uconnected.com.au" class="">
<div class="">
<div class="">
<div class="">root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class=""><br class="">
</div>
<div class="">The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby
and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.</div>
<div class=""><br class="">
</div>
<div class="">It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.</div>
</div>
</div>
</blockquote>
<p class="">Have a look to the output of corex.ps or core.psx - the worker childs are named "receiver".<br class="">
</p>
<p class="">Cheers,</p>
<p class="">Henning<br class="">
</p>
<p class=""><br class="">
</p>
<p class=""><br class="">
</p>
<blockquote type="cite" cite="mid:0D657D17-5035-425C-AD84-7812EA64C536@uconnected.com.au" class="">
<div class="">
<div class="">
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" class="" moz-do-not-send="true">miconda@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Hello,</p>
<p class="">if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.</p>
<p class="">Once you indentify a process that is leaking pkg, execute the rpc command:</p>
<p class=""> - <a href="https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary" class="" moz-do-not-send="true">
https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary</a></p>
<p class="">When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.</p>
<p class="">Cheers,<br class="">
Daniel<br class="">
</p>
<p class=""><br class="">
</p>
<div class="moz-cite-prefix">On 01.08.19 03:43, Andrew White wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:1834350B-938B-4745-ACE4-051F63E45122@uconnected.com.au" class="">
<div dir="auto" style="word-wrap:
break-word; -webkit-nbsp-mode: space;
line-break: after-white-space;" class="">
Hi all,
<div class=""><br class="">
</div>
<div class="">I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.</div>
<div class=""><br class="">
</div>
<div class="">Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory
going on, mostly increasing during our busier daytime hours.</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://i.imgur.com/UTzx2k1.png" class="" moz-do-not-send="true">https://i.imgur.com/UTzx2k1.png</a></div>
<div class=""><br class="">
</div>
<div class="">Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.</div>
<div class=""><br class="">
</div>
<div class="">I’ve been reading through <a href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory" class="" moz-do-not-send="true">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a>, but I’m a bit nervous, as I’m not really a C/deep
memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com/" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Henning Westerholt - <a class="moz-txt-link-freetext" href="https://skalatan.de/blog/" moz-do-not-send="true">https://skalatan.de/blog/</a>
Kamailio services - <a class="moz-txt-link-freetext" href="https://skalatan.de/services" moz-do-not-send="true">https://skalatan.de/services</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Henning Westerholt - <a class="moz-txt-link-freetext" href="https://skalatan.de/blog/">https://skalatan.de/blog/</a>
Kamailio services - <a class="moz-txt-link-freetext" href="https://skalatan.de/services">https://skalatan.de/services</a></pre>
</body>
</html>