<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Daniel/Henning,<div class=""><br class=""></div><div class="">Thanks so much for your quick responses, they’re very much appreciated!</div><div class=""><br class=""></div><div class="">Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module. </div><div class=""><br class=""></div><div class="">Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):</div><div class=""><br class=""></div><div class=""><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">KSR::PV.sets(</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">"$du"</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">, destination[</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">'sip_uri’</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">])</span></div><div class=""><font color="#333333" face="Consolas, Liberation Mono, Menlo, Courier, monospace" class=""><span style="caret-color: rgb(51, 51, 51); white-space: pre; background-color: rgb(255, 255, 255);" class=""><br class=""></span></font></div><div class="">For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?</div><div class=""><span style="background-color: rgb(255, 255, 255);" class=""><br class=""></span></div><div class=""><span style="background-color: rgb(255, 255, 255);" class="">Thanks!</span></div><div class=""><span style="background-color: rgb(255, 255, 255);" class=""><br class=""></span></div><div class=""><span style="background-color: rgb(255, 255, 255);" class="">Andrew<br class=""></span><div class="">
<div><br class=""><blockquote type="cite" class=""><div class="">On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" class="">miconda@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hello,</p><p class="">are you defining a lot of kamailio variables in your ruby script?
In other words, are you using variables with different name, like
$var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in
ruby script and is changing depending of the sip message?</p><p class="">Cheers,<br class="">
Daniel<br class="">
</p>
<div class="moz-cite-prefix">On 01.08.19 15:34, Andrew White wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:0D657D17-5035-425C-AD84-7812EA64C536@uconnected.com.au" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
Thanks Daniel, you’re fantastic!
<div class=""><br class="">
</div>
<div class="">I have 4 children/workers configured with -m 128 -M
32. The machine in question has 512MB of memory, 1 core and 1GB
swap on an SSD.</div>
<div class=""><br class="">
</div>
<div class="">I restarted Kamailio with memlog=1 and I’ve been
sending batches of 30 calls in. I’ve noticed 4 of the 13
Kamailio processes going up in memory after each batch, which I
suspect to be the primary children/workers. Immediately post
restart:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root 28531 0.7 5.5 329368 27196 ? Sl
22:48 0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28532 0.6 4.9 329368 24528 ? Sl
22:48 0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28533 0.6 5.5 329368 27244 ? Sl
22:48 0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28534 0.7 5.4 329368 26788 ? Sl
22:48 0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class=""><br class="">
</div>
<div class="">After about 90 calls:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root 28531 0.0 6.7 330688 32948 ?
Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28532 0.0 6.5 330560 32264 ?
Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28533 0.0 6.5 330556 32272 ?
Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28534 0.0 6.6 330564 32592 ?
Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
</div>
<div class=""><br class="">
</div>
<div class="">None of the other 9 Kamailio processes are increasing at
all.</div>
<div class=""><br class="">
</div>
<div class="">I ran corex.pkg_summary against one of them and got the
following dump:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/SqTF3K5knK/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/SqTF3K5knK/</a></div>
<div class=""><br class="">
</div>
<div class="">I can see a lot of allocation to pvapi.c, does this
indicate maybe I’m setting PVs that need to be unset?</div>
<div class=""><br class="">
</div>
<div class="">Here’s another after another 60 calls:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/9WQXqZtfT2/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/9WQXqZtfT2/</a></div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root 28531 0.0 6.9 330820 33928 ? Sl 22:48
0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28532 0.0 6.7 330692 33352 ? Sl 22:48
0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28533 0.0 6.7 330688 33280 ? Sl 22:48
0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root 28534 0.0 6.7 330696 33192 ? Sl 22:48
0:00 /usr/local/sbin/kamailio -DD -P
/var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class=""><br class="">
</div>
<div class="">The only changes I’ve made on this config over the last
couple of weeks (since I saw this issue) is removing the
dispatcher module and adding in a small function in app_ruby
(which I already use) to query redis (which I also already
use from app_ruby and make a heap of queries per call) for
some values and write $du manually. I also added in the
topoh module.</div>
<div class=""><br class="">
</div>
<div class="">It also makes a lot of sense to me to monitor the
individual processes rather than the aggregate. Is there a
way to identify simply from bash what processes are workers
programmatically? I’d like to monitor just those
individually in my monitoring.</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 1 Aug 2019, at 8:24 pm, Daniel-Constantin
Mierla <<a href="mailto:miconda@gmail.com" class="" moz-do-not-send="true">miconda@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hello,</p><p class="">if it is pkg, then you have to see which
process is increasing the use of memory, because it is
private memory, specific for each process. The sum is
an indicator, but the debugging has to be done for a
specific process/pid.</p><p class="">Once you indentify a process that is leaking
pkg, execute the rpc command:</p><p class=""> - <a href="https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary" class="" moz-do-not-send="true">https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary</a></p><p class="">When that process is doing some runtime work
(e.g., handling of a sip message), the syslog will get
a summary with used pkg chunks. Send those log
messages here for analysis. You have to set memlog
core parameter to a value smaller than debug.</p><p class="">Cheers,<br class="">
Daniel<br class="">
</p><p class=""><br class="">
</p>
<div class="moz-cite-prefix">On 01.08.19 03:43, Andrew
White wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:1834350B-938B-4745-ACE4-051F63E45122@uconnected.com.au" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div dir="auto" style="word-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;" class="">Hi all,
<div class=""><br class="">
</div>
<div class="">I had a Kamailio crash the other day,
and some debugging showed I ran out of PKG memory.</div>
<div class=""><br class="">
</div>
<div class="">Since then I’ve run a simple bash
script to compile the amount of memory used by all
child processes, effective /usr/local/sbin/kamcmd
pkg.stats | grep real_used summed together. I’ve
graphed out the data, and there’s a clear growth
of PKG memory going on, mostly increasing during
our busier daytime hours.</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://i.imgur.com/UTzx2k1.png" class="" moz-do-not-send="true">https://i.imgur.com/UTzx2k1.png</a></div>
<div class=""><br class="">
</div>
<div class="">Based on this, I suspect either a
module loaded or something within my app_ruby conf
is leaking memory.</div>
<div class=""><br class="">
</div>
<div class="">I’ve been reading through <a href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory" class="" moz-do-not-send="true">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a>,
but I’m a bit nervous, as I’m not really a C/deep
memory type of guy. I can see a GDB script I can
attach to Kamailio, but is that going to use
significant resources to run or impact the running
process? Is there a newer/better/alternative way
to do this, and to help me break this down?</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com/" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com/">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a></pre>
</div>
</div></blockquote></div><br class=""></div></div></body></html>