<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Very interesting results!<div class=""><br class=""></div><div class="">11pm last night:</div><div class=""><a href="https://paste.ubuntu.com/p/qMz2tN7bcR/" class="">https://paste.ubuntu.com/p/qMz2tN7bcR/</a></div><div class=""><br class=""></div><div class="">2pm today:</div><div class=""><a href="https://paste.ubuntu.com/p/QDmTwpV83X/" class="">https://paste.ubuntu.com/p/QDmTwpV83X/</a></div><div class=""><br class=""></div><div class="">The growth is pretty small. The only variance is disabling the topoh module.</div><div class=""><br class=""></div><div class="">I’m not big on C, but I think this indicates a leak in the topoh module, does this sound right?</div><div class=""><br class=""></div><div class="">I’m happy to put in an issue on GitHub if that’s necessary here.</div><div class=""><br class=""></div><div class="">Cheers,</div><div class=""><br class=""></div><div class="">Andrew<br class="">
<div><br class=""><blockquote type="cite" class=""><div class="">On 5 Aug 2019, at 11:30 pm, Andrew White <<a href="mailto:andrew@uconnected.com.au" class="">andrew@uconnected.com.au</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hey Henning,<div class=""><br class=""></div><div class="">Unfortunately I’ve had some troubles with app_ruby in dev-7. Interestingly, dev-7 shows as “dirty” too:</div><div class=""><br class=""></div><div class="">version: kamailio 5.3.0-dev7 (x86_64/linux) bd477f-dirty</div><div class=""><br class=""></div><div class="">Here’s the ansible playbook I wrote to pull the specific version, there’s no fancy patching or anything going on:</div><div class=""><br class=""></div><div class=""><div style="color: rgb(212, 212, 212); background-color: rgb(30, 30, 30); font-family: Menlo, Monaco, "Courier New", monospace; line-height: 18px; white-space: pre;" class=""><div class=""> - <span style="color: #569cd6;" class="">name</span>: <span style="color: #ce9178;" class="">Download kamailio</span></div><div class=""> <span style="color: #569cd6;" class="">git</span>: </div><div class=""> <span style="color: #569cd6;" class="">repo</span>: <span style="color: #ce9178;" class=""><a href="https://github.com/kamailio/kamailio.git" class="">https://github.com/kamailio/kamailio.git</a></span></div><div class=""> <span style="color: #569cd6;" class="">version</span>: <span style="color: #ce9178;" class="">"{{ kamailio_version }}"</span></div><div class=""> <span style="color: #569cd6;" class="">dest</span>: <span style="color: #ce9178;" class="">/usr/src/kamailio</span></div><br class=""><div class=""> - <span style="color: #569cd6;" class="">name</span>: <span style="color: #ce9178;" class="">Roll back to specific commit</span></div><div class=""> <span style="color: #569cd6;" class="">command</span>: <span style="color: #ce9178;" class="">"git reset --hard {{ kamailio_commit }}"</span></div><div class=""> <span style="color: #569cd6;" class="">args</span>:</div><div class=""> <span style="color: #569cd6;" class="">chdir</span>: <span style="color: #ce9178;" class="">/usr/src/kamailio</span></div><div class=""> <span style="color: #569cd6;" class="">when</span>: <span style="color: #ce9178;" class="">kamailio_commit is defined</span> </div><br class=""><div class=""> - <span style="color: #569cd6;" class="">name</span>: <span style="color: #ce9178;" class="">Build kamailio from source</span></div><div class=""> <span style="color: #569cd6;" class="">command</span>: <span style="color: #ce9178;" class="">"{{ item }} chdir=/usr/src/kamailio"</span></div><div class=""> <span style="color: #569cd6;" class="">with_items</span>:</div><div class=""> - <span style="color: #ce9178;" class="">make include_modules="app_ruby db_redis" cfg</span></div><div class=""> - <span style="color: #ce9178;" class="">make all</span></div><div class=""> - <span style="color: #ce9178;" class="">make install</span></div><div class=""> <span style="color: #569cd6;" class="">args</span>:</div><div class=""> <span style="color: #569cd6;" class="">creates</span>: <span style="color: #ce9178;" class="">/usr/local/sbin/kamailio</span></div></div><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class=""></div></div></div>
</div>
<div class="">Either way, I’ve removed the topoh module from my production (dev-4) machine, and re-taken memory information. I’ll give it about 12 hours and see where memory is sitting. I figure this should isolate if it’s a topoh issue or not.</div><div class=""><br class=""></div><div class="">Thanks!</div><div class=""><br class=""></div><div class="">Andrew</div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 5 Aug 2019, at 10:46 am, Andrew White <<a href="mailto:andrew@uconnected.com.au" class="">andrew@uconnected.com.au</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Henning,<div class=""><br class=""></div><div class="">That is interesting, I don’t know what the dirty version is. My deploy script I use for Kamailio pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as this had a fix in place Daniel did for app_ruby a while back. I don’t recall ever putting on any custom patches or modules. I compile from source on build from that git commit using:</div><div class=""><br class=""></div><div class=""><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">make</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class=""> </span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">include_modules="app_ruby</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class=""> </span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">db_redis"</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class=""> </span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">cfg</span><br class="">
<div class=""><br class=""></div><div class="">I’ll deploy dev7 tonight and report back!</div><div class=""><br class=""></div><div class="">Thanks,</div><div class=""><br class=""></div><div class="">Andrew</div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 4 Aug 2019, at 8:28 pm, Henning Westerholt <<a href="mailto:hw@skalatan.de" class="">hw@skalatan.de</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hi Andrew,</p><p class="">dev4 version - I see. I quickly looked into the topos module commits, but there have been not that much changes there. But can you try nevertheless to update to the latest git master (e.g. dev7) version on your test setup?</p><p class="">What about the "dirty" version flag, do you have some own patches or modules that you use?</p><p class="">Cheers,</p><p class="">Henning<br class="">
</p>
<div class="moz-cite-prefix">Am 04.08.19 um 01:42 schrieb Andrew White:<br class="">
</div>
<blockquote type="cite" cite="mid:3BF06C6B-18E7-4068-BE21-5D87F6D7E8A0@uconnected.com.au" class="">
Hey Henning,
<div class=""><br class="">
</div>
<div class="">It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty. </div>
<div class=""><br class="">
</div>
<div class="">I don’t think there’s any extra messages. The actual dispatching is simple, here’s the code:</div>
<div class=""><br class="">
</div>
<div class="">
<ol class="linenums" style="box-sizing: inherit; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; background-color: rgb(255, 255, 255); margin: 0px; caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); white-space: pre; padding: 0px !important;">
<li class="L216" rel="L216" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
<span class="hljs-keyword" style="box-sizing: inherit; font-weight: bold; line-height: 20px;">if</span> destination
<span class="hljs-keyword" style="box-sizing: inherit; font-weight: bold; line-height: 20px;">
then</span> </li><li class="L217" rel="L217" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
KSR::PV.sets(<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">"$du"</span>, destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'sip_uri'</span>])
</li><li class="L218" rel="L218" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
pbx = destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'display_name'</span>]
</li><li class="L219" rel="L219" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
<a href="http://ksr.info/" class="" moz-do-not-send="true">KSR.info</a>(<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">"Chose
<span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">
#{pbx}</span> as inbound PBX, as it's received <span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">
#{destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'count'</span>].to_i}</span>/<span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">#{destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'total_count'</span>].to_i}</span>
(<span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">#{destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'calculated_percentage'</span>].to_i}</span>%)
of calls in this ruleset, where its max is <span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">
#{destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'percentage'</span>].to_i}</span>%"</span>)
</li><li class="L220" rel="L220" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
<span class="hljs-keyword" style="box-sizing: inherit; font-weight: bold; line-height: 20px;">end</span></li></ol>
<div class=""><br class="">
</div>
<div class="">The code before that just pulls data from redis and returns a ruby hash with where we should send the call, so no kamailio variables set at all.</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 3 Aug 2019, at 8:03 pm, Henning Westerholt <<a href="mailto:hw@skalatan.de" class="" moz-do-not-send="true">hw@skalatan.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hi Andrew,</p><p class="">interesting insight from the logs indeed. According to your changes this looks indeed like related to the topoh module or custom dispatcher code. Which version of Kamailio do you use?
<br class="">
</p><p class="">Are you sending out some messages /replies as well from your custom dispatcher app_ruby logic?</p><p class="">Cheers,</p><p class="">Henning<br class="">
</p>
<div class="moz-cite-prefix">Am 03.08.19 um 09:32 schrieb Andrew White:<br class="">
</div>
<blockquote type="cite" cite="mid:079E416B-4B04-45AC-BAE4-313F4CD3E8FB@uconnected.com.au" class="">
Hi all,
<div class=""><br class="">
</div>
<div class="">I’ve done two outputs about 8 hours/3200 calls in between:</div>
<div class=""><br class="">
</div>
<div class="">First one (about 5 hours since last restart):</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/Y6Wvd9hsV2/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/Y6Wvd9hsV2/</a></div>
<div class="">
<div class="">
<div dir="auto" style="text-align: start;
text-indent: 0px; word-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;" class="">
<div dir="auto" style="word-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;" class="">
<div style="caret-color: rgb(0, 0, 0);
letter-spacing: normal; text-transform:
none; white-space: normal; word-spacing:
0px; text-decoration: none;
-webkit-text-stroke-width: 0px; font-family:
Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal;
font-weight: normal; text-align: start;
text-indent: 0px;" class="">
<br class="">
Second one (about 8 hours after the first, 3200 odd cals later):</div>
<div style="caret-color: rgb(0, 0, 0);
letter-spacing: normal; text-transform:
none; white-space: normal; word-spacing:
0px; text-decoration: none;
-webkit-text-stroke-width: 0px; font-family:
Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal;
font-weight: normal; text-align: start;
text-indent: 0px;" class="">
<br class="">
</div>
<div style="text-align: start; text-indent:
0px;" class="">
<a href="https://paste.ubuntu.com/p/85KYg7gD2z/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/85KYg7gD2z/</a></div>
<div style="text-align: start; text-indent:
0px;" class="">
<br class="">
</div>
<div style="text-align: start; text-indent:
0px;" class="">
It looks like roughly 7MB of extra memory is being used on that process, which is 1/4 active workers (I can see all of them have grown roughly the same amount in that time). Looking at the memory status, there appear to be about 14,000 more lines. A cursory
glance shows about 6k of those lines with msg_translator.c (compared to 1k in the first).</div>
<div style="text-align: start; text-indent:
0px;" class="">
<br class="">
</div>
<div style="text-align: start; text-indent:
0px;" class="">
I wonder if maybe this could be related to my use of the topoh module, given every message is now being touched by it to mask the origin?</div>
<div style="text-align: start; text-indent:
0px;" class="">
<br class="">
</div>
<div style="text-align: start; text-indent:
0px;" class="">
Thanks!</div>
<div style="text-align: start; text-indent:
0px;" class="">
<br class="">
</div>
<div style="text-align: start; text-indent:
0px;" class="">
Andrew</div>
</div>
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 3 Aug 2019, at 10:07 am, Andrew White <<a href="mailto:andrew@uconnected.com.au" class="" moz-do-not-send="true">andrew@uconnected.com.au</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;" class="">
Hi Daniel/Henning,
<div class=""><br class="">
</div>
<div class="">Thanks so much for your quick responses, they’re very much appreciated!</div>
<div class=""><br class="">
</div>
<div class="">Daniel: I don’t believe so. I’ve just checked over the code, and I only have 9 instances of PV.sets throughout the entire script. Additionally, this issue appears to have only cropped up in the last week or two (I didn’t monitor PKG memory at
the time, but we didn’t have any memory related crashes before that point), which leads me to believe it’s related to my config code, the adding of the topoh module or the removal of the dispatcher module. </div>
<div class=""><br class="">
</div>
<div class="">Within that config code, there’s only one additional Kamailio variable set, which is part of my replacement/custom dispatcher (app_ruby):</div>
<div class=""><br class="">
</div>
<div class=""><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">KSR::PV.sets(</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">"$du"</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">,
destination[</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">'sip_uri’</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">])</span></div>
<div class=""><font class="" face="Consolas,
Liberation Mono, Menlo, Courier,
monospace" color="#333333"><span style="caret-color: rgb(51, 51, 51); white-space: pre; background-color: rgb(255, 255, 255);" class=""></span></font></div>
<div class="">For now, I’ve taken the output of a mem dump of one of the production workers, ps output, etc. I’ll leave that few a few hours, monitor to ensure there is a measurable memory increase, and send through the logs if that’s ok?</div>
<div class=""><span style="background-color:
rgb(255, 255, 255);" class=""><br class="">
</span></div>
<div class=""><span style="background-color:
rgb(255, 255, 255);" class="">Thanks!</span></div>
<div class=""><span style="background-color:
rgb(255, 255, 255);" class=""><br class="">
</span></div>
<div class=""><span style="background-color:
rgb(255, 255, 255);" class="">Andrew<br class="">
</span>
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" class="" moz-do-not-send="true">miconda@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hello,</p><p class="">are you defining a lot of kamailio variables in your ruby script? In other words, are you using variables with different name, like $var(xyz) or $sht(a=>xyz), where xyz is passed from/computed in ruby script and is changing depending of the sip
message?</p><p class="">Cheers,<br class="">
Daniel<br class="">
</p>
<div class="moz-cite-prefix">On 01.08.19 15:34, Andrew White wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:0D657D17-5035-425C-AD84-7812EA64C536@uconnected.com.au" class="">
Thanks Daniel, you’re fantastic!
<div class=""><br class="">
</div>
<div class="">I have 4 children/workers configured with -m 128 -M 32. The machine in question has 512MB of memory, 1 core and 1GB swap on an SSD.</div>
<div class=""><br class="">
</div>
<div class="">I restarted Kamailio with memlog=1 and I’ve been sending batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio processes going up in memory after each batch, which I suspect to be the primary children/workers. Immediately post restart:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root 28531 0.7 5.5 329368 27196 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28532 0.6 4.9 329368 24528 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28533 0.6 5.5 329368 27244 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28534 0.7 5.4 329368 26788 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class=""><br class="">
</div>
<div class="">After about 90 calls:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root 28531 0.0 6.7 330688 32948 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28532 0.0 6.5 330560 32264 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28533 0.0 6.5 330556 32272 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28534 0.0 6.6 330564 32592 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
</div>
<div class=""><br class="">
</div>
<div class="">None of the other 9 Kamailio processes are increasing at all.</div>
<div class=""><br class="">
</div>
<div class="">I ran corex.pkg_summary against one of them and got the following dump:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/SqTF3K5knK/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/SqTF3K5knK/</a></div>
<div class=""><br class="">
</div>
<div class="">I can see a lot of allocation to pvapi.c, does this indicate maybe I’m setting PVs that need to be unset?</div>
<div class=""><br class="">
</div>
<div class="">Here’s another after another 60 calls:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://paste.ubuntu.com/p/9WQXqZtfT2/" class="" moz-do-not-send="true">https://paste.ubuntu.com/p/9WQXqZtfT2/</a></div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root 28531 0.0 6.9 330820 33928 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28532 0.0 6.7 330692 33352 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28533 0.0 6.7 330688 33280 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class="">root 28534 0.0 6.7 330696 33192 ? Sl 22:48 0:00 /usr/local/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg -m 128 -M 32</div>
<div class=""><br class="">
</div>
<div class="">The only changes I’ve made on this config over the last couple of weeks (since I saw this issue) is removing the dispatcher module and adding in a small function in app_ruby (which I already use) to query redis (which I also already use from app_ruby
and make a heap of queries per call) for some values and write $du manually. I also added in the topoh module.</div>
<div class=""><br class="">
</div>
<div class="">It also makes a lot of sense to me to monitor the individual processes rather than the aggregate. Is there a way to identify simply from bash what processes are workers programmatically? I’d like to monitor just those individually in my monitoring.</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" class="" moz-do-not-send="true">miconda@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hello,</p><p class="">if it is pkg, then you have to see which process is increasing the use of memory, because it is private memory, specific for each process. The sum is an indicator, but the debugging has to be done for a specific process/pid.</p><p class="">Once you indentify a process that is leaking pkg, execute the rpc command:</p><p class=""> - <a href="https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary" class="" moz-do-not-send="true">
https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary</a></p><p class="">When that process is doing some runtime work (e.g., handling of a sip message), the syslog will get a summary with used pkg chunks. Send those log messages here for analysis. You have to set memlog core parameter to a value smaller than debug.</p><p class="">Cheers,<br class="">
Daniel<br class="">
</p><p class=""><br class="">
</p>
<div class="moz-cite-prefix">On 01.08.19 03:43, Andrew White wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:1834350B-938B-4745-ACE4-051F63E45122@uconnected.com.au" class="">
<div dir="auto" style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
line-break:
after-white-space;" class="">
Hi all,
<div class=""><br class="">
</div>
<div class="">I had a Kamailio crash the other day, and some debugging showed I ran out of PKG memory.</div>
<div class=""><br class="">
</div>
<div class="">Since then I’ve run a simple bash script to compile the amount of memory used by all child processes, effective /usr/local/sbin/kamcmd pkg.stats | grep real_used summed together. I’ve graphed out the data, and there’s a clear growth of PKG memory
going on, mostly increasing during our busier daytime hours.</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://i.imgur.com/UTzx2k1.png" class="" moz-do-not-send="true">https://i.imgur.com/UTzx2k1.png</a></div>
<div class=""><br class="">
</div>
<div class="">Based on this, I suspect either a module loaded or something within my app_ruby conf is leaking memory.</div>
<div class=""><br class="">
</div>
<div class="">I’ve been reading through <a href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory" class="" moz-do-not-send="true">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a>, but I’m a bit nervous, as I’m not really a C/deep
memory type of guy. I can see a GDB script I can attach to Kamailio, but is that going to use significant resources to run or impact the running process? Is there a newer/better/alternative way to do this, and to help me break this down?</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com/" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com/" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Henning Westerholt - <a class="moz-txt-link-freetext" href="https://skalatan.de/blog/" moz-do-not-send="true">https://skalatan.de/blog/</a>
Kamailio services - <a class="moz-txt-link-freetext" href="https://skalatan.de/services" moz-do-not-send="true">https://skalatan.de/services</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Henning Westerholt - <a class="moz-txt-link-freetext" href="https://skalatan.de/blog/">https://skalatan.de/blog/</a>
Kamailio services - <a class="moz-txt-link-freetext" href="https://skalatan.de/services">https://skalatan.de/services</a></pre>
</div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></body></html>