<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>To close this discussion - patches for fixing this issue are now
in master and 5.2 branches.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 09.08.19 09:12, Andrew White wrote:<br>
</div>
<blockquote type="cite"
cite="mid:1299AE03-ADB5-4FDC-8C76-A3696E9A0580@uconnected.com.au">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
If anyone is following this in the future and looking for further
information, here is the GitHub issue:
<div class=""><br class="">
</div>
<div class=""><a
href="https://github.com/kamailio/kamailio/issues/2027"
class="" moz-do-not-send="true">https://github.com/kamailio/kamailio/issues/2027</a></div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 6 Aug 2019, at 5:23 pm, Henning Westerholt
<<a href="mailto:hw@skalatan.de" class=""
moz-do-not-send="true">hw@skalatan.de</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Hello Andrew,</p>
<p class="">indeed, the difference is only 1.4k or so
memory. This is probably just an effect of the
different time when you did the dump.</p>
<p class="">Yes, an Github issue would be great. Add
this dump as well <a class="moz-txt-link-freetext"
href="https://paste.ubuntu.com/p/85KYg7gD2z/"
moz-do-not-send="true">
https://paste.ubuntu.com/p/85KYg7gD2z/</a></p>
<p class="">Would be perfect if you could reproduce it
with current git master, though (if the app_ruby
problem is also solved).</p>
<p class="">Cheers,</p>
<p class="">Henning<br class="">
</p>
<div class="moz-cite-prefix">Am 06.08.19 um 09:17
schrieb Andrew White:<br class="">
</div>
<blockquote type="cite"
cite="mid:A5A2A885-05EB-438C-8EB0-D8D01DDA17CE@uconnected.com.au"
class="">
Very interesting results!
<div class=""><br class="">
</div>
<div class="">11pm last night:</div>
<div class=""><a
href="https://paste.ubuntu.com/p/qMz2tN7bcR/"
class="" moz-do-not-send="true">https://paste.ubuntu.com/p/qMz2tN7bcR/</a></div>
<div class=""><br class="">
</div>
<div class="">2pm today:</div>
<div class=""><a
href="https://paste.ubuntu.com/p/QDmTwpV83X/"
class="" moz-do-not-send="true">https://paste.ubuntu.com/p/QDmTwpV83X/</a></div>
<div class=""><br class="">
</div>
<div class="">The growth is pretty small. The only
variance is disabling the topoh module.</div>
<div class=""><br class="">
</div>
<div class="">I’m not big on C, but I think this
indicates a leak in the topoh module, does this
sound right?</div>
<div class=""><br class="">
</div>
<div class="">I’m happy to put in an issue on GitHub
if that’s necessary here.</div>
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class=""><br class="">
</div>
<div class="">Andrew<br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 5 Aug 2019, at 11:30 pm, Andrew
White <<a
href="mailto:andrew@uconnected.com.au"
class="" moz-do-not-send="true">andrew@uconnected.com.au</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; line-break:
after-white-space;" class="">
Hey Henning,
<div class=""><br class="">
</div>
<div class="">Unfortunately I’ve had some
troubles with app_ruby in dev-7.
Interestingly, dev-7 shows as “dirty” too:</div>
<div class=""><br class="">
</div>
<div class="">version: kamailio 5.3.0-dev7
(x86_64/linux) bd477f-dirty</div>
<div class=""><br class="">
</div>
<div class="">Here’s the ansible playbook I
wrote to pull the specific version,
there’s no fancy patching or anything
going on:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="color: rgb(212, 212, 212); background-color: rgb(30, 30, 30); font-family: Menlo, Monaco, "Courier New", monospace; line-height: 18px; white-space: pre;" class="">
<div class="">- <span style="color: #569cd6;" class="">name</span>: <span style="color: #ce9178;" class="">
Download kamailio</span></div>
<div class=""><span style="color: #569cd6;" class="">git</span>: </div>
<div class=""><span style="color: #569cd6;" class="">repo</span>: <span style="color: #ce9178;" class="">
<a href="https://github.com/kamailio/kamailio.git" class="" moz-do-not-send="true">https://github.com/kamailio/kamailio.git</a></span></div>
<div class=""><span style="color: #569cd6;" class="">version</span>: <span style="color: #ce9178;" class="">
"{{ kamailio_version }}"</span></div>
<div class=""><span style="color: #569cd6;" class="">dest</span>: <span style="color: #ce9178;" class="">
/usr/src/kamailio</span></div>
<div class="">- <span style="color: #569cd6;" class="">name</span>: <span style="color: #ce9178;" class="">
Roll back to specific commit</span></div>
<div class=""><span style="color: #569cd6;" class="">command</span>: <span style="color: #ce9178;" class="">
"git reset --hard {{ kamailio_commit }}"</span></div>
<div class=""><span style="color: #569cd6;" class="">args</span>:</div>
<div class=""><span style="color: #569cd6;" class="">chdir</span>: <span style="color: #ce9178;" class="">
/usr/src/kamailio</span></div>
<div class=""><span style="color: #569cd6;" class="">when</span>: <span style="color: #ce9178;" class="">
kamailio_commit is defined</span> </div>
<div class="">- <span style="color: #569cd6;" class="">name</span>: <span style="color: #ce9178;" class="">
Build kamailio from source</span></div>
<div class=""><span style="color: #569cd6;" class="">command</span>: <span style="color: #ce9178;" class="">
"{{ item }} chdir=/usr/src/kamailio"</span></div>
<div class=""><span style="color: #569cd6;" class="">with_items</span>:</div>
<div class="">- <span style="color: #ce9178;" class="">make include_modules="app_ruby db_redis" cfg</span></div>
<div class="">- <span style="color: #ce9178;" class="">make all</span></div>
<div class="">- <span style="color: #ce9178;" class="">make install</span></div>
<div class=""><span style="color: #569cd6;" class="">args</span>:</div>
<div class=""><span style="color: #569cd6;" class="">creates</span>: <span style="color: #ce9178;" class="">
/usr/local/sbin/kamailio</span></div>
</div>
<div class="">
<div dir="auto" style="caret-color:
rgb(0, 0, 0); letter-spacing: normal;
text-align: start; text-indent: 0px;
text-transform: none; white-space:
normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px;
text-decoration: none; word-wrap:
break-word; -webkit-nbsp-mode: space;
line-break: after-white-space;"
class="">
<div dir="auto" style="word-wrap:
break-word; -webkit-nbsp-mode:
space; line-break:
after-white-space;" class="">
<div style="caret-color: rgb(0, 0,
0); font-family: Helvetica;
font-size: 12px; font-style:
normal; font-variant-caps: normal;
font-weight: normal;
letter-spacing: normal;
text-align: start; text-indent:
0px; text-transform: none;
white-space: normal; word-spacing:
0px; -webkit-text-stroke-width:
0px; text-decoration: none;"
class="">
<br class="">
</div>
</div>
</div>
</div>
<div class="">Either way, I’ve removed the
topoh module from my production (dev-4)
machine, and re-taken memory
information. I’ll give it about 12 hours
and see where memory is sitting. I
figure this should isolate if it’s a
topoh issue or not.</div>
<div class=""><br class="">
</div>
<div class="">Thanks!</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 5 Aug 2019, at 10:46
am, Andrew White <<a
href="mailto:andrew@uconnected.com.au"
class="" moz-do-not-send="true">andrew@uconnected.com.au</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
line-break: after-white-space;"
class="">
Hi Henning,
<div class=""><br class="">
</div>
<div class="">That is interesting,
I don’t know what the dirty
version is. My deploy script I
use for Kamailio
pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6
from GitHub, as this had a fix
in place Daniel did for app_ruby
a while back. I don’t recall
ever putting on any custom
patches or modules. I compile
from source on build from that
git commit using:</div>
<div class=""><br class="">
</div>
<div class=""><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">make</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">
</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">include_modules="app_ruby</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">
</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">db_redis"</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">
</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">cfg</span><br
class="">
<div class=""><br class="">
</div>
<div class="">I’ll deploy dev7
tonight and report back!</div>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class=""><br class="">
</div>
<div class="">Andrew</div>
<div class=""><br class="">
<blockquote type="cite"
class="">
<div class="">On 4 Aug 2019,
at 8:28 pm, Henning
Westerholt <<a
href="mailto:hw@skalatan.de"
class=""
moz-do-not-send="true">hw@skalatan.de</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div text="#000000"
bgcolor="#FFFFFF"
class="">
<p class="">Hi Andrew,</p>
<p class="">dev4 version
- I see. I quickly
looked into the topos
module commits, but
there have been not
that much changes
there. But can you try
nevertheless to update
to the latest git
master (e.g. dev7)
version on your test
setup?</p>
<p class="">What about
the "dirty" version
flag, do you have some
own patches or modules
that you use?</p>
<p class="">Cheers,</p>
<p class="">Henning<br
class="">
</p>
<div
class="moz-cite-prefix">Am
04.08.19 um 01:42
schrieb Andrew White:<br
class="">
</div>
<blockquote type="cite"
cite="mid:3BF06C6B-18E7-4068-BE21-5D87F6D7E8A0@uconnected.com.au"
class="">
Hey Henning,
<div class=""><br
class="">
</div>
<div class="">It’s
currently version:
kamailio 5.3.0-dev4
(x86_64/linux)
6d43ea-dirty. </div>
<div class=""><br
class="">
</div>
<div class="">I don’t
think there’s any
extra messages. The
actual dispatching
is simple, here’s
the code:</div>
<div class=""><br
class="">
</div>
<div class="">
<ol class="linenums" style="box-sizing: inherit; font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; background-color: rgb(255, 255, 255); margin: 0px; caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); white-space: pre; padding: 0px !important;">
<li class="L216" rel="L216" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
<span class="hljs-keyword" style="box-sizing: inherit; font-weight: bold; line-height: 20px;">if</span> destination
<span class="hljs-keyword" style="box-sizing: inherit; font-weight: bold; line-height: 20px;">
then</span> </li><li class="L217" rel="L217" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
KSR::PV.sets(<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">"$du"</span>, destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'sip_uri'</span>])
</li><li class="L218" rel="L218" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
pbx = destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'display_name'</span>]
</li><li class="L219" rel="L219" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
<a href="http://ksr.info/" class="" moz-do-not-send="true">KSR.info</a>(<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">"Chose
<span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">
#{pbx}</span> as inbound PBX, as it's received <span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">
#{destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'count'</span>].to_i}</span>/<span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">#{destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'total_count'</span>].to_i}</span>
(<span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">#{destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'calculated_percentage'</span>].to_i}</span>%)
of calls in this ruleset, where its max is <span class="hljs-subst" style="box-sizing: inherit; color: rgb(51, 51, 51); line-height: 20px;">
#{destination[<span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); line-height: 20px;">'percentage'</span>].to_i}</span>%"</span>)
</li><li class="L220" rel="L220" style="box-sizing: inherit; line-height: 20px; display: block; width: 1835.0625px;">
<span class="hljs-keyword" style="box-sizing: inherit; font-weight: bold; line-height: 20px;">end</span></li></ol>
<div class=""><br
class="">
</div>
<div class="">The
code before that
just pulls data
from redis and
returns a ruby
hash with where we
should send the
call, so no
kamailio variables
set at all.</div>
<div class=""><br
class="">
</div>
<div class="">Thanks!</div>
<div class=""><br
class="">
</div>
<div class="">Andrew</div>
<div class=""><br
class="">
<blockquote
type="cite"
class="">
<div class="">On
3 Aug 2019, at
8:03 pm,
Henning
Westerholt
<<a
href="mailto:hw@skalatan.de"
class=""
moz-do-not-send="true">hw@skalatan.de</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div
text="#000000"
bgcolor="#FFFFFF" class="">
<p class="">Hi
Andrew,</p>
<p class="">interesting
insight from
the logs
indeed.
According to
your changes
this looks
indeed like
related to the
topoh module
or custom
dispatcher
code. Which
version of
Kamailio do
you use?
<br class="">
</p>
<p class="">Are
you sending
out some
messages
/replies as
well from your
custom
dispatcher
app_ruby
logic?</p>
<p class="">Cheers,</p>
<p class="">Henning<br
class="">
</p>
<div
class="moz-cite-prefix">Am
03.08.19 um
09:32 schrieb
Andrew White:<br
class="">
</div>
<blockquote
type="cite"
cite="mid:079E416B-4B04-45AC-BAE4-313F4CD3E8FB@uconnected.com.au"
class="">
Hi all,
<div class=""><br
class="">
</div>
<div class="">I’ve
done two
outputs about
8 hours/3200
calls in
between:</div>
<div class=""><br
class="">
</div>
<div class="">First
one (about 5
hours since
last restart):</div>
<div class=""><br
class="">
</div>
<div class=""><a
href="https://paste.ubuntu.com/p/Y6Wvd9hsV2/" class=""
moz-do-not-send="true">https://paste.ubuntu.com/p/Y6Wvd9hsV2/</a></div>
<div class="">
<div class="">
<div
dir="auto"
style="text-align:
start;
text-indent:
0px;
word-wrap:
break-word;
-webkit-nbsp-mode:
space;
line-break:
after-white-space;"
class="">
<div
dir="auto"
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
line-break:
after-white-space;"
class="">
<div
style="caret-color:
rgb(0, 0, 0);
letter-spacing: normal; text-transform: none; white-space: normal;
word-spacing:
0px;
text-decoration:
none;
-webkit-text-stroke-width:
0px;
font-family:
Helvetica;
font-size:
12px;
font-style:
normal;
font-variant-caps:
normal;
font-weight:
normal;
text-align:
start;
text-indent:
0px;" class="">
<br class="">
Second one
(about 8 hours
after the
first, 3200
odd cals
later):</div>
<div
style="caret-color:
rgb(0, 0, 0);
letter-spacing: normal; text-transform: none; white-space: normal;
word-spacing:
0px;
text-decoration:
none;
-webkit-text-stroke-width:
0px;
font-family:
Helvetica;
font-size:
12px;
font-style:
normal;
font-variant-caps:
normal;
font-weight:
normal;
text-align:
start;
text-indent:
0px;" class="">
<br class="">
</div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
<a
href="https://paste.ubuntu.com/p/85KYg7gD2z/"
class=""
moz-do-not-send="true">https://paste.ubuntu.com/p/85KYg7gD2z/</a></div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
<br class="">
</div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
It looks like
roughly 7MB of
extra memory
is being used
on that
process, which
is 1/4 active
workers (I can
see all of
them have
grown roughly
the same
amount in that
time). Looking
at the memory
status, there
appear to be
about 14,000
more lines. A
cursory glance
shows about 6k
of those lines
with msg_translator.c (compared to 1k in the first).</div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
<br class="">
</div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
I wonder if
maybe this
could be
related to my
use of the
topoh module,
given every
message is now
being touched
by it to mask
the origin?</div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
<br class="">
</div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
Thanks!</div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
<br class="">
</div>
<div
style="text-align:
start;
text-indent:
0px;" class="">
Andrew</div>
</div>
</div>
</div>
<div class=""><br
class="">
<blockquote
type="cite"
class="">
<div class="">On
3 Aug 2019, at
10:07 am,
Andrew White
<<a
href="mailto:andrew@uconnected.com.au"
class=""
moz-do-not-send="true">andrew@uconnected.com.au</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
line-break:
after-white-space;"
class="">
Hi
Daniel/Henning,
<div class=""><br
class="">
</div>
<div class="">Thanks
so much for
your quick
responses,
they’re very
much
appreciated!</div>
<div class=""><br
class="">
</div>
<div class="">Daniel:
I don’t
believe so.
I’ve just
checked over
the code, and
I only have 9
instances of
PV.sets
throughout the
entire script.
Additionally,
this issue
appears to
have only
cropped up in
the last week
or two (I
didn’t monitor
PKG memory at
the time, but
we didn’t have
any memory
related
crashes before
that point),
which leads me
to believe
it’s related
to my config
code, the
adding of the
topoh module
or the removal
of the
dispatcher
module. </div>
<div class=""><br
class="">
</div>
<div class="">Within
that config
code, there’s
only one
additional
Kamailio
variable set,
which is part
of my
replacement/custom
dispatcher
(app_ruby):</div>
<div class=""><br
class="">
</div>
<div class=""><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">KSR::PV.sets(</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">"$du"</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">,
destination[</span><span class="hljs-string" style="box-sizing: inherit; color: rgb(221, 17, 68); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; line-height: 20px; white-space: pre;">'sip_uri’</span><span style="caret-color: rgb(51, 51, 51); color: rgb(51, 51, 51); font-family: Consolas, "Liberation Mono", Menlo, Courier, monospace; white-space: pre; background-color: rgb(255, 255, 255);" class="">])</span></div>
<div class=""><font
class=""
face="Consolas,
Liberation
Mono, Menlo,
Courier,
monospace"
color="#333333"><span style="caret-color: rgb(51, 51, 51); white-space: pre; background-color: rgb(255, 255, 255);" class=""></span></font></div>
<div class="">For
now, I’ve
taken the
output of a
mem dump of
one of the
production
workers, ps
output, etc.
I’ll leave
that few a few
hours, monitor
to ensure
there is a
measurable
memory
increase, and
send through
the logs if
that’s ok?</div>
<div class=""><span
style="background-color: rgb(255, 255, 255);" class=""><br class="">
</span></div>
<div class=""><span
style="background-color: rgb(255, 255, 255);" class="">Thanks!</span></div>
<div class=""><span
style="background-color: rgb(255, 255, 255);" class=""><br class="">
</span></div>
<div class=""><span
style="background-color: rgb(255, 255, 255);" class="">Andrew<br
class="">
</span>
<div class="">
<div class=""><br
class="">
<blockquote
type="cite"
class="">
<div class="">On
2 Aug 2019, at
11:44 pm,
Daniel-Constantin
Mierla <<a
href="mailto:miconda@gmail.com" class="" moz-do-not-send="true">miconda@gmail.com</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div
text="#000000"
bgcolor="#FFFFFF" class="">
<p class="">Hello,</p>
<p class="">are
you defining a
lot of
kamailio
variables in
your ruby
script? In
other words,
are you using
variables with
different
name, like
$var(xyz) or
$sht(a=>xyz),
where xyz is
passed
from/computed
in ruby script
and is
changing
depending of
the sip
message?</p>
<p class="">Cheers,<br
class="">
Daniel<br
class="">
</p>
<div
class="moz-cite-prefix">On
01.08.19
15:34, Andrew
White wrote:<br
class="">
</div>
<blockquote
type="cite"
cite="mid:0D657D17-5035-425C-AD84-7812EA64C536@uconnected.com.au"
class="">
Thanks Daniel,
you’re
fantastic!
<div class=""><br
class="">
</div>
<div class="">I
have 4
children/workers
configured
with -m 128 -M
32. The
machine in
question has
512MB of
memory, 1 core
and 1GB swap
on an SSD.</div>
<div class=""><br
class="">
</div>
<div class="">I
restarted
Kamailio with
memlog=1 and
I’ve been
sending
batches of 30
calls in. I’ve
noticed 4 of
the 13
Kamailio
processes
going up in
memory after
each batch,
which I
suspect to be
the primary
children/workers.
Immediately
post restart:</div>
<div class=""><br
class="">
</div>
<div class="">
<div class="">root
28531 0.7
5.5 329368
27196 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28532 0.6
4.9 329368
24528 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28533 0.6
5.5 329368
27244 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28534 0.7
5.4 329368
26788 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class=""><br
class="">
</div>
<div class="">After
about 90
calls:</div>
<div class=""><br
class="">
</div>
<div class="">
<div class="">root
28531 0.0
6.7 330688
32948 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28532 0.0
6.5 330560
32264 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28533 0.0
6.5 330556
32272 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28534 0.0
6.6 330564
32592 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
</div>
<div class=""><br
class="">
</div>
<div class="">None
of the other 9
Kamailio
processes are
increasing at
all.</div>
<div class=""><br
class="">
</div>
<div class="">I
ran corex.pkg_summary against one of them and got the following dump:</div>
<div class=""><br
class="">
</div>
<div class=""><a
href="https://paste.ubuntu.com/p/SqTF3K5knK/" class=""
moz-do-not-send="true">https://paste.ubuntu.com/p/SqTF3K5knK/</a></div>
<div class=""><br
class="">
</div>
<div class="">I
can see a lot
of allocation
to pvapi.c,
does this
indicate maybe
I’m setting
PVs that need
to be unset?</div>
<div class=""><br
class="">
</div>
<div class="">Here’s
another after
another 60
calls:</div>
<div class=""><br
class="">
</div>
<div class=""><a
href="https://paste.ubuntu.com/p/9WQXqZtfT2/" class=""
moz-do-not-send="true">https://paste.ubuntu.com/p/9WQXqZtfT2/</a></div>
<div class=""><br
class="">
</div>
<div class="">
<div class="">root
28531 0.0
6.9 330820
33928 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28532 0.0
6.7 330692
33352 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28533 0.0
6.7 330688
33280 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class="">root
28534 0.0
6.7 330696
33192 ?
Sl 22:48
0:00
/usr/local/sbin/kamailio
-DD -P
/var/run/kamailio/kamailio.pid
-f
/etc/kamailio/kamailio.cfg
-m 128 -M 32</div>
<div class=""><br
class="">
</div>
<div class="">The
only changes
I’ve made on
this config
over the last
couple of
weeks (since I
saw this
issue) is
removing the
dispatcher
module and
adding in a
small function
in app_ruby
(which I
already use)
to query redis
(which I also
already use
from app_ruby
and make a
heap of
queries per
call) for some
values and
write $du
manually. I
also added in
the topoh
module.</div>
<div class=""><br
class="">
</div>
<div class="">It
also makes a
lot of sense
to me to
monitor the
individual
processes
rather than
the aggregate.
Is there a way
to identify
simply from
bash what
processes are
workers
programmatically?
I’d like to
monitor just
those
individually
in my
monitoring.</div>
<div class=""><br
class="">
</div>
<div class="">Thanks!</div>
<div class=""><br
class="">
</div>
<div class="">Andrew</div>
</div>
<div class=""><br
class="">
</div>
<div class=""><br
class="">
<blockquote
type="cite"
class="">
<div class="">On
1 Aug 2019, at
8:24 pm,
Daniel-Constantin
Mierla <<a
href="mailto:miconda@gmail.com" class="" moz-do-not-send="true">miconda@gmail.com</a>>
wrote:</div>
<br
class="Apple-interchange-newline">
<div class="">
<div
text="#000000"
bgcolor="#FFFFFF" class="">
<p class="">Hello,</p>
<p class="">if
it is pkg,
then you have
to see which
process is
increasing the
use of memory,
because it is
private
memory,
specific for
each process.
The sum is an
indicator, but
the debugging
has to be done
for a specific
process/pid.</p>
<p class="">Once
you indentify
a process that
is leaking
pkg, execute
the rpc
command:</p>
<p class="">
- <a
href="https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary"
class=""
moz-do-not-send="true">
https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pkg_summary</a></p>
<p class="">When
that process
is doing some
runtime work
(e.g.,
handling of a
sip message),
the syslog
will get a
summary with
used pkg
chunks. Send
those log
messages here
for analysis.
You have to
set memlog
core parameter
to a value
smaller than
debug.</p>
<p class="">Cheers,<br
class="">
Daniel<br
class="">
</p>
<p class=""><br
class="">
</p>
<div
class="moz-cite-prefix">On
01.08.19
03:43, Andrew
White wrote:<br
class="">
</div>
<blockquote
type="cite"
cite="mid:1834350B-938B-4745-ACE4-051F63E45122@uconnected.com.au"
class="">
<div
dir="auto"
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
line-break:
after-white-space;"
class="">
Hi all,
<div class=""><br
class="">
</div>
<div class="">I
had a Kamailio
crash the
other day, and
some debugging
showed I ran
out of PKG
memory.</div>
<div class=""><br
class="">
</div>
<div class="">Since
then I’ve run
a simple bash
script to
compile the
amount of
memory used by
all child
processes,
effective /usr/local/sbin/kamcmd
pkg.stats |
grep real_used
summed
together. I’ve
graphed out
the data, and
there’s a
clear growth
of PKG memory
going on,
mostly
increasing
during our
busier daytime
hours.</div>
<div class=""><br
class="">
</div>
<div class=""><a
href="https://i.imgur.com/UTzx2k1.png" class="" moz-do-not-send="true">https://i.imgur.com/UTzx2k1.png</a></div>
<div class=""><br
class="">
</div>
<div class="">Based
on this, I
suspect either
a module
loaded or
something
within my
app_ruby conf
is leaking
memory.</div>
<div class=""><br
class="">
</div>
<div class="">I’ve
been reading
through <a
href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory"
class=""
moz-do-not-send="true">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a>,
but I’m a bit
nervous, as
I’m not really
a C/deep
memory type of
guy. I can see
a GDB script I
can attach to
Kamailio, but
is that going
to use
significant
resources to
run or impact
the running
process? Is
there a
newer/better/alternative
way to do
this, and to
help me break
this down?</div>
<div class=""><br
class="">
</div>
<div class="">Thanks!</div>
<div class=""><br
class="">
</div>
<div class="">Andrew</div>
</div>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com/" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com/" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Henning Westerholt - <a class="moz-txt-link-freetext" href="https://skalatan.de/blog/" moz-do-not-send="true">https://skalatan.de/blog/</a>
Kamailio services - <a class="moz-txt-link-freetext" href="https://skalatan.de/services" moz-do-not-send="true">https://skalatan.de/services</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Henning Westerholt - <a class="moz-txt-link-freetext" href="https://skalatan.de/blog/" moz-do-not-send="true">https://skalatan.de/blog/</a>
Kamailio services - <a class="moz-txt-link-freetext" href="https://skalatan.de/services" moz-do-not-send="true">https://skalatan.de/services</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Henning Westerholt - <a class="moz-txt-link-freetext" href="https://skalatan.de/blog/" moz-do-not-send="true">https://skalatan.de/blog/</a>
Kamailio services - <a class="moz-txt-link-freetext" href="https://skalatan.de/services" moz-do-not-send="true">https://skalatan.de/services</a></pre>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a></pre>
</body>
</html>