<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello,</p>
<p>if you do dmq replication between kamailio systems running
different major versions, then it is likely to get memory leaks
due to replication of data and most probably cannot be fixed. This
is because internal structures of modules (also dmq commands) can
change, practically what an instance does is not ensured to happen
on the other instance. Just for example, from my mind, htable got
some changes during past releases, dmq also has significant
enhancements by getting support for more transport protocols.</p>
<p>If you get memory leaks when you run same Kamailio major version
on all Kamailio nodes, then that can be troubleshoot and fixed.</p>
<p>Happy new year,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 07.01.22 11:33, George
Diamantopoulos wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAPcKEYOszsZzS7zCVydiMQMohLweeUQg2x=9eyxfOZeYgApLwA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Hello all and happy new year,</div>
<div><br>
</div>
<div>I have some new information to share regarding this issue.
I believe the previous metrics I sent to the list might not be
indicative of the way the problem manifests. Here's what I
believe so far:</div>
<div> - Issue is exacerbated (or manifests) during
moderate-to-high cps, or grows linearly with total traffic
processed since last restart<br>
</div>
<div> - shm stats show a lot of memory consumed by
sip_msg_shm_clone</div>
<div> - also reproduced this time on bullseye with kamailio
5.5.3</div>
<div><br>
</div>
<div>Here's some more meaningful stats taken at more appropriate
times (i.e. after more traffic has been processed) than the
previous ones. These two kamailio instances have identical
configuration and traffic patterns:<br>
</div>
<div>- <a href="https://pastebin.com/gHa803kB"
moz-do-not-send="true" class="moz-txt-link-freetext">https://pastebin.com/gHa803kB</a>
for kamailio 5.5.3 showing high sip_msg_shm_clone on debian
bullseye</div>
<div>- <a href="https://pastebin.com/JbcZbbSQ"
moz-do-not-send="true" class="moz-txt-link-freetext">https://pastebin.com/JbcZbbSQ</a>
for kamailio 5.4.6 on debian buster</div>
<div><br>
</div>
<div>There is still DMQ use for these instances despite the
version mismatch. Unfortunately I can't migrate all DMQ nodes
to 5.5.x at this time, not unless I can have assurances that
it is DMQ that causes this issue with shm memory exhaustion...<br>
</div>
<div><br>
</div>
<div>After shmem was exhausted on 5.5.3, it stopped processing
traffic. I issued a kamctl trap at that time but I'm assuming
the backtrace won't show much except for the inability to
allocate shm? If you think the backtrace at that point would
be useful in any way, let me know and I'll try to share it
privately. In case it isn't useful, what other debugging
information can be gathered to dissect this issue? Thanks!</div>
<div><br>
</div>
<div>BR,</div>
<div>George<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 30 Jun 2021 at 19:20,
Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com" moz-do-not-send="true"
class="moz-txt-link-freetext">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>for the sake of completion: the autoexpire should clean
the items if they are not used during the expiration
interval. If you want to get them deleted after first
expiration interval always, see the updateexpire attribute
for htable modparam.</p>
<p>Also, note that replication should be done only between
Kamailio instances with same major version, because there
can be internal differences between major versions that
can lead to unexpected behaviour. In other words, if you
replicate, doing between two kamailio with version 5.5.x
or between two kamailio with version 5.4.x, but not
between a kamailkio 5.5.x and a kamailio 5.4.x.</p>
<p>The total amount of used memory in the stats file for 5.5
does not seem to be high as a rough estimation. The
highest by module is in htable, but it is around 20MB.
Maybe you took the stats too early, quickly after a
restart?<br>
</p>
<div>Cheers,<br>
Daniel</div>
<div><br>
</div>
<div>On 30.06.21 17:20, George Diamantopoulos wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Daniel,</div>
<div><br>
</div>
<div>Thanks for the feedback. I think I might have been
too quick to blame htable for this behaviour. In fact,
version 5.4 seems to consume more memory than 5.5
(175129776 bytes vs 20581096), which makes sense since
it has been running for longer (I missed the extra
digit previously).</div>
<div><br>
</div>
<div>So I'm not sure htable is to blame. On the other
hand, I don't see any other modules using up too much
of shmem either, so maybe memory stats can't provide
the answer here?</div>
<div><br>
</div>
<div>To answer your question, though, I do use DMQ and
both tables that use it have autoexpire set to the
same value on both 5.4 and 5.5:</div>
<div><br>
</div>
<div><span style="font-family:monospace"><span
style="color:rgb(0,0,0);background-color:rgb(255,255,255)">/etc/kamailio#
grep dmq kamailio-module-params.cfg </span><br>
modparam("dmq", "server_address", "sip:<a
href="http://172.30.43.1:5090" target="_blank"
moz-do-not-send="true">172.30.43.1:5090</a>") <br>
modparam("dmq", "notification_address", "sip:<a
href="http://dmq.services.mydomain.com:5090"
target="_blank" moz-do-not-send="true">dmq.services.mydomain.com:5090</a>")
<br>
modparam("dmq", "multi_notify", 1) <br>
modparam("htable", "enable_dmq", 1) <br>
modparam("htable", "htable",
'cid2hi=>size=8;autoexpire=600;dmqreplicate=1') <br>
modparam("htable", "htable",
'xcid2count=>size=8;autoexpire=600;dmqreplicate=1')<br>
</span></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 30 Jun 2021 at
17:43, Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>do you replicate items in the htable via dmq?
Does the htable have autoexpire value set?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 30.06.21 13:54, George Diamantopoulos wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Forwarding my reply to the list,
using gmail's reply button set Henning as the
sole recipient :-\<br>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">----------
Forwarded message ---------<br>
From: <b class="gmail_sendername"
dir="auto">George Diamantopoulos</b> <span
dir="auto"><<a
href="mailto:georgediam@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">georgediam@gmail.com</a>></span><br>
Date: Sat, 26 Jun 2021 at 02:25<br>
Subject: Re: [SR-Users] Possible memory
leak on 5.5.x (new)?<br>
To: Henning Westerholt <<a
href="mailto:hw@skalatan.de"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">hw@skalatan.de</a>><br>
</div>
<br>
<br>
<div dir="ltr">
<div>Hello Henning,</div>
<div><br>
</div>
<div>Thanks for your reply. Here's what
has come up after a few hours:</div>
<div><br>
</div>
<div>shm55: <a
href="https://pastebin.com/h9JCePmc"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://pastebin.com/h9JCePmc</a></div>
<div>shm54: <a
href="https://pastebin.com/Nx5xEEnA"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://pastebin.com/Nx5xEEnA</a></div>
<div><br>
</div>
<div>It seems to me htable is the culprit?
Are you seeing anything different? 54
has been running for 77020 seconds, 55
for 28521 (significantly less).</div>
<div><br>
</div>
<div>I'm going to turn it off until we
figure something out...</div>
<div><br>
</div>
<div>BR,</div>
<div>George<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri,
25 Jun 2021 at 18:17, Henning Westerholt
<<a href="mailto:hw@skalatan.de"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">hw@skalatan.de</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div lang="DE">
<div>
<p class="MsoNormal"><span
lang="EN-GB">Hello,</span></p>
<p class="MsoNormal"><span
lang="EN-GB"> </span></p>
<p class="MsoNormal"><span
lang="EN-GB">Good observation.
Please run the memory statistics
CLI commands to get more hints
about the module that might
cause it (as per below link).
Then please report more details.
If you can point to a particular
module, you can also open an
issue on our tracker.</span></p>
<p class="MsoNormal"><span
lang="EN-GB"> </span></p>
<p class="MsoNormal"><span
lang="EN-GB"><a
href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a></span></p>
<p class="MsoNormal"><span
lang="EN-GB"> </span></p>
<p class="MsoNormal"><span
lang="EN-GB">Cheers,</span></p>
<p class="MsoNormal"><span
lang="EN-GB"> </span></p>
<p class="MsoNormal"><span
lang="EN-GB">Henning</span></p>
<p class="MsoNormal"><span
lang="EN-GB"> </span></p>
<div
style="border-color:rgb(225,225,225)
currentcolor
currentcolor;border-style:solid
none none;border-width:1pt medium
medium;padding:3pt 0cm 0cm">
<p class="MsoNormal"
style="margin-left:35.4pt"><b>From:</b>
sr-users <<a
href="mailto:sr-users-bounces@lists.kamailio.org"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">sr-users-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>George
Diamantopoulos<br>
<b>Sent:</b> Friday, June 25,
2021 4:53 PM<br>
<b>To:</b> Kamailio (SER) -
Users Mailing List <<a
href="mailto:sr-users@lists.kamailio.org"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">sr-users@lists.kamailio.org</a>><br>
<b>Subject:</b> [SR-Users]
Possible memory leak on 5.5.x
(new)?</p>
</div>
<p class="MsoNormal"
style="margin-left:35.4pt"> </p>
<div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">Hello
all,</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">I'm
still investigating the (most
likely non-kamailio-related)
memory leak of my previous
message to the list, there
have been no developments so
far. I'll update if anything
changes.</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">This
concerns a new finding which
seems to affect kamailio
5.5.x. I have two kamailio
instances receiving the same
traffic via round-robin. I
upgraded only one of them to
5.5.1 and left the other to
5.4.6 as I feared of any
issues arising. I was lucky to
do so, because with identical
configuration, 5.5.x seems to
run out of SHM very quickly.
Here are links to graphs
produced by our monitoring
system:</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">Old
kamailio (no memory leak): <a
href="https://pasteboard.co/K8fVBiD.png" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">
https://pasteboard.co/K8fVBiD.png</a></p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">New
kamailio (possible leak): <a
href="https://pasteboard.co/K8fVS9N.png" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">
https://pasteboard.co/K8fVS9N.png</a></p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">The
configuration uses mtree,
htable, vars and vns
extensively. Has anyone come
across anything similar? Let
me know if I can provide any
further information to help
disect this. Thanks!</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">BR,</p>
</div>
<div>
<p class="MsoNormal"
style="margin-left:35.4pt">George</p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" moz-do-not-send="true" class="moz-txt-link-freetext">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online
Feb 21-24, 2022 (America Timezone)
* <a class="moz-txt-link-freetext" href="https://www.asipto.com/sw/kamailio-advanced-training-online/">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</body>
</html>