<div dir="ltr"><div>Hello again,</div><div><br></div><div>Indeed it must have been the version mismatch. I have upgraded all instances to 5.5.3 and shm now reports reasonable numbers. Thanks!</div><div><br></div><div>BR,</div><div>George<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 7 Jan 2022 at 14:10, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,<br>
</p>
<div>On 07.01.22 13:04, George
Diamantopoulos wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Daniel,</div>
<div><br>
</div>
<div>I see, thanks for the response. I guess I'll try upgrading
all instances to 5.5.x for now, and hopefully that will fix
it. If not, I'll revert to 5.4.x and post here again. If the
issue does manifest with all instances on 5.5.x however, what
information should I collect to investigate this further? I'm
guessing shm stats is a start, but is it enough? <br>
</div>
</div>
</blockquote>
<p>it is hard to say, starting with the stats and shm summary is
good!</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Lastly, is sip_msg_shm_clone growing consistent with DMQ
incompatibility between 5.5.x <-> 5.4.x, or should such
memory leaks manifest in some other way in shm stats?<br>
</div>
</div>
</blockquote>
<p>If the leak manifests only when dmq is used, then troubleshooting
has to be done with the same Kamailio version on all nodes,
otherwise can be waste of time. The cloning function is used for
other features as well, not only for dmq, could be other reasons
to leak from it.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Cheers,</div>
<div>George</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 7 Jan 2022 at 13:20,
Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>if you do dmq replication between kamailio systems
running different major versions, then it is likely to get
memory leaks due to replication of data and most probably
cannot be fixed. This is because internal structures of
modules (also dmq commands) can change, practically what
an instance does is not ensured to happen on the other
instance. Just for example, from my mind, htable got some
changes during past releases, dmq also has significant
enhancements by getting support for more transport
protocols.</p>
<p>If you get memory leaks when you run same Kamailio major
version on all Kamailio nodes, then that can be
troubleshoot and fixed.</p>
<p>Happy new year,<br>
Daniel<br>
</p>
<div>On 07.01.22 11:33, George Diamantopoulos wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello all and happy new year,</div>
<div><br>
</div>
<div>I have some new information to share regarding this
issue. I believe the previous metrics I sent to the
list might not be indicative of the way the problem
manifests. Here's what I believe so far:</div>
<div> - Issue is exacerbated (or manifests) during
moderate-to-high cps, or grows linearly with total
traffic processed since last restart<br>
</div>
<div> - shm stats show a lot of memory consumed by
sip_msg_shm_clone</div>
<div> - also reproduced this time on bullseye with
kamailio 5.5.3</div>
<div><br>
</div>
<div>Here's some more meaningful stats taken at more
appropriate times (i.e. after more traffic has been
processed) than the previous ones. These two kamailio
instances have identical configuration and traffic
patterns:<br>
</div>
<div>- <a href="https://pastebin.com/gHa803kB" target="_blank">https://pastebin.com/gHa803kB</a>
for kamailio 5.5.3 showing high sip_msg_shm_clone on
debian bullseye</div>
<div>- <a href="https://pastebin.com/JbcZbbSQ" target="_blank">https://pastebin.com/JbcZbbSQ</a>
for kamailio 5.4.6 on debian buster</div>
<div><br>
</div>
<div>There is still DMQ use for these instances despite
the version mismatch. Unfortunately I can't migrate
all DMQ nodes to 5.5.x at this time, not unless I can
have assurances that it is DMQ that causes this issue
with shm memory exhaustion...<br>
</div>
<div><br>
</div>
<div>After shmem was exhausted on 5.5.3, it stopped
processing traffic. I issued a kamctl trap at that
time but I'm assuming the backtrace won't show much
except for the inability to allocate shm? If you think
the backtrace at that point would be useful in any
way, let me know and I'll try to share it privately.
In case it isn't useful, what other debugging
information can be gathered to dissect this issue?
Thanks!</div>
<div><br>
</div>
<div>BR,</div>
<div>George<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 30 Jun 2021 at
19:20, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>for the sake of completion: the autoexpire should
clean the items if they are not used during the
expiration interval. If you want to get them
deleted after first expiration interval always,
see the updateexpire attribute for htable
modparam.</p>
<p>Also, note that replication should be done only
between Kamailio instances with same major
version, because there can be internal differences
between major versions that can lead to unexpected
behaviour. In other words, if you replicate, doing
between two kamailio with version 5.5.x or between
two kamailio with version 5.4.x, but not between a
kamailkio 5.5.x and a kamailio 5.4.x.</p>
<p>The total amount of used memory in the stats file
for 5.5 does not seem to be high as a rough
estimation. The highest by module is in htable,
but it is around 20MB. Maybe you took the stats
too early, quickly after a restart?<br>
</p>
<div>Cheers,<br>
Daniel</div>
<div><br>
</div>
<div>On 30.06.21 17:20, George Diamantopoulos wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Daniel,</div>
<div><br>
</div>
<div>Thanks for the feedback. I think I might
have been too quick to blame htable for this
behaviour. In fact, version 5.4 seems to
consume more memory than 5.5 (175129776 bytes
vs 20581096), which makes sense since it has
been running for longer (I missed the extra
digit previously).</div>
<div><br>
</div>
<div>So I'm not sure htable is to blame. On the
other hand, I don't see any other modules
using up too much of shmem either, so maybe
memory stats can't provide the answer here?</div>
<div><br>
</div>
<div>To answer your question, though, I do use
DMQ and both tables that use it have
autoexpire set to the same value on both 5.4
and 5.5:</div>
<div><br>
</div>
<div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">/etc/kamailio#
grep dmq kamailio-module-params.cfg </span><br>
modparam("dmq", "server_address", "sip:<a href="http://172.30.43.1:5090" target="_blank">172.30.43.1:5090</a>")
<br>
modparam("dmq", "notification_address",
"sip:<a href="http://dmq.services.mydomain.com:5090" target="_blank">dmq.services.mydomain.com:5090</a>")
<br>
modparam("dmq", "multi_notify", 1) <br>
modparam("htable", "enable_dmq", 1) <br>
modparam("htable", "htable",
'cid2hi=>size=8;autoexpire=600;dmqreplicate=1')
<br>
modparam("htable", "htable",
'xcid2count=>size=8;autoexpire=600;dmqreplicate=1')<br>
</span></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 30 Jun
2021 at 17:43, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>do you replicate items in the htable via
dmq? Does the htable have autoexpire value
set?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 30.06.21 13:54, George
Diamantopoulos wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Forwarding my reply to the
list, using gmail's reply button set
Henning as the sole recipient :-\<br>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">----------
Forwarded message ---------<br>
From: <b class="gmail_sendername" dir="auto">George Diamantopoulos</b>
<span dir="auto"><<a href="mailto:georgediam@gmail.com" target="_blank">georgediam@gmail.com</a>></span><br>
Date: Sat, 26 Jun 2021 at 02:25<br>
Subject: Re: [SR-Users] Possible
memory leak on 5.5.x (new)?<br>
To: Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>><br>
</div>
<br>
<br>
<div dir="ltr">
<div>Hello Henning,</div>
<div><br>
</div>
<div>Thanks for your reply. Here's
what has come up after a few
hours:</div>
<div><br>
</div>
<div>shm55: <a href="https://pastebin.com/h9JCePmc" target="_blank">https://pastebin.com/h9JCePmc</a></div>
<div>shm54: <a href="https://pastebin.com/Nx5xEEnA" target="_blank">https://pastebin.com/Nx5xEEnA</a></div>
<div><br>
</div>
<div>It seems to me htable is the
culprit? Are you seeing anything
different? 54 has been running
for 77020 seconds, 55 for 28521
(significantly less).</div>
<div><br>
</div>
<div>I'm going to turn it off
until we figure something out...</div>
<div><br>
</div>
<div>BR,</div>
<div>George<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Fri, 25 Jun 2021 at 18:17,
Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DE">
<div>
<p class="MsoNormal"><span lang="EN-GB">Hello,</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">Good
observation. Please run
the memory statistics
CLI commands to get more
hints about the module
that might cause it (as
per below link). Then
please report more
details. If you can
point to a particular
module, you can also
open an issue on our
tracker.</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB"><a href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory" target="_blank">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">Cheers,</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">Henning</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b>
sr-users <<a href="mailto:sr-users-bounces@lists.kamailio.org" target="_blank">sr-users-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>George
Diamantopoulos<br>
<b>Sent:</b> Friday,
June 25, 2021 4:53 PM<br>
<b>To:</b> Kamailio
(SER) - Users Mailing
List <<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>><br>
<b>Subject:</b>
[SR-Users] Possible
memory leak on 5.5.x
(new)?</p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hello
all,</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I'm
still investigating
the (most likely
non-kamailio-related)
memory leak of my
previous message to
the list, there have
been no developments
so far. I'll update if
anything changes.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">This
concerns a new finding
which seems to affect
kamailio 5.5.x. I have
two kamailio instances
receiving the same
traffic via
round-robin. I
upgraded only one of
them to 5.5.1 and left
the other to 5.4.6 as
I feared of any issues
arising. I was lucky
to do so, because with
identical
configuration, 5.5.x
seems to run out of
SHM very quickly. Here
are links to graphs
produced by our
monitoring system:</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Old
kamailio (no memory
leak): <a href="https://pasteboard.co/K8fVBiD.png" target="_blank">
https://pasteboard.co/K8fVBiD.png</a></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">New
kamailio (possible
leak): <a href="https://pasteboard.co/K8fVS9N.png" target="_blank">
https://pasteboard.co/K8fVS9N.png</a></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">The
configuration uses
mtree, htable, vars
and vns extensively.
Has anyone come across
anything similar? Let
me know if I can
provide any further
information to help
disect this. Thanks!</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">BR,</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">George</p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online
Feb 21-24, 2022 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online
Feb 21-24, 2022 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote></div>