<div dir="ltr"><div>Hello all and happy new year,</div><div><br></div><div>I have some new information to share regarding this issue. I believe the previous metrics I sent to the list might not be indicative of the way the problem manifests. Here's what I believe so far:</div><div> - Issue is exacerbated (or manifests) during moderate-to-high cps, or grows linearly with total traffic processed since last restart<br></div><div> - shm stats show a lot of memory consumed by sip_msg_shm_clone</div><div> - also reproduced this time on bullseye with kamailio 5.5.3</div><div><br></div><div>Here's some more meaningful stats taken at more appropriate times (i.e. after more traffic has been processed) than the previous ones. These two kamailio instances have identical configuration and traffic patterns:<br></div><div>- <a href="https://pastebin.com/gHa803kB">https://pastebin.com/gHa803kB</a> for kamailio 5.5.3 showing high sip_msg_shm_clone on debian bullseye</div><div>- <a href="https://pastebin.com/JbcZbbSQ">https://pastebin.com/JbcZbbSQ</a> for kamailio 5.4.6 on debian buster</div><div><br></div><div>There is still DMQ use for these instances despite the version mismatch. Unfortunately I can't migrate all DMQ nodes to 5.5.x at this time, not unless I can have assurances that it is DMQ that causes this issue with shm memory exhaustion...<br></div><div><br></div><div>After shmem was exhausted on 5.5.3, it stopped processing traffic. I issued a kamctl trap at that time but I'm assuming the backtrace won't show much except for the inability to allocate shm? If you think the backtrace at that point would be useful in any way, let me know and I'll try to share it privately. In case it isn't useful, what other debugging information can be gathered to dissect this issue? Thanks!</div><div><br></div><div>BR,</div><div>George<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 30 Jun 2021 at 19:20, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>for the sake of completion: the autoexpire should clean the items
if they are not used during the expiration interval. If you want
to get them deleted after first expiration interval always, see
the updateexpire attribute for htable modparam.</p>
<p>Also, note that replication should be done only between Kamailio
instances with same major version, because there can be internal
differences between major versions that can lead to unexpected
behaviour. In other words, if you replicate, doing between two
kamailio with version 5.5.x or between two kamailio with version
5.4.x, but not between a kamailkio 5.5.x and a kamailio 5.4.x.</p>
<p>The total amount of used memory in the stats file for 5.5 does
not seem to be high as a rough estimation. The highest by module
is in htable, but it is around 20MB. Maybe you took the stats too
early, quickly after a restart?<br>
</p>
<div>Cheers,<br>
Daniel</div>
<div><br>
</div>
<div>On 30.06.21 17:20, George
Diamantopoulos wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Daniel,</div>
<div><br>
</div>
<div>Thanks for the feedback. I think I might have been too
quick to blame htable for this behaviour. In fact, version 5.4
seems to consume more memory than 5.5 (175129776 bytes vs
20581096), which makes sense since it has been running for
longer (I missed the extra digit previously).</div>
<div><br>
</div>
<div>So I'm not sure htable is to blame. On the other hand, I
don't see any other modules using up too much of shmem either,
so maybe memory stats can't provide the answer here?</div>
<div><br>
</div>
<div>To answer your question, though, I do use DMQ and both
tables that use it have autoexpire set to the same value on
both 5.4 and 5.5:</div>
<div><br>
</div>
<div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">/etc/kamailio#
grep dmq kamailio-module-params.cfg
</span><br>
modparam("dmq", "server_address", "sip:<a href="http://172.30.43.1:5090" target="_blank">172.30.43.1:5090</a>")
<br>
modparam("dmq", "notification_address", "sip:<a href="http://dmq.services.mydomain.com:5090" target="_blank">dmq.services.mydomain.com:5090</a>")
<br>
modparam("dmq", "multi_notify", 1)
<br>
modparam("htable", "enable_dmq", 1)
<br>
modparam("htable", "htable",
'cid2hi=>size=8;autoexpire=600;dmqreplicate=1')
<br>
modparam("htable", "htable",
'xcid2count=>size=8;autoexpire=600;dmqreplicate=1')<br>
</span></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 30 Jun 2021 at 17:43,
Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>do you replicate items in the htable via dmq? Does the
htable have autoexpire value set?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 30.06.21 13:54, George Diamantopoulos wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Forwarding my reply to the list, using
gmail's reply button set Henning as the sole recipient
:-\<br>
<div><br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">----------
Forwarded message ---------<br>
From: <b class="gmail_sendername" dir="auto">George
Diamantopoulos</b> <span dir="auto"><<a href="mailto:georgediam@gmail.com" target="_blank">georgediam@gmail.com</a>></span><br>
Date: Sat, 26 Jun 2021 at 02:25<br>
Subject: Re: [SR-Users] Possible memory leak on
5.5.x (new)?<br>
To: Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>><br>
</div>
<br>
<br>
<div dir="ltr">
<div>Hello Henning,</div>
<div><br>
</div>
<div>Thanks for your reply. Here's what has come
up after a few hours:</div>
<div><br>
</div>
<div>shm55: <a href="https://pastebin.com/h9JCePmc" target="_blank">https://pastebin.com/h9JCePmc</a></div>
<div>shm54: <a href="https://pastebin.com/Nx5xEEnA" target="_blank">https://pastebin.com/Nx5xEEnA</a></div>
<div><br>
</div>
<div>It seems to me htable is the culprit? Are you
seeing anything different? 54 has been running
for 77020 seconds, 55 for 28521 (significantly
less).</div>
<div><br>
</div>
<div>I'm going to turn it off until we figure
something out...</div>
<div><br>
</div>
<div>BR,</div>
<div>George<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, 25 Jun
2021 at 18:17, Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DE">
<div>
<p class="MsoNormal"><span lang="EN-GB">Hello,</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">Good
observation. Please run the memory
statistics CLI commands to get more
hints about the module that might cause
it (as per below link). Then please
report more details. If you can point to
a particular module, you can also open
an issue on our tracker.</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB"><a href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory" target="_blank">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">Cheers,</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">Henning</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b>
sr-users <<a href="mailto:sr-users-bounces@lists.kamailio.org" target="_blank">sr-users-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>George
Diamantopoulos<br>
<b>Sent:</b> Friday, June 25, 2021 4:53
PM<br>
<b>To:</b> Kamailio (SER) - Users
Mailing List <<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>><br>
<b>Subject:</b> [SR-Users] Possible
memory leak on 5.5.x (new)?</p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hello all,</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I'm still
investigating the (most likely
non-kamailio-related) memory leak of
my previous message to the list, there
have been no developments so far. I'll
update if anything changes.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">This
concerns a new finding which seems to
affect kamailio 5.5.x. I have two
kamailio instances receiving the same
traffic via round-robin. I upgraded
only one of them to 5.5.1 and left the
other to 5.4.6 as I feared of any
issues arising. I was lucky to do so,
because with identical configuration,
5.5.x seems to run out of SHM very
quickly. Here are links to graphs
produced by our monitoring system:</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Old
kamailio (no memory leak): <a href="https://pasteboard.co/K8fVBiD.png" target="_blank">
https://pasteboard.co/K8fVBiD.png</a></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">New
kamailio (possible leak): <a href="https://pasteboard.co/K8fVS9N.png" target="_blank">
https://pasteboard.co/K8fVS9N.png</a></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">The
configuration uses mtree, htable, vars
and vns extensively. Has anyone come
across anything similar? Let me know
if I can provide any further
information to help disect this.
Thanks!</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">BR,</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">George</p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a></pre>
</div>
</blockquote></div>