<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>first, I would recommend to upgrade to the latest version in
v5.1.x series, the 5.1.0 was the first there and there were many
issues fixed in the 5.1 branch that will ensure smoother run in
long term.</p>
<p>The issue reported is likely related to fragmentation, try to set
global parameter:</p>
<p>mem_join=1</p>
<p>As the instance needs to deal with large chunks for xcap
documents, I would also suggest to increase a bit the pool of pkg
memory via -M command line parameter -- I see that now is 8MB,
make it 12 or 16.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 19.09.19 01:07, Cody Herzog wrote:<br>
</div>
<blockquote type="cite"
cite="mid:MN2PR16MB30553686FC46224188F673ACA58E0@MN2PR16MB3055.namprd16.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal">Hello.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Recently, for the first time, we
experienced an apparent memory issue in our production
environment. Here's an example of the relevant log messages:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Sep 13 18:55:22 SIPCOMM
/usr/local/sbin/kamailio[2302]: ERROR: <core>
[core/mem/q_malloc.c:286]: qm_find_free():
qm_find_free(0x7fd8bbcfc010, 134648); Free fragment not found!<o:p></o:p></p>
<p class="MsoNormal">Sep 13 18:55:22 SIPCOMM
/usr/local/sbin/kamailio[2302]: ERROR: <core>
[core/mem/q_malloc.c:419]: qm_malloc():
qm_malloc(0x7fd8bbcfc010, 134648) called from xcap_server:
xcap_server.c: ki_xcaps_put(549), module: xcap_server; Free
fragment not found!<o:p></o:p></p>
<p class="MsoNormal">Sep 13 18:55:22 SIPCOMM
/usr/local/sbin/kamailio[2302]: ERROR: xcap_server
[xcap_server.c:552]: ki_xcaps_put(): no more pkg<o:p></o:p></p>
<p class="MsoNormal">Sep 13 18:55:22 SIPCOMM
/usr/local/sbin/kamailio[2252]: ERROR: app_perl
[kamailioxs.xs:487]: XS_Kamailio_log(): 500 Server error<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The failed operation was the XCAP server
module trying to generate a very large RLS contact list for a
user. The issue only impacted users with very large lists, as
though a large contiguous block of memory could not be found,
whereas other smaller allocations continued to work fine. We
believe the requested allocation was around 112 KB in size.<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">The server had been up for 14 days. We were
able to work around the issue temporarily by just restarting
the Kamailio service. It's unusual, because our production
server is often up for months, and we've never seen this issue
before. The load on production is increasing slowly due to an
increased concurrent user count, so that might be related.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Before restarting the service on
production, we captured the output of the following commands:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">kamctl stats shmem<o:p></o:p></p>
<p class="MsoNormal">kamcmd mod.stats all shm<o:p></o:p></p>
<p class="MsoNormal">kamcmd pkg.stats<o:p></o:p></p>
<p class="MsoNormal">kamcmd mod.stats all pkg<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here's the shared mem output:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">{<o:p></o:p></p>
<p class="MsoNormal"> "jsonrpc": "2.0",<o:p></o:p></p>
<p class="MsoNormal"> "result": [<o:p></o:p></p>
<p class="MsoNormal"> "shmem:fragments = 27240",<o:p></o:p></p>
<p class="MsoNormal"> "shmem:free_size = 447203296",<o:p></o:p></p>
<p class="MsoNormal"> "shmem:max_used_size = 116175576",<o:p></o:p></p>
<p class="MsoNormal"> "shmem:real_used_size = 89667616",<o:p></o:p></p>
<p class="MsoNormal"> "shmem:total_size = 536870912",<o:p></o:p></p>
<p class="MsoNormal"> "shmem:used_size = 68824240"<o:p></o:p></p>
<p class="MsoNormal"> ],<o:p></o:p></p>
<p class="MsoNormal"> "id": 6934<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here's the pkg output for the particular
PID which was throwing the errors:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">{<o:p></o:p></p>
<p class="MsoNormal"> entry: 34<o:p></o:p></p>
<p class="MsoNormal"> pid: 2302<o:p></o:p></p>
<p class="MsoNormal"> rank: 14<o:p></o:p></p>
<p class="MsoNormal"> used: 2415864<o:p></o:p></p>
<p class="MsoNormal"> free: 4949688<o:p></o:p></p>
<p class="MsoNormal"> real_used: 3438920<o:p></o:p></p>
<p class="MsoNormal"> total_size: 8388608<o:p></o:p></p>
<p class="MsoNormal"> total_frags: 1951<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We didn't see anything obvious in the stats
output which explains the issue.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We've been trying to reproduce the issue in
a dev environment using simulated higher than production load
running for many days, but so far we've had no luck. We've
been monitoring memory stats over time, but we don't see any
obvious leaks or issues.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We've searched various past threads, but
didn't find any obvious answers. Here are some of the
documents and the threads we've been reading:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://www.kamailio.org/wiki/tutorials/troubleshooting/memory">https://www.kamailio.org/wiki/tutorials/troubleshooting/memory</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://www.kamailio.org/wiki/cookbooks/3.3.x/core#mem_join">https://www.kamailio.org/wiki/cookbooks/3.3.x/core#mem_join</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://sr-users.sip-router.narkive.com/3TEDs3ga/tcp-free-fragment-not-found">https://sr-users.sip-router.narkive.com/3TEDs3ga/tcp-free-fragment-not-found</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://lists.kamailio.org/pipermail/sr-users/2012-June/073552.html">https://lists.kamailio.org/pipermail/sr-users/2012-June/073552.html</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://lists.kamailio.org/pipermail/sr-users/2017-February/096132.html">https://lists.kamailio.org/pipermail/sr-users/2017-February/096132.html</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://lists.kamailio.org/pipermail/sr-users/2017-September/098607.html">https://lists.kamailio.org/pipermail/sr-users/2017-September/098607.html</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://github.com/kamailio/kamailio/issues/1001">https://github.com/kamailio/kamailio/issues/1001</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://lists.kamailio.org/pipermail/sr-users/2016-April/092592.html">https://lists.kamailio.org/pipermail/sr-users/2016-April/092592.html</a><o:p></o:p></p>
<p class="MsoNormal"><a class="moz-txt-link-freetext" href="https://lists.kamailio.org/pipermail/sr-users/2010-July/064832.html">https://lists.kamailio.org/pipermail/sr-users/2010-July/064832.html</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regarding our Kamailio version and build
options, here's the output of 'kamailio -v':<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">----<o:p></o:p></p>
<p class="MsoNormal">version: kamailio 5.1.0 (x86_64/linux)<o:p></o:p></p>
<p class="MsoNormal">flags: STATS: Off, USE_TCP, USE_TLS,
USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC,
TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES<o:p></o:p></p>
<p class="MsoNormal">ADAPTIVE_WAIT_LOOPS=1024,
MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB<o:p></o:p></p>
<p class="MsoNormal">poll method support: poll, epoll_lt,
epoll_et, sigio_rt, select.<o:p></o:p></p>
<p class="MsoNormal">id: unknown<o:p></o:p></p>
<p class="MsoNormal">compiled on 20:07:31 Jan 4 2018 with gcc
5.4.0<o:p></o:p></p>
<p class="MsoNormal">----<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are running on a 64-bit Ubuntu Server
virtual machine.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any help would be greatly appreciated.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks very much.<o:p></o:p></p>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- <a class="moz-txt-link-freetext" href="https://asipto.com/u/kat">https://asipto.com/u/kat</a></pre>
</body>
</html>