<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.E-MailFormatvorlage18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Please also give the Kamailio version that you are using, just to sort out eventual already fixed memory leaks.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">About the question regarding the 2GB memory mapping – according to your configuration this should be shared memory (not assuming a memory leak in system memory because of a library etc..).
So, it should be not 2GB per process, but 2GB in total.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">And the memory is also not completely used, only 855MB of it:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">shmem:real_used_size = 855.574.648<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">Maximum private per process memory is according to your cfg 12 MB.</span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">-- <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning Westerholt –
</span><span style="mso-fareast-language:EN-US"><a href="https://skalatan.de/blog/"><span lang="EN-GB">https://skalatan.de/blog/</span></a></span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Kamailio services –
</span><span style="mso-fareast-language:EN-US"><a href="https://gilawa.com/"><span lang="EN-GB">https://gilawa.com</span></a>
</span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><b><span lang="EN-GB">From:</span></b><span lang="EN-GB"> sr-dev <sr-dev-bounces@lists.kamailio.org>
<b>On Behalf Of </b>Andrey Deykunov<br>
<b>Sent:</b> Wednesday, April 8, 2020 12:19 PM<br>
<b>To:</b> Kamailio (SER) - Development Mailing List <sr-dev@lists.kamailio.org><br>
<b>Subject:</b> [sr-dev] High memory consumption during load tests<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-GB"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I've noticed more than 14 Gb RSS memory consumption during load test (8000 WSS clients send registrations):<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">[…]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"># ps aux | grep kamailio<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:35.4pt">
hpbx 15948 0.0 0.1 2232328 22920 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15949 1.8 3.8 2232460 789212 ? S 01:40 1:23 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15950 1.8 3.8 2232460 790164 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15951 1.7 3.8 2232460 790656 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15952 1.8 3.8 2232460 790180 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15953 1.7 3.8 2232460 788192 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15954 1.8 3.8 2232460 789484 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15955 1.7 3.8 2232460 789640 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15959 1.8 3.8 2232460 788104 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15960 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15961 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15963 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15964 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15966 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15968 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15969 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15971 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15972 0.0 0.0 2232328 8152 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15973 0.0 0.0 2232328 8152 ? S 01:40 0:02 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15975 0.0 0.0 2232328 8156 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15977 4.8 4.0 2232428 835908 ? S 01:40 3:43 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15979 0.0 2.5 2232452 517192 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15980 0.0 2.5 2232328 515344 ? S 01:40 0:03 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15984 0.0 0.0 2232328 8004 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15986 0.0 0.0 2232328 8276 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15987 2.4 3.9 2232492 817888 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15989 2.4 3.9 2232492 816604 ? S 01:40 1:52 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15991 2.4 3.9 2232492 816952 ? S 01:40 1:51 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15992 2.3 3.9 2232492 817236 ? S 01:40 1:49 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15993 2.4 3.9 2232492 818900 ? S 01:40 1:51 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15994 2.4 3.9 2232492 817692 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15996 2.4 3.9 2232492 816832 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15998 2.3 3.9 2232492 814336 ? S 01:40 1:49 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 16001 2.1 3.1 2232328 655908 ? S 01:40 1:37 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Shared memory consumption is:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"># /var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics shmem:<br>
shmem:fragments = 4<br>
shmem:free_size = 1291909000<br>
shmem:max_used_size = 855663448<br>
shmem:real_used_size = 855574648<br>
shmem:total_size = 2147483648<br>
shmem:used_size = 818055392<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Also, I don't understand why kamailio maps more than 2 GB per every child process:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"># pmap -x $(pidof kamailio) | grep total<br>
total kB 2232332 659732 653680<br>
total kB 2232496 816132 802908<br>
total kB 2232496 818680 805456<br>
total kB 2232496 818544 805320<br>
total kB 2232496 818900 805676<br>
total kB 2232496 817888 804664<br>
total kB 2232496 818436 805212<br>
total kB 2232496 818124 804900<br>
total kB 2232496 818508 805284<br>
total kB 2232332 8344 3648<br>
total kB 2232332 8072 3580<br>
total kB 2232332 517360 512444<br>
total kB 2232456 519752 514580<br>
total kB 2232432 835996 829552<br>
total kB 2232332 8428 3576<br>
total kB 2232332 8832 3852<br>
total kB 2232332 10380 4604<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232464 789152 777596<br>
total kB 2232464 790128 778484<br>
total kB 2232464 790904 779004<br>
total kB 2232464 788796 777240<br>
total kB 2232464 790284 778384<br>
total kB 2232464 790732 779176<br>
total kB 2232464 790360 778716<br>
total kB 2232464 790512 778612<br>
total kB 2232332 22920 8992<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Any ideas what is happening here?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Andrey<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>