<div dir="ltr">Thank you Henning. I theorized it and you proved my point. <div>We're using 5.3.1.</div><div><br></div><div>Andrey</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">чт, 9 апр. 2020 г. в 09:59, Henning Westerholt <<a href="mailto:hw@skalatan.de">hw@skalatan.de</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DE">
<div class="gmail-m_4593333364172145547WordSection1">
<p class="MsoNormal"><span lang="EN-GB">Hello,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Please also give the Kamailio version that you are using, just to sort out eventual already fixed memory leaks.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">About the question regarding the 2GB memory mapping – according to your configuration this should be shared memory (not assuming a memory leak in system memory because of a library etc..).
So, it should be not 2GB per process, but 2GB in total.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">And the memory is also not completely used, only 855MB of it:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">shmem:real_used_size = 855.574.648<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Maximum private per process memory is according to your cfg 12 MB.</span><span lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Cheers,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Henning<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- <u></u>
<u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Henning Westerholt –
</span><span><a href="https://skalatan.de/blog/" target="_blank"><span lang="EN-GB">https://skalatan.de/blog/</span></a></span><span lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Kamailio services –
</span><span><a href="https://gilawa.com/" target="_blank"><span lang="EN-GB">https://gilawa.com</span></a>
</span><span lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><b><span lang="EN-GB">From:</span></b><span lang="EN-GB"> sr-dev <<a href="mailto:sr-dev-bounces@lists.kamailio.org" target="_blank">sr-dev-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>Andrey Deykunov<br>
<b>Sent:</b> Wednesday, April 8, 2020 12:19 PM<br>
<b>To:</b> Kamailio (SER) - Development Mailing List <<a href="mailto:sr-dev@lists.kamailio.org" target="_blank">sr-dev@lists.kamailio.org</a>><br>
<b>Subject:</b> [sr-dev] High memory consumption during load tests<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><span lang="EN-GB"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi,<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I've noticed more than 14 Gb RSS memory consumption during load test (8000 WSS clients send registrations):<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">[…]<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"># ps aux | grep kamailio<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:12pt;margin-left:35.4pt">
hpbx 15948 0.0 0.1 2232328 22920 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15949 1.8 3.8 2232460 789212 ? S 01:40 1:23 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15950 1.8 3.8 2232460 790164 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15951 1.7 3.8 2232460 790656 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15952 1.8 3.8 2232460 790180 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15953 1.7 3.8 2232460 788192 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15954 1.8 3.8 2232460 789484 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15955 1.7 3.8 2232460 789640 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15959 1.8 3.8 2232460 788104 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15960 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15961 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15963 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15964 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15966 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15968 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15969 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15971 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15972 0.0 0.0 2232328 8152 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15973 0.0 0.0 2232328 8152 ? S 01:40 0:02 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15975 0.0 0.0 2232328 8156 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15977 4.8 4.0 2232428 835908 ? S 01:40 3:43 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15979 0.0 2.5 2232452 517192 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15980 0.0 2.5 2232328 515344 ? S 01:40 0:03 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15984 0.0 0.0 2232328 8004 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15986 0.0 0.0 2232328 8276 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15987 2.4 3.9 2232492 817888 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15989 2.4 3.9 2232492 816604 ? S 01:40 1:52 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15991 2.4 3.9 2232492 816952 ? S 01:40 1:51 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15992 2.3 3.9 2232492 817236 ? S 01:40 1:49 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15993 2.4 3.9 2232492 818900 ? S 01:40 1:51 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15994 2.4 3.9 2232492 817692 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15996 2.4 3.9 2232492 816832 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 15998 2.3 3.9 2232492 814336 ? S 01:40 1:49 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<br>
hpbx 16001 2.1 3.1 2232328 655908 ? S 01:40 1:37 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Shared memory consumption is:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"># /var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics shmem:<br>
shmem:fragments = 4<br>
shmem:free_size = 1291909000<br>
shmem:max_used_size = 855663448<br>
shmem:real_used_size = 855574648<br>
shmem:total_size = 2147483648<br>
shmem:used_size = 818055392<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Also, I don't understand why kamailio maps more than 2 GB per every child process:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"># pmap -x $(pidof kamailio) | grep total<br>
total kB 2232332 659732 653680<br>
total kB 2232496 816132 802908<br>
total kB 2232496 818680 805456<br>
total kB 2232496 818544 805320<br>
total kB 2232496 818900 805676<br>
total kB 2232496 817888 804664<br>
total kB 2232496 818436 805212<br>
total kB 2232496 818124 804900<br>
total kB 2232496 818508 805284<br>
total kB 2232332 8344 3648<br>
total kB 2232332 8072 3580<br>
total kB 2232332 517360 512444<br>
total kB 2232456 519752 514580<br>
total kB 2232432 835996 829552<br>
total kB 2232332 8428 3576<br>
total kB 2232332 8832 3852<br>
total kB 2232332 10380 4604<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232332 8480 3564<br>
total kB 2232464 789152 777596<br>
total kB 2232464 790128 778484<br>
total kB 2232464 790904 779004<br>
total kB 2232464 788796 777240<br>
total kB 2232464 790284 778384<br>
total kB 2232464 790732 779176<br>
total kB 2232464 790360 778716<br>
total kB 2232464 790512 778612<br>
total kB 2232332 22920 8992<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Any ideas what is happening here?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Andrey<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote></div>