<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello,</p>
<p>first, it is better if you test with latest version from the
branch 5.3, because 5.3.2 is already outdated in that series.
Otherwise, we may troubleshoot a side effect of an issue already
fixed.</p>
<p>Second, the function that allocated most of the shm memory is
related to transaction, cloning the message in shared memory. Can
you fetch the stats and see how many transactions are listed when
you face that issues?</p>
<p>kamctl stats</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 25.06.20 11:11, José Seabra wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAEVZRWvuz6i3Rs-2xq3KaoHujdhfwdWrUNN+GJRaHiLOawUG3g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hello there.
<div><br>
</div>
<div>During my stress tests against our kamailio servers I
detected that it was running out of shm memory.</div>
<div><br>
</div>
<div><br>
</div>
<div>edge-sip-proxy[6119]: ERROR: ESP_LOG: <a
href="mailto:10405249-1580@10.225.121.206"
moz-do-not-send="true">10405249-1580@10.225.121.206</a>:
<core> [core/mem/q_malloc.c:298]: qm_find_free():
qm_find_free(0x7ff370920000, 4224); Free fragment not found!<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a
href="mailto:10405249-1580@10.225.121.206"
moz-do-not-send="true">10405249-1580@10.225.121.206</a>:
<core> [core/mem/q_malloc.c:432]: qm_malloc():
qm_malloc(0x7ff370920000, 4224) called from core:
core/sip_msg_clone.c: sip_msg_shm_clone(496), module: core;
Free fragment not found!<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a
href="mailto:10405249-1580@10.225.121.206"
moz-do-not-send="true">10405249-1580@10.225.121.206</a>:
<core> [core/sip_msg_clone.c:499]: sip_msg_shm_clone():
could not allocate shared memory from shm pool<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a
href="mailto:10405249-1580@10.225.121.206"
moz-do-not-send="true">10405249-1580@10.225.121.206</a>: tm
[t_lookup.c:1293]: new_t(): out of mem:<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a
href="mailto:10405249-1580@10.225.121.206"
moz-do-not-send="true">10405249-1580@10.225.121.206</a>: tm
[t_lookup.c:1439]: t_newtran(): new_t failed<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a
href="mailto:10405249-1580@10.225.121.206"
moz-do-not-send="true">10405249-1580@10.225.121.206</a>: sl
[sl_funcs.c:392]: sl_reply_error(): stateless error reply
used: I'm terribly sorry, server error occurred (1/SL)<br>
<div>
<div><br>
</div>
<div><br>
</div>
<div>The output of <span
style="background-color:rgb(251,250,249);color:rgb(51,51,51);font-family:Consolas,"Andale
Mono WT","Andale Mono","Bitstream
Vera Sans Mono","Nimbus Mono
L",Monaco,"Courier
New",monospace;font-size:14px">kamctl stats shmem </span>confirms
that kamailio is running out of memory:</div>
<div>{<br>
"jsonrpc": "2.0",<br>
"result": [<br>
"shmem:fragments = 36427",<br>
"shmem:free_size = 14123552",<br>
"shmem:max_used_size = 268435456",<br>
"shmem:real_used_size = 254311904",<br>
"shmem:total_size = 268435456",<br>
"shmem:used_size = 215787904"<br>
],<br>
"id": 10901<br>
}<br>
</div>
<div><br>
</div>
<div>Then I was check what was the module that were
consuming more memory with <span
style="background-color:rgb(251,250,249);color:rgb(51,51,51);font-family:Consolas,"Andale
Mono WT","Andale Mono","Bitstream
Vera Sans Mono","Nimbus Mono
L",Monaco,"Courier
New",monospace;font-size:14px">kamcmd mod.stats all
shm </span>and the output of this command showed me that
Core module was consuming most of shm memory, as you can
see bellow is the output of this command shows that Core -
build_req_buf_from_sip_req is consuming most of shm.</div>
<div> </div>
<div>Module: core<br>
{<br>
sip_msg_shm_clone(496): 4872856<br>
create_avp(175): 24768<br>
msg_lump_cloner(986): 154376<br>
xavp_new_value(106): 780896<br>
build_req_buf_from_sip_req(2187): 183984624<br>
counters_prefork_init(211): 36864<br>
cfg_clone_str(130): 96<br>
cfg_shmize(217): 848<br>
main_loop(1303): 8<br>
init_pt(106): 8<br>
init_pt(105): 8<br>
init_pt(104): 5256<br>
register_timer(995): 232<br>
cfg_register_ctx(47): 64<br>
init_tcp(5021): 8192<br>
init_tcp(5015): 32768<br>
init_tcp(5007): 8<br>
init_tcp(5000): 8<br>
init_tcp(4993): 8<br>
init_tcp(4987): 8<br>
init_tcp(4975): 8<br>
init_avps(90): 8<br>
init_avps(89): 8<br>
init_dst_blacklist(438): 16384<br>
init_dst_blacklist(430): 8<br>
timer_alloc(498): 96<br>
init_dns_cache(361): 8<br>
init_dns_cache(352): 16384<br>
init_dns_cache(344): 16<br>
init_dns_cache(336): 8<br>
init_timer(267): 8<br>
init_timer(266): 16384<br>
init_timer(265): 8<br>
init_timer(264): 8<br>
init_timer(253): 8<br>
init_timer(221): 8<br>
init_timer(210): 278544<br>
init_timer(209): 8<br>
init_timer(197): 8<br>
cfg_child_cb_new(829): 64<br>
sr_cfg_init(361): 8<br>
sr_cfg_init(354): 8<br>
sr_cfg_init(347): 8<br>
sr_cfg_init(335): 8<br>
sr_cfg_init(323): 8<br>
qm_shm_lock_init(1202): 8<br>
Total: 190229920<br>
}<br>
</div>
<div><br>
</div>
<div>Then I stopped the sipp that was running
against kamailio in order to see if the memory consumed
by <i>build_req_buf_from_sip_req</i> decreased but it
didn't, so, it seems that there is a memleak in this
version.</div>
<div> </div>
<div>If the information that I'm sending in this email is
not enough to identify the root cause of this, please let
me know if there is anything else that I can do here to
help identify better what is the main reason.</div>
<div>Than you</div>
<div><br>
</div>
<div>Best Regards</div>
-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">Cumprimentos
<div>José Seabra</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Kamailio (SER) - Users Mailing List
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Funding: <a class="moz-txt-link-freetext" href="https://www.paypal.me/dcmierla">https://www.paypal.me/dcmierla</a></pre>
</body>
</html>