<div dir="ltr">Hi Daniel,<div>Thank you for your reply.</div><div><br><div>I will do stress tests with the latest version from branch 5.3.</div><div>If it is still happening I will let you know.</div></div><div><br></div><div>Cheers, </div><div>José</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> escreveu no dia quinta, 25/06/2020 à(s) 14:44:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>first, it is better if you test with latest version from the
branch 5.3, because 5.3.2 is already outdated in that series.
Otherwise, we may troubleshoot a side effect of an issue already
fixed.</p>
<p>Second, the function that allocated most of the shm memory is
related to transaction, cloning the message in shared memory. Can
you fetch the stats and see how many transactions are listed when
you face that issues?</p>
<p>kamctl stats</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 25.06.20 11:11, José Seabra wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello there.
<div><br>
</div>
<div>During my stress tests against our kamailio servers I
detected that it was running out of shm memory.</div>
<div><br>
</div>
<div><br>
</div>
<div>edge-sip-proxy[6119]: ERROR: ESP_LOG: <a href="mailto:10405249-1580@10.225.121.206" target="_blank">10405249-1580@10.225.121.206</a>:
<core> [core/mem/q_malloc.c:298]: qm_find_free():
qm_find_free(0x7ff370920000, 4224); Free fragment not found!<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a href="mailto:10405249-1580@10.225.121.206" target="_blank">10405249-1580@10.225.121.206</a>:
<core> [core/mem/q_malloc.c:432]: qm_malloc():
qm_malloc(0x7ff370920000, 4224) called from core:
core/sip_msg_clone.c: sip_msg_shm_clone(496), module: core;
Free fragment not found!<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a href="mailto:10405249-1580@10.225.121.206" target="_blank">10405249-1580@10.225.121.206</a>:
<core> [core/sip_msg_clone.c:499]: sip_msg_shm_clone():
could not allocate shared memory from shm pool<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a href="mailto:10405249-1580@10.225.121.206" target="_blank">10405249-1580@10.225.121.206</a>: tm
[t_lookup.c:1293]: new_t(): out of mem:<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a href="mailto:10405249-1580@10.225.121.206" target="_blank">10405249-1580@10.225.121.206</a>: tm
[t_lookup.c:1439]: t_newtran(): new_t failed<br>
edge-sip-proxy[6119]: ERROR: ESP_LOG: <a href="mailto:10405249-1580@10.225.121.206" target="_blank">10405249-1580@10.225.121.206</a>: sl
[sl_funcs.c:392]: sl_reply_error(): stateless error reply
used: I'm terribly sorry, server error occurred (1/SL)<br>
<div>
<div><br>
</div>
<div><br>
</div>
<div>The output of <span>kamctl stats shmem </span>confirms
that kamailio is running out of memory:</div>
<div>{<br>
"jsonrpc": "2.0",<br>
"result": [<br>
"shmem:fragments = 36427",<br>
"shmem:free_size = 14123552",<br>
"shmem:max_used_size = 268435456",<br>
"shmem:real_used_size = 254311904",<br>
"shmem:total_size = 268435456",<br>
"shmem:used_size = 215787904"<br>
],<br>
"id": 10901<br>
}<br>
</div>
<div><br>
</div>
<div>Then I was check what was the module that were
consuming more memory with <span>kamcmd mod.stats all
shm </span>and the output of this command showed me that
Core module was consuming most of shm memory, as you can
see bellow is the output of this command shows that Core -
build_req_buf_from_sip_req is consuming most of shm.</div>
<div> </div>
<div>Module: core<br>
{<br>
sip_msg_shm_clone(496): 4872856<br>
create_avp(175): 24768<br>
msg_lump_cloner(986): 154376<br>
xavp_new_value(106): 780896<br>
build_req_buf_from_sip_req(2187): 183984624<br>
counters_prefork_init(211): 36864<br>
cfg_clone_str(130): 96<br>
cfg_shmize(217): 848<br>
main_loop(1303): 8<br>
init_pt(106): 8<br>
init_pt(105): 8<br>
init_pt(104): 5256<br>
register_timer(995): 232<br>
cfg_register_ctx(47): 64<br>
init_tcp(5021): 8192<br>
init_tcp(5015): 32768<br>
init_tcp(5007): 8<br>
init_tcp(5000): 8<br>
init_tcp(4993): 8<br>
init_tcp(4987): 8<br>
init_tcp(4975): 8<br>
init_avps(90): 8<br>
init_avps(89): 8<br>
init_dst_blacklist(438): 16384<br>
init_dst_blacklist(430): 8<br>
timer_alloc(498): 96<br>
init_dns_cache(361): 8<br>
init_dns_cache(352): 16384<br>
init_dns_cache(344): 16<br>
init_dns_cache(336): 8<br>
init_timer(267): 8<br>
init_timer(266): 16384<br>
init_timer(265): 8<br>
init_timer(264): 8<br>
init_timer(253): 8<br>
init_timer(221): 8<br>
init_timer(210): 278544<br>
init_timer(209): 8<br>
init_timer(197): 8<br>
cfg_child_cb_new(829): 64<br>
sr_cfg_init(361): 8<br>
sr_cfg_init(354): 8<br>
sr_cfg_init(347): 8<br>
sr_cfg_init(335): 8<br>
sr_cfg_init(323): 8<br>
qm_shm_lock_init(1202): 8<br>
Total: 190229920<br>
}<br>
</div>
<div><br>
</div>
<div>Then I stopped the sipp that was running
against kamailio in order to see if the memory consumed
by <i>build_req_buf_from_sip_req</i> decreased but it
didn't, so, it seems that there is a memleak in this
version.</div>
<div> </div>
<div>If the information that I'm sending in this email is
not enough to identify the root cause of this, please let
me know if there is anything else that I can do here to
help identify better what is the main reason.</div>
<div>Than you</div>
<div><br>
</div>
<div>Best Regards</div>
-- <br>
<div dir="ltr">Cumprimentos
<div>José Seabra</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Funding: <a href="https://www.paypal.me/dcmierla" target="_blank">https://www.paypal.me/dcmierla</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Cumprimentos<div>José Seabra</div></div>