<div dir="ltr"><div>Nice finding ! however Rogelio is saying there was no errors in the logs, looking at the patch he would have seen some.</div><div><br></div><div>LM_ERR("running job failed\n");</div><div><br></div><div>Hope I am wrong.</div><div><br></div><div>Rogerio is this a slow leak of you mean suddenly it is using 1.5G ?</div><div><br></div><div>There can be a lot of shm used by sip_msg_shm_clone if Kamailio starts to re transmit.</div><div>2K seems very small.</div><div>Normally this kind of high memory usage can take place at startup when DMQ userloc is getting a full sync from all other nodes.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 31, 2018 at 5:59 AM, Charles Chance <span dir="ltr"><<a href="mailto:charles.chance@sipcentric.com" target="_blank">charles.chance@sipcentric.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="auto">Hi Daniel,</div><div dir="auto"><br></div><div dir="auto">Nice spot! I had tried to reproduce locally, but had not considered the possibility that jobs may be failing somewhere in Rogelio’s setup.</div></div><div dir="auto"><br></div><div dir="auto">Most likely your patch will resolve it but I’m happy to take a look further if not.<br></div><div dir="auto"><br></div><div dir="auto">Cheers,</div><div dir="auto"><br></div><div dir="auto">Charles</div><div class="HOEnZb"><div class="h5"><div><br><div class="gmail_quote"><div dir="ltr">On Tue, 31 Jul 2018 at 13:05, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>not using dmq much, but at a quick look in the code, I noticed that there are some cases when the job fields were not released if the processing was not completely done for various reasons.</div><div><br></div><div>I pushed the commit <wbr>a1f5fbe2c18246d4afefa44fd8a526<wbr>12a5182a46, can you try with it and see the results?</div><div><br></div><div>Maybe Charles Chance can also do a bit of review here, being the one doing most of the work lately for dmq.</div><div><br></div><div>Cheers,</div><div>Daniel</div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 31, 2018 at 6:58 AM, Rogelio Perez <span dir="ltr"><<a href="mailto:rogelio@telnyx.com" target="_blank">rogelio@telnyx.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div><div>We're running three instances of Kamailo v5.14 as registrars handling registrations from ~2000 SIP clients, with one instance being primary and the other two as backups.</div><div><br></div><div>The three of them are using the dmq and dmq_usrloc modules to synchronize user locations, however after a couple of days of operation the two failover instances show memory leak behaviors, with mem usage assigned to the core taking all available resources.<br></div><div><br></div><div>When this happens we've noticed that:<br></div> - The shared memory used by the function "<span class="m_-3936293401878097385m_8371728938109241847m_3667832859572217976m_-3181406981141791056gmail-m_4123646824836054726m_4959446694813274363gmail-m_-5601345256008515377gmail-m_8524775435516132372gmail-pl-en">sip_msg_shm_clone</span>" spikes (from 1kb to 1.5GB).<br> - The shared memory used by the function
"dmq:worker.c:job_queue_push" also increases, but not as much (from 1kb to 1MB)<br> - DMQ request are not being answered
(with a 200 OK) by the affected instance during this memory leak, which make us
think that DMQ module becomes unresponsive.<div><br></div><div>A few more notes:</div><div> - The failover instances are doing nothing except receiving replicated contacts.</div><div> - The shared memory grows at the same rate on both instances, but the critical behavior never happens at the same time.</div><div> - We are allocating 1GB memory on startup to each instance.</div><div> - We store the location DB in a psql DB and we load it at startup.<br></div><div> - We didn't find any errors in syslog, even at debug level.</div><div><div><div><br></div><div>Has anyone experienced a similar issue who can suggest a possible solution?</div><div><br></div><div>Thanks,</div><div>Rogelio Perez</div><div>Telnyx<br></div></div></div></div>
<br>______________________________<wbr>_________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/<wbr>cgi-bin/mailman/listinfo/sr-<wbr>users</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div><div class="gmail_extra">-- <br><div class="m_-3936293401878097385m_8371728938109241847gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a></div><div><a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/<wbr>miconda</a></div></div></div></div></div>
</div></blockquote></div></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div dir="ltr" class="m_-3936293401878097385gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><font face="arial, helvetica, sans-serif"><b><font size="2">Charles Chance</font></b><br><font size="2">Managing Director</font></font><br><div><font face="arial, helvetica, sans-serif"><font size="2"><br></font></font></div><div><font face="arial, helvetica, sans-serif"><font size="2">t. 0330 120 1200 m. 07932 063 891</font></font></div></div></div></div></div>
<br>
<div><font color="gray" style="font-size:10pt;font-family:Helvetica,Arial,sans-serif">Sipcentric Ltd.
Company registered in England & Wales no. 7365592.</font><span style="font-size:10pt;font-family:Helvetica,Arial,sans-serif"> </span><font color="gray" style="font-size:10pt;font-family:Helvetica,Arial,sans-serif">Registered
office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.</font></div></font></span><br>______________________________<wbr>_________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/<wbr>cgi-bin/mailman/listinfo/sr-<wbr>users</a><br>
<br></blockquote></div><br></div>