<div dir="ltr">Hello<div>Yes I bet more in the system race condition, but yes it's hard to identify it because I tried several times to replicate it and it didn't happen.</div><div>When it failed the restart was performed by Chef, and I'm trying to replicate it doing the restarts by myself without using chef, not sure if chef might be helping for this cause or not, need to investigate.</div><div><br></div><div>Thank you all</div><div><br></div><div>Best Regards</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Henning Westerholt <<a href="mailto:hw@skalatan.de">hw@skalatan.de</a>> escreveu no dia sexta, 16/10/2020 à(s) 15:13:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DE" style="overflow-wrap: break-word;">
<div class="gmail-m_-5453948142017087075WordSection1">
<p class="MsoNormal"><span>Hello,<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">It might be something during the restart, that some of the processes did not started correctly due to still allocated resources from the “old” processes still running. Maybe a race condition
in this particular system environment, but hard to say if you cannot reproduce it.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Cheers,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Henning<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">-- <u></u>
<u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Henning Westerholt –
</span><span><a href="https://skalatan.de/blog/" target="_blank"><span lang="EN-GB" style="color:rgb(5,99,193)">https://skalatan.de/blog/</span></a></span><span lang="EN-GB"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Kamailio services –
</span><span><a href="https://gilawa.com/" target="_blank"><span lang="EN-GB" style="color:rgb(5,99,193)">https://gilawa.com</span></a></span><span>
<span lang="EN-GB"><u></u><u></u></span></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b> sr-users <<a href="mailto:sr-users-bounces@lists.kamailio.org" target="_blank">sr-users-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>José Seabra<br>
<b>Sent:</b> Friday, October 16, 2020 4:02 PM<br>
<b>To:</b> Kamailio (SER) - Users Mailing List <<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>><br>
<b>Subject:</b> Re: [SR-Users] usrloc_dmq - Suddenly stopped usrloc replication<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">no, I'm pretty sure that didn't happen, I don't have any other config files there.<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Regards<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Ovidiu Sas <<a href="mailto:osas@voipembedded.com" target="_blank">osas@voipembedded.com</a>> escreveu no dia sexta, 16/10/2020 à(s) 13:47:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Maybe you restarted with a different config file?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">-ovidiu<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Fri, Oct 16, 2020 at 04:54 José Seabra <<a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi Ovidiu,<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I compared both logs and when kamailio didn't started all processes, there is missing the log msg from usrloc_dmq:<u></u><u></u></p>
</div>
<div>
<pre style="margin-left:35.4pt;background:rgb(244,245,247);max-height:30em;white-space:pre-wrap;outline:none;overflow:auto"><span style="font-size:9pt;color:rgb(23,43,77)">INFO: dmq_usrloc [dmq_usrloc.c:77]: mod_init(): dmq usrloc replication mode = 1<u></u><u></u></span></pre>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">But I still have doubts why it happened, do you guys know any possible reason that could make this happening?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">Thank you<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Best regards<u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">José<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Ovidiu Sas <<a href="mailto:osas@voipembedded.com" target="_blank">osas@voipembedded.com</a>> escreveu no dia quinta, 15/10/2020 à(s) 23:18:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-left:35.4pt">Compare the logs for the two cases. Also, check if there are any error<br>
logs for the case with missing processes.<br>
<br>
Regards,<br>
Ovidiu Sas<br>
<br>
On Thu, Oct 15, 2020 at 5:51 PM José Seabra <<a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>> wrote:<br>
><br>
> Hi there,<br>
> so, I guess that I found the reason why it's not working, I noticed that it stopped working at the same time that kamailio restarted, then I went check the number of processes and workers and I realised that we were missing a few processes and workers, restarted
kamailio and it started working again, all processes and workers were created.<br>
> Anyway I still don't know why the restart didn't start all the processes.<br>
><br>
> Regards<br>
><br>
> José Seabra <<a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>> escreveu no dia quinta, 15/10/2020 à(s) 22:12:<br>
>><br>
>> Hello there,<br>
>> I forgot to mention that I'm running kamailio 5.3.4<br>
>><br>
>> Best Regards<br>
>><br>
>> José Seabra <<a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>> escreveu no dia quinta, 15/10/2020 à(s) 22:11:<br>
>>><br>
>>> Hello there,<br>
>>><br>
>>> Suddenly the usrloc replication through DMQ stopped, but the interesting thing is that DMQ cluster status is OK, htable replication still working only usrloc_dmq stopped working.<br>
>>><br>
>>> I increased the kamailio loglevel to 3 in order to see if there was some log related and then registered a peer, but the only DMQ logs that I see are related with notification_peer and htable replication, nothing related with usrloc_dmq.<br>
>>><br>
>>> What I can do more here to try to understand what went wrong?<br>
>>><br>
>>> I haven't restarted the cluster yet, because I would like to try to understand the root cause of this issue.<br>
>>><br>
>>> Any advice would be appreciated<br>
>>><br>
>>> Thank you<br>
>>><br>
>>> --<br>
>>> Cumprimentos<br>
>>> José Seabra<br>
>><br>
>><br>
>><br>
>> --<br>
>> Cumprimentos<br>
>> José Seabra<br>
><br>
><br>
><br>
> --<br>
> Cumprimentos<br>
> José Seabra<br>
> _______________________________________________<br>
> Kamailio (SER) - Users Mailing List<br>
> <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
> <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
<br>
<br>
<br>
-- <br>
VoIP Embedded, Inc.<br>
<a href="http://www.voipembedded.com" target="_blank">http://www.voipembedded.com</a><br>
<br>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">-- <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Cumprimentos<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">José Seabra<u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><u></u><u></u></p>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">-- <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">VoIP Embedded, Inc.<br>
<a href="http://www.voipembedded.com" target="_blank">http://www.voipembedded.com</a><u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><u></u> <u></u></p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">-- <u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Cumprimentos<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">José Seabra<u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Cumprimentos<div>José Seabra</div></div>