<div dir="auto"><div>Hi,<div dir="auto"><br></div><div dir="auto">Thank you all for the suggestions. It makes sense the ARP to be the issue as we have homer on the same network. So we will relocate it to a another subnet and we will also consider adding the proxy in-between but again not on the same subnet.</div><div dir="auto">We will also install the gdb to have some backtraces when needed.</div><div dir="auto"><br></div><div dir="auto">I will update you with results when ready.</div><div dir="auto"><br></div>Kind regards,</div><div dir="auto">Angelo<br><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Thu, Feb 3, 2022, 6:57 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p>Hello,</p>
<p>when Kamailio is not processing sip messages, so it looks like
stuck, you can grab the backtraces with gdb to see what each
kamailio process is executing at that moment. You need gdb
installed and then run:</p>
<p>kamctl trap</p>
<p>It writes to a file in /tmp, the file name is printed by the
command.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 03.02.22 12:22, James Browne wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>I've encountered a very similar problem in the past.The
siptrace module uses UDP and does not expect a response, so it
can be confusing that kamailio's behavior changes depending on
whether the homer server is online.</div>
<div><br>
</div>
<div>The problem I've seen in the past is that when kamailio and
a statsd server are in the same IP subnet, and the statds
server goes offline, an ARP request is made whenever traffic
is to be sent, and this ARP request is blocking until it times
out (a few seconds). It blocks all workers because they all
want to send a UDP packet and all have to wait.<br>
</div>
<div><br>
</div>
<div>If this is what's going on, then moving the siptrace server
to a different subnet should resolve the problem.</div>
<div><br>
</div>
<div>James<br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 2 Feb 2022 at 21:43,
Angelo Sipper <<a href="mailto:sippro97@gmail.com" target="_blank" rel="noreferrer">sippro97@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
<div>Hi,</div>
<div dir="auto"><br>
</div>
<div dir="auto">I have the exact same configuration with two
workers and same syntax.</div>
<div dir="auto">It works fine until I shut down homer. After
that Kamailio stop precessing any traffic and all clients
try to register as they are disconnected.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Kind Regards,</div>
<div dir="auto">Angelo<br>
<br>
<div class="gmail_quote" dir="auto">
<div dir="ltr" class="gmail_attr">On Wed, Feb 2, 2022,
8:45 PM Karsten Horsmann <<a href="mailto:khorsmann@gmail.com" target="_blank" rel="noreferrer">khorsmann@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
<div dir="auto">Hi, </div>
<div dir="auto"><br>
</div>
<div dir="auto">try to define an extra sendsocket. </div>
<div dir="auto">
<table>
<tbody>
</tbody>
</table>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">And add an listen directive for that
sendsocket. Sorry for the formating wrote this on
my smartphone. </div>
<div dir="auto"><br>
</div>
<div dir="auto">Works for me with an similar setup
and 5.3 </div>
<div dir="auto">
<table>
<tbody>
</tbody>
</table>
</div>
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto">
<div dir="auto"># capture node or webhomer
target</div>
<div dir="auto">#!ifdef WITH_HOMER</div>
<div dir="auto">#!substdef
"!MY_HOMER_CAPTURE!<a rel="noreferrer">sip:MY_HOMER_IP_ADDR:MY_HOMER_PORT!g</a>"</div>
<div dir="auto">#!substdef
"!MY_LOCAL_SENDSOCK!<a rel="noreferrer">sip:MY_PRV_IP:MY_LOCAL_SEND_PORT!g</a>"</div>
<div dir="auto">#!substdef
"!MY_LOCAL_SENDLISTEN!<a rel="noreferrer">udp:MY_PRV_IP:MY_LOCAL_SEND_PORT!g</a>"</div>
<div dir="auto">#!endif</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
</div>
#!ifdef WITH_HOMER
<div dir="auto">#Siptrace</div>
<div dir="auto">modparam("siptrace",
"force_send_sock", "MY_LOCAL_SENDSOCK")</div>
<div dir="auto"># local interface from where to send
the duplicated traffic</div>
<div dir="auto">modparam("siptrace",
"duplicate_uri", "MY_HOMER_CAPTURE")</div>
<div dir="auto">modparam("siptrace", "hep_mode_on",
1) # enable/disable Homer encapsulate mode</div>
<div dir="auto">modparam("siptrace", "hep_version",
3) # HEPv3 the timestamp and capture agent ID will
be included in the HEP header</div>
<div dir="auto">modparam("siptrace",
"hep_capture_id", MY_HOMER_CAPTURE_ID) # capture
agent id Limitation: 32-bit for HEPv3.</div>
<div dir="auto">modparam("siptrace",
"trace_to_database", 0) # trace to database</div>
<div dir="auto">modparam("siptrace", "trace_on", 1)
# enable/disable trace</div>
<div dir="auto">modparam("siptrace", "trace_mode",
1) # default 0, if 1 then you dont need call
siptrace flag or sip_trace()</div>
<div dir="auto">#!endif</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div dir="auto"># sendsocket for homer </div>
<div dir="auto">#!ifdef WITH_HOMER</div>
<div dir="auto">socket_workers=2</div>
<div dir="auto">listen=MY_LOCAL_SENDLISTEN</div>
<div dir="auto">#!endif</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Henning Westerholt
<<a href="mailto:hw@gilawa.com" rel="noreferrer
noreferrer noreferrer noreferrer" target="_blank">hw@gilawa.com</a>>
schrieb am Mi., 2. Feb. 2022, 18:44:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="DE">
<div>
<p class="MsoNormal"><span>Hello,</span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal"><span lang="EN-GB">any
feedback regarding my previous question
about log files, errors?</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">Cheers,</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">Henning</span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<p class="MsoNormal"><span lang="EN-GB">-- </span></p>
<p class="MsoNormal"><span lang="EN-GB">Henning
Westerholt –
</span><span><a href="https://skalatan.de/blog/" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank"><span style="color:rgb(5,99,193)" lang="EN-GB">https://skalatan.de/blog/</span></a></span><span lang="EN-GB"></span></p>
<p class="MsoNormal"><span lang="EN-GB">Kamailio
services –
</span><span><a href="https://gilawa.com/" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank"><span style="color:rgb(5,99,193)" lang="EN-GB">https://gilawa.com</span></a></span><span lang="EN-GB"></span></p>
<p class="MsoNormal"><span lang="EN-GB"> </span></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b>
sr-users <<a href="mailto:sr-users-bounces@lists.kamailio.org" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">sr-users-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>Angelo Sipper<br>
<b>Sent:</b> Wednesday, February 2, 2022
5:45 PM<br>
<b>To:</b> Kamailio (SER) - Users Mailing
List <<a href="mailto:sr-users@lists.kamailio.org" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">sr-users@lists.kamailio.org</a>><br>
<b>Subject:</b> Re: [SR-Users] Homer7 and
Kamailio 5.3</p>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi,</p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Memory on the
system is 8GB and on Kamailio we have
SHM 4096 and PKG 32. Each of these VM
handles only a few hundred of concurrent
calls and each VM has a few hundred of
TLS user registrations.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Kamailio uses
IP to send the traffic to Homer and not
domain name.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Any ideas how
to solve this issue?</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Kind Regards,</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Angelo</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Tue, Feb
1, 2022, 8:43 PM Nick F <<a href="mailto:nik.oloakt@gmail.com" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">nik.oloakt@gmail.com</a>>
wrote:</p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hello, </p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Do use
domain name as destination for
homer? </p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"> </p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Mon,
Jan 31, 2022, 23:12 Henning
Westerholt <<a href="mailto:hw@gilawa.com" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">hw@gilawa.com</a>>
wrote:</p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">
Hello,</p>
<p class="MsoNormal" style="margin-left:35.4pt">
</p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB">Interesting
observation. Do you see
anything related to this error
period in the log files? Maybe
out of memory errors etc..?</span></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB"> </span></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB">Cheers,</span></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB"> </span></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB">Henning</span></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB"> </span></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB">-- </span></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB">Henning
Westerholt – </span><a href="https://skalatan.de/blog/" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank"><span style="color:rgb(5,99,193)" lang="EN-GB">https://skalatan.de/blog/</span></a></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB">Kamailio
services – </span><a href="https://gilawa.com/" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank"><span style="color:rgb(5,99,193)" lang="EN-GB">https://gilawa.com</span></a></p>
<p class="MsoNormal" style="margin-left:35.4pt">
<span lang="EN-GB"> </span></p>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-left:70.8pt">
<b>From:</b> sr-users <<a href="mailto:sr-users-bounces@lists.kamailio.org" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">sr-users-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>Angelo
Sipper<br>
<b>Sent:</b> Monday, January
31, 2022 12:32 PM<br>
<b>To:</b> Kamailio (SER) -
Users Mailing List <<a href="mailto:sr-users@lists.kamailio.org" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">sr-users@lists.kamailio.org</a>><br>
<b>Subject:</b> Re: [SR-Users]
Homer7 and Kamailio 5.3</p>
</div>
<p class="MsoNormal" style="margin-left:70.8pt">
</p>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Hi,</p>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
</p>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Any ideas on this topic?</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Anyone with same behavior?</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Regards,</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Angelo</p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:70.8pt">
</p>
<div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
On Thu, Jan 27, 2022, 9:18
PM Angelo Sipper <<a href="mailto:sippro97@gmail.com" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">sippro97@gmail.com</a>>
wrote:</p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Hi to all,</p>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
We have a Kamailio 5.3
that sends all traffic
to a homer7 using
siptrace module with
parameters
force_send_sock and
hep_mode_on,1 and
hep_version,3 and
trace_mode,1. Everything
works fine getting all
traffic to Homer7 but,
in case Homer7 is not
reachable, Kamailio then
no longer accepts
traffic or registrations
and it is like it
crashes. When we revive
Homer7 and is it again
reachable accepting
traffic, Kamailio then
works again fine without
any restart or actions.</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Any ideas why this may
be happening? Is there
anything we need to add
to the config to avoid
this behavior as having
Homer7 now is a single
point of failure for
Kamailio?</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Thank you!</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Kind Regards,</p>
</div>
<div>
<p class="MsoNormal" style="margin-left:70.8pt">
Angelo</p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">__________________________________________________________<br>
Kamailio - Users Mailing List - Non
Commercial Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in
the recipients, do not reply only to
the sender!<br>
Edit mailing list options or
unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer noreferrer
noreferrer noreferrer noreferrer" target="_blank">
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:35.4pt">__________________________________________________________<br>
Kamailio - Users Mailing List - Non
Commercial Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the
recipients, do not reply only to the
sender!<br>
Edit mailing list options or
unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a></p>
</blockquote>
</div>
</div>
</div>
__________________________________________________________<br>
Kamailio - Users Mailing List - Non Commercial
Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" rel="noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the
recipients, do not reply only to the sender!<br>
Edit mailing list options or unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer noreferrer noreferrer noreferrer
noreferrer noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote>
</div>
__________________________________________________________<br>
Kamailio - Users Mailing List - Non Commercial
Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" rel="noreferrer noreferrer" target="_blank">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the recipients, do
not reply only to the sender!<br>
Edit mailing list options or unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote>
</div>
</div>
</div>
__________________________________________________________<br>
Kamailio - Users Mailing List - Non Commercial Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank" rel="noreferrer">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the recipients, do not
reply only to the sender!<br>
Edit mailing list options or unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote>
</div>
<br>
<fieldset></fieldset>
<pre>__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank" rel="noreferrer">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" rel="noreferrer">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" rel="noreferrer">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" rel="noreferrer">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" rel="noreferrer">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online
Feb 21-24, 2022 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank" rel="noreferrer">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
__________________________________________________________<br>
Kamailio - Users Mailing List - Non Commercial Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank" rel="noreferrer">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the recipients, do not reply only to the sender!<br>
Edit mailing list options or unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div></div></div>