<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hello,</p>
<p>actually your expectation that packets should come in order is
"naive", just think about how internet is constructed and IP
routing works. In the past it was easy to reproduce on mobile
networks scenarios when sending CANCEL very quickly after the
INVITE resulted in CANCEL arriving first at the proxy, then the
INVITE.</p>
<p>Probably you don't get it in your lab environment where you have
sipp on the same system as the sip server or one network segment
away, but over the internet the packets can get in different order
because of network transmission (different IP routing paths). It
is the reason in some cases there are jitter buffers and sequence
numbers (e.g., in RTP and SIP (CSeq)). In other words, the
protocols like RTP or SIP were designed to deal with these
out-of-order packets.</p>
<p>Ans here is a wikipedia short article enumerating what can cause
out of order:</p>
<p> - <a class="moz-txt-link-freetext" href="https://en.wikipedia.org/wiki/Out-of-order_delivery">https://en.wikipedia.org/wiki/Out-of-order_delivery</a></p>
<p>That said, if you missed in the message from mailing list archive
that you linked to, there is a global parameter that should reduce
the risk of sending out of order sip packets to the minimum that
can be done from SIP processing point of view based on call-id,
but there are still chances that on multi-cpu systems the packets
read from the network can get to be processed in different order
because of how read on udp sockets is done by kernel/libc and how
cpu scheduler allocate cycles to running application processes.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 05.12.22 19:34, Jawaid Bazyar wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CBCA38F3-68B8-4CF3-8EB2-164438D13571@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}@font-face
{font-family:Roboto;
panose-1:2 11 6 4 2 2 2 2 2 4;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
font-size:8.5pt;
font-family:Menlo;
color:black;}span.s2
{mso-style-name:s2;
color:#400BD9;}span.s1
{mso-style-name:s1;}span.apple-converted-space
{mso-style-name:apple-converted-space;}.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}div.WordSection1
{page:WordSection1;}</style>
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have experienced out of order packet
processing when testing a simple Kamailio config.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The configuration is as follows, basically:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">request_route{</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span
class="s1">record_route()</span><span class="s2">;</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span
class="s1">enum_query()</span><span class="s2">;</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span
class="s1">xlog("INVITE ENUM query - To URI $tU")</span><span
class="s2">;</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span
class="s1">forward()</span><span class="s2">;</span><o:p></o:p></p>
<p class="p1"><span class="s1">}</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I saw this thread from 2020:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:black"><a
href="https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html"
moz-do-not-send="true"><span
style="font-size:10.5pt;font-family:Roboto;color:#1155CC">https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">The issue seems
to be that kamailio process scheduling is naïve – i.e.,
incoming SIP packets are processed without regard to whether
packets received before this one, are currently being
processed. This means any packets after the initial INVITE
that require more processing, can get reordered. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">In my test lab I
have:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">
SIPp – UAC<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">
Kamailio Proxy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">
SIPp – UAS<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">The proxy uses
enum NAPR lookups to route calls to +13038151000 to the UAS.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">Now, if I do SIPp
UAC o SIPp UAS directly, I have no problems – no out of
order packets.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">It is only when I
introduce Kamailio in the middle that I get OOO packets.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">See the images
attached: uac-to-proxy, proxy, and proxy-to-uas.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal">In this example, Kamailio OOO causes SIPp
UAC to fail to “generate audio” – because UAC does not see
packets in the correct order, I never turns on the simulated
audio. Calls that have in-order dialogs, work fine, and SIPP
UAC “pauses” 10 seconds to simulate a phone call.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So far, the error rate runs from 1/1000 to
around 1/200 – pretty bad.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">In the thread, a
few things were suggested.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">Have fewer
kamailio processes than cores:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">
Did not resolve issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">Try
route_locks_size = 256<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">
Did not resolve issue. Though, it did alter it somewhat.
But, it is not clear to me how this works – would this
setting restrict the number of open calls to 256?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">Have only one
kamailio process: (“children=1”)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">
This works. “Works”, I should say, because this would
greatly restrict total platform scalability to a point where
it is probably useless for my application.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">I saw the same
issue discussed in the OpenSIPS mailing list from 2010, and
the response was “this is not a bug”.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:black">Well, I
respectfully beg to differ with both OpenSIPS and Kamailio –
it IS a bug. I don’t think a proxy should reorder packets
involved in a call in a non-deterministic way. In the
Kamailio list thread, Alex Balashov discusses the
contortions he has to go through to avoid repercussions from
this issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoNormal">Kamailio as-is probably works fine for
relatively low-volume operations. And a lot of the feedback is
“why are out of order packets a problem?” OK, sure, in the
very specific example given in the 2020 thread, maybe who
cares. But in my thinking, there is absolutely nothing
preventing Kamailio from generating much more serious OOO
scenarios that would cause calls to fail. In my example,
Kamailio OOO causes SIPp to fail to “generate audio”. Who
knows how other SIP stacks will behave?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But the more one will try to scale
Kamailio, the more significantly this out of order processing
issue will become.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The solution to this seems very simple and
straightforward – put packets to be processed into a queue PER
Call-ID, or something along those lines.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In short, the parallelism should be by
call, not by packet.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What say ye? Have I misunderstood something
here?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Cheers,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Jawaid<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
<a class="moz-txt-link-abbreviated" href="mailto:sr-users@lists.kamailio.org">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a></pre>
</body>
</html>