<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Roboto;
panose-1:2 11 6 4 2 2 2 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.m-1847061220120970495p1, li.m-1847061220120970495p1, div.m-1847061220120970495p1
{mso-style-name:m_-1847061220120970495p1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.m-1847061220120970495s1
{mso-style-name:m_-1847061220120970495s1;}
span.m-1847061220120970495apple-converted-space
{mso-style-name:m_-1847061220120970495apple-converted-space;}
span.m-1847061220120970495s2
{mso-style-name:m_-1847061220120970495s2;}
p.m-1847061220120970495p2, li.m-1847061220120970495p2, div.m-1847061220120970495p2
{mso-style-name:m_-1847061220120970495p2;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal>Hello Sergey,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>That did seem to help serialize the messages at 1000 cps – using t_relay_to_tcp for outbound routing. At higher cps rates (2000 cps and up) I did get some call failures again. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I will have to give some consideration as to implications of this model – in my application, I will have a relatively static community of SIP agents talking to the proxy (maximum of a few thousand), with pretty high volume from each speaker. This would mean a relatively manageable “thousands” of TCP connections spread out over several clusters.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>sr-users <sr-users-bounces@lists.kamailio.org> on behalf of Sergey Safarov <s.safarov@gmail.com><br><b>Reply-To: </b>"Kamailio (SER) - Users Mailing List" <sr-users@lists.kamailio.org><br><b>Date: </b>Monday, December 5, 2022 at 11:59 AM<br><b>To: </b>"Kamailio (SER) - Users Mailing List" <sr-users@lists.kamailio.org><br><b>Subject: </b>Re: [SR-Users] Packet processing order<o:p></o:p></span></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Could you try TCP/TLS transport?<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>In this case, packets will be ordered back at the TCP/TLS transport level.<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Mon, Dec 5, 2022 at 9:35 PM Jawaid Bazyar <<a href="mailto:bazyar@gmail.com">bazyar@gmail.com</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hi,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I have experienced out of order packet processing when testing a simple Kamailio config.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The configuration is as follows, basically:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=m-1847061220120970495p1><span class=m-1847061220120970495s1>request_route{</span><o:p></o:p></p><p class=m-1847061220120970495p1><span class=m-1847061220120970495apple-converted-space> </span><span class=m-1847061220120970495s1>record_route()</span><span class=m-1847061220120970495s2>;</span><o:p></o:p></p><p class=m-1847061220120970495p2> <o:p></o:p></p><p class=m-1847061220120970495p1><span class=m-1847061220120970495apple-converted-space> </span><span class=m-1847061220120970495s1>enum_query()</span><span class=m-1847061220120970495s2>;</span><o:p></o:p></p><p class=m-1847061220120970495p1><span class=m-1847061220120970495apple-converted-space> </span><span class=m-1847061220120970495s1>xlog("INVITE ENUM query - To URI $tU")</span><span class=m-1847061220120970495s2>;</span><o:p></o:p></p><p class=m-1847061220120970495p1><span class=m-1847061220120970495apple-converted-space> </span><span class=m-1847061220120970495s1>forward()</span><span class=m-1847061220120970495s2>;</span><o:p></o:p></p><p class=m-1847061220120970495p1><span class=m-1847061220120970495s1>}</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I saw this thread from 2020:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'><a href="https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html" target="_blank"><span style='font-size:10.5pt;font-family:Roboto;color:#1155CC'>https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html</span></a></span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP packets are processed without regard to whether packets received before this one, are currently being processed. This means any packets after the initial INVITE that require more processing, can get reordered. </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>In my test lab I have:</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> SIPp – UAC</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> Kamailio Proxy</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> SIPp – UAS</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order packets.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>It is only when I introduce Kamailio in the middle that I get OOO packets.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>See the images attached: uac-to-proxy, proxy, and proxy-to-uas.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC does not see packets in the correct order, I never turns on the simulated audio. Calls that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a phone call.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>In the thread, a few things were suggested.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>Have fewer kamailio processes than cores:</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> Did not resolve issue.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>Try route_locks_size = 256</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> Did not resolve issue. Though, it did alter it somewhat. But, it is not clear to me how this works – would this setting restrict the number of open calls to 256?</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>Have only one kamailio process: (“children=1”)</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> This works. “Works”, I should say, because this would greatly restrict total platform scalability to a point where it is probably useless for my application.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response was “this is not a bug”.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'>Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I don’t think a proxy should reorder packets involved in a call in a non-deterministic way. In the Kamailio list thread, Alex Balashov discusses the contortions he has to go through to avoid repercussions from this issue.</span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:black'> </span><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Kamailio as-is probably works fine for relatively low-volume operations. And a lot of the feedback is “why are out of order packets a problem?” OK, sure, in the very specific example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely nothing preventing Kamailio from generating much more serious OOO scenarios that would cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”. Who knows how other SIP stacks will behave?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>But the more one will try to scale Kamailio, the more significantly this out of order processing issue will become.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The solution to this seems very simple and straightforward – put packets to be processed into a queue PER Call-ID, or something along those lines.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>In short, the parallelism should be by call, not by packet.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>What say ye? Have I misunderstood something here?<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Cheers,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Jawaid<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p></div></div><p class=MsoNormal>__________________________________________________________<br>Kamailio - Users Mailing List - Non Commercial Discussions<br><a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>Important: keep the mailing list in the recipients, do not reply only to the sender!<br>Edit mailing list options or unsubscribe:<br><a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></p></div></blockquote></div><p class=MsoNormal>__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <o:p></o:p></p></div></body></html>