[SR-Users] Packet processing order

[Jawaid Bazyar]

Hi,

 

I have experienced out of order packet processing when testing a simple Kamailio config.

 

The configuration is as follows, basically:

 

request_route{

        record_route();

 

        enum_query();

        xlog("INVITE ENUM query - To URI $tU");

        forward();

}

 

I saw this thread from 2020:

 

https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html

 

The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP packets are processed without regard to whether packets received before this one, are currently being processed. This means any packets after the initial INVITE that require more processing, can get reordered. 

 

In my test lab I have:

                SIPp – UAC

                Kamailio Proxy

                SIPp – UAS

 

The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.

 

Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order packets.

 

It is only when I introduce Kamailio in the middle that I get OOO packets.

 

See the images attached: uac-to-proxy, proxy, and proxy-to-uas.

 

In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC does not see packets in the correct order, I never turns on the simulated audio. Calls that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a phone call.

 

So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.

 

 

In the thread, a few things were suggested.

 

Have fewer kamailio processes than cores:

                Did not resolve issue.

 

Try route_locks_size = 256

                Did not resolve issue. Though, it did alter it somewhat. But, it is not clear to me how this works – would this setting restrict the number of open calls to 256?

 

Have only one kamailio process: (“children=1”)

                This works. “Works”, I should say, because this would greatly restrict total platform scalability to a point where it is probably useless for my application.

 

 

I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response was “this is not a bug”.

 

Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I don’t think a proxy should reorder packets involved in a call in a non-deterministic way. In the Kamailio list thread, Alex Balashov discusses the contortions he has to go through to avoid repercussions from this issue.

 

Kamailio as-is probably works fine for relatively low-volume operations. And a lot of the feedback is “why are out of order packets a problem?” OK, sure, in the very specific example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely nothing preventing Kamailio from generating much more serious OOO scenarios that would cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”. Who knows how other SIP stacks will behave?

 

But the more one will try to scale Kamailio, the more significantly this out of order processing issue will become.

 

The solution to this seems very simple and straightforward – put packets to be processed into a queue PER Call-ID, or something along those lines.

 

In short, the parallelism should be by call, not by packet.

 

What say ye? Have I misunderstood something here?

 

Cheers,

 

Jawaid

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20221205/8fcd625f/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: proxy.png
Type: image/png
Size: 33411 bytes
Desc: not available
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20221205/8fcd625f/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: proxy-to-uas.png
Type: image/png
Size: 30838 bytes
Desc: not available
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20221205/8fcd625f/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: uac-to-proxy.png
Type: image/png
Size: 30410 bytes
Desc: not available
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20221205/8fcd625f/attachment-0002.png>