5 Dec
2022
5 Dec
'22
11:34 p.m.
Hi,
I have experienced out of order packet processing when testing a simple Kamailio config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html
The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP
packets are processed without regard to whether packets received before this one, are
currently being processed. This means any packets after the initial INVITE that require
more processing, can get reordered.
In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC
does not see packets in the correct order, I never turns on the simulated audio. Calls
that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a
phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat. But, it is not
clear to me how this works – would this setting restrict the number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would greatly restrict
total platform scalability to a point where it is probably useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response
was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I don’t
think a proxy should reorder packets involved in a call in a non-deterministic way. In the
Kamailio list thread, Alex Balashov discusses the contortions he has to go through to
avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations. And a lot of the
feedback is “why are out of order packets a problem?” OK, sure, in the very specific
example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely
nothing preventing Kamailio from generating much more serious OOO scenarios that would
cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this out of order
processing issue will become.
The solution to this seems very simple and straightforward – put packets to be processed
into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
5 Dec
5 Dec
11:58 p.m.
Could you try TCP/TLS transport?
In this case, packets will be ordered back at the TCP/TLS transport level.
On Mon, Dec 5, 2022 at 9:35 PM Jawaid Bazyar <bazyar(a)gmail.com> wrote:
Hi,
I have experienced out of order packet processing when testing a simple
Kamailio config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html
The issue seems to be that kamailio process scheduling is naïve – i.e.,
incoming SIP packets are processed without regard to whether packets
received before this one, are currently being processed. This means any
packets after the initial INVITE that require more processing, can get
reordered.
In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of
order packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio”
– because UAC does not see packets in the correct order, I never turns on
the simulated audio. Calls that have in-order dialogs, work fine, and SIPP
UAC “pauses” 10 seconds to simulate a phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat.
But, it is not clear to me how this works – would this setting restrict the
number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would
greatly restrict total platform scalability to a point where it is probably
useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and
the response was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS
a bug. I don’t think a proxy should reorder packets involved in a call in a
non-deterministic way. In the Kamailio list thread, Alex Balashov discusses
the contortions he has to go through to avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations.
And a lot of the feedback is “why are out of order packets a problem?” OK,
sure, in the very specific example given in the 2020 thread, maybe who
cares. But in my thinking, there is absolutely nothing preventing Kamailio
from generating much more serious OOO scenarios that would cause calls to
fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this
out of order processing issue will become.
The solution to this seems very simple and straightforward – put packets
to be processed into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
6 Dec
6 Dec
12:25 a.m.
Hello Sergey,
That did seem to help serialize the messages at 1000 cps – using t_relay_to_tcp for
outbound routing. At higher cps rates (2000 cps and up) I did get some call failures
again.
I will have to give some consideration as to implications of this model – in my
application, I will have a relatively static community of SIP agents talking to the proxy
(maximum of a few thousand), with pretty high volume from each speaker. This would mean a
relatively manageable “thousands” of TCP connections spread out over several clusters.
From: sr-users <sr-users-bounces(a)lists.kamailio.org> on behalf of Sergey Safarov
<s.safarov(a)gmail.com>
Reply-To: "Kamailio (SER) - Users Mailing List"
<sr-users(a)lists.kamailio.org>
Date: Monday, December 5, 2022 at 11:59 AM
To: "Kamailio (SER) - Users Mailing List" <sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Packet processing order
Could you try TCP/TLS transport?
In this case, packets will be ordered back at the TCP/TLS transport level.
On Mon, Dec 5, 2022 at 9:35 PM Jawaid Bazyar <bazyar(a)gmail.com> wrote:
Hi,
I have experienced out of order packet processing when testing a simple Kamailio config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html
The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP
packets are processed without regard to whether packets received before this one, are
currently being processed. This means any packets after the initial INVITE that require
more processing, can get reordered.
In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC
does not see packets in the correct order, I never turns on the simulated audio. Calls
that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a
phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat. But, it is not
clear to me how this works – would this setting restrict the number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would greatly restrict
total platform scalability to a point where it is probably useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response
was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I don’t
think a proxy should reorder packets involved in a call in a non-deterministic way. In the
Kamailio list thread, Alex Balashov discusses the contortions he has to go through to
avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations. And a lot of the
feedback is “why are out of order packets a problem?” OK, sure, in the very specific
example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely
nothing preventing Kamailio from generating much more serious OOO scenarios that would
cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this out of order
processing issue will become.
The solution to this seems very simple and straightforward – put packets to be processed
into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________ Kamailio - Users Mailing List -
Non Commercial Discussions sr-users(a)lists.kamailio.org Important: keep the mailing list in
the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
1:54 p.m.
Could you show messge flow in the sngrep?
Which command used to start sipp for UAC and UAS? (-t t1 or -t tn)
On Mon, Dec 5, 2022 at 10:26 PM Jawaid Bazyar <bazyar(a)gmail.com> wrote:
Hello Sergey,
That did seem to help serialize the messages at 1000 cps – using
t_relay_to_tcp for outbound routing. At higher cps rates (2000 cps and up)
I did get some call failures again.
I will have to give some consideration as to implications of this model –
in my application, I will have a relatively static community of SIP agents
talking to the proxy (maximum of a few thousand), with pretty high volume
from each speaker. This would mean a relatively manageable “thousands” of
TCP connections spread out over several clusters.
*From: *sr-users <sr-users-bounces(a)lists.kamailio.org> on behalf of
Sergey Safarov <s.safarov(a)gmail.com>
*Reply-To: *"Kamailio (SER) - Users Mailing List" <
sr-users(a)lists.kamailio.org>
*Date: *Monday, December 5, 2022 at 11:59 AM
*To: *"Kamailio (SER) - Users Mailing List"
<sr-users(a)lists.kamailio.org>
*Subject: *Re: [SR-Users] Packet processing order
Could you try TCP/TLS transport?
In this case, packets will be ordered back at the TCP/TLS transport level.
On Mon, Dec 5, 2022 at 9:35 PM Jawaid Bazyar <bazyar(a)gmail.com> wrote:
Hi,
I have experienced out of order packet processing when testing a simple
Kamailio config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html
The issue seems to be that kamailio process scheduling is naïve – i.e.,
incoming SIP packets are processed without regard to whether packets
received before this one, are currently being processed. This means any
packets after the initial INVITE that require more processing, can get
reordered.
In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of
order packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio”
– because UAC does not see packets in the correct order, I never turns on
the simulated audio. Calls that have in-order dialogs, work fine, and SIPP
UAC “pauses” 10 seconds to simulate a phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat.
But, it is not clear to me how this works – would this setting restrict the
number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would
greatly restrict total platform scalability to a point where it is probably
useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and
the response was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS
a bug. I don’t think a proxy should reorder packets involved in a call in a
non-deterministic way. In the Kamailio list thread, Alex Balashov discusses
the contortions he has to go through to avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations.
And a lot of the feedback is “why are out of order packets a problem?” OK,
sure, in the very specific example given in the 2020 thread, maybe who
cares. But in my thinking, there is absolutely nothing preventing Kamailio
from generating much more serious OOO scenarios that would cause calls to
fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this
out of order processing issue will become.
The solution to this seems very simple and straightforward – put packets
to be processed into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________ Kamailio -
Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org Important: keep the mailing list in the
recipients, do not reply only to the sender! Edit mailing list options or
unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to
the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
2:15 p.m.
Hello,
actually your expectation that packets should come in order is "naive",
just think about how internet is constructed and IP routing works. In
the past it was easy to reproduce on mobile networks scenarios when
sending CANCEL very quickly after the INVITE resulted in CANCEL arriving
first at the proxy, then the INVITE.
Probably you don't get it in your lab environment where you have sipp on
the same system as the sip server or one network segment away, but over
the internet the packets can get in different order because of network
transmission (different IP routing paths). It is the reason in some
cases there are jitter buffers and sequence numbers (e.g., in RTP and
SIP (CSeq)). In other words, the protocols like RTP or SIP were designed
to deal with these out-of-order packets.
Ans here is a wikipedia short article enumerating what can cause out of
order:
- https://en.wikipedia.org/wiki/Out-of-order_delivery
That said, if you missed in the message from mailing list archive that
you linked to, there is a global parameter that should reduce the risk
of sending out of order sip packets to the minimum that can be done from
SIP processing point of view based on call-id, but there are still
chances that on multi-cpu systems the packets read from the network can
get to be processed in different order because of how read on udp
sockets is done by kernel/libc and how cpu scheduler allocate cycles to
running application processes.
Cheers,
Daniel
On 05.12.22 19:34, Jawaid Bazyar wrote:
Hi,
I have experienced out of order packet processing when testing a
simple Kamailio config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html
<https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html>
The issue seems to be that kamailio process scheduling is naïve –
i.e., incoming SIP packets are processed without regard to whether
packets received before this one, are currently being processed. This
means any packets after the initial INVITE that require more
processing, can get reordered.
In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the
UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out
of order packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate
audio” – because UAC does not see packets in the correct order, I
never turns on the simulated audio. Calls that have in-order dialogs,
work fine, and SIPP UAC “pauses” 10 seconds to simulate a phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it
somewhat. But, it is not clear to me how this works – would this
setting restrict the number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would
greatly restrict total platform scalability to a point where it is
probably useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010,
and the response was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio –
it IS a bug. I don’t think a proxy should reorder packets involved in
a call in a non-deterministic way. In the Kamailio list thread, Alex
Balashov discusses the contortions he has to go through to avoid
repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume
operations. And a lot of the feedback is “why are out of order packets
a problem?” OK, sure, in the very specific example given in the 2020
thread, maybe who cares. But in my thinking, there is absolutely
nothing preventing Kamailio from generating much more serious OOO
scenarios that would cause calls to fail. In my example, Kamailio OOO
causes SIPp to fail to “generate audio”. Who knows how other SIP
stacks will behave?
But the more one will try to scale Kamailio, the more significantly
this out of order processing issue will become.
The solution to this seems very simple and straightforward – put
packets to be processed into a queue PER Call-ID, or something along
those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
6:10 p.m.
I suspect that what Jawaid meant by "naive" was that, in Kamailio, there is not
a central distributor thread which aims to buffer, serialise and otherwise order packets
prior to distribution into worker processes or threads, as is common in many other
multiprocess systems.
I agree that "naive" is perhaps not the best choice of vocabulary for it, and of
course there would be performance downsides to such a centralised pipeline. But I think
that's what was probably meant.
-- Alex
On Dec 6, 2022, at 4:15 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
actually your expectation that packets should come in order is "naive", just
think about how internet is constructed and IP routing works. In the past it was easy to
reproduce on mobile networks scenarios when sending CANCEL very quickly after the INVITE
resulted in CANCEL arriving first at the proxy, then the INVITE.
Probably you don't get it in your lab environment where you have sipp on the same
system as the sip server or one network segment away, but over the internet the packets
can get in different order because of network transmission (different IP routing paths).
It is the reason in some cases there are jitter buffers and sequence numbers (e.g., in RTP
and SIP (CSeq)). In other words, the protocols like RTP or SIP were designed to deal with
these out-of-order packets.
Ans here is a wikipedia short article enumerating what can cause out of order:
- https://en.wikipedia.org/wiki/Out-of-order_delivery
That said, if you missed in the message from mailing list archive that you linked to,
there is a global parameter that should reduce the risk of sending out of order sip
packets to the minimum that can be done from SIP processing point of view based on
call-id, but there are still chances that on multi-cpu systems the packets read from the
network can get to be processed in different order because of how read on udp sockets is
done by kernel/libc and how cpu scheduler allocate cycles to running application
processes.
Cheers,
Daniel
On 05.12.22 19:34, Jawaid Bazyar wrote:
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Hi,
I have experienced out of order packet processing when testing a simple Kamailio
config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html
The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP
packets are processed without regard to whether packets received before this one, are
currently being processed. This means any packets after the initial INVITE that require
more processing, can get reordered. In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order
packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC
does not see packets in the correct order, I never turns on the simulated audio. Calls
that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a
phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat. But, it is not
clear to me how this works – would this setting restrict the number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would greatly restrict
total platform scalability to a point where it is probably useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response
was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I
don’t think a proxy should reorder packets involved in a call in a non-deterministic way.
In the Kamailio list thread, Alex Balashov discusses the contortions he has to go through
to avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations. And a lot of
the feedback is “why are out of order packets a problem?” OK, sure, in the very specific
example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely
nothing preventing Kamailio from generating much more serious OOO scenarios that would
cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this out of order
processing issue will become.
The solution to this seems very simple and straightforward – put packets to be processed
into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
6:44 p.m.
Hi Alex,
Yes that's what I meant exactly. Some multi-process systems are truly stateless (DNS,
for example).
I meant no insult by use of the term, just its sense of "simplistic".
I appreciate Daniel's comments about the Internet creating out of order packets. While
there is truth to that, operators go to great lengths to avoid designs that reorder
packets, as it does strain stacks. TCP treats out of order much the same way it handles
packet loss and it can cause retransmissions and loss of goodput. Surely in SIP there is
the potential to cause timeouts and retransmissions yes?
I guess my point would be, sure, sometimes it can't be helped; but if it can be
helped, why exacerbate it?
I was doing some thinking about performance, and you can maintain performance by the
following approach:
Create N queues, one assigned to each worker thread
Hash incoming messages to a queue based on call ID
Each worker thread works on its assigned queue.
This takes care of the issue - from the perspective of a single dialog, it's the same
as setting children=1. But this approach also ought to enable you to have many more worker
threads. Currently, the more worker threads, the more likely reordering will occur. With
the algorithm above, you can have many more worker threads and not worry about increasing
reordering.
Cheers,
Jawaid
On 12/6/22, 6:11 AM, "sr-users on behalf of Alex Balashov"
<sr-users-bounces(a)lists.kamailio.org on behalf of abalashov(a)evaristesys.com> wrote:
I suspect that what Jawaid meant by "naive" was that, in Kamailio, there is
not a central distributor thread which aims to buffer, serialise and otherwise order
packets prior to distribution into worker processes or threads, as is common in many other
multiprocess systems.
I agree that "naive" is perhaps not the best choice of vocabulary for it,
and of course there would be performance downsides to such a centralised pipeline. But I
think that's what was probably meant.
-- Alex
On Dec 6, 2022, at 4:15 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
Hello,
actually your expectation that packets should come in order is "naive", just
think about how internet is constructed and IP routing works. In the past it was easy to
reproduce on mobile networks scenarios when sending CANCEL very quickly after the INVITE
resulted in CANCEL arriving first at the proxy, then the INVITE.
Probably you don't get it in your lab environment where you have sipp on the same
system as the sip server or one network segment away, but over the internet the packets
can get in different order because of network transmission (different IP routing paths).
It is the reason in some cases there are jitter buffers and sequence numbers (e.g., in RTP
and SIP (CSeq)). In other words, the protocols like RTP or SIP were designed to deal with
these out-of-order packets.
Ans here is a wikipedia short article enumerating what can cause out of order:
- https://en.wikipedia.org/wiki/Out-of-order_delivery
That said, if you missed in the message from mailing list archive that you linked to,
there is a global parameter that should reduce the risk of sending out of order sip
packets to the minimum that can be done from SIP processing point of view based on
call-id, but there are still chances that on multi-cpu systems the packets read from the
network can get to be processed in different order because of how read on udp sockets is
done by kernel/libc and how cpu scheduler allocate cycles to running application
processes.
Cheers,
Daniel
On 05.12.22 19:34, Jawaid Bazyar wrote:
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi,
I have experienced out of order packet processing when testing a simple Kamailio
config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
sr-users(a)lists.kamailio.org <https://www.mail-archive.com/<a
href=>/msg11786.html">https://www.mail-archive.com/sr-users@list…
The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP
packets are processed without regard to whether packets received before this one, are
currently being processed. This means any packets after the initial INVITE that require
more processing, can get reordered. In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order
packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC
does not see packets in the correct order, I never turns on the simulated audio. Calls
that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a
phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat. But, it is not
clear to me how this works – would this setting restrict the number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would greatly restrict
total platform scalability to a point where it is probably useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response
was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I
don’t think a proxy should reorder packets involved in a call in a non-deterministic way.
In the Kamailio list thread, Alex Balashov discusses the contortions he has to go through
to avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations. And a lot of
the feedback is “why are out of order packets a problem?” OK, sure, in the very specific
example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely
nothing preventing Kamailio from generating much more serious OOO scenarios that would
cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this out of order
processing issue will become.
The solution to this seems very simple and straightforward – put packets to be processed
into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
7:09 p.m.
On Dec 6, 2022, at 8:44 AM, Jawaid Bazyar
<bazyar(a)gmail.com> wrote:
Create N queues, one assigned to each worker thread
Hash incoming messages to a queue based on call ID
Each worker thread works on its assigned queue.
This is, in essence, what the `route_locks_size` setting speaks to:
https://www.kamailio.org/wikidocs/cookbooks/5.6.x/core/#route_locks_size
... with the difference that the queues aren't mapped to a particular worker thread.
That would be hard to accomplish in Kamailio's preforked process model, since the
processes are just fork()s that all call recvfrom() / accept() / whatever. It's the
kernel that actually decides which process gets any given message or connection. You'd
have to literally add a distributor thread to an architecture where none exists.
Good news! You can devise your own -- with a little work. ;-)
1) Start by creating (let's say) 4 mqueues[1]:
modparam("mqueue", "mqueue", "name=proc1")
modparam("mqueue", "mqueue", "name=proc2")
modparam("mqueue", "mqueue", "name=proc3")
modparam("mqueue", "mqueue", "name=proc4")
2) Create 4 task routes (rtimer[2] tasks) to consume these routes:
modparam("rtimer", "timer",
"name=proc1;interval=10u;mode=1")
modparam("rtimer", "exec", "timer=proc1;route=PROC1")
modparam("rtimer", "timer",
"name=proc2;interval=10u;mode=1")
modparam("rtimer", "exec", "timer=proc2;route=PROC2")
modparam("rtimer", "timer",
"name=proc3;interval=10u;mode=1")
modparam("rtimer", "exec", "timer=proc3;route=PROC3")
modparam("rtimer", "timer",
"name=proc4;interval=10u;mode=1")
modparam("rtimer", "exec", "timer=proc4;route=PROC4")
3) Have each PROC[x] route consume its respective mqueue. Don't have to needlessly
duplicate, just create some wrappers that use an intermediate variable:
route[ONWARD] {
# In so many words.
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
route[THE_REAL_PROC] {
while(mq_fetch("proc$var(qnum)")) {
$var(id) = $(mqk(term_proc){s.select,0,:}{s.int});
$var(label) = $(mqk(term_proc){s.select,1,:}{s.int});
# Resume transaction in this worker process (rtimer process).
t_continue("$var(id)", "$var(label)", "ONWARD");
}
}
route[PROC1] {
$var(qnum) = '1';
route(THE_REAL_PROC);
}
route[PROC2] {
$var(qnum) = '2';
route(THE_REAL_PROC);
}
route[PROC3] {
$var(qnum) = '3';
route(THE_REAL_PROC);
}
route[PROC4] {
$var(qnum) = '4';
route(THE_REAL_PROC);
}
4) Set children=1 / socket_workers=1 for the applicable listener.
5) Devise a request_route which will hash over the Call-ID, suspend the transaction[3],
and send the transaction index and label identifiers to one of these 4 queues.
core_hash() from cfgutils[4] is a great fit for that, but requires that the hash domain
size be a power of 2 -- which makes a number of queues like 4 or 8 a great choice. :-)
request_route {
...
$var(q) = core_hash("$ci", "", 2); # 2^2, aka 4 queues
if(!t_suspend()) {
sl_send_reply("500", "Internal server error - cannot
suspend");
exit;
}
mq_add("proc$var(q)", "$T(id_index):$T(id_label)");
}
6) This setup will guarantee that messages associated with one Call-ID will always go to
the exact same worker process.
Caveat emptor, of course: haven't tested this exactly as rendered, and it creates a
single bottleneck at the worker process that handles the listening.
-- Alex
[1] https://kamailio.org/docs/modules/5.6.x/modules/mqueue.html
[2] https://kamailio.org/docs/modules/5.6.x/modules/rtimer.html
[3] https://kamailio.org/docs/modules/5.6.x/modules/tmx.html#tmx.f.t_suspend
https://kamailio.org/docs/modules/5.6.x/modules/tmx.html#tmx.f.t_continue
[4] https://kamailio.org/docs/modules/5.6.x/modules/cfgutils.html
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
7:13 p.m.
On Dec 6, 2022, at 9:09 AM, Alex Balashov
<abalashov(a)evaristesys.com> wrote:
route[THE_REAL_PROC] {
while(mq_fetch("proc$var(qnum)")) {
$var(id) = $(mqk(term_proc){s.select,0,:}{s.int});
$var(label) = $(mqk(term_proc){s.select,1,:}{s.int});
My bad. Replace `$(mqk(term_proc)` here with `$(mqk(proc$var(qnum))`
I don't know if PVs can be interpolated there in a mixed way. If they can't, easy
fix:
route[PROC1] {
$var(qnum) = 'proc1';
route(THE_REAL_PROC);
}
route[PROC2] {
$var(qnum) = 'proc2';
route(THE_REAL_PROC);
}
route[PROC3] {
$var(qnum) = 'proc3';
route(THE_REAL_PROC);
}
route[PROC4] {
$var(qnum) = 'proc4';
route(THE_REAL_PROC);
}
route[THE_REAL_PROC] {
while(mq_fetch("$var(qnum)")) {
$var(id) = $(mqk($var(qnum)){s.select,0,:}{s.int});
$var(label) = $(mqk($var(qnum)){s.select,1,:}{s.int});
# Resume transaction in this worker process (rtimer process).
t_continue("$var(id)", "$var(label)", "ONWARD");
}
}
I know that works.
-- Alex
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
8:59 p.m.
On 06.12.22 14:44, Jawaid Bazyar wrote:
Hi Alex,
Yes that's what I meant exactly. Some multi-process systems are truly stateless (DNS,
for example).
I meant no insult by use of the term, just its sense of "simplistic".
I appreciate Daniel's comments about the Internet creating out of order packets.
While there is truth to that, operators go to great lengths to avoid designs that reorder
packets, as it does strain stacks. TCP treats out of order much the same way it handles
packet loss and it can cause retransmissions and loss of goodput.
Well, here you refer to tcp stack implementation, which is at endpoints
of the tcp connection (A<=>B), and they deal with packets order, not the
intermediary IP routers/nodes, what I said in my previous response.
Feel free to make pull requests with patches that you think are useful,
they will be reviewed and merged if they are useful, it's an open source
project and contributions are encouraged. Just have in mind that
performance is a major characteristic expected from a sip proxy.
Surely in SIP there is the potential to cause
timeouts and retransmissions yes?
I guess my point would be, sure, sometimes it can't be helped; but if it can be
helped, why exacerbate it?
I was doing some thinking about performance, and you can maintain performance by the
following approach:
Create N queues, one assigned to each worker thread
Hash incoming messages to a queue based on call ID
Each worker thread works on its assigned queue.
Which does not work on all cases .. like (imagining) for a system used
for large file transfers done with MESSAGE/INFO requests using same
Call-Id and incremented CSeq per file. You can end up with having same
hash id reminder for many call-ids, pushing most of the packets in the
same queue, the associated worker process becoming incapable of handling
the assigned traffic, packets then being dropped from network/socket
buffer or fill available memory.
This takes care of the issue - from the perspective of a single dialog, it's the same
as setting children=1. But this approach also ought to enable you to have many more worker
threads. Currently, the more worker threads, the more likely reordering will occur. With
the algorithm above, you can have many more worker threads and not worry about increasing
reordering.
I pointed to the core parameters for an option that is now available and
Alex named it. He also provided hints about other queuing features. The
sworker module can be leveraged for another variant. But again, it does
not solve it overall.
Cheers,
Daniel
Cheers,
Jawaid
On 12/6/22, 6:11 AM, "sr-users on behalf of Alex Balashov"
<sr-users-bounces(a)lists.kamailio.org on behalf of abalashov(a)evaristesys.com> wrote:
I suspect that what Jawaid meant by "naive" was that, in Kamailio, there is
not a central distributor thread which aims to buffer, serialise and otherwise order
packets prior to distribution into worker processes or threads, as is common in many other
multiprocess systems.
I agree that "naive" is perhaps not the best choice of vocabulary for it,
and of course there would be performance downsides to such a centralised pipeline. But I
think that's what was probably meant.
-- Alex
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
On Dec 6, 2022, at 4:15 AM, Daniel-Constantin
Mierla <miconda(a)gmail.com> wrote:
Hello,
actually your expectation that packets should come in order is "naive", just
think about how internet is constructed and IP routing works. In the past it was easy to
reproduce on mobile networks scenarios when sending CANCEL very quickly after the INVITE
resulted in CANCEL arriving first at the proxy, then the INVITE.
Probably you don't get it in your lab environment where you have sipp on the same
system as the sip server or one network segment away, but over the internet the packets
can get in different order because of network transmission (different IP routing paths).
It is the reason in some cases there are jitter buffers and sequence numbers (e.g., in RTP
and SIP (CSeq)). In other words, the protocols like RTP or SIP were designed to deal with
these out-of-order packets.
Ans here is a wikipedia short article enumerating what can cause out of order:
- https://en.wikipedia.org/wiki/Out-of-order_delivery
That said, if you missed in the message from mailing list archive that you linked to,
there is a global parameter that should reduce the risk of sending out of order sip
packets to the minimum that can be done from SIP processing point of view based on
call-id, but there are still chances that on multi-cpu systems the packets read from the
network can get to be processed in different order because of how read on udp sockets is
done by kernel/libc and how cpu scheduler allocate cycles to running application
processes.
Cheers,
Daniel
On 05.12.22 19:34, Jawaid Bazyar wrote:
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users Hi,
I have experienced out of order packet processing when testing a simple Kamailio
config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
sr-users(a)lists.kamailio.org <https://www.mail-archive.com/<a
href=>/msg11786.html">https://www.mail-archive.com/sr-users@list…
The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP
packets are processed without regard to whether packets received before this one, are
currently being processed. This means any packets after the initial INVITE that require
more processing, can get reordered. In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order
packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC
does not see packets in the correct order, I never turns on the simulated audio. Calls
that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a
phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat. But, it is not
clear to me how this works – would this setting restrict the number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would greatly restrict
total platform scalability to a point where it is probably useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response
was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I
don’t think a proxy should reorder packets involved in a call in a non-deterministic way.
In the Kamailio list thread, Alex Balashov discusses the contortions he has to go through
to avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations. And a lot of
the feedback is “why are out of order packets a problem?” OK, sure, in the very specific
example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely
nothing preventing Kamailio from generating much more serious OOO scenarios that would
cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this out of order
processing issue will become.
The solution to this seems very simple and straightforward – put packets to be processed
into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
8:14 p.m.
I did get it that the complain was about Kamailio processing style, but
overall is about getting the SIP traffic from A to B. With or without
proxy in the middle, B has to be able to deal with packets our of order.
It can be a SIP proxy that changes the order or it can be IP routers on
different paths that lead to different order, for B is irrelevant. The
SIP proxy may send in the order of receiving, still B can get them out
of order.
It is like implementing a TCP stack and then report dysfunctionality to
ISPs/carriers when the IP packets come out of order because IP packets
went on different paths or IP routers made different routing decisions.
Daniel
On 06.12.22 14:10, Alex Balashov wrote:
I suspect that what Jawaid meant by "naive"
was that, in Kamailio, there is not a central distributor thread which aims to buffer,
serialise and otherwise order packets prior to distribution into worker processes or
threads, as is common in many other multiprocess systems.
I agree that "naive" is perhaps not the best choice of vocabulary for it, and
of course there would be performance downsides to such a centralised pipeline. But I think
that's what was probably meant.
-- Alex
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
On Dec 6, 2022, at 4:15 AM, Daniel-Constantin
Mierla <miconda(a)gmail.com> wrote:
Hello,
actually your expectation that packets should come in order is "naive", just
think about how internet is constructed and IP routing works. In the past it was easy to
reproduce on mobile networks scenarios when sending CANCEL very quickly after the INVITE
resulted in CANCEL arriving first at the proxy, then the INVITE.
Probably you don't get it in your lab environment where you have sipp on the same
system as the sip server or one network segment away, but over the internet the packets
can get in different order because of network transmission (different IP routing paths).
It is the reason in some cases there are jitter buffers and sequence numbers (e.g., in RTP
and SIP (CSeq)). In other words, the protocols like RTP or SIP were designed to deal with
these out-of-order packets.
Ans here is a wikipedia short article enumerating what can cause out of order:
- https://en.wikipedia.org/wiki/Out-of-order_delivery
That said, if you missed in the message from mailing list archive that you linked to,
there is a global parameter that should reduce the risk of sending out of order sip
packets to the minimum that can be done from SIP processing point of view based on
call-id, but there are still chances that on multi-cpu systems the packets read from the
network can get to be processed in different order because of how read on udp sockets is
done by kernel/libc and how cpu scheduler allocate cycles to running application
processes.
Cheers,
Daniel
On 05.12.22 19:34, Jawaid Bazyar wrote:
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users Hi,
I have experienced out of order packet processing when testing a simple Kamailio
config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html
The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP
packets are processed without regard to whether packets received before this one, are
currently being processed. This means any packets after the initial INVITE that require
more processing, can get reordered. In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order
packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC
does not see packets in the correct order, I never turns on the simulated audio. Calls
that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a
phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat. But, it is not
clear to me how this works – would this setting restrict the number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would greatly restrict
total platform scalability to a point where it is probably useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response
was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I
don’t think a proxy should reorder packets involved in a call in a non-deterministic way.
In the Kamailio list thread, Alex Balashov discusses the contortions he has to go through
to avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations. And a lot of
the feedback is “why are out of order packets a problem?” OK, sure, in the very specific
example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely
nothing preventing Kamailio from generating much more serious OOO scenarios that would
cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this out of order
processing issue will become.
The solution to this seems very simple and straightforward – put packets to be processed
into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
8:23 p.m.
Yeah, I agree with that.
And of course, my proposed in-script solution might generate some very undesirable TCP/TLS
bottlenecks.
On Dec 6, 2022, at 10:14 AM, Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
I did get it that the complain was about Kamailio processing style, but
overall is about getting the SIP traffic from A to B. With or without
proxy in the middle, B has to be able to deal with packets our of order.
It can be a SIP proxy that changes the order or it can be IP routers on
different paths that lead to different order, for B is irrelevant. The
SIP proxy may send in the order of receiving, still B can get them out
of order.
It is like implementing a TCP stack and then report dysfunctionality to
ISPs/carriers when the IP packets come out of order because IP packets
went on different paths or IP routers made different routing decisions.
Daniel
On 06.12.22 14:10, Alex Balashov wrote:
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
I suspect that what Jawaid meant by
"naive" was that, in Kamailio, there is not a central distributor thread which
aims to buffer, serialise and otherwise order packets prior to distribution into worker
processes or threads, as is common in many other multiprocess systems.
I agree that "naive" is perhaps not the best choice of vocabulary for it, and
of course there would be performance downsides to such a centralised pipeline. But I think
that's what was probably meant.
-- Alex
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
On Dec 6, 2022, at 4:15 AM, Daniel-Constantin
Mierla <miconda(a)gmail.com> wrote:
Hello,
actually your expectation that packets should come in order is "naive", just
think about how internet is constructed and IP routing works. In the past it was easy to
reproduce on mobile networks scenarios when sending CANCEL very quickly after the INVITE
resulted in CANCEL arriving first at the proxy, then the INVITE.
Probably you don't get it in your lab environment where you have sipp on the same
system as the sip server or one network segment away, but over the internet the packets
can get in different order because of network transmission (different IP routing paths).
It is the reason in some cases there are jitter buffers and sequence numbers (e.g., in RTP
and SIP (CSeq)). In other words, the protocols like RTP or SIP were designed to deal with
these out-of-order packets.
Ans here is a wikipedia short article enumerating what can cause out of order:
- https://en.wikipedia.org/wiki/Out-of-order_delivery
That said, if you missed in the message from mailing list archive that you linked to,
there is a global parameter that should reduce the risk of sending out of order sip
packets to the minimum that can be done from SIP processing point of view based on
call-id, but there are still chances that on multi-cpu systems the packets read from the
network can get to be processed in different order because of how read on udp sockets is
done by kernel/libc and how cpu scheduler allocate cycles to running application
processes.
Cheers,
Daniel
On 05.12.22 19:34, Jawaid Bazyar wrote:
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users Hi,
I have experienced out of order packet processing when testing a simple Kamailio config.
The configuration is as follows, basically:
request_route{
record_route();
enum_query();
xlog("INVITE ENUM query - To URI $tU");
forward();
}
I saw this thread from 2020:
https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html
The issue seems to be that kamailio process scheduling is naïve – i.e., incoming SIP
packets are processed without regard to whether packets received before this one, are
currently being processed. This means any packets after the initial INVITE that require
more processing, can get reordered. In my test lab I have:
SIPp – UAC
Kamailio Proxy
SIPp – UAS
The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS.
Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of order packets.
It is only when I introduce Kamailio in the middle that I get OOO packets.
See the images attached: uac-to-proxy, proxy, and proxy-to-uas.
In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” – because UAC
does not see packets in the correct order, I never turns on the simulated audio. Calls
that have in-order dialogs, work fine, and SIPP UAC “pauses” 10 seconds to simulate a
phone call.
So far, the error rate runs from 1/1000 to around 1/200 – pretty bad.
In the thread, a few things were suggested.
Have fewer kamailio processes than cores:
Did not resolve issue.
Try route_locks_size = 256
Did not resolve issue. Though, it did alter it somewhat. But, it is not
clear to me how this works – would this setting restrict the number of open calls to 256?
Have only one kamailio process: (“children=1”)
This works. “Works”, I should say, because this would greatly restrict
total platform scalability to a point where it is probably useless for my application.
I saw the same issue discussed in the OpenSIPS mailing list from 2010, and the response
was “this is not a bug”.
Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS a bug. I don’t
think a proxy should reorder packets involved in a call in a non-deterministic way. In the
Kamailio list thread, Alex Balashov discusses the contortions he has to go through to
avoid repercussions from this issue.
Kamailio as-is probably works fine for relatively low-volume operations. And a lot of the
feedback is “why are out of order packets a problem?” OK, sure, in the very specific
example given in the 2020 thread, maybe who cares. But in my thinking, there is absolutely
nothing preventing Kamailio from generating much more serious OOO scenarios that would
cause calls to fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”.
Who knows how other SIP stacks will behave?
But the more one will try to scale Kamailio, the more significantly this out of order
processing issue will become.
The solution to this seems very simple and straightforward – put packets to be processed
into a queue PER Call-ID, or something along those lines.
In short, the parallelism should be by call, not by packet.
What say ye? Have I misunderstood something here?
Cheers,
Jawaid
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
518
days inactive
519
days old
11 comments
4 participants
participants (4)
-
Alex Balashov -
Daniel-Constantin Mierla -
Jawaid Bazyar -
Sergey Safarov