[SR-Users] Kamailio propagates 180 and 200 OK OUT OF ORDER

Daniel-Constantin Mierla miconda at gmail.com
Thu Apr 9 18:17:39 CEST 2020


Hello,

then the overtaking is in between reading from the socket and getting to
parsing the call-id value -- the cpu is lost by first reader after
recvfrom() and the second process get enough cpu time to go ahead
further. I haven't encountered this case, but as I said previously, it
is very unlikely, but still possible. I added the route_locks_size
because in the past I had cases when processing of some messages took
longer executing config (e.g., due to authentication, accounting, ..)
and I needed to be sure they are processed in the order they enter
config execution.

Then the option is to see if a single process with stateless sending out
(using forward()) gives the capacity, if you don't do any other complex
processing. Or if you do more complex processing, use a dispatcher
process with forwarding to local host or in a similar manner try to use
mqueue+rtimer for dispatching using shared memory queues.

Of course, it is open source and there is also the C coding way, to add
a synchronizing mechanism to protect against parallel execution of the
code from recvfrom() till call-id lock is acquired.

Cheers,
Daniel

On 09.04.20 17:37, Luis Rojas G. wrote:
> Hello,
>
> Well, it did not work at all. Exactly same behavior, with random out
> of order messages.
>
> Best regards,
>
> Luis
>
> On 4/9/20 11:28 AM, Daniel-Constantin Mierla wrote:
>>
>> 	
>> MICONDA at GMAIL.COM appears similar to someone who previously sent you
>> email, but may not be that person. Learn why this could be a risk
>> <http://aka.ms/LearnAboutSenderIdentification>
>> 	Feedback <http://aka.ms/SafetyTipsFeedback>
>>
>> Hello,
>>
>> the sip messages belonging to the same dialog have the same value for
>> Call-Id header. The locking is done based on hashing the Call-Id, so
>> it doesn't need at all the dialog module for its purpose.
>>
>> Cheers,
>> Daniel
>>
>> On 09.04.20 14:19, Luis Rojas G. wrote:
>>> Hello, Daniel,
>>>
>>> I am not so sure. I first tried adding that parameter, but it did
>>> not work at all.  Same behavior. Then I read the documentation more
>>> carefully :
>>>
>>> https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size
>>>
>>>
>>>       route_locks_size
>>>
>>> Set the number of mutex locks to be used for synchronizing the
>>> execution of messages sharing the same Call-Id. In other words,
>>> enables Kamailio to execute the config script sequentially for the
>>> requests and replies received *within the same dialog* – a new
>>> message received *within the same dialog* waits until the previous
>>> one is routed out.
>>>
>>> Locks to execute sequentially messages belonging to same dialog. How
>>> will Kamailio be aware that messages belong to same dialog, without
>>> the dialog module?. With just stateless proxy it has no idea about
>>> dialogs, it just forwards messages. I guess that's why just adding
>>> that parameter did not work.
>>>
>>> Am I wrong?
>>>
>>> Luis
>>>
>>>
>>>
>>> On 4/9/20 3:47 AM, Daniel-Constantin Mierla wrote:
>>>>
>>>> Hello,
>>>>
>>>> On 08.04.20 23:03, Luis Rojas G. wrote:
>>>>> Hello, Daniel,
>>>>>
>>>>> I looked into that parameter, but  I need to use with the dialog
>>>>> module, and I'm pretty afraid to use that.
>>>>
>>>> who said or where is written than you need to load the dialog
>>>> module? You definitely don't.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>>> I was looking more into the stateless proxy, because I need to
>>>>> process a lot of traffic.
>>>>>
>>>>> My target is 4200CAPS. with duration between 90s and 210. Let's
>>>>> say, 150 seconds. That would mean 630.000 simultaneous dialogs. I
>>>>> don't think the solution can go that way.
>>>>>
>>>>> it would really help me to be able to use completely stateless
>>>>> proxy plus Async in reply_route(), to introduce an artificial
>>>>> delay before forwarding 200 OK to Invite.. As someone mentioned,
>>>>> it would help me on request_route(), for race conditions between
>>>>> ACK and Re-Invite.
>>>>>
>>>>> Any idea why Async is not allowed in reply_route()?
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Luis
>>>>>
>>>>>
>>>>> On 4/8/20 1:07 PM, Daniel-Constantin Mierla wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> you have to keep in mind that Kamailio is a SIP packet router,
>>>>>> not a telephony engine. If 180 and 200 replies are part of a call
>>>>>> is not something that Kamailio recognize at its core. Its main
>>>>>> goal is to route out as fast as possible what is received, by
>>>>>> executing the configuration file script. Now, a matter of your
>>>>>> configuration file, processing of some SIP messages can take
>>>>>> longer than processing other. And the processing is done in
>>>>>> parallel, a matter of children parameter (and tcp_children,
>>>>>> sctp_children).
>>>>>>
>>>>>> With that in mind, a way to try to cope better with the issue you
>>>>>> face is to set route_locks_size parameter, see:
>>>>>>
>>>>>>   *
>>>>>> https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size
>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.kamailio.org%2Fwiki%2Fcookbooks%2Fdevel%2Fcore%23route_locks_size&data=02%7C01%7C%7C8eb82088a4b243f9d11308d7dc9a99a4%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220428894461050&sdata=b0xkSt%2FrLOGMd%2F0PTZCSzarfWW8jc5%2FABbIrNTaappc%3D&reserved=0>
>>>>>>
>>>>>> Probably is what you look for.
>>>>>>
>>>>>> But if you want more tight constraints, like when receiving a 180
>>>>>> after a 200ok and not route it out, you have to make the logic in
>>>>>> configuration file by combining modules such as dialog or htable
>>>>>> (as already suggested).
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>>
>>>>>> On 08.04.20 16:04, Luis Rojas G. wrote:
>>>>>>> Hi, Henning,
>>>>>>>
>>>>>>> No need to be ironic. As I mentioned on my first post, I tried
>>>>>>> stateful proxy and I observed the same behavior.
>>>>>>>
>>>>>>> /"I tried using stateful proxy and I obtained the same result."/
>>>>>>>
>>>>>>> The asynchronous sleep seems promising. I will look into it.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Luis
>>>>>>>
>>>>>>>
>>>>>>> On 4/8/20 9:30 AM, Henning Westerholt wrote:
>>>>>>>>
>>>>>>>> Hi Luis,
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> I see. Well, you want to use Kamailio as a stateless proxy, on
>>>>>>>> the other hand it should do things that are inherently stateful. 😉
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> As mentioned, have a look to the dialog module to track the
>>>>>>>> state of dialogs that you process. This will not work in a
>>>>>>>> stateless mode, though.
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> You can also use the htable module to just store some data
>>>>>>>> about the processed messages in a shared memory table and use
>>>>>>>> this to enforce your ordering. There is also the option to do
>>>>>>>> an asynchronous sleep (with the async) module on the message
>>>>>>>> that you want to delay but still processing other messages
>>>>>>>> during it.
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> Henning
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> -- 
>>>>>>>>
>>>>>>>> Henning Westerholt – https://skalatan.de/blog/
>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskalatan.de%2Fblog%2F&data=02%7C01%7C%7C8eb82088a4b243f9d11308d7dc9a99a4%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220428894471045&sdata=%2FPz8nYYTRYQUs8P7ssC1VcMCv6DWD6V9fe6IoDF4Bmo%3D&reserved=0>
>>>>>>>>
>>>>>>>> Kamailio services – https://gilawa.com
>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgilawa.com%2F&data=02%7C01%7C%7C8eb82088a4b243f9d11308d7dc9a99a4%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220428894481042&sdata=v6WrHhIxQNEPijRXbqZTB%2Fh%2BFdtrDFsPVn9qDGFUsB4%3D&reserved=0>
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> *From:* Luis Rojas G. <luis.rojas at sixbell.com>
>>>>>>>> *Sent:* Wednesday, April 8, 2020 3:00 PM
>>>>>>>> *To:* Henning Westerholt <hw at skalatan.de>; Kamailio (SER) -
>>>>>>>> Users Mailing List <sr-users at lists.kamailio.org>
>>>>>>>> *Subject:* Re: [SR-Users] Kamailio propagates 180 and 200 OK
>>>>>>>> OUT OF ORDER
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> Hello, Henning,
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> I am worried about this scenario, because it's a symptom of
>>>>>>>> what may happen in other cases. For instance, I've seen that
>>>>>>>> this operator usually sends re-invites immediate after sending
>>>>>>>> ACK.   This may create race conditions like 3.1.5 of RFC5407
>>>>>>>>
>>>>>>>> https://tools.ietf.org/html/rfc5407#page-22
>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc5407%23page-22&data=02%7C01%7C%7C8eb82088a4b243f9d11308d7dc9a99a4%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220428894481042&sdata=us7z8AaVoVsqprOMW97D6PqF5Bl6C3oBu%2Fp7pCXt7%2Bo%3D&reserved=0>
>>>>>>>>
>>>>>>>> I'd understand that one happens because of packet loss, as it's
>>>>>>>> in UDP's nature, but in this case it would be artificially
>>>>>>>> created by Kamailio. if there was no problem at network level
>>>>>>>> (packet loss, packets following different path on the network
>>>>>>>> and arriving out of order), why Kamailio creates it?
>>>>>>>>
>>>>>>>> I'd expect that the shared memory is used precisely for this.
>>>>>>>> If an instance of kamailio receives a 200 OK, it could check on
>>>>>>>> the shm and say "hey, another instance is processing a 180 for
>>>>>>>> this call. Let's wait for it to finish" (*). I know there could
>>>>>>>> still be a problem, the instance processing the 180 undergoes a
>>>>>>>> context switch just after it receives the message, but before
>>>>>>>> writing to shm, but it would greatly reduce the chance.
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> In our applications we use a SIP stack that always sends
>>>>>>>> messages to the application in the same order it receives them,
>>>>>>>> even though is multi-threaded and messages from the network are
>>>>>>>> received by different threads. So, they really syncronize
>>>>>>>> between them. Why Kamailio instances don't?
>>>>>>>>
>>>>>>>> I am evaluating kamailio to use it as a dispatcher to balance
>>>>>>>> load against our several Application Servers, to present to the
>>>>>>>> operator just a couple of entrance points to our platform (they
>>>>>>>> don't want to establish connections to each one of our
>>>>>>>> servers). This operator is very difficult to deal with. I am
>>>>>>>> sure they will complain something like "why are you sending
>>>>>>>> messages out of order? Fix that". The operator will be able to
>>>>>>>> see traces and check that messages entered the Kamailio nodes
>>>>>>>> in order and left out of order. They will not accept it.
>>>>>>>>
>>>>>>>> (*) Not really "wait", as it would introduce a delay in
>>>>>>>> processing all messages. it should be like putting it on a
>>>>>>>> queue, continue processing other messages, and go back to the
>>>>>>>> queue later.
>>>>>>>>
>>>>>>>> Well, thanks for your answer.
>>>>>>>>
>>>>>>>> Luis
>>>>>>>>
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/8/20 3:01 AM, Henning Westerholt wrote:
>>>>>>>>
>>>>>>>>     Hello Luis,
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     as the 1xx responses are usually send unreliable (unless
>>>>>>>>     you use PRACK), you should not make any assumption on the
>>>>>>>>     order or even the arrival of this messages. It can also
>>>>>>>>     happens on a network level, if send by UDP.
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     Can you elaborate why you think this re-ordering is a
>>>>>>>>     problem for you?
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     One idea to enforce some ordering would be to use the
>>>>>>>>     dialog module in combination with reply routes and the
>>>>>>>>     textops(x)  module.
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     About the shared memory question – Kamailio implement its
>>>>>>>>     own memory manager (private memory and shared memory pool).
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     Cheers,
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     Henning
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     -- 
>>>>>>>>
>>>>>>>>     Henning Westerholt – https://skalatan.de/blog/
>>>>>>>>     <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskalatan.de%2Fblog%2F&data=02%7C01%7C%7C8eb82088a4b243f9d11308d7dc9a99a4%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220428894491034&sdata=xGN%2BtLb0lh84qCzNUUZAdqPqW14q7eE03W881jYxXDQ%3D&reserved=0>
>>>>>>>>
>>>>>>>>     Kamailio services – https://gilawa.com
>>>>>>>>     <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgilawa.com%2F&data=02%7C01%7C%7C8eb82088a4b243f9d11308d7dc9a99a4%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220428894501031&sdata=xSoqNmnm%2Bfwv9AP30GyY6BIFLLeNGXPhuJNuStpjBzU%3D&reserved=0>
>>>>>>>>
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     *From:* sr-users <sr-users-bounces at lists.kamailio.org>
>>>>>>>>     <mailto:sr-users-bounces at lists.kamailio.org> *On Behalf Of
>>>>>>>>     *Luis Rojas G.
>>>>>>>>     *Sent:* Tuesday, April 7, 2020 10:43 PM
>>>>>>>>     *To:* sr-users at lists.kamailio.org
>>>>>>>>     <mailto:sr-users at lists.kamailio.org>
>>>>>>>>     *Subject:* [SR-Users] Kamailio propagates 180 and 200 OK
>>>>>>>>     OUT OF ORDER
>>>>>>>>
>>>>>>>>      
>>>>>>>>
>>>>>>>>     Good day,
>>>>>>>>
>>>>>>>>     I am testing the dispatcher module, using Kamailio as
>>>>>>>>     stateless proxy. I have a pool of UAC (scripts in SIPP) and
>>>>>>>>     a pool of UAS (also scripts in SIPP) for the destinations.
>>>>>>>>     Kamailio version is kamailio-5.3.3-4.1.x86_64.
>>>>>>>>
>>>>>>>>     Problem I have is, if UAS responds 180 and 200 OK to Invite
>>>>>>>>     immediately, sometimes they are propagated out of order.
>>>>>>>>     200 OK before 180, like this :
>>>>>>>>
>>>>>>>>     UAS is 172.30.4.195:5061. UAC is 172.30.4.195:5080.
>>>>>>>>     Kamailio is 192.168.253.4:5070
>>>>>>>>
>>>>>>>>     Difference between 180 and 200 is just about 50 microseconds.
>>>>>>>>
>>>>>>>>     My guess is that both messages are received by different
>>>>>>>>     instances of Kamailio, and then because of context
>>>>>>>>     switches, even though the 180 is received before, that
>>>>>>>>     process ends after the processing of 200. However, I had
>>>>>>>>     the idea that in order to avoid these problems the kamailio
>>>>>>>>     processes synchronized with each other using a shared
>>>>>>>>     memory. I tried using stateful proxy and I obtained the
>>>>>>>>     same result.
>>>>>>>>
>>>>>>>>     By the way, anyone has any idea about how Kamailio's share
>>>>>>>>     memory is implemented? It clearly does not use the typical
>>>>>>>>     system calls shmget(), shmat(), because they are not shown
>>>>>>>>     by ipcs command.
>>>>>>>>
>>>>>>>>     Before posting here I googled, but I couldn't find anything
>>>>>>>>     related to this. I can't believe I am the only one who ever
>>>>>>>>     had this problem, so I guess I am doing something wrong...
>>>>>>>>
>>>>>>>>     Please, any help. I'm really stuck on this.
>>>>>>>>
>>>>>>>>     Thanks.
>>>>>>>>
>>>>>>>>     -- 
>>>>>>>>
>>>>>>>>  
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Luis Rojas
>>>>>>>> Software Architect
>>>>>>>> Sixbell
>>>>>>>> Los Leones 1200
>>>>>>>> Providencia
>>>>>>>> Santiago, Chile
>>>>>>>> Phone: (+56-2) 22001288
>>>>>>>> mailto:luis.rojas at sixbell.com
>>>>>>>> http://www.sixbell.com <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sixbell.com%2F&data=02%7C01%7C%7C8eb82088a4b243f9d11308d7dc9a99a4%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220428894501031&sdata=4V5xZCU4szWuyvrARjPBaaToAmm8eowL7GhYPthkCjQ%3D&reserved=0>
>>>>>>>
>>>>>>>
>>>>>>> -- 
>>>>>>> Luis Rojas
>>>>>>> Software Architect
>>>>>>> Sixbell
>>>>>>> Los Leones 1200
>>>>>>> Providencia
>>>>>>> Santiago, Chile
>>>>>>> Phone: (+56-2) 22001288
>>>>>>> mailto:luis.rojas at sixbell.com
>>>>>>> http://www.sixbell.com
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Kamailio (SER) - Users Mailing List
>>>>>>> sr-users at lists.kamailio.org
>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>> -- 
>>>>>> Daniel-Constantin Mierla -- www.asipto.com
>>>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>>>
>>>>>
>>>>> -- 
>>>>> Luis Rojas
>>>>> Software Architect
>>>>> Sixbell
>>>>> Los Leones 1200
>>>>> Providencia
>>>>> Santiago, Chile
>>>>> Phone: (+56-2) 22001288
>>>>> mailto:luis.rojas at sixbell.com
>>>>> http://www.sixbell.com
>>>> -- 
>>>> Daniel-Constantin Mierla -- www.asipto.com
>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>
>>>
>>> -- 
>>> Luis Rojas
>>> Software Architect
>>> Sixbell
>>> Los Leones 1200
>>> Providencia
>>> Santiago, Chile
>>> Phone: (+56-2) 22001288
>>> mailto:luis.rojas at sixbell.com
>>> http://www.sixbell.com
>> -- 
>> Daniel-Constantin Mierla -- www.asipto.com
>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>
>
> -- 
> Luis Rojas
> Software Architect
> Sixbell
> Los Leones 1200
> Providencia
> Santiago, Chile
> Phone: (+56-2) 22001288
> mailto:luis.rojas at sixbell.com
> http://www.sixbell.com

-- 
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200409/de042b63/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 49792 bytes
Desc: not available
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200409/de042b63/attachment.png>


More information about the sr-users mailing list