[SR-Users] Kamailio propagates 180 and 200 OK OUT OF ORDER
Luis Rojas G.
luis.rojas at sixbell.com
Thu Apr 9 19:48:49 CEST 2020
Hello,
I have a lot of experience developing mutithreaded applications, and I
don't see it so unlikely at all that a process loses cpu just after
recvfrom(). It's just as probable as to lose it just before, or when
writing on a cache or just before of after sendto(). If there are many
messages going through, some of them will fall in this scenario. if I
try sending a burst of 100 messages, I see two or three presenting the
scenario.
Just forward() with a single process does not give the capacity. I'm
getting almost 1000caps. More than that and start getting errores,
retransmissions, etc. And this is just one way. I need to receive the
call to go back to the network (our application is a B2BUA), so I will
be down to 500caps, with a simple scenario, with no reliable responses,
reinvites, updates, etc. I will end up having as many standalone
kamailio processes as the current servers I do have now.
I really think the simplest way would be to add a small delay to 200 OK.
Very small, like 10ms, should be enough. Simple and it should work. As
Alex Balashov commented he did for the case with ACK-Re-Invite.
I have to figure out how to make async_ms_sleep() work in reply_route().
Thanks for all the comments and ideas
Best regards,
Luis
. On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote:
>
>
> MICONDA at GMAIL.COM appears similar to someone who previously sent you
> email, but may not be that person. Learn why this could be a risk
> <http://aka.ms/LearnAboutSenderIdentification>
> Feedback <http://aka.ms/SafetyTipsFeedback>
>
> Hello,
>
> then the overtaking is in between reading from the socket and getting
> to parsing the call-id value -- the cpu is lost by first reader after
> recvfrom() and the second process get enough cpu time to go ahead
> further. I haven't encountered this case, but as I said previously, it
> is very unlikely, but still possible. I added the route_locks_size
> because in the past I had cases when processing of some messages took
> longer executing config (e.g., due to authentication, accounting, ..)
> and I needed to be sure they are processed in the order they enter
> config execution.
>
> Then the option is to see if a single process with stateless sending
> out (using forward()) gives the capacity, if you don't do any other
> complex processing. Or if you do more complex processing, use a
> dispatcher process with forwarding to local host or in a similar
> manner try to use mqueue+rtimer for dispatching using shared memory
> queues.
>
> Of course, it is open source and there is also the C coding way, to
> add a synchronizing mechanism to protect against parallel execution of
> the code from recvfrom() till call-id lock is acquired.
>
> Cheers,
> Daniel
>
> On 09.04.20 17:37, Luis Rojas G. wrote:
>> Hello,
>>
>> Well, it did not work at all. Exactly same behavior, with random out
>> of order messages.
>>
>> Best regards,
>>
>> Luis
>>
>> On 4/9/20 11:28 AM, Daniel-Constantin Mierla wrote:
>>>
>>>
>>> MICONDA at GMAIL.COM appears similar to someone who previously sent you
>>> email, but may not be that person. Learn why this could be a risk
>>> <http://aka.ms/LearnAboutSenderIdentification>
>>> Feedback <http://aka.ms/SafetyTipsFeedback>
>>>
>>> Hello,
>>>
>>> the sip messages belonging to the same dialog have the same value
>>> for Call-Id header. The locking is done based on hashing the
>>> Call-Id, so it doesn't need at all the dialog module for its purpose.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 09.04.20 14:19, Luis Rojas G. wrote:
>>>> Hello, Daniel,
>>>>
>>>> I am not so sure. I first tried adding that parameter, but it did
>>>> not work at all. Same behavior. Then I read the documentation more
>>>> carefully :
>>>>
>>>> https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size
>>>>
>>>>
>>>> route_locks_size
>>>>
>>>> Set the number of mutex locks to be used for synchronizing the
>>>> execution of messages sharing the same Call-Id. In other words,
>>>> enables Kamailio to execute the config script sequentially for the
>>>> requests and replies received *within the same dialog* – a new
>>>> message received *within the same dialog* waits until the previous
>>>> one is routed out.
>>>>
>>>> Locks to execute sequentially messages belonging to same dialog.
>>>> How will Kamailio be aware that messages belong to same dialog,
>>>> without the dialog module?. With just stateless proxy it has no
>>>> idea about dialogs, it just forwards messages. I guess that's why
>>>> just adding that parameter did not work.
>>>>
>>>> Am I wrong?
>>>>
>>>> Luis
>>>>
>>>>
>>>>
>>>> On 4/9/20 3:47 AM, Daniel-Constantin Mierla wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> On 08.04.20 23:03, Luis Rojas G. wrote:
>>>>>> Hello, Daniel,
>>>>>>
>>>>>> I looked into that parameter, but I need to use with the dialog
>>>>>> module, and I'm pretty afraid to use that.
>>>>>
>>>>> who said or where is written than you need to load the dialog
>>>>> module? You definitely don't.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>>
>>>>>> I was looking more into the stateless proxy, because I need to
>>>>>> process a lot of traffic.
>>>>>>
>>>>>> My target is 4200CAPS. with duration between 90s and 210. Let's
>>>>>> say, 150 seconds. That would mean 630.000 simultaneous dialogs. I
>>>>>> don't think the solution can go that way.
>>>>>>
>>>>>> it would really help me to be able to use completely stateless
>>>>>> proxy plus Async in reply_route(), to introduce an artificial
>>>>>> delay before forwarding 200 OK to Invite.. As someone mentioned,
>>>>>> it would help me on request_route(), for race conditions between
>>>>>> ACK and Re-Invite.
>>>>>>
>>>>>> Any idea why Async is not allowed in reply_route()?
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Luis
>>>>>>
>>>>>>
>>>>>> On 4/8/20 1:07 PM, Daniel-Constantin Mierla wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> you have to keep in mind that Kamailio is a SIP packet router,
>>>>>>> not a telephony engine. If 180 and 200 replies are part of a
>>>>>>> call is not something that Kamailio recognize at its core. Its
>>>>>>> main goal is to route out as fast as possible what is received,
>>>>>>> by executing the configuration file script. Now, a matter of
>>>>>>> your configuration file, processing of some SIP messages can
>>>>>>> take longer than processing other. And the processing is done in
>>>>>>> parallel, a matter of children parameter (and tcp_children,
>>>>>>> sctp_children).
>>>>>>>
>>>>>>> With that in mind, a way to try to cope better with the issue
>>>>>>> you face is to set route_locks_size parameter, see:
>>>>>>>
>>>>>>> *
>>>>>>> https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size
>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.kamailio.org%2Fwiki%2Fcookbooks%2Fdevel%2Fcore%23route_locks_size&data=02%7C01%7C%7C2be8e9d786c6497d159b08d7dca18936%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220458688617500&sdata=6%2FMFcu9NfJzxmNqGRFppJjaSlm2iSYGhXdKargmv6BM%3D&reserved=0>
>>>>>>>
>>>>>>> Probably is what you look for.
>>>>>>>
>>>>>>> But if you want more tight constraints, like when receiving a
>>>>>>> 180 after a 200ok and not route it out, you have to make the
>>>>>>> logic in configuration file by combining modules such as dialog
>>>>>>> or htable (as already suggested).
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>> On 08.04.20 16:04, Luis Rojas G. wrote:
>>>>>>>> Hi, Henning,
>>>>>>>>
>>>>>>>> No need to be ironic. As I mentioned on my first post, I tried
>>>>>>>> stateful proxy and I observed the same behavior.
>>>>>>>>
>>>>>>>> /"I tried using stateful proxy and I obtained the same result."/
>>>>>>>>
>>>>>>>> The asynchronous sleep seems promising. I will look into it.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Luis
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/8/20 9:30 AM, Henning Westerholt wrote:
>>>>>>>>>
>>>>>>>>> Hi Luis,
>>>>>>>>>
>>>>>>>>> I see. Well, you want to use Kamailio as a stateless proxy, on
>>>>>>>>> the other hand it should do things that are inherently
>>>>>>>>> stateful. 😉
>>>>>>>>>
>>>>>>>>> As mentioned, have a look to the dialog module to track the
>>>>>>>>> state of dialogs that you process. This will not work in a
>>>>>>>>> stateless mode, though.
>>>>>>>>>
>>>>>>>>> You can also use the htable module to just store some data
>>>>>>>>> about the processed messages in a shared memory table and use
>>>>>>>>> this to enforce your ordering. There is also the option to do
>>>>>>>>> an asynchronous sleep (with the async) module on the message
>>>>>>>>> that you want to delay but still processing other messages
>>>>>>>>> during it.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Henning
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Henning Westerholt – https://skalatan.de/blog/
>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskalatan.de%2Fblog%2F&data=02%7C01%7C%7C2be8e9d786c6497d159b08d7dca18936%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220458688617500&sdata=R7VybNsI08Eb61gYPn80iHeHkrIx9wGqcoW%2B4yy%2BGHo%3D&reserved=0>
>>>>>>>>>
>>>>>>>>> Kamailio services – https://gilawa.com
>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgilawa.com%2F&data=02%7C01%7C%7C2be8e9d786c6497d159b08d7dca18936%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220458688627448&sdata=j51s%2BxSqhe%2BeM3syV0wda50nkON0FsEuLoKugcqOu3Q%3D&reserved=0>
>>>>>>>>>
>>>>>>>>> *From:* Luis Rojas G. <luis.rojas at sixbell.com>
>>>>>>>>> *Sent:* Wednesday, April 8, 2020 3:00 PM
>>>>>>>>> *To:* Henning Westerholt <hw at skalatan.de>; Kamailio (SER) -
>>>>>>>>> Users Mailing List <sr-users at lists.kamailio.org>
>>>>>>>>> *Subject:* Re: [SR-Users] Kamailio propagates 180 and 200 OK
>>>>>>>>> OUT OF ORDER
>>>>>>>>>
>>>>>>>>> Hello, Henning,
>>>>>>>>>
>>>>>>>>> I am worried about this scenario, because it's a symptom of
>>>>>>>>> what may happen in other cases. For instance, I've seen that
>>>>>>>>> this operator usually sends re-invites immediate after sending
>>>>>>>>> ACK. This may create race conditions like 3.1.5 of RFC5407
>>>>>>>>>
>>>>>>>>> https://tools.ietf.org/html/rfc5407#page-22
>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc5407%23page-22&data=02%7C01%7C%7C2be8e9d786c6497d159b08d7dca18936%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220458688627448&sdata=QYLEaLP0l36cF9jntgvyrilRakwpWXlPw7kdIMlG%2BpY%3D&reserved=0>
>>>>>>>>>
>>>>>>>>> I'd understand that one happens because of packet loss, as
>>>>>>>>> it's in UDP's nature, but in this case it would be
>>>>>>>>> artificially created by Kamailio. if there was no problem at
>>>>>>>>> network level (packet loss, packets following different path
>>>>>>>>> on the network and arriving out of order), why Kamailio
>>>>>>>>> creates it?
>>>>>>>>>
>>>>>>>>> I'd expect that the shared memory is used precisely for this.
>>>>>>>>> If an instance of kamailio receives a 200 OK, it could check
>>>>>>>>> on the shm and say "hey, another instance is processing a 180
>>>>>>>>> for this call. Let's wait for it to finish" (*). I know there
>>>>>>>>> could still be a problem, the instance processing the 180
>>>>>>>>> undergoes a context switch just after it receives the message,
>>>>>>>>> but before writing to shm, but it would greatly reduce the chance.
>>>>>>>>>
>>>>>>>>> In our applications we use a SIP stack that always sends
>>>>>>>>> messages to the application in the same order it receives
>>>>>>>>> them, even though is multi-threaded and messages from the
>>>>>>>>> network are received by different threads. So, they really
>>>>>>>>> syncronize between them. Why Kamailio instances don't?
>>>>>>>>>
>>>>>>>>> I am evaluating kamailio to use it as a dispatcher to balance
>>>>>>>>> load against our several Application Servers, to present to
>>>>>>>>> the operator just a couple of entrance points to our platform
>>>>>>>>> (they don't want to establish connections to each one of our
>>>>>>>>> servers). This operator is very difficult to deal with. I am
>>>>>>>>> sure they will complain something like "why are you sending
>>>>>>>>> messages out of order? Fix that". The operator will be able to
>>>>>>>>> see traces and check that messages entered the Kamailio nodes
>>>>>>>>> in order and left out of order. They will not accept it.
>>>>>>>>>
>>>>>>>>> (*) Not really "wait", as it would introduce a delay in
>>>>>>>>> processing all messages. it should be like putting it on a
>>>>>>>>> queue, continue processing other messages, and go back to the
>>>>>>>>> queue later.
>>>>>>>>>
>>>>>>>>> Well, thanks for your answer.
>>>>>>>>>
>>>>>>>>> Luis
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 4/8/20 3:01 AM, Henning Westerholt wrote:
>>>>>>>>>
>>>>>>>>> Hello Luis,
>>>>>>>>>
>>>>>>>>> as the 1xx responses are usually send unreliable (unless
>>>>>>>>> you use PRACK), you should not make any assumption on the
>>>>>>>>> order or even the arrival of this messages. It can also
>>>>>>>>> happens on a network level, if send by UDP.
>>>>>>>>>
>>>>>>>>> Can you elaborate why you think this re-ordering is a
>>>>>>>>> problem for you?
>>>>>>>>>
>>>>>>>>> One idea to enforce some ordering would be to use the
>>>>>>>>> dialog module in combination with reply routes and the
>>>>>>>>> textops(x) module.
>>>>>>>>>
>>>>>>>>> About the shared memory question – Kamailio implement its
>>>>>>>>> own memory manager (private memory and shared memory pool).
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Henning
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Henning Westerholt – https://skalatan.de/blog/
>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskalatan.de%2Fblog%2F&data=02%7C01%7C%7C2be8e9d786c6497d159b08d7dca18936%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220458688637406&sdata=68oasuEqgZUreKO2Fl3LTEuGedrc5n6lqS4ECPJbgzQ%3D&reserved=0>
>>>>>>>>>
>>>>>>>>> Kamailio services – https://gilawa.com
>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgilawa.com%2F&data=02%7C01%7C%7C2be8e9d786c6497d159b08d7dca18936%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220458688637406&sdata=6dnbrKgWIlvwDKZycwoDTyjMAlYAGJy1TL3QqSbmHDw%3D&reserved=0>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *From:* sr-users <sr-users-bounces at lists.kamailio.org>
>>>>>>>>> <mailto:sr-users-bounces at lists.kamailio.org> *On Behalf Of
>>>>>>>>> *Luis Rojas G.
>>>>>>>>> *Sent:* Tuesday, April 7, 2020 10:43 PM
>>>>>>>>> *To:* sr-users at lists.kamailio.org
>>>>>>>>> <mailto:sr-users at lists.kamailio.org>
>>>>>>>>> *Subject:* [SR-Users] Kamailio propagates 180 and 200 OK
>>>>>>>>> OUT OF ORDER
>>>>>>>>>
>>>>>>>>> Good day,
>>>>>>>>>
>>>>>>>>> I am testing the dispatcher module, using Kamailio as
>>>>>>>>> stateless proxy. I have a pool of UAC (scripts in SIPP)
>>>>>>>>> and a pool of UAS (also scripts in SIPP) for the
>>>>>>>>> destinations. Kamailio version is kamailio-5.3.3-4.1.x86_64.
>>>>>>>>>
>>>>>>>>> Problem I have is, if UAS responds 180 and 200 OK to
>>>>>>>>> Invite immediately, sometimes they are propagated out of
>>>>>>>>> order. 200 OK before 180, like this :
>>>>>>>>>
>>>>>>>>> UAS is 172.30.4.195:5061. UAC is 172.30.4.195:5080.
>>>>>>>>> Kamailio is 192.168.253.4:5070
>>>>>>>>>
>>>>>>>>> Difference between 180 and 200 is just about 50 microseconds.
>>>>>>>>>
>>>>>>>>> My guess is that both messages are received by different
>>>>>>>>> instances of Kamailio, and then because of context
>>>>>>>>> switches, even though the 180 is received before, that
>>>>>>>>> process ends after the processing of 200. However, I had
>>>>>>>>> the idea that in order to avoid these problems the
>>>>>>>>> kamailio processes synchronized with each other using a
>>>>>>>>> shared memory. I tried using stateful proxy and I obtained
>>>>>>>>> the same result.
>>>>>>>>>
>>>>>>>>> By the way, anyone has any idea about how Kamailio's share
>>>>>>>>> memory is implemented? It clearly does not use the typical
>>>>>>>>> system calls shmget(), shmat(), because they are not shown
>>>>>>>>> by ipcs command.
>>>>>>>>>
>>>>>>>>> Before posting here I googled, but I couldn't find
>>>>>>>>> anything related to this. I can't believe I am the only
>>>>>>>>> one who ever had this problem, so I guess I am doing
>>>>>>>>> something wrong...
>>>>>>>>>
>>>>>>>>> Please, any help. I'm really stuck on this.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Luis Rojas
>>>>>>>>> Software Architect
>>>>>>>>> Sixbell
>>>>>>>>> Los Leones 1200
>>>>>>>>> Providencia
>>>>>>>>> Santiago, Chile
>>>>>>>>> Phone: (+56-2) 22001288
>>>>>>>>> mailto:luis.rojas at sixbell.com
>>>>>>>>> http://www.sixbell.com <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sixbell.com%2F&data=02%7C01%7C%7C2be8e9d786c6497d159b08d7dca18936%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220458688637406&sdata=WZUsCO7APh4dMg6o6D7L0etdFjgrLhFchGMoYipf6HQ%3D&reserved=0>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Luis Rojas
>>>>>>>> Software Architect
>>>>>>>> Sixbell
>>>>>>>> Los Leones 1200
>>>>>>>> Providencia
>>>>>>>> Santiago, Chile
>>>>>>>> Phone: (+56-2) 22001288
>>>>>>>> mailto:luis.rojas at sixbell.com
>>>>>>>> http://www.sixbell.com
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Kamailio (SER) - Users Mailing List
>>>>>>>> sr-users at lists.kamailio.org
>>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla --www.asipto.com
>>>>>>> www.twitter.com/miconda --www.linkedin.com/in/miconda
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Luis Rojas
>>>>>> Software Architect
>>>>>> Sixbell
>>>>>> Los Leones 1200
>>>>>> Providencia
>>>>>> Santiago, Chile
>>>>>> Phone: (+56-2) 22001288
>>>>>> mailto:luis.rojas at sixbell.com
>>>>>> http://www.sixbell.com
>>>>> --
>>>>> Daniel-Constantin Mierla --www.asipto.com
>>>>> www.twitter.com/miconda --www.linkedin.com/in/miconda
>>>>
>>>>
>>>> --
>>>> Luis Rojas
>>>> Software Architect
>>>> Sixbell
>>>> Los Leones 1200
>>>> Providencia
>>>> Santiago, Chile
>>>> Phone: (+56-2) 22001288
>>>> mailto:luis.rojas at sixbell.com
>>>> http://www.sixbell.com
>>> --
>>> Daniel-Constantin Mierla --www.asipto.com
>>> www.twitter.com/miconda --www.linkedin.com/in/miconda
>>
>>
>> --
>> Luis Rojas
>> Software Architect
>> Sixbell
>> Los Leones 1200
>> Providencia
>> Santiago, Chile
>> Phone: (+56-2) 22001288
>> mailto:luis.rojas at sixbell.com
>> http://www.sixbell.com
> --
> Daniel-Constantin Mierla --www.asipto.com
> www.twitter.com/miconda --www.linkedin.com/in/miconda
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.rojas at sixbell.com
http://www.sixbell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200409/d3bac829/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 49792 bytes
Desc: not available
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200409/d3bac829/attachment.png>
More information about the sr-users
mailing list