[SR-Users] Kamailio propagates 180 and 200 OK OUT OF ORDER
Luis Rojas G.
luis.rojas at sixbell.com
Fri Apr 10 14:21:17 CEST 2020
Hi, Alex,
It's 16 virtual cores (8 physical plus HyperThreading) and 48GB of RAM.
Luis
On 4/10/20 8:02 AM, Alex Balashov wrote:
> Luis,
>
> I wonder, how many CPU cores/available hardware threads (taking into
> account HyperThreading and all that—so just the number of CPUs in
> /proc/cpuinfo) are available here? It almost sounds like something
> which would occur with maybe 1 or 2 CPUs being contended over. Perhaps
> the simplest thing for this kind of call volume is to run Kamailio on
> a host with 8 or 12+ CPU cores?
>
> — Alex
>
> —
> Sent from my iPad
>
>> On Apr 10, 2020, at 2:20 AM, Luis Rojas G. <luis.rojas at sixbell.com>
>> wrote:
>>
>>
>> Hello,
>>
>> I have a lot of experience developing mutithreaded applications, and
>> I don't see it so unlikely at all that a process loses cpu just after
>> recvfrom(). It's just as probable as to lose it just before, or when
>> writing on a cache or just before of after sendto(). If there are
>> many messages going through, some of them will fall in this scenario.
>> if I try sending a burst of 100 messages, I see two or three
>> presenting the scenario.
>>
>> Just forward() with a single process does not give the capacity. I'm
>> getting almost 1000caps. More than that and start getting errores,
>> retransmissions, etc. And this is just one way. I need to receive the
>> call to go back to the network (our application is a B2BUA), so I
>> will be down to 500caps, with a simple scenario, with no reliable
>> responses, reinvites, updates, etc. I will end up having as many
>> standalone kamailio processes as the current servers I do have now.
>>
>> I really think the simplest way would be to add a small delay to 200
>> OK. Very small, like 10ms, should be enough. Simple and it should
>> work. As Alex Balashov commented he did for the case with ACK-Re-Invite.
>>
>> I have to figure out how to make async_ms_sleep() work in reply_route().
>>
>> Thanks for all the comments and ideas
>>
>> Best regards,
>>
>> Luis
>>
>>
>>
>> . On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote:
>>>
>>>
>>> MICONDA at GMAIL.COM appears similar to someone who previously sent you
>>> email, but may not be that person. Learn why this could be a risk
>>> <http://aka.ms/LearnAboutSenderIdentification>
>>> Feedback <http://aka.ms/SafetyTipsFeedback>
>>>
>>> Hello,
>>>
>>> then the overtaking is in between reading from the socket and
>>> getting to parsing the call-id value -- the cpu is lost by first
>>> reader after recvfrom() and the second process get enough cpu time
>>> to go ahead further. I haven't encountered this case, but as I said
>>> previously, it is very unlikely, but still possible. I added the
>>> route_locks_size because in the past I had cases when processing of
>>> some messages took longer executing config (e.g., due to
>>> authentication, accounting, ..) and I needed to be sure they are
>>> processed in the order they enter config execution.
>>>
>>> Then the option is to see if a single process with stateless sending
>>> out (using forward()) gives the capacity, if you don't do any other
>>> complex processing. Or if you do more complex processing, use a
>>> dispatcher process with forwarding to local host or in a similar
>>> manner try to use mqueue+rtimer for dispatching using shared memory
>>> queues.
>>>
>>> Of course, it is open source and there is also the C coding way, to
>>> add a synchronizing mechanism to protect against parallel execution
>>> of the code from recvfrom() till call-id lock is acquired.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 09.04.20 17:37, Luis Rojas G. wrote:
>>>> Hello,
>>>>
>>>> Well, it did not work at all. Exactly same behavior, with random
>>>> out of order messages.
>>>>
>>>> Best regards,
>>>>
>>>> Luis
>>>>
>>>> On 4/9/20 11:28 AM, Daniel-Constantin Mierla wrote:
>>>>>
>>>>>
>>>>> MICONDA at GMAIL.COM appears similar to someone who previously sent
>>>>> you email, but may not be that person. Learn why this could be a
>>>>> risk <http://aka.ms/LearnAboutSenderIdentification>
>>>>> Feedback <http://aka.ms/SafetyTipsFeedback>
>>>>>
>>>>> Hello,
>>>>>
>>>>> the sip messages belonging to the same dialog have the same value
>>>>> for Call-Id header. The locking is done based on hashing the
>>>>> Call-Id, so it doesn't need at all the dialog module for its purpose.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>> On 09.04.20 14:19, Luis Rojas G. wrote:
>>>>>> Hello, Daniel,
>>>>>>
>>>>>> I am not so sure. I first tried adding that parameter, but it did
>>>>>> not work at all. Same behavior. Then I read the documentation
>>>>>> more carefully :
>>>>>>
>>>>>> https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size
>>>>>>
>>>>>>
>>>>>> route_locks_size
>>>>>>
>>>>>> Set the number of mutex locks to be used for synchronizing the
>>>>>> execution of messages sharing the same Call-Id. In other words,
>>>>>> enables Kamailio to execute the config script sequentially for
>>>>>> the requests and replies received *within the same dialog* – a
>>>>>> new message received *within the same dialog* waits until the
>>>>>> previous one is routed out.
>>>>>>
>>>>>> Locks to execute sequentially messages belonging to same dialog.
>>>>>> How will Kamailio be aware that messages belong to same dialog,
>>>>>> without the dialog module?. With just stateless proxy it has no
>>>>>> idea about dialogs, it just forwards messages. I guess that's why
>>>>>> just adding that parameter did not work.
>>>>>>
>>>>>> Am I wrong?
>>>>>>
>>>>>> Luis
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/9/20 3:47 AM, Daniel-Constantin Mierla wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> On 08.04.20 23:03, Luis Rojas G. wrote:
>>>>>>>> Hello, Daniel,
>>>>>>>>
>>>>>>>> I looked into that parameter, but I need to use with the
>>>>>>>> dialog module, and I'm pretty afraid to use that.
>>>>>>>
>>>>>>> who said or where is written than you need to load the dialog
>>>>>>> module? You definitely don't.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Daniel
>>>>>>>
>>>>>>>
>>>>>>>> I was looking more into the stateless proxy, because I need to
>>>>>>>> process a lot of traffic.
>>>>>>>>
>>>>>>>> My target is 4200CAPS. with duration between 90s and 210. Let's
>>>>>>>> say, 150 seconds. That would mean 630.000 simultaneous dialogs.
>>>>>>>> I don't think the solution can go that way.
>>>>>>>>
>>>>>>>> it would really help me to be able to use completely stateless
>>>>>>>> proxy plus Async in reply_route(), to introduce an artificial
>>>>>>>> delay before forwarding 200 OK to Invite.. As someone
>>>>>>>> mentioned, it would help me on request_route(), for race
>>>>>>>> conditions between ACK and Re-Invite.
>>>>>>>>
>>>>>>>> Any idea why Async is not allowed in reply_route()?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> Luis
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/8/20 1:07 PM, Daniel-Constantin Mierla wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> you have to keep in mind that Kamailio is a SIP packet router,
>>>>>>>>> not a telephony engine. If 180 and 200 replies are part of a
>>>>>>>>> call is not something that Kamailio recognize at its core. Its
>>>>>>>>> main goal is to route out as fast as possible what is
>>>>>>>>> received, by executing the configuration file script. Now, a
>>>>>>>>> matter of your configuration file, processing of some SIP
>>>>>>>>> messages can take longer than processing other. And the
>>>>>>>>> processing is done in parallel, a matter of children parameter
>>>>>>>>> (and tcp_children, sctp_children).
>>>>>>>>>
>>>>>>>>> With that in mind, a way to try to cope better with the issue
>>>>>>>>> you face is to set route_locks_size parameter, see:
>>>>>>>>>
>>>>>>>>> *
>>>>>>>>> https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size
>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.kamailio.org%2Fwiki%2Fcookbooks%2Fdevel%2Fcore%23route_locks_size&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477633477&sdata=P5gWkVhKbY0EJj6J5jpUZYHL09xAARlgOpvgETTZJGo%3D&reserved=0>
>>>>>>>>>
>>>>>>>>> Probably is what you look for.
>>>>>>>>>
>>>>>>>>> But if you want more tight constraints, like when receiving a
>>>>>>>>> 180 after a 200ok and not route it out, you have to make the
>>>>>>>>> logic in configuration file by combining modules such as
>>>>>>>>> dialog or htable (as already suggested).
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Daniel
>>>>>>>>>
>>>>>>>>> On 08.04.20 16:04, Luis Rojas G. wrote:
>>>>>>>>>> Hi, Henning,
>>>>>>>>>>
>>>>>>>>>> No need to be ironic. As I mentioned on my first post, I
>>>>>>>>>> tried stateful proxy and I observed the same behavior.
>>>>>>>>>>
>>>>>>>>>> /"I tried using stateful proxy and I obtained the same result."/
>>>>>>>>>>
>>>>>>>>>> The asynchronous sleep seems promising. I will look into it.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Luis
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 4/8/20 9:30 AM, Henning Westerholt wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Luis,
>>>>>>>>>>>
>>>>>>>>>>> I see. Well, you want to use Kamailio as a stateless proxy,
>>>>>>>>>>> on the other hand it should do things that are inherently
>>>>>>>>>>> stateful. 😉
>>>>>>>>>>>
>>>>>>>>>>> As mentioned, have a look to the dialog module to track the
>>>>>>>>>>> state of dialogs that you process. This will not work in a
>>>>>>>>>>> stateless mode, though.
>>>>>>>>>>>
>>>>>>>>>>> You can also use the htable module to just store some data
>>>>>>>>>>> about the processed messages in a shared memory table and
>>>>>>>>>>> use this to enforce your ordering. There is also the option
>>>>>>>>>>> to do an asynchronous sleep (with the async) module on the
>>>>>>>>>>> message that you want to delay but still processing other
>>>>>>>>>>> messages during it.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Henning
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Henning Westerholt – https://skalatan.de/blog/
>>>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskalatan.de%2Fblog%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477643469&sdata=7wVK%2BNcbz0LZBJKlnUgvt0XqOYLQh2xZoasaU0q29qI%3D&reserved=0>
>>>>>>>>>>>
>>>>>>>>>>> Kamailio services – https://gilawa.com
>>>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgilawa.com%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477643469&sdata=CHNJCNYzwqEkgdLXXyfKcv8McuEutS4PPhCX5Oe9FGE%3D&reserved=0>
>>>>>>>>>>>
>>>>>>>>>>> *From:* Luis Rojas G. <luis.rojas at sixbell.com>
>>>>>>>>>>> *Sent:* Wednesday, April 8, 2020 3:00 PM
>>>>>>>>>>> *To:* Henning Westerholt <hw at skalatan.de>; Kamailio (SER) -
>>>>>>>>>>> Users Mailing List <sr-users at lists.kamailio.org>
>>>>>>>>>>> *Subject:* Re: [SR-Users] Kamailio propagates 180 and 200 OK
>>>>>>>>>>> OUT OF ORDER
>>>>>>>>>>>
>>>>>>>>>>> Hello, Henning,
>>>>>>>>>>>
>>>>>>>>>>> I am worried about this scenario, because it's a symptom of
>>>>>>>>>>> what may happen in other cases. For instance, I've seen that
>>>>>>>>>>> this operator usually sends re-invites immediate after
>>>>>>>>>>> sending ACK. This may create race conditions like 3.1.5 of
>>>>>>>>>>> RFC5407
>>>>>>>>>>>
>>>>>>>>>>> https://tools.ietf.org/html/rfc5407#page-22
>>>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc5407%23page-22&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477653469&sdata=57Iwh49x3vUdl5541kBc1mvD50u4w5wQQCovMcvE030%3D&reserved=0>
>>>>>>>>>>>
>>>>>>>>>>> I'd understand that one happens because of packet loss, as
>>>>>>>>>>> it's in UDP's nature, but in this case it would be
>>>>>>>>>>> artificially created by Kamailio. if there was no problem at
>>>>>>>>>>> network level (packet loss, packets following different path
>>>>>>>>>>> on the network and arriving out of order), why Kamailio
>>>>>>>>>>> creates it?
>>>>>>>>>>>
>>>>>>>>>>> I'd expect that the shared memory is used precisely for
>>>>>>>>>>> this. If an instance of kamailio receives a 200 OK, it could
>>>>>>>>>>> check on the shm and say "hey, another instance is
>>>>>>>>>>> processing a 180 for this call. Let's wait for it to finish"
>>>>>>>>>>> (*). I know there could still be a problem, the instance
>>>>>>>>>>> processing the 180 undergoes a context switch just after it
>>>>>>>>>>> receives the message, but before writing to shm, but it
>>>>>>>>>>> would greatly reduce the chance.
>>>>>>>>>>>
>>>>>>>>>>> In our applications we use a SIP stack that always sends
>>>>>>>>>>> messages to the application in the same order it receives
>>>>>>>>>>> them, even though is multi-threaded and messages from the
>>>>>>>>>>> network are received by different threads. So, they really
>>>>>>>>>>> syncronize between them. Why Kamailio instances don't?
>>>>>>>>>>>
>>>>>>>>>>> I am evaluating kamailio to use it as a dispatcher to
>>>>>>>>>>> balance load against our several Application Servers, to
>>>>>>>>>>> present to the operator just a couple of entrance points to
>>>>>>>>>>> our platform (they don't want to establish connections to
>>>>>>>>>>> each one of our servers). This operator is very difficult to
>>>>>>>>>>> deal with. I am sure they will complain something like "why
>>>>>>>>>>> are you sending messages out of order? Fix that". The
>>>>>>>>>>> operator will be able to see traces and check that messages
>>>>>>>>>>> entered the Kamailio nodes in order and left out of order.
>>>>>>>>>>> They will not accept it.
>>>>>>>>>>>
>>>>>>>>>>> (*) Not really "wait", as it would introduce a delay in
>>>>>>>>>>> processing all messages. it should be like putting it on a
>>>>>>>>>>> queue, continue processing other messages, and go back to
>>>>>>>>>>> the queue later.
>>>>>>>>>>>
>>>>>>>>>>> Well, thanks for your answer.
>>>>>>>>>>>
>>>>>>>>>>> Luis
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 4/8/20 3:01 AM, Henning Westerholt wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello Luis,
>>>>>>>>>>>
>>>>>>>>>>> as the 1xx responses are usually send unreliable (unless
>>>>>>>>>>> you use PRACK), you should not make any assumption on
>>>>>>>>>>> the order or even the arrival of this messages. It can
>>>>>>>>>>> also happens on a network level, if send by UDP.
>>>>>>>>>>>
>>>>>>>>>>> Can you elaborate why you think this re-ordering is a
>>>>>>>>>>> problem for you?
>>>>>>>>>>>
>>>>>>>>>>> One idea to enforce some ordering would be to use the
>>>>>>>>>>> dialog module in combination with reply routes and the
>>>>>>>>>>> textops(x) module.
>>>>>>>>>>>
>>>>>>>>>>> About the shared memory question – Kamailio implement
>>>>>>>>>>> its own memory manager (private memory and shared memory
>>>>>>>>>>> pool).
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Henning
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Henning Westerholt – https://skalatan.de/blog/
>>>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskalatan.de%2Fblog%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477663459&sdata=kUCddL3S%2FQx8%2B1eFbrlSJZy3GSj2OmLHdnXncvdDKfo%3D&reserved=0>
>>>>>>>>>>>
>>>>>>>>>>> Kamailio services – https://gilawa.com
>>>>>>>>>>> <https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgilawa.com%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477663459&sdata=sZ3pmtBa9noTCpm6mp5P1Mzelc%2Byxz1XFOB0tvPHRCI%3D&reserved=0>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *From:* sr-users <sr-users-bounces at lists.kamailio.org>
>>>>>>>>>>> <mailto:sr-users-bounces at lists.kamailio.org> *On Behalf
>>>>>>>>>>> Of *Luis Rojas G.
>>>>>>>>>>> *Sent:* Tuesday, April 7, 2020 10:43 PM
>>>>>>>>>>> *To:* sr-users at lists.kamailio.org
>>>>>>>>>>> <mailto:sr-users at lists.kamailio.org>
>>>>>>>>>>> *Subject:* [SR-Users] Kamailio propagates 180 and 200 OK
>>>>>>>>>>> OUT OF ORDER
>>>>>>>>>>>
>>>>>>>>>>> Good day,
>>>>>>>>>>>
>>>>>>>>>>> I am testing the dispatcher module, using Kamailio as
>>>>>>>>>>> stateless proxy. I have a pool of UAC (scripts in SIPP)
>>>>>>>>>>> and a pool of UAS (also scripts in SIPP) for the
>>>>>>>>>>> destinations. Kamailio version is kamailio-5.3.3-4.1.x86_64.
>>>>>>>>>>>
>>>>>>>>>>> Problem I have is, if UAS responds 180 and 200 OK to
>>>>>>>>>>> Invite immediately, sometimes they are propagated out of
>>>>>>>>>>> order. 200 OK before 180, like this :
>>>>>>>>>>>
>>>>>>>>>>> <image001.png>
>>>>>>>>>>>
>>>>>>>>>>> UAS is 172.30.4.195:5061. UAC is 172.30.4.195:5080.
>>>>>>>>>>> Kamailio is 192.168.253.4:5070
>>>>>>>>>>>
>>>>>>>>>>> Difference between 180 and 200 is just about 50
>>>>>>>>>>> microseconds.
>>>>>>>>>>>
>>>>>>>>>>> My guess is that both messages are received by different
>>>>>>>>>>> instances of Kamailio, and then because of context
>>>>>>>>>>> switches, even though the 180 is received before, that
>>>>>>>>>>> process ends after the processing of 200. However, I had
>>>>>>>>>>> the idea that in order to avoid these problems the
>>>>>>>>>>> kamailio processes synchronized with each other using a
>>>>>>>>>>> shared memory. I tried using stateful proxy and I
>>>>>>>>>>> obtained the same result.
>>>>>>>>>>>
>>>>>>>>>>> By the way, anyone has any idea about how Kamailio's
>>>>>>>>>>> share memory is implemented? It clearly does not use the
>>>>>>>>>>> typical system calls shmget(), shmat(), because they are
>>>>>>>>>>> not shown by ipcs command.
>>>>>>>>>>>
>>>>>>>>>>> Before posting here I googled, but I couldn't find
>>>>>>>>>>> anything related to this. I can't believe I am the only
>>>>>>>>>>> one who ever had this problem, so I guess I am doing
>>>>>>>>>>> something wrong...
>>>>>>>>>>>
>>>>>>>>>>> Please, any help. I'm really stuck on this.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Luis Rojas
>>>>>>>>>>> Software Architect
>>>>>>>>>>> Sixbell
>>>>>>>>>>> Los Leones 1200
>>>>>>>>>>> Providencia
>>>>>>>>>>> Santiago, Chile
>>>>>>>>>>> Phone: (+56-2) 22001288
>>>>>>>>>>> mailto:luis.rojas at sixbell.com
>>>>>>>>>>> http://www.sixbell.com <https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sixbell.com%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477673456&sdata=FPhaFAUYu%2BEg8%2B6R4%2BgpvgG12f3MQJ%2FFJFvhTT2ITP0%3D&reserved=0>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Luis Rojas
>>>>>>>>>> Software Architect
>>>>>>>>>> Sixbell
>>>>>>>>>> Los Leones 1200
>>>>>>>>>> Providencia
>>>>>>>>>> Santiago, Chile
>>>>>>>>>> Phone: (+56-2) 22001288
>>>>>>>>>> mailto:luis.rojas at sixbell.com
>>>>>>>>>> http://www.sixbell.com
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Kamailio (SER) - Users Mailing List
>>>>>>>>>> sr-users at lists.kamailio.org
>>>>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>>>>> --
>>>>>>>>> Daniel-Constantin Mierla --www.asipto.com
>>>>>>>>> www.twitter.com/miconda --www.linkedin.com/in/miconda
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Luis Rojas
>>>>>>>> Software Architect
>>>>>>>> Sixbell
>>>>>>>> Los Leones 1200
>>>>>>>> Providencia
>>>>>>>> Santiago, Chile
>>>>>>>> Phone: (+56-2) 22001288
>>>>>>>> mailto:luis.rojas at sixbell.com
>>>>>>>> http://www.sixbell.com
>>>>>>> --
>>>>>>> Daniel-Constantin Mierla --www.asipto.com
>>>>>>> www.twitter.com/miconda --www.linkedin.com/in/miconda
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Luis Rojas
>>>>>> Software Architect
>>>>>> Sixbell
>>>>>> Los Leones 1200
>>>>>> Providencia
>>>>>> Santiago, Chile
>>>>>> Phone: (+56-2) 22001288
>>>>>> mailto:luis.rojas at sixbell.com
>>>>>> http://www.sixbell.com
>>>>> --
>>>>> Daniel-Constantin Mierla --www.asipto.com
>>>>> www.twitter.com/miconda --www.linkedin.com/in/miconda
>>>>
>>>>
>>>> --
>>>> Luis Rojas
>>>> Software Architect
>>>> Sixbell
>>>> Los Leones 1200
>>>> Providencia
>>>> Santiago, Chile
>>>> Phone: (+56-2) 22001288
>>>> mailto:luis.rojas at sixbell.com
>>>> http://www.sixbell.com
>>> --
>>> Daniel-Constantin Mierla --www.asipto.com
>>> www.twitter.com/miconda --www.linkedin.com/in/miconda
>>
>>
>> --
>> Luis Rojas
>> Software Architect
>> Sixbell
>> Los Leones 1200
>> Providencia
>> Santiago, Chile
>> Phone: (+56-2) 22001288
>> mailto:luis.rojas at sixbell.com
>> http://www.sixbell.com
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.kamailio.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsr-users&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477823379&sdata=9SJPhjaKPNJyIVJJy6Ag43u1GYl6RbDn2faPAwbHXn4%3D&reserved=0
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.rojas at sixbell.com
http://www.sixbell.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20200410/68c97dd3/attachment.html>
More information about the sr-users
mailing list