I'm looking for documentation//howto/advice/best practices for running Kamailio under Kubernetes on AWS via Rancher. Specifically, how to manage the public/Elastic IP. This is a load balancer (dispatcher) scenario with rtpengine for receiving inbound calls from a DID provider, but a similar configuration might be used for clients. I'm open to using an AWS load balancer, and would like to scale with multiple Kamailio pods behind a single public IP. I'd rather use Rancher-managed EC2 than pay extra for EKS, unless EKS can cancel out its own cost. This sounds like it should be a solved problem, just complex. I see blogs and videos out there, but it's not always obvious if their content is current. Anyone want to help me peel this onion?
-Calvin
Hello,
I usually recommend considering if you really need to do it, the (added) large complexity is in most cases not worth the (small) benefit.
If you want to go for it anyway, one way of doing it is just binding the public IP(s) to individual Kamailio instances. Using an IP load balancer to have multiple Kamailio server behind a single IP only works in limited scenarios (stateless). In the end it boils down the usual deployment and configuration challenges one has to deal on k8s.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Calvin E. calvine@gmail.com Sent: Dienstag, 28. März 2023 23:20 To: sr-users@lists.kamailio.org Subject: [SR-Users] Rancher, AWS, and Elastic IP
I'm looking for documentation//howto/advice/best practices for running Kamailio under Kubernetes on AWS via Rancher. Specifically, how to manage the public/Elastic IP. This is a load balancer (dispatcher) scenario with rtpengine for receiving inbound calls from a DID provider, but a similar configuration might be used for clients. I'm open to using an AWS load balancer, and would like to scale with multiple Kamailio pods behind a single public IP. I'd rather use Rancher-managed EC2 than pay extra for EKS, unless EKS can cancel out its own cost. This sounds like it should be a solved problem, just complex. I see blogs and videos out there, but it's not always obvious if their content is current. Anyone want to help me peel this onion?
-Calvin
I usually recommend considering if you really need to do it, the (added) large complexity is in most cases not worth the (small) benefit.
I don't disagree. There's an ongoing "containerize all the things" effort here, and Kamailio may need to be exempt from that considering how much traffic a single instance can potentially handle. The minimum requirement is active/backup failover with a floating IP, despite the chance that an Elastic IP might not move "instantly". It's the internal network Kamailio is talking to that actually benefits from containerization.
If you want to go for it anyway, one way of doing it is just binding the public IP(s) to individual Kamailio instances. Using an IP load balancer to have multiple Kamailio server behind a single IP only works in limited scenarios (stateless).
Is there an AWS-ism that lets an IP follow a pod but doesn't assign it to the worker node itself? The thought of putting a worker node in a public subnet makes me uncomfortable.
It's been my understanding that Kamailio clustering would solve life behind a load balancer? This is the part where I'm hesitant to follow guidance from ten year old blog posts and YouTube videos and need to grok the current state of clustering. I'm working with the presumption that call state can be stored externally, i.e. Redis, and any node could handle any messages from any dialog. Similarly for rtpengine. "Is this the real life? / Is this just fantasy?"
On Mar 29, 2023, at 12:42 PM, calvine@gmail.com wrote:
It's been my understanding that Kamailio clustering would solve life behind a load balancer? This is the part where I'm hesitant to follow guidance from ten year old blog posts and YouTube videos and need to grok the current state of clustering. I'm working with the presumption that call state can be stored externally, i.e. Redis, and any node could handle any messages from any dialog. Similarly for rtpengine. "Is this the real life? / Is this just fantasy?"
No, this contains vastly far more fantasy than reality.
Some parts are true, but not because of any clustering, but just by the very nature of how proxies operate. For instance, Kamailio can indeed handle any in-dialog message, but not because it stores call state somewhere; call state is just not necessary to route an in-dialog request.
On the other hand, you can’t send a CANCEL message to any proxy, because that’s a “hop-by-hop” request that requires transaction state (not to be confused with dialog state/call state) to deal with, and that kind of state _cannot_ be replicated.
RTPEngine is reasonably clusterable, using Redis as an intermediary.
But no, on the whole, the idea that you can just replicate all necessary state and send anything anywhere any time is rather fantastical. It works in bits and pieces. I wouldn’t get overly attached to this model, it’s a bit romantic. :-)
— Alex
In thinking about it further: the other problem is that SIP, despite its IETF heritage, ended up in a place where network and transport-layer reachability information (in the OSI burrito sense) is just part of protocol messages, depriving SIP of the cleaner logical translation layer - and the resulting separation - between its logical addressing and lower-level endpoint reachability information.
Unlike something like HTTP, SIP also has a bunch of persistent state. This makes it a lot harder to put SIP behind generic/protocol-agnostic load balancers, deal with NAT, etc. It’s not impossible, it’s just much harder.
Henning said it: the juice probably isn’t worth the squeeze.
I find it helpful to think of Kamailio as a piece of very low-level infrastructure more akin to a router than something like an application gateway/server. The latter type of network element makes sense to cluster, containerise, etc. Kamailio less so. That’s not to say there aren’t use-cases for it — there certainly are — but this doesn’t necessarily sound like one of them.
— Alex
On Mar 29, 2023, at 1:22 PM, Alex Balashov abalashov@evaristesys.com wrote:
On Mar 29, 2023, at 12:42 PM, calvine@gmail.com wrote:
It's been my understanding that Kamailio clustering would solve life behind a load balancer? This is the part where I'm hesitant to follow guidance from ten year old blog posts and YouTube videos and need to grok the current state of clustering. I'm working with the presumption that call state can be stored externally, i.e. Redis, and any node could handle any messages from any dialog. Similarly for rtpengine. "Is this the real life? / Is this just fantasy?"
No, this contains vastly far more fantasy than reality.
Some parts are true, but not because of any clustering, but just by the very nature of how proxies operate. For instance, Kamailio can indeed handle any in-dialog message, but not because it stores call state somewhere; call state is just not necessary to route an in-dialog request.
On the other hand, you can’t send a CANCEL message to any proxy, because that’s a “hop-by-hop” request that requires transaction state (not to be confused with dialog state/call state) to deal with, and that kind of state _cannot_ be replicated.
RTPEngine is reasonably clusterable, using Redis as an intermediary.
But no, on the whole, the idea that you can just replicate all necessary state and send anything anywhere any time is rather fantastical. It works in bits and pieces. I wouldn’t get overly attached to this model, it’s a bit romantic. :-)
— Alex
-- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800
Hello,
Alex already provided more context - just some additional notes:
You could of course put Kamailio in a container without using k8s, this is something that several people are doing. Just using it as deployment tool, to have a uniform management for operations etc.. You usually have it run in host network mode.
You can also do that for rtpengine. For full performance you want to operate it in kernel mode, which means that you need to run it as privileged container with host network. The rtpengine will need to load kernel modules on the host. In the future there might be something like media streaming mesh, but this seems to be still in development.
These are of course no blocker against deploying it on k8s, just making it harder to get the benefits.
Cheers,
Henning
-----Original Message----- From: calvine@gmail.com calvine@gmail.com Sent: Mittwoch, 29. März 2023 18:43 To: sr-users@lists.kamailio.org Subject: [SR-Users] Re: Rancher, AWS, and Elastic IP
I usually recommend considering if you really need to do it, the (added) large complexity is in most cases not worth the (small) benefit.
I don't disagree. There's an ongoing "containerize all the things" effort here, and Kamailio may need to be exempt from that considering how much traffic a single instance can potentially handle. The minimum requirement is active/backup failover with a floating IP, despite the chance that an Elastic IP might not move "instantly". It's the internal network Kamailio is talking to that actually benefits from containerization.
If you want to go for it anyway, one way of doing it is just binding the public IP(s) to individual Kamailio instances. Using an IP load balancer to have multiple Kamailio server behind a single IP only works in limited scenarios (stateless).
Is there an AWS-ism that lets an IP follow a pod but doesn't assign it to the worker node itself? The thought of putting a worker node in a public subnet makes me uncomfortable.
It's been my understanding that Kamailio clustering would solve life behind a load balancer? This is the part where I'm hesitant to follow guidance from ten year old blog posts and YouTube videos and need to grok the current state of clustering. I'm working with the presumption that call state can be stored externally, i.e. Redis, and any node could handle any messages from any dialog. Similarly for rtpengine. "Is this the real life? / Is this just fantasy?" __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
-
Alex Balashov -
Calvin E. -
calvine＠gmail.com
-
Henning Westerholt