[Serusers] request for comments

[Greg Fausak]

More food :-)

We distribute our SER boxes.  We did it by:

1) using the domain name for a login id to the postgres database
2) slicing each table based on domain name, that is, instead of
      subscriber I have a view calls av_subscriber which only shows 
records
      from the current domain.
3) Each ser proxy serves a domain.

Using this technique I can run dozens of SER proxies each with
its own view of the database...however, if any one domain gets too
big I will have a problem (I haven't had that problem yet, I'll let you 
know!).

I have always thought that the way to solve the distribution
problem is to relax the in-memory caching of registrations.
Everytime some UAC registers the database is updated, and everytime
a call is to be delivered the location table is queried.  Using this 
technique
will tax the database more, but it would allow multiple SER proxies
without the need for a sticky bit set, that is a round robin or least 
used
SLB model.

At the current time because of the caching you can't have two SER 
proxies
serving the same REGISTERed customer base because the location table
gets tromped.

---greg

On Jun 30, 2004, at 1:06 PM, Jev wrote:

>
>
> Zeus Ng wrote:
>> I may share some of my experience on a similar concept. Notice that 
>> it's not
>> a solution but more an idea sharing.
>
> Solutions are nice, but I really want to hear ideas :) Provides food 
> for thought! :)
>
>> From the experiment, I found that there
>> is a fundamental weakness in ser (plus UDP plus NAT) to support a
>> distributed SIP environment. I'm not saying it can't be done. 
>> However, to
>> make ser more distributed, I think there is a need to redesign the 
>> way ser
>> handle user location. The lab environment I have is 4 ser proxies and 
>> 2 ser location servers. The
>> 4 ser proxies were used as front end for proxying SIP requests. They 
>> have a
>> SRV record in the DNS server for UAs which understand this record. 
>> For UA
>> that doesn't understand SRV, the DNS also reply the proxies IP in a 
>> round
>> robin fashion.
>> When a UA lookup the IP of the proxy, it get one from either the SRV 
>> record
>> or round robin A record.
>> All REGISTER requests are forwarded from the proxies to the primary 
>> location
>> server. This is than replicated to the secondary location server by
>> t_replicate. So, the proxies has no knowledge of UA location. Only the
>> location servers know where to reach the UA.
>> For other SIP requests, I have tried two different methods to handle 
>> them.
>> 1. Forward all requests to location server and use record_route to 
>> keep the
>> proxy in the path:
>> This works great to maintain dialogue as INVITE, reINVITE, BYE, 
>> CANCEL will
>> all proxy back to the location server which has the transaction 
>> state. OTOH,
>> it is poor in NAT handling since the location server was never 
>> directly
>> contacted by the NAT device. The nat ping will not keep a hole in the 
>> NAT
>> device. Also, it has no performance improvement over one single
>> "proxy+location" server as all requests end up in location server.
>
> So you had the backend location server contacting the UAC directly? 
> I'm attempting to route the invite back through the originating front 
> end proxy that has the nat session already established with the natted 
> UAC. At the moment this only works because I am rewriting the 
> (hardcoded) hostname in my config, but I'm looking at doing this 
> dynamically so that any requests to the user location server will have 
> their hostname rewritten to the previous hop.
>
>
>> 2. Proxy querying UA location via SQL
>> In this method, I've written a small SQL script to be run by the 
>> proxy via
>> exec_dst to check the UA location from the location server DB 
>> backend. (I
>> know that DB is not the best place to check location but it is easier 
>> than
>> writing C program to query the memory USRLOC on the location server.) 
>> This
>> works best for performance as the proxies are sharing the requests as 
>> well
>> as RTP proxying. However, it is relatively poor in NAT and 
>> transaction as
>> the INVITE, BYE and CANCEL can be handled by different proxy due DNS
>> resolution.
>
> I really want to keep my operations within SIP messaging only, and not 
> having to rely on external mechanisms such as sql queries. This 
> maintains our flexibility to use any SIP compliant device. It's a 
> great idea thogh! :)
>
>> One way I see ser going distributed is to follow the idea of squid 
>> plus some
>> enhancement. The group of proxies are put into partnership. When the 
>> proxy
>> receive a REGISTER request, it check whether one of its partner has a 
>> record
>> of that UA or not. If yes, it forward the request to the other proxy 
>> and
>> forget it. Otherwise, it save the location in its memory, do NAT 
>> stuff and
>> becomes the authoritive proxy for that UA until the REGISTER expires. 
>> When
>> other request comes in, the proxy do the same check with its partner 
>> again
>> and forward the request to the authoritive proxy. This way, the 
>> authoritive
>> proxy maintains the nat ping, shares the RTP proxying and keep trace 
>> of
>> transactions.
>> When a new proxy comes in, we just need to tell ser that there is a 
>> new
>> member in the partnership. (Though, we need to find a way to tell ser 
>> about
>> this without restarting so that it maintains the USRLOC in memory)
>> Instantly, this proxy can serve new UA that was never seen before or 
>> its
>> REGISTER has expires somewhere.
>
> This sounds like a cool idea, I'm not familiar with squids proxiy 
> partnership model, but what you explain seems sound to me. Perhaps the 
> ser proxies could use SRV records to learn about new 'partner' ser 
> proxies? Or would this be a miss-aplication of the SRV feature?
>
>> The only thing I haven't figured out a solution would be how to pick 
>> up UA
>> location when one of the proxy fails. I don't like the way 
>> t_replicate works
>> as it requires hard coding other proxies in the script and needs 
>> restarting
>> ser for failover.
>
> If a proxy that is maintaing a NAT session with a UAC goes away, I see 
> no way of passing off this session/location to another server except 
> just waiting for the UAC to re-register.
>
>> Zeus
>>> -----Original Message-----
>>> From: serusers-bounces at iptel.org [mailto:serusers-bounces at lists.iptel.org] 
>>> On Behalf Of Jev
>>> Sent: Wednesday, 30 June 2004 8:53 AM
>>> To: Andrei Pelinescu-Onciul
>>> Cc: serusers at lists.iptel.org
>>> Subject: Re: [Serusers] request for comments
>>>
>>>
>>> Andrei Pelinescu-Onciul wrote:
>>> [snip]
>>>
>>>> So all the packets comming from the same ip will be sent to
>>>
>>> the same
>>>> fron end SER? (hashing after src ip)?
>>>
>>> Yes, using ciscos "Sticky IP" which I admit, I do not know about, 
>>> but I'm told it will do this job properly.
>>>
>>>
>>>> Anyway there are some problems related to the nat traversal:
>>>>
>>>> 1. nat ping - nat ping needs to access usrloc, so that it
>>>
>>> would know
>>>> which users to ping. However on your setup the front-end
>>>
>>> servers have
>>>> no ideea about this, so they wouldn't be able to nat ping.
>>>
>>> The "main"
>>>> server (User accounts) knows who to ping but its ping won't
>>>
>>> traverse a
>>>> symmetric nat (the nat will have an open binding only with the 
>>>> outbound proxy, which would be one of the load balanced 
>>>> front-ends).
>>>
>>> I do realize this now, so I'm considering running a non-persistent 
>>> usr_loc (no mysql back end) on all the front end servers, and using 
>>> t_replicate between all of them. I admit I have not verified if this 
>>> is possible, so please forgive me if I'm talking non-sense here at 
>>> this stage. My concern here, as I mentioned in my reply to Klaus's 
>>> post, is that if I use t_replicate will all my front end ser 
>>> servers, will they all spit udp at a single natted client when the 
>>> client has only one udp session with one front end server?
>>>
>>>
>>>
>>>> 2. consider user A calling user B, where at least B is
>>>
>>> behind a nat.
>>>> The invite would reach the "main" server which will look up
>>>
>>> B and will
>>>> try to send the message to B's address. Unfortunately B's nat will 
>>>> drop the packet, because it has an open binding only
>>>
>>> between B and the
>>>> load balanced ip. (this will work only if B has a full cone
>>>
>>> nat which
>>>> is very very unlikely)
>>>
>>> I'm not sure on the solution here. I will need to make the call go 
>>> via the front end ser server that has the active udp session with 
>>> the client. I'm going to sleep on this!
>>>
>>>
>>>
>>>> 3. assuming the above stuff will work somehow, you still have to be 
>>>> very carefull to open only one rtp proxy session (since
>>>
>>> each front end
>>>> has its own rtp proxy you should make sure you use
>>>
>>> force_rtp_proxy on
>>>> only one of them, for the same call)
>>>
>>>
>>> I agree, and I realize that I'm making some challenging issues for 
>>> myself :)
>>> Thank you Andrei for your comments!
>>>
>>> -Jev
>>>
>>> _______________________________________________
>>> Serusers mailing list
>>> serusers at lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
>>>
>
> _______________________________________________
> Serusers mailing list
> serusers at lists.iptel.org
> http://lists.iptel.org/mailman/listinfo/serusers
>
>
Greg Fausak
www.AddaBrand.com
(US) 469-546-1265