Hi Tina, Thanks :-) I believe the load balancer can be in front and transparantly translate addresses in the SIP messages: User agent always sees a.b.c.d | Load balancer with public IP a.b.c.d | | ser1 (10.0.0.2) ser2 (10.0.0.3) Paul: You have a similar load-balancing solution, right? Do you use a commerical load balancer that you can reveal the name of? :-) The load balancer need to understand SIP (sort of) and use something (ex. IP address of user agent) to select SER server to send to and rewrite incoming to 10.0.0.2 and 10.0.0.3 respectively and outgoing to a.b.c.d. ser1 and ser2 will happily believe they are alone and communicate directly with user agent using their own private addresses. Both should know about all users (usrloc). Paul just recently posted a small patch for a setup where they use two-way replication between two mysql servers and where each SER instance will save all REGISTERs it receives from user agents, but only save to memory those replicated from the other SER peer. Klaus pointed out that the same can be done by not doing mysql replication, but have one DB for each SER and just use t_replicate() and save_noreply() (for two servers, for more than two you need to use forward_tcp()). I have heard complaints that this approach will generate too much network traffic. Also, t_replicate() does not guarantee delivery of the replication (forward_tcp is better, but not fool proof). So, the "best" solution would probably be to hope that Paul's recent TCP/UDP patch for network access to FIFO commands get included in the CVS. Then, a new replication module should use XMLRPC or whatever protocol the patch is using to forward (in a guaranteed mode) the usrloc information, which then would be injected directly into SER's memory, just like save() would, and of course in whatever db_mode you would like. The replicaton module would have to keep a queue (of course, flushed to a DB or something) in case of network problems, so that the replication would be done (sooner or later) regardless of problems. Of course, this is a major undertaking, so for most replication scenarioes, I would think the forward_tcp() + save_noreply() would work just fine. g-)

That's excellent news, Andreas! I'm not able to see immediately how the two can work nicely together, but I'm sure it should be possible to get SER's FIFO net-centric now ;-) For some reason, however, I think an alternative to SIP-based replication is somewhere down the line. Paul relies on mysql two-way replication and I follow his reasoning that replication should be done at the DB layer (which we do as well, but using LDAP. The usrloc replication is still SIP-based...). However, the solution where forward_tcp() and save_memory() are used, you get both SIP replication traffic, as well as DB replication. If one has full TCP/UDP-based FIFO access, shouldn't it be easy to write a small module that will send a ul_add command to another server? I wonder why Paul and you haven't done that yet, but still rely on SIP-based updates? g-) Andreas Granig wrote:

Paul, we are using LVS as well. It creates template connection according to source Ip (e.g. UAC) and forwards to one of the tunneled IP. I tested this a couple of weeks ago, ser was happy and inserted VIP into via header, the only issue is to route the calls into right server. I agree, there is no such thing as SIP-level clustering. However, how would you solve location problem by DB replication - 1) usrloc does not update from DB....2) even though, I am afraid it can become bottleneck and lock down users ... KRs, Tina Java Rockx <javarockx(a)gmail.com> wrote: See my inline comments. Regards, Paul On Apr 8, 2005 1:08 PM, Greger V. Teigre <greger(a)teigre.com> wrote:Hi Tina, Thanks :-) I believe the load balancer can be in front and transparantly translate addresses in the SIP messages: User agent always sees a.b.c.d | Load balancer with public IP a.b.c.d | | ser1 (10.0.0.2) ser2 (10.0.0.3) Paul: You have a similar load-balancing solution, right? Do you use a commerical load balancer that you can reveal the name of? :-) Our platform is 100% open source. Our load balancer is LVS. I have no idea how it actually works. One if our engineers from RedHat set all that networking stuff up and it's way beyond me. The load balancer need to understand SIP (sort of) and use something (ex. IP address of user agent) to select SER server to send to and rewrite incoming to 10.0.0.2 and 10.0.0.3 respectively and outgoing to a.b.c.d. ser1 and ser2 will happily believe they are alone and communicate directly with user agent using their own private addresses. Both should know about all users (usrloc). Paul just recently posted a small patch for a setup where they use two-way replication between two That small patch will need a slight revision. We discovered through further testing that that patch only fixed attempts to INSERT into the location table. We didn't cover the UPDATE route, unfortunately. I believe Andrew has that patch done and is testing it. So I'll post it to serdev as soon as we know it works - possibly later today. mysql servers and where each SER instance will save all REGISTERs it receives from user agents, but only save to memory those replicated from the other SER peer. Klaus pointed out that the same can be done by not doing mysql replication, but have one DB for each SER and just use t_replicate() and save_noreply() (for two servers, for more than two you need to use forward_tcp()). I have heard complaints that this approach will generate too much network traffic. Also, t_replicate() does not guarantee delivery of the replication (forward_tcp is better, but not fool proof). So, the "best" solution would probably be to hope that Paul's I fully agree. Replication at the sip proxy level, IMHO, is not a good idea. From a purist point of view, replication is not a function of a SIP router - nor should it be. It is a function of the database and we believe that the MySQL replication is a better approach from a speed and reliabilty point of view - not to mention that it simplifies the tasks of the sip router. recent TCP/UDP patch for network access to FIFO commands get included in the CVS. Then, a new replication module should use XMLRPC or whatever protocol the patch is using to forward (in a guaranteed mode) the usrloc information, which then would be injected directly into SER's memory, just like save() would, and of course in whatever db_mode you would like. The replicaton module would have to keep a queue (of course, flushed to a DB or something) in case of network problems, so that the replication would be done (sooner or later) regardless of problems. Of course, this is a major undertaking, so for most replication scenarioes, I would think the forward_tcp() + save_noreply() would work just fine. g-) _______________________________________________ Serusers mailing list serusers(a)lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers --------------------------------- Do you Yahoo!? Make Yahoo! your home page

Paul, thanks a lot, it helps a lot. The thing I do not understand - how you are going to solve call-id stickness (see my another post). Since we're also use LVS I had to find something proprietary, I am afraid this is not a best solution, but the only I have today. KRs, Tina Java Rockx <javarockx(a)gmail.com> wrote: Tina, I really don't know how the LVS server is configured because our network guy that we acquired from RedHat set all that stuff up. I do know that we basically have this set up: I hope the formatting goes well :-) +-----------+ +----------+ +-------+ +---------+ +--------------------+ | internet |------| Cisco |------| LVS |-----| Cisco |----| Application | | cloud | | 3600 | | | | 3600 | | & DB Servers | +-----------+ +----------+ +--------+ +---------+ +--------------------+ So the "Application & DB Servers" box represents multiple servers (all are dedicated to their service). These include SER, MySQL, Apache, configuration management, monitoring, RTP proxies, etc. MySQL is active-active so we have two-way replication. We only have 2 MySQL servers. And this is all we plan on ever having as MySQL is more than capable of handling anything ser can throw at it (given the right hardware). Each ser server is 100% oblivious to other SER servers except for when handling REGISTER messages. Here is basically the ser.cfg for each SER server: listen=10.3.0.221 # this will be 10.3.0.222 on the peer sip proxy modparam("usrloc", "db_mode", 1) # write-through route { if (method=="REGISTER") { if (src_ip==10.3.0.221) { # ip of peer ser proxy save_memory("location"); break; } else { save("location"); t_replicate("10.3.0.222"); break; }; }; } We have identifed a deficiency in the usrloc module urecord.c file whereby when using write-though mode the save_memory() function still attempt to update MySQL. This is a very bad thing because it causes a primary key violation on the location table. Yesterday I posted a patch to serdev to fix this, but today we identified another case where save_memory() tried to insert in to the location table. I'll post a revised patch to serdev as soon as I get a chance. So here is what happens. Assume I have two sip routers and two MySQL servers as: sip-01 is 10.3.0.221 sip-02 is 10.3.0.222 db-01 is 10.2.0.21 db-02 is 10.2.0.22 Now let's just assume that sip-01 is connected to db-01 and sip-02 is connected to db-02. When a SIP UA connects to a sip router (via LVS), assume sip-01 in this example and REGISTERs, sip-01 will call save("location") which updates/inserts to the location table. sip-01 then calls t_replicate to sip-02. sip-02 recieves the REGISTER message and calls save_memory("location") to update its in-memory cache only. It should never hit db-02. Now db-01 eventually (within a second or two) replicates to db-02 and now sip-02 is good to go, even in the event of a restart. Our difficulties mostly surrounded the fact that ser was inserting the physical IP of eth0 on the server as the top VIA rather than the VIP as required. We solved this by using record_route_preset() and passing the VIP as the IP. We also had to bring up a dummy interface in order to get ser to honor the VIP. Hope this helps. Regards, Paul On Apr 8, 2005 7:19 PM, Tina <kramarv(a)yahoo.com> wrote:Paul, we are using LVS as well. It creates template connection according to source Ip (e.g. UAC) and forwards to one of the tunneled IP. I tested this a couple of weeks ago, ser was happy and inserted VIP into via header, the only issue is to route the calls into right server. I agree, there is no such thing as SIP-level clustering. However, how would you solve location problem by DB replication - 1) usrloc does not update from DB....2) even though, I am afraid it can become bottleneck and lock down users ... KRs, Tina Java Rockx <javarockx(a)gmail.com> wrote: See my inline comments. Regards, Paul On Apr 8, 2005 1:08 PM, Greger V. Teigre <greger(a)teigre.com> wrote: Hi Tina, Thanks :-) I believe the load balancer can be in front and transparantly translate addresses in the SIP messages: User agent always sees a.b.c.d | Load balancer with public IP a.b.c.d | | ser1 (10.0.0.2) ser2 (10.0.0.3) Paul: You have a similar load-balancing solution, right? Do you use a commerical load balancer that you can reveal the name of? :-) Our platform is 100% open source. Our load balancer is LVS. I have no idea how it actually works. One if our engineers from RedHat set all that networking stuff up and it's way beyond me. The load balancer need to understand SIP (sort of) and use something (ex. IP address of user agent) to select SER server to send to and rewrite incoming to 10.0.0.2 and 10.0.0.3 respectively and outgoing to a.b.c.d. ser1 and ser2 will happily believe they are alone and communicate directly with user agent using their own private addresses. Both should know about all users (usrloc). Paul just recently posted a small patch for a setup where they use two-way replication between two That small patch will need a slight revision. We discovered through further testing that that patch only fixed attempts to INSERT into the location table. We didn't cover the UPDATE route, unfortunately. I believe Andrew has that patch done and is testing it. So I'll post it to serdev as soon as we know it works - possibly later today. mysql servers and where each SER instance will save all REGISTERs it receives from user agents, but only save to memory those replicated from the other SER peer. Klaus pointed out that the same can be done by not doing mysql replication, but have one DB for each SER and just use t_replicate() and save_noreply() (for two servers, for more than two you need to use forward_tcp()). I have heard complaints that this approach will generate too much network traffic. Also, t_replicate() does not guarantee delivery of the replication (forward_tcp is better, but not fool proof). So, the "best" solution would probably be to hope that Paul's I fully agree. Replication at the sip proxy level, IMHO, is not a good idea. From a purist point of view, replication is not a function of a SIP router - nor should it be. It is a function of the database and we believe that the MySQL replication is a better approach from a speed and reliabilty point of view - not to mention that it simplifies the tasks of the sip router. recent TCP/UDP patch for network access to FIFO commands get included in the CVS. Then, a new replication module should use XMLRPC or whatever protocol the patch is using to forward (in a guaranteed mode) the usrloc information, which then would be injected directly into SER's memory, just like save() would, and of course in whatever db_mode you would like. The replicaton module would have to keep a queue (of course, flushed to a DB or something) in case of network problems, so that the replication would be done (sooner or later) regardless of problems. Of course, this is a major undertaking, so for most replication scenarioes, I would think the forward_tcp() + save_noreply() would work just fine. g-) _______________________________________________ Serusers mailing list serusers(a)lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers --------------------------------- Do you Yahoo!? Make Yahoo! your home page __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

Yes, if you use LVS with direct routing or tunneling, that is what you experience. What I described was a "generic" SIP-aware load balancer where SIP messages would be rewritten and stickiness implemented based on ex. UA IP address (or call-id like vovida's load balancer). Well, IMO, DNS SRV in itself is not bad. It's just that many user clients do not support DNS SRV yet. Except that, I like the concept and it will give you a geographical redundancy and load balancing. Are you referring to the load balancer proxy? IMHO, the SIP-aware load balancer makes things a bit messy. It sounds to me that the LVS + tunneling/direct routing + virtual IP on dummy adapter is a better solution. I'm not sure if I understand. So, you have 2 servers at two location, each location with a shared DB and then replication across an IPsec tunnel?? IMHO, mysql 3.23.x two-way replication is quite shaky and dangerous to rely on. With no locking, you will easily get overwrites and you have to be very sure that your application doesn't mess up the DB. I haven't looked at mysql 4.1 clustering, but from the little I have seen, it looks good. Is that what you use? I believe Paul uses forward_tcp() and save_memory() to save the location to the replicated server's memory, while the save("location") on the primary server will store to the DB (which then replicates on the DB level). g-)

Paul, I appreciate a lot this information, and am interested in your patch a lot. Tina Java Rockx <javarockx(a)gmail.com> wrote: See inline. On Apr 10, 2005 5:32 AM, Greger V. Teigre <greger(a)teigre.com> wrote:> Greger, thanks a lot. Yes, if you use LVS with direct routing or tunneling, that is what you experience. What I described was a "generic" SIP-aware load balancer where SIP messages would be rewritten and stickiness implemented based on ex. UA IP address (or call-id like vovida's load balancer). Well, IMO, DNS SRV in itself is not bad. It's just that many user clients do not support DNS SRV yet. Except that, I like the concept and it will give you a geographical redundancy and load balancing. Are you referring to the load balancer proxy? IMHO, the SIP-aware load balancer makes things a bit messy. It sounds to me that the LVS + tunneling/direct routing + virtual IP on dummy adapter is a better solution. I'm not sure if I understand. So, you have 2 servers at two location, each location with a shared DB and then replication across an IPsec tunnel?? IMHO, mysql 3.23.x two-way replication is quite shaky and dangerous to rely on. With no locking, you will easily get overwrites and you have to be very sure that your application doesn't mess up the DB. I haven't looked at mysql 4.1 clustering, but from the little I have seen, it looks good. Is that what you use? I agree. That is why we use MySQL 4.1. I believe Paul uses forward_tcp() and save_memory() to save the location to the replicated server's memory, while the save("location") on the primary server will store to the DB (which then replicates on the DB level). My ser.cfg uses t_replicate() and save_memory() with our urecord.c patch applied. save_memory() does not work when the db_mode is write-through. What happens is save() is called from SIP-01 which does either an INSERT or UPDATE sql on the location table. But save_memory() attempts to do the same thing on SIP-02 (only in write-through mode). So we had to patch usrloc's urecord.c file to have it honor a flag called FL_MEM because if I call save_memory("location") I demand that ser only save to memory - but it doesn't so this, IMHO is a bug. I posted a patch for this last Thursday to serdev, but unfortunately we identified another related bug in ser, so a revised patch is needed, which I'll try to post tomorrow. Regards, Paul g-) _______________________________________________ Serusers mailing list serusers(a)lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers --------------------------------- Do you Yahoo!? Yahoo! Small Business - Try our new resources site!

Thx!!! Java Rockx <javarockx(a)gmail.com> wrote:Well it's going on 1:30AM here in the US so when our Red Hat engineer gets in I'll have him give me the final patch for the save_memory() bug. I'll post it to serdev then. It should be about 12PM local time on the US east coast. Regards, Paul On Apr 10, 2005 11:10 PM, Tina <kramarv(a)yahoo.com> wrote:Paul, I appreciate a lot this information, and am interested in your patch a lot. Tina Java Rockx <javarockx(a)gmail.com> wrote: See inline. On Apr 10, 2005 5:32 AM, Greger V. Teigre <greger(a)teigre.com> wrote: > Greger, thanks a lot. Yes, if you use LVS with direct routing or tunneling, that is what you experience. What I described was a "generic" SIP-aware load balancer where SIP messages would be rewritten and stickiness implemented based on ex. UA IP address (or call-id like vovida's load balancer). Well, IMO, DNS SRV in itself is not bad. It's just that many user clients do not support DNS SRV yet. Except that, I like the concept and it will give you a geographical redundancy and load balancing. Are you referring to the load balancer proxy? IMHO, the SIP-aware load balancer makes things a bit messy. It sounds to me that the LVS + tunneling/direct routing + virtual IP on dummy adapter is a better solution. I'm not sure if I understand. So, you have 2 servers at two location, each location with a shared DB and then replication across an IPsec tunnel?? IMHO, mysql 3.23.x two-way replication is quite shaky and dangerous to rely on. With no locking, you will easily get overwrites and you have to be very sure that your application doesn't mess up the DB. I haven't looked at mysql 4.1 clustering, but from the little I have seen, it looks good. Is that what you use? I agree. That is why we use MySQL 4.1. I believe Paul uses forward_tcp() and save_memory() to save the location to the replicated server's memory, while the save("location") on the primary server will store to the DB (which then replicates on the DB level). My ser.cfg uses t_replicate() and save_memory() with our urecord.c patch applied. save_memory() does not work when the db_mode is write-through. What happens is save() is called from SIP-01 which does either an INSERT or UPDATE sql on the location table. But save_memory() attempts to do the same thing on SIP-02 (only in write-through mode). So we had to patch usrloc's urecord.c file to have it honor a flag called FL_MEM because if I call save_memory("location") I demand that ser only save to memory - but it doesn't so this, IMHO is a bug. I posted a patch for this last Thursday to serdev, but unfortunately we identified another related bug in ser, so a revised patch is needed, which I'll try to post tomorrow. Regards, Paul g-) _______________________________________________ Serusers mailing list serusers(a)lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers --------------------------------- Do you Yahoo!? Yahoo! Small Business - Try our new resources site! __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

Tina, I'm not really sure what you would like to do. Is it related to replication? The problem of getting through a NAT from a SER server (replicated), but where the client has not registered, is that you really don't get the redundancy you want. If the server where one client is registered goes down, that client cannot be reached even if you stand on your head yelling "please!" (of course, unless you do load balancing as described in my just recent email). The restricted NAT in front of the client will look at the IP address of the SER replication server and say "nope." Anyway, have you looked at the new LCR module? (for the HEAD CVS, but backported, you can find the backport at http://onsip.org) g-) ---- Original Message ---- From: Tina To: Greger V. Teigre Cc: serusers(a)lists.iptel.org Sent: Friday, April 08, 2005 02:11 AM Subject: Re: [Serusers] still no help - usrloc synchronization