[Serusers] still no help - usrloc synchronization
Matt Schulte
mschulte at netlogic.net
Tue Apr 12 14:31:22 CEST 2005
Forgive my ignorance, why is NAT a bottleneck? Are you refering to CPU
load or managibility? With alot of load balancers, NAT is the only way
to go.
-----Original Message-----
From: Tina [mailto:kramarv at yahoo.com]
Sent: Monday, April 11, 2005 11:37 AM
To: Java Rockx
Cc: serusers at lists.iptel.org
Subject: Re: [Serusers] still no help - usrloc synchronization
Pretty good news, Paul. I've only tested IP tunneling. It worked
for me. The problem I described was born in my thoughts. I've also read
some complaints from SIP-LVS users. The only configuration where Call-Id
stickness does not necessary is LVS-NAT. Unfortunately, NAT becomes
bottleneck very fast.
Java Rockx <javarockx at gmail.com> wrote:
Tina,
I wish I had more information on that, but that LVS
stuff is a black art to me. One of our engineers whipped up an LVS
configuration that seems quite happy with the Call-ID "stickness" you
described. All I can say right now is that it is apparently possible to
do this with 100% open source. (i'm keeping my fingers crossed) :-)
Regards,
Paul
On Apr 10, 2005 11:03 PM, Tina <kramarv at yahoo.com>
wrote:
Paul,
thanks a lot, it helps a lot. The thing I do not
understand - how you are going to solve call-id stickness (see my
another post). Since we're also use LVS I had to find something
proprietary, I am afraid this is not a best solution, but the only I
have today.
KRs,
Tina
Java Rockx <javarockx at gmail.com> wrote:
Tina,
I really don't know how the LVS server
is configured because our network guy that we acquired from RedHat set
all that stuff up. I do know that we basically have this set up:
I hope the formatting goes well :-)
+-----------+ +----------+
+-------+ +---------+ +--------------------+
| internet |------| Cisco |------| LVS
|-----| Cisco |----| Application |
| cloud | | 3600 | |
| | 3600 | | & DB Servers |
+-----------+ +----------+
+--------+ +---------+ +--------------------+
So the "Application & DB Servers" box
represents multiple servers (all are dedicated to their service). These
include SER, MySQL, Apache, configuration management, monitoring, RTP
proxies, etc.
MySQL is active-active so we have
two-way replication. We only have 2 MySQL servers. And this is all we
plan on ever having as MySQL is more than capable of handling anything
ser can throw at it (given the right hardware).
Each ser server is 100% oblivious to
other SER servers except for when handling REGISTER messages. Here is
basically the ser.cfg for each SER server:
listen=10.3.0.221 <http://10.3.0.221/>
# this will be 10.3.0.222 <http://10.3.0.222/> on the peer sip proxy
modparam("usrloc", "db_mode", 1) #
write-through
route {
if (method=="REGISTER") {
if (src_ip==10.3.0.221
<http://10.3.0.221/> ) { # ip of peer ser proxy
save_memory("location");
break;
} else {
save("location");
t_replicate("10.3.0.222 <http://10.3.0.222/> ");
break;
};
};
}
We have identifed a deficiency in the
usrloc module urecord.c file whereby when using write-though mode the
save_memory() function still attempt to update MySQL. This is a very bad
thing because it causes a primary key violation on the location table.
Yesterday I posted a patch to serdev to
fix this, but today we identified another case where save_memory() tried
to insert in to the location table. I'll post a revised patch to serdev
as soon as I get a chance.
So here is what happens. Assume I have
two sip routers and two MySQL servers as:
sip-01 is 10.3.0.221
<http://10.3.0.221/>
sip-02 is 10.3.0.222
<http://10.3.0.222/>
db-01 is 10.2.0.21 <http://10.2.0.21/>
db-02 is 10.2.0.22 <http://10.2.0.22/>
Now let's just assume that sip-01 is
connected to db-01 and sip-02 is connected to db-02.
When a SIP UA connects to a sip router
(via LVS), assume sip-01 in this example and REGISTERs, sip-01 will call
save("location") which updates/inserts to the location table. sip-01
then calls t_replicate to sip-02.
sip-02 recieves the REGISTER message and
calls save_memory("location") to update its in-memory cache only. It
should never hit db-02.
Now db-01 eventually (within a second or
two) replicates to db-02 and now sip-02 is good to go, even in the event
of a restart.
Our difficulties mostly surrounded the
fact th at ser was inserting the physical IP of eth0 on the server as
the top VIA rather than the VIP as required. We solved this by using
record_route_preset() and passing the VIP as the IP. We also had to
bring up a dummy interface in order to get ser to honor the VIP.
Hope this helps.
Regards,
Paul
On Apr 8, 2005 7:19 PM, Tina
<kramarv at yahoo.com> wrote:
Paul,
we are using LVS as well. It creates
template connection according to source Ip (e.g. UAC) and forwards to
one of the tunneled IP. I tested this a couple of weeks ago, ser was
happy and inserted VIP into via header, the only issue is to route the
calls into right server.
I agree, there is no such thing as
SIP-level clustering. However, how would you solve location problem by
DB replication - 1) usrloc does not update from DB....2) even though, I
am afraid it can become bottleneck and lock down users ...
KRs,
Tina
Java Rockx <javarockx at gmail.com> wrote:
See my inline comments.
Regards,
Paul
On Apr 8, 2005 1:08 PM, Greger V. Teigre
<greger at teigre.com> wrote:
Hi Tina,
> I enjoy reading your posts, thanks a
lot for the great work you are
> doing for ser users!
Thanks :-)
> I am going to consider that kind of
scenario as well (load balancer
> returning Virtual IP address). I think
for such configuration server
> side has to keep session information
and always forward a call to the
> right server.
I believe the load balancer can be in
front and transparantly translate addresses in the SIP messages:
User agent always sees a.b.c.d
|
Load balancer with public IP a.b.c.d
| |
ser1 (10.0.0.2 <http://10.0.0.2/> )
ser2 (10.0.0.3 <http://10.0.0.3/> )
Paul: You have a similar load-balancing
solution, right? Do you use a commerical load balancer that you can
reveal the name of? :-)
Our platform is 100% open source. Our
load balancer is LVS. I have no idea how it actually works. One if our
engineers from RedHat set all that networking stuff up and it's way
beyond me.
The load balancer need to understand SIP
(sort of) and use something (ex. IP address of user agent) to select SER
server to send to and rewrite incoming to 10.0.0.2 <http://10.0.0.2/>
and 10.0.0.3 <http://10.0.0.3/> respectively and outgoing to a.b.c.d.
ser1 and ser2 will happily believe they are alone and communicate
directly with user agent using their own private addresses. Both should
know about all users (usrloc). Paul just recently posted a small patch
for a setup where they use two-way replication between two
That small patch will need a slight
revision. We discovered through further testing that that patch only
fixed attempts to INSERT into the location table. We didn't cover the
UPDATE route, unfortunately. I believe Andrew has that patch done and is
testing it. So I'll post it to serdev as soon as we know it works -
possibly later today.
mysql servers and where each SER
instance will save all REGISTERs it receives from user agents, but only
save to memory those replicated from the other SER peer. Klaus pointed
out that the same can be done by not doing mysql replication, but have
one DB for each SER and just use t_replicate() and save_noreply() (for
two servers, for more than two you need to use forward_tcp()).
I have heard complaints that this
approach will generate too much network traffic. Also, t_replicate()
does not guarantee delivery of the replication (forward_tcp is better,
but not fool proof). So, the "best" solution would probably be to hope
that Paul's
I fully agree. Replication at the sip
proxy level, IMHO, is not a good idea. From a purist point of view,
replication is not a function of a SIP router - nor should it be. It is
a function of the database and we believe that the MySQL replication is
a better approach from a speed and reliabilty point of view - not to
mention that it simplifies the tasks of the sip router.
recent TCP/UDP patch for network access
to FIFO commands get included in the CVS. Then, a new replication module
should use XMLRPC or whatever protocol the patch is using to forward (in
a guaranteed mode) the usrloc information, which then would be injected
directly into SER's memory, just like save() would, and of course in
whatever db_mode you would like. The replicaton module would have to
keep a queue (of course, flushed to a DB or something) in case of
network problems, so that the replication would be done (sooner or
later) regardless of problems.
Of course, this is a major
undertaking, so for most replication scenarioes, I would think the
forward_tcp() + save_noreply() would work just fine.
g-)
> "Greger V. Teigre" <greger at teigre.com>
wrote:
> I was thinking about a load balancing
scenario where the load
> balancer will replace the IP
addresses.
> g-)
>
> ---- Original Message ----
> From: Tina
> To: Greger V. Teigre
> Sent: Wednesday, April 06, 2005 09:45
PM
> Subject: Re: [Serusers] still no help
- usrloc synchronization
>
>> Thank you for givingme the scenario
with "restricted IP" NAT, I am
>> starting to find some acceptable
solution.
>> Unfortunately, "one-public-IP"
approach is not free from problems
>> also. If your SIP router inserts this
"one-public-IP" into the VIA
>> header, the reply routing goes via
wrong SIP server...
>> If your SIP server inserts its
real-IP-address - the scenario
>> mentioned above is s till not
resolved.
>> Any comments?
>> Tina
>>
>> "Greger V. Teigre"
<greger at teigre.com> wrote:
>> See inline.
>>
>>> If you use DNS server for load
balancing... the client receives one
>>> of your domain IP addresses
according to SRV. I don't see the
>>> problem
>>> with a call here, cause UAC asks t!
he address only once (before
>>> sending INVITE). UAC already has the
IP for BYE/reINVITEs. So why
>>> would you replicate INVITEs?
>>
>> I would never replicate INVITEs, I
would just make sure that they are
>> proxied through the correct SER
server (i.e. IP).
>>
>> The problems depends on your setup.
If you have SERs with different
>> IPs, ex UA1 has registered with
server A and UA2 ha s registered with
>> server B: If UA2 wants to call UA1
and UA is behind an IP restricted
>> NAT, server A is stored in the NAT
table of the NAT in front of UA1.
>> If server B sends an INVITE to UA1,
the INVITE will be refused by
>> UA1's NAT.
>> This is why a "one public IP" in
front of a load balancing
>> cluster probably is a good way to go.
>>
>>> If you use IPVS/LVS... I believe you
can force SER to insert it's
>>> public IP into VIA,! so there is no
problem with replies. With
>>> regard
>>> to another requests, I believe load
balancer keeps connection
>>> template, then when another request
comes it would be forwarded to
>>> the same ser.
>>
>> Yes. There are different "keys" to
use to load balance SIP messages.
>> One good way from a NAT point of view
is to use originating IP
> > address. What you must remember is
that the problem is not on the
>> server side, but on the client side.
The NAT will in many situations
>> stop incoming UDP packets if the
originating ip:port is not already
>> stored in the NAT table. The Via
header does not matter for the NAT.
>> g-)
>>
>>> Any comments?
>>>
>>> "Greger V. Teigre"
<greger at teigre.com> wrote:
>>> Yes, I believe that is so. But still
you get a problem if the NAT is
>>> restricted, port-restricted or
symmetric... The best would be to
>>> load>> balance and always make sure
that a given client is handled
>>> through a given
>>> SER (REGISTER and INVITEs). That
includes forwarding INVITEs from
>>> one
>>> SER to
>>> another... OR you must load balance
in front of your servers with
>>> one
>>> common
>>> public IP.
>>> g-)
>>>
>>> Matt Schulte wrote:
>>>> Ack, I didn't even think about NAT.
Would these be added before it
>>>> gets sent off to the second proxy?
ie:
>>>>
>>>> if (!src_ip==blah.netlogic.net
<http://blah.netlogic.net/> ) {
>>>> add_rcv_param();
>>>> t_replicate("blah.netlogic.net
<http://blah.netlogic.net/> ", "999");
>>>> };
>>>>
>>>> -----Original Message-----
>>>> From: Greger V. Teigre
[mailto:greger at teigre.com <mailto:greger at tei gre.com> ]
>>>> Sent: Tuesday, April 05, 2005 7:49
AM
>>>> To: Matt Schulte; kramarv at yahoo.com
>>>> ! ; Cc: serusers at lists.iptel.org
>>>> Subject: ! Re: [Serusers] still no
help - usrloc synchronization
>>>>
>>>>
>>>> Well, you still have the NAT issues
unless you do load balancing
>>>> and your
>>>> SER servers have the same public
IP.
>>>> Have you looked at 0.9.0 nathelper
function add_rcv_param() ? It
>>>> will add received info to the
contact header for the other SER to
>>>> process. Haven't really tried
yet...
>>>> g-)
>>>>
>>>> Matt Schulte wrote:
>>>>> I'm starting to lean this
direction, using t_replicate and all. I
>>>>> could never get usrloc (db mode)
to function properly..
>>>>> t_replicate is
>>>>
>>>>> a dirty but very effective
workaround.
>>>>>
>>>>> -----Original Message-----
>>>>> From: Greger V. Teigre
[mailto:greger at teigre.com]
>>>>> Sent: Saturday, April 02,! 2005
1:33 AM
>>>>> To: kramarv at yahoo.com
>>>>> Cc: serusers at lists.iptel.org
>>>>> Subject: Re: [Serusers] still ! no
help - usrloc synchronization
>>>>>
>>>>>
>>>>> Have a look at this thread:
>>>>>
http://lists.iptel.org/pipermail/serusers/2005-January/014669.html
>>>>> g-)
>>>>>
>>>>> Java Rockx wrote:
>>>>>> Tina,
>>>>>>
>>>>>> I thought I saw you post the
other day that you did not want to
>>>>>> use t_replicate(), however, this
is probably your best bet to
>>>>>> getting this
>>>>>
>>>>>> to work, IMHO.
>>>>>>
>>>>>> Regards,
>>>>>> Paul
>>>>>>
>>>>>> On Apr 1, 2005 4:08 PM, Tina
wrote:
>>>>> ! >>
>>>>>> ! ;> Hi, please help me, I'm
stuck with it!!!!!
>>>>>>> I am trying to set up several
sers with a shared MySQL database
>>>>>>> for location service.
>>>>>>>
>>>>>>> I set in each ser.cfg:
>>>>>>>
>>>>>>> modparam("usrloc", "db_mode", 2)
>>>>>>> modparam("usrloc",
>>>>>>>
"db_url","sql://ser:heslo@192.168.25.163/ser")
>>>>>>>
>>>>>>> and the servers are not
synchronized.
>>>>>>> The I set
>>>>>>> modparam("usrloc", "db_mode", 2)
>>>>>>>
>>>>>>>
>>>>>>> made UAC (Xlite) register to one
of the servers.
>>>>>>> I see it via usrloc, but there
is no record in "location" mySQL
>>>>>>> table....So others do not see
the client and I'm unable to make
>>>>>>> calls....
>>>>>>>
>>>>>>>
>>>>>>> Please help how to work with
usrloc and mySQL...
>>>>>>>
>>>>>>> Tina,
>>>>>>> software engineer
>>>>>>>
>>>>>>> ________________________________
>>>>>>> Do you Yahoo!?
>>>>>>> Better first dates. More second
dates. Yahoo! Personals
>>>>>>>
>>>>>>>
>>>>>>>
_______________________________________________
>>>>>>> Serusers mailing list
>>>>>>> serusers at lists.iptel.org
>>>>>>>
http://lists.iptel.org/mailman/listinfo/serusers
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
_______________________________________________
>>>>>> Serusers mailing list
>>>>>> serusers at lists.iptel.org
>>>>>>
http://lists.iptel.org/mailman/listinfo/serusers
>>>>>
>>>>>
_______________________________________________
>>>>> Serusers mailing list
>>>>> serusers at lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serusers
>>>
>>>
>>>
>>>
>>> Yahoo! Messenger
>>> Show us what our next emoticon
should look like. Join the fun.
>>
>>
>> Do you Yahoo!?
>> Better first dates. More second
dates. Yahoo! Personals
>
>
> Yahoo! Messenger
> Show us what our next emoticon should
look like. Join the fun.
_______________________________________________
Serusers mailing list
serusers at lists.iptel.org
http://lists.iptel.org/mailman/listinfo/serusers
_____
Do you Yahoo!?
Make Yahoo! your home page
<http://us.rd.yahoo.com/my/navbar/sethp/*http://www.yahoo.com/r/hs>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam
protection around
http://mail.yahoo.com <http://mail.yahoo.com/>
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20050412/9392d8b2/attachment.htm>
More information about the sr-users
mailing list