Hey All,
This is a long email, so please bare with me...
Up until now we have used our primary SER serve as both registrar and call routing engine. However, we've recently hit a growth point where the registration load on our servers is high enough that it's affecting our call routing.
We currently have about 4,000 IAD's all registering every 5 minutes. This works out to be almost 13.5 regs every second, which in itself is a lot .. If they were sequential. However, at times we will get hundreds of IADs all attempting to register simultaneously (within seconds of each other) and it will keep the server busy for a few moments and cause SIP timeouts on calls that are in the process of being setup.
Currently I am using the auth_db module with MySQL for client registration authentication. I believe that the MySQL authentication is what is causing the high load on the server.
What I want to do is try to offload the the registrations from our primary routing server, yet, I need the primary server to be stateful of the all registered clients. -or- Find a more efficient way to authenticate client registrations so that the load is not so high.
Problem is that most of our customers are home-based VoIP telephone customers behind a Linksys cable/DSL router. So, in order to avoid NAT issues we must have all of our traffic which terminates to the IAD from our PSTN provider come from the same IP address that the the IAD registered from. So I can't use another server and then t_replicate the requests back to the routing server .. Because then the registrar / IP address that will be open on the customers router NAT gateway / firewall will differ from the IP that will terminate the traffic to the IAD from our system.
So - my question is .. How do I take load off of the primary server??
Would implementing radius for authentication fix this problem? I am very familiar with radius (using Freeradius) and could easily set this up, but I would prefer not to go to the trouble if it's not going to help with the load.
Has anyone else had an issue with client authentication via MySQL and auth_db causing a high load with SER?
I am hoping that you fine folks might be able to point me in the right direction. Thanks so much!
Darren Nay VoIP Network Development Ionosphere, Inc dnay@ionosphere.net
serusers-bounces@lists.iptel.org wrote:
Hey All,
This is a long email, so please bare with me...
Up until now we have used our primary SER serve as both registrar and call routing engine. However, we've recently hit a growth point where the registration load on our servers is high enough that it's affecting our call routing.
We currently have about 4,000 IAD's all registering every 5 minutes.
why every 5 minutes?
If you set it to highr figure, that will automatically reduce registration load !
So - my question is .. How do I take load off of the primary server??
Would implementing radius for authentication fix this problem? I am very familiar with radius (using Freeradius) and could easily set this up, but I would prefer not to go to the trouble if it's not going to help with the load.
Has anyone else had an issue with client authentication via MySQL and auth_db causing a high load with SER?
is your MYSQL on same host as SER?
Senad, Thanks for your response.
There are several reasons.
The 2 biggest are..
1 - We offer our customers the option to forward calls when an IAD is not registered .. In order to do that the registration interval needs to be relatively low.
2 - We've had some issues with setting registration intervals higher than 10 minutes. It seems to work about 99.5% of the time, but occaisionally an endpoint won't reregister before the reg interval on SER times out. I believe this is a bug in our endpoint and have been working with them on that. Setting the reg interval below 10 minutes has eliminated that problem for the time being .. However, even if this problem was fixed by the IAD manufacturer we still have the first reason (above) that would keep us from being able to increase the interval.
Darren
On 5/16/05 1:23 PM, "Senad J" senad@bicom.us wrote:
serusers-bounces@lists.iptel.org wrote:
Hey All,
This is a long email, so please bare with me...
Up until now we have used our primary SER serve as both registrar and call routing engine. However, we've recently hit a growth point where the registration load on our servers is high enough that it's affecting our call routing.
We currently have about 4,000 IAD's all registering every 5 minutes.
why every 5 minutes?
If you set it to highr figure, that will automatically reduce registration load !
So - my question is .. How do I take load off of the primary server??
Would implementing radius for authentication fix this problem? I am very familiar with radius (using Freeradius) and could easily set this up, but I would prefer not to go to the trouble if it's not going to help with the load.
Has anyone else had an issue with client authentication via MySQL and auth_db causing a high load with SER?
is your MYSQL on same host as SER?
Darren Nay wrote:
Senad, Thanks for your response.
There are several reasons.
The 2 biggest are..
1 - We offer our customers the option to forward calls when an IAD is not registered .. In order to do that the registration interval needs to be relatively low.
Fine... But still I am curious why u need to do in order to forward the calls.
2 - We've had some issues with setting registration intervals higher than 10 minutes. It seems to work about 99.5% of the time, but occaisionally an endpoint won't reregister before the reg interval on SER times out. I believe this is a bug in our endpoint and have been working with them on that. Setting the reg interval below 10 minutes has eliminated that problem for the time being .. However, even if this problem was fixed by the IAD manufacturer we still have the first reason (above) that would keep us from being able to increase the interval.
Well. apart from fixing the issue with your IAD, you could use DNS SRV record. Each time, IAD wants to re-register it will use DNS SRV record, hence "hitting" different server.
This way, u will need more than one SER server each getting its data from central network database. If you need cluster file system, then you could use GFS or similar.
Regards,
Senad
PSS!!! Which IAD are you using ?
SER should be able to handle more than that, especially (am guessing here, since I dont have any stats) since after REGISTER there are far more MESSAGES which hit the server, and even possibly hit the acc DB that the register, eg for a call setup, you have all those INVITES and there ACK's BYEs etc, which all go through ser, and require it to process far more based upon what the request is, as opposed to ser.cfg just looking up REGISTER method.
When u say u get timeouts, have u checked mysql at that point, although it shouldnt have a problem, bu maybe max_connects on mysql maybe causing a problem, worth a quick look.
As for separate REGISTER server, can you change the configs on the devices, just (this is a poor mans solutions...hence i would use it :-)) ask some to register on sip1.domain.com and the otheres sip2.domain.com, or if they come from blocks of IP address which you know, route them locally to different ser servers based upon IP address.
I am still hedging my best on Mysql though
Iqbal
Darren Nay wrote:
Hey All,
This is a long email, so please bare with me...
Up until now we have used our primary SER serve as both registrar and call routing engine. However, we've recently hit a growth point where the registration load on our servers is high enough that it's affecting our call routing.
We currently have about 4,000 IAD's all registering every 5 minutes. This works out to be almost 13.5 regs every second, which in itself is a lot .. If they were sequential. However, at times we will get hundreds of IADs all attempting to register simultaneously (within seconds of each other) and it will keep the server busy for a few moments and cause SIP timeouts on calls that are in the process of being setup.
Currently I am using the auth_db module with MySQL for client registration authentication. I believe that the MySQL authentication is what is causing the high load on the server.
What I want to do is try to offload the the registrations from our primary routing server, yet, I need the primary server to be stateful of the all registered clients. -or- Find a more efficient way to authenticate client registrations so that the load is not so high.
Problem is that most of our customers are home-based VoIP telephone customers behind a Linksys cable/DSL router. So, in order to avoid NAT issues we must have all of our traffic which terminates to the IAD from our PSTN provider come from the same IP address that the the IAD registered from. So I can't use another server and then t_replicate the requests back to the routing server .. Because then the registrar / IP address that will be open on the customers router NAT gateway / firewall will differ from the IP that will terminate the traffic to the IAD from our system.
So - my question is .. How do I take load off of the primary server??
Would implementing radius for authentication fix this problem? I am very familiar with radius (using Freeradius) and could easily set this up, but I would prefer not to go to the trouble if it's not going to help with the load.
Has anyone else had an issue with client authentication via MySQL and auth_db causing a high load with SER?
I am hoping that you fine folks might be able to point me in the right direction. Thanks so much!
Darren Nay VoIP Network Development Ionosphere, Inc dnay@ionosphere.net
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
.
Any modern day server should be able to handle your SER requirements quite easily. I would try to increase the SER children to see if it helps.
I have heard of stats as high as 2000 call setups per minute on a high end box. So a few hundred registers per second should be easy to handle. Unless of course your MySQL database is many milliseconds away.
As several people have pointed out, you have a bottleneck somewhere in your system. I have not seen numbers on the penalty you get for using mysql for authentication, I wouldn't think it is much. However, if you get hundreds if registration attempts simultaneously, you will reach the limit, sooner or later. I would guess you get your problems with the database lookups. Try hitting mysql with a high load directly and performance tune mysql. Take a look at this: http://dev.mysql.com/doc/mysql/en/server-parameters.html You should most definitely increase max_connections as mentioned earlier. Also, with 4,000 accounts, mysql should be able to cache all the users.
Would appreciate you reporting back to your list on your results! g-)
Darren Nay wrote:
Hey All,
This is a long email, so please bare with me...
Up until now we have used our primary SER serve as both registrar and call routing engine. However, we've recently hit a growth point where the registration load on our servers is high enough that it's affecting our call routing.
We currently have about 4,000 IAD's all registering every 5 minutes. This works out to be almost 13.5 regs every second, which in itself is a lot .. If they were sequential. However, at times we will get hundreds of IADs all attempting to register simultaneously (within seconds of each other) and it will keep the server busy for a few moments and cause SIP timeouts on calls that are in the process of being setup.
Currently I am using the auth_db module with MySQL for client registration authentication. I believe that the MySQL authentication is what is causing the high load on the server.
What I want to do is try to offload the the registrations from our primary routing server, yet, I need the primary server to be stateful of the all registered clients. -or- Find a more efficient way to authenticate client registrations so that the load is not so high.
Problem is that most of our customers are home-based VoIP telephone customers behind a Linksys cable/DSL router. So, in order to avoid NAT issues we must have all of our traffic which terminates to the IAD from our PSTN provider come from the same IP address that the the IAD registered from. So I can't use another server and then t_replicate the requests back to the routing server .. Because then the registrar / IP address that will be open on the customers router NAT gateway / firewall will differ from the IP that will terminate the traffic to the IAD from our system.
So - my question is .. How do I take load off of the primary server??
Would implementing radius for authentication fix this problem? I am very familiar with radius (using Freeradius) and could easily set this up, but I would prefer not to go to the trouble if it's not going to help with the load.
Has anyone else had an issue with client authentication via MySQL and auth_db causing a high load with SER?
I am hoping that you fine folks might be able to point me in the right direction. Thanks so much!
Darren Nay VoIP Network Development Ionosphere, Inc dnay@ionosphere.net
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Also if possible do a quick debug on slow queries in mysql, to see if any of the tables might be ndexed incorrectly.
Iqbal
Greger V. Teigre wrote:
As several people have pointed out, you have a bottleneck somewhere in your system. I have not seen numbers on the penalty you get for using mysql for authentication, I wouldn't think it is much. However, if you get hundreds if registration attempts simultaneously, you will reach the limit, sooner or later. I would guess you get your problems with the database lookups. Try hitting mysql with a high load directly and performance tune mysql. Take a look at this: http://dev.mysql.com/doc/mysql/en/server-parameters.html You should most definitely increase max_connections as mentioned earlier. Also, with 4,000 accounts, mysql should be able to cache all the users.
Would appreciate you reporting back to your list on your results! g-)
Darren Nay wrote:
Hey All,
This is a long email, so please bare with me...
Up until now we have used our primary SER serve as both registrar and call routing engine. However, we've recently hit a growth point where the registration load on our servers is high enough that it's affecting our call routing.
We currently have about 4,000 IAD's all registering every 5 minutes. This works out to be almost 13.5 regs every second, which in itself is a lot .. If they were sequential. However, at times we will get hundreds of IADs all attempting to register simultaneously (within seconds of each other) and it will keep the server busy for a few moments and cause SIP timeouts on calls that are in the process of being setup.
Currently I am using the auth_db module with MySQL for client registration authentication. I believe that the MySQL authentication is what is causing the high load on the server.
What I want to do is try to offload the the registrations from our primary routing server, yet, I need the primary server to be stateful of the all registered clients. -or- Find a more efficient way to authenticate client registrations so that the load is not so high.
Problem is that most of our customers are home-based VoIP telephone customers behind a Linksys cable/DSL router. So, in order to avoid NAT issues we must have all of our traffic which terminates to the IAD from our PSTN provider come from the same IP address that the the IAD registered from. So I can't use another server and then t_replicate the requests back to the routing server .. Because then the registrar / IP address that will be open on the customers router NAT gateway / firewall will differ from the IP that will terminate the traffic to the IAD from our system.
So - my question is .. How do I take load off of the primary server??
Would implementing radius for authentication fix this problem? I am very familiar with radius (using Freeradius) and could easily set this up, but I would prefer not to go to the trouble if it's not going to help with the load.
Has anyone else had an issue with client authentication via MySQL and auth_db causing a high load with SER?
I am hoping that you fine folks might be able to point me in the right direction. Thanks so much!
Darren Nay VoIP Network Development Ionosphere, Inc dnay@ionosphere.net
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
.
-
Andres -
Darren Nay -
Greger V. Teigre -
Iqbal -
Senad J