I have four different x86_64 boxes which perform identically. I don't think it is a hardware issue... although it could be a driver issue of some kind. All the hardware on those four are similar (Dell rack mount servers, not all the same model but similar).
If I get a chance this weekend I'll load the i386 distro on the Opteron based server I took out of production and see what that does.
Michael
-----Original Message----- From: Mik Cheez [mailto:michael_bulk@wildgate.com] Sent: Thursday, May 29, 2008 7:36 PM To: Michael Young Cc: users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it
Wow...that's interesting. Have you tried running i386 on the first box? If that works you may be correct, but if the problem persists you may have to consider a hardware issue. If not that, then perhaps a different 64 bit Linux distribution...
Michael Young wrote:
More information on the performance problems I have been fighting:
I replaced one of my OpenSER production boxes with a new server today. The old one was running x86_64 version of CentOS 5 (two dual core Opteron processors). The new one is running the i386 flavor of CentOS 5 (one quad core Xeon processor). I copied the configs from the old box to the new
one.
The boxes were built identically using the same process and applications.
The new box is now matching the performance I was seeing on the 1.2.1 box
of
my provider. This new box is easily handling 50 new calls per second.
So this indicates that something is wrong with the combination of Linux x86_64 and OpenSER 1.3.X.
NOTE: I am NOT blaming OpenSER or saying that the problem lies with the OpenSER code. But something in the combination of those is causing a huge performance problem. For all I know the problem could be in an Ethernet driver in the x86_64 package.
Those of you who have done benchmarking lately -- were your benchmarks run on x86_64? I would be curious to know if others who have reported performance problems lately were on x86_64 (I thought at least one other report indicated they were using that build).
Michael Young
-----Original Message----- From: users-bounces@lists.openser.org [mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young Sent: Tuesday, May 27, 2008 8:16 AM To: 'Henning Westerholt'; users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it
Here is what I know (disclaimer: I am not a coder... as far as I am concerned C is for Cookie, and that's good enough for me):
One of my providers is a company I used to work for. They have a primary OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new INVITES per second. I have their ENUM configured to route half of those calls to my proxy1, and the other half to my proxy2.
At 25 cps (new calls per second), the load on their SER box is .1 to .2 at most. The processor rarely gets to 1% utilized. OpenSER is using between
200
and 400 Mb of RAM.
At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets
to
1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1 chokes and stops responding to invites. It is not the processor or the RAM that are limiting... there is something else involved that takes the load
up
to 2.0, at which point it stops responding.
I have stripped down my config, trying to find out where the problem is.
At
this point I have wild theories, but few cold hard facts.
Their OpenSER box is running ACC, and writing records to a MYSQL cluster that lives on a different box. Mine is not running ACC at all anymore.
They
use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor,
I
am using CentOS x86_64. I actually have a Xeon-based server on order, wondering if the problem is only on x86_64 based systems.
Increasing the UDP buffer size is the only thing that has gotten me to 10 cps, before I was dying at 3. My OpenSER proxies route calls to a server farm of ~70 Asterisk boxes. If one of those asterisk servers goes off
line,
my proxy's load will immediately go up to 2.0, the processor level will go to 100%, and it stops processing many (but not all) new calls. It seems to block some of the processes at this point, if I try to shut OpenSER down I have to manually kill 6 to 8 of the child processes.
Here are the modules I'm loading: loadmodule "mysql.so" loadmodule "sl.so" loadmodule "tm.so" loadmodule "rr.so" loadmodule "maxfwd.so" loadmodule "textops.so" loadmodule "mi_fifo.so" loadmodule "uri_db.so" loadmodule "uri.so" loadmodule "xlog.so" loadmodule "permissions.so" loadmodule "alias_db.so" loadmodule "domain.so"
Maybe this information can help someone else who is running into a similar problem.
Michael
-----Original Message----- From: Henning Westerholt [mailto:henning.westerholt@1und1.de] Sent: Tuesday, May 27, 2008 3:22 AM To: users@lists.openser.org Cc: Michael Young Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it
On Saturday 24 May 2008, Michael Young wrote:
I posted a message earlier this week, that said that my OpenSER install
is
not correctly calculating the buffer size as far as I can tell, but have not gotten a response on that yet. Performance is still not as good as previous OpenSER versions with this setup (we have a 1.2.1 server that is running circles around the three 1.3.2 servers).
Hi Michael,
i can confirm a (albeit minor) slowdown from 0.9 branch to actual
releases,
probably caused e.g. from the added pseudo-variable stuff, more DNS
lookups
and further abstractions. But a slowdown like this sounds more like a bug
to
me.
Do you can share more details about the problem?
Cheers,
Henning
No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM
No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM
Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM
No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM
Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM
No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM