[OpenSER-Users] Openser not processing requests at rate being sent to it

[Michael Young]

More information on the performance problems I have been fighting:

I replaced one of my OpenSER production boxes with a new server today. The
old one was running x86_64 version of CentOS 5 (two dual core Opteron
processors). The new one is running the i386 flavor of CentOS 5 (one quad
core Xeon processor). I copied the configs from the old box to the new one.
The boxes were built identically using the same process and applications.

The new box is now matching the performance I was seeing on the 1.2.1 box of
my provider. This new box is easily handling 50 new calls per second. 

So this indicates that something is wrong with the combination of Linux
x86_64 and OpenSER 1.3.X.

NOTE: I am NOT blaming OpenSER or saying that the problem lies with the
OpenSER code. But something in the combination of those is causing a huge
performance problem. For all I know the problem could be in an Ethernet
driver in the x86_64 package.

Those of you who have done benchmarking lately -- were your benchmarks run
on x86_64? I would be curious to know if others who have reported
performance problems lately were on x86_64 (I thought at least one other
report indicated they were using that build).

Michael Young



-----Original Message-----
From: users-bounces at lists.openser.org
[mailto:users-bounces at lists.openser.org] On Behalf Of Michael Young
Sent: Tuesday, May 27, 2008 8:16 AM
To: 'Henning Westerholt'; users at lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it

Here is what I know (disclaimer: I am not a coder... as far as I am
concerned C is for Cookie, and that's good enough for me):

One of my providers is a company I used to work for. They have a primary
OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new
INVITES per second. I have their ENUM configured to route half of those
calls to my proxy1, and the other half to my proxy2.

At 25 cps (new calls per second), the load on their SER box is .1 to .2 at
most. The processor rarely gets to 1% utilized. OpenSER is using between 200
and 400 Mb of RAM.

At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to
1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1
chokes and stops responding to invites. It is not the processor or the RAM
that are limiting... there is something else involved that takes the load up
to 2.0, at which point it stops responding.

I have stripped down my config, trying to find out where the problem is. At
this point I have wild theories, but few cold hard facts.

Their OpenSER box is running ACC, and writing records to a MYSQL cluster
that lives on a different box. Mine is not running ACC at all anymore. They
use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor, I
am using CentOS x86_64. I actually have a Xeon-based server on order,
wondering if the problem is only on x86_64 based systems.

Increasing the UDP buffer size is the only thing that has gotten me to 10
cps, before I was dying at 3. My OpenSER proxies route calls to a server
farm of ~70 Asterisk boxes. If one of those asterisk servers goes off line,
my proxy's load will immediately go up to 2.0, the processor level will go
to 100%, and it stops processing many (but not all) new calls. It seems to
block some of the processes at this point, if I try to shut OpenSER down I
have to manually kill 6 to 8 of the child processes.

Here are the modules I'm loading:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "mi_fifo.so"
loadmodule "uri_db.so"
loadmodule "uri.so"
loadmodule "xlog.so"
loadmodule "permissions.so"
loadmodule "alias_db.so"
loadmodule "domain.so"

Maybe this information can help someone else who is running into a similar
problem.

Michael


-----Original Message-----
From: Henning Westerholt [mailto:henning.westerholt at 1und1.de] 
Sent: Tuesday, May 27, 2008 3:22 AM
To: users at lists.openser.org
Cc: Michael Young
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it

On Saturday 24 May 2008, Michael Young wrote:
> I posted a message earlier this week, that said that my OpenSER install is
> not correctly calculating the buffer size as far as I can tell, but have
> not gotten a response on that yet. Performance is still not as good as
> previous OpenSER versions with this setup (we have a 1.2.1 server that is
> running circles around the three 1.3.2 servers).

Hi Michael,

i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases, 
probably caused e.g. from the added pseudo-variable stuff, more DNS lookups 
and further abstractions. But a slowdown like this sounds more like a bug to

me.

Do you can share more details about the problem?

Cheers,

Henning

No virus found in this incoming message.
Checked by AVG. 
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
 

No virus found in this outgoing message.
Checked by AVG. 
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
 


_______________________________________________
Users mailing list
Users at lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users

No virus found in this incoming message.
Checked by AVG. 
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
 

No virus found in this outgoing message.
Checked by AVG. 
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM