[OpenSER-Users] Openser not processing requests at rate being sent to it

[Michael Young]

I have four different x86_64 boxes which perform identically. I don't think
it is a hardware issue... although it could be a driver issue of some kind.
All the hardware on those four are similar (Dell rack mount servers, not all
the same model but similar).

If I get a chance this weekend I'll load the i386 distro on the Opteron
based server I took out of production and see what that does.

Michael

-----Original Message-----
From: Mik Cheez [mailto:michael_bulk at wildgate.com] 
Sent: Thursday, May 29, 2008 7:36 PM
To: Michael Young
Cc: users at lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it

Wow...that's interesting.  Have you tried running i386 on the first box? 
  If that works you may be correct, but if the problem persists you may 
have to consider a hardware issue.  If not that, then perhaps a 
different 64 bit Linux distribution...

Michael Young wrote:
> More information on the performance problems I have been fighting:
> 
> I replaced one of my OpenSER production boxes with a new server today. The
> old one was running x86_64 version of CentOS 5 (two dual core Opteron
> processors). The new one is running the i386 flavor of CentOS 5 (one quad
> core Xeon processor). I copied the configs from the old box to the new
one.
> The boxes were built identically using the same process and applications.
> 
> The new box is now matching the performance I was seeing on the 1.2.1 box
of
> my provider. This new box is easily handling 50 new calls per second. 
> 
> So this indicates that something is wrong with the combination of Linux
> x86_64 and OpenSER 1.3.X.
> 
> NOTE: I am NOT blaming OpenSER or saying that the problem lies with the
> OpenSER code. But something in the combination of those is causing a huge
> performance problem. For all I know the problem could be in an Ethernet
> driver in the x86_64 package.
> 
> Those of you who have done benchmarking lately -- were your benchmarks run
> on x86_64? I would be curious to know if others who have reported
> performance problems lately were on x86_64 (I thought at least one other
> report indicated they were using that build).
> 
> Michael Young
> 
> 
> 
> -----Original Message-----
> From: users-bounces at lists.openser.org
> [mailto:users-bounces at lists.openser.org] On Behalf Of Michael Young
> Sent: Tuesday, May 27, 2008 8:16 AM
> To: 'Henning Westerholt'; users at lists.openser.org
> Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
> sent to it
> 
> Here is what I know (disclaimer: I am not a coder... as far as I am
> concerned C is for Cookie, and that's good enough for me):
> 
> One of my providers is a company I used to work for. They have a primary
> OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new
> INVITES per second. I have their ENUM configured to route half of those
> calls to my proxy1, and the other half to my proxy2.
> 
> At 25 cps (new calls per second), the load on their SER box is .1 to .2 at
> most. The processor rarely gets to 1% utilized. OpenSER is using between
200
> and 400 Mb of RAM.
> 
> At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets
to
> 1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1
> chokes and stops responding to invites. It is not the processor or the RAM
> that are limiting... there is something else involved that takes the load
up
> to 2.0, at which point it stops responding.
> 
> I have stripped down my config, trying to find out where the problem is.
At
> this point I have wild theories, but few cold hard facts.
> 
> Their OpenSER box is running ACC, and writing records to a MYSQL cluster
> that lives on a different box. Mine is not running ACC at all anymore.
They
> use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor,
I
> am using CentOS x86_64. I actually have a Xeon-based server on order,
> wondering if the problem is only on x86_64 based systems.
> 
> Increasing the UDP buffer size is the only thing that has gotten me to 10
> cps, before I was dying at 3. My OpenSER proxies route calls to a server
> farm of ~70 Asterisk boxes. If one of those asterisk servers goes off
line,
> my proxy's load will immediately go up to 2.0, the processor level will go
> to 100%, and it stops processing many (but not all) new calls. It seems to
> block some of the processes at this point, if I try to shut OpenSER down I
> have to manually kill 6 to 8 of the child processes.
> 
> Here are the modules I'm loading:
> loadmodule "mysql.so"
> loadmodule "sl.so"
> loadmodule "tm.so"
> loadmodule "rr.so"
> loadmodule "maxfwd.so"
> loadmodule "textops.so"
> loadmodule "mi_fifo.so"
> loadmodule "uri_db.so"
> loadmodule "uri.so"
> loadmodule "xlog.so"
> loadmodule "permissions.so"
> loadmodule "alias_db.so"
> loadmodule "domain.so"
> 
> Maybe this information can help someone else who is running into a similar
> problem.
> 
> Michael
> 
> 
> -----Original Message-----
> From: Henning Westerholt [mailto:henning.westerholt at 1und1.de] 
> Sent: Tuesday, May 27, 2008 3:22 AM
> To: users at lists.openser.org
> Cc: Michael Young
> Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
> sent to it
> 
> On Saturday 24 May 2008, Michael Young wrote:
>> I posted a message earlier this week, that said that my OpenSER install
is
>> not correctly calculating the buffer size as far as I can tell, but have
>> not gotten a response on that yet. Performance is still not as good as
>> previous OpenSER versions with this setup (we have a 1.2.1 server that is
>> running circles around the three 1.3.2 servers).
> 
> Hi Michael,
> 
> i can confirm a (albeit minor) slowdown from 0.9 branch to actual
releases, 
> probably caused e.g. from the added pseudo-variable stuff, more DNS
lookups 
> and further abstractions. But a slowdown like this sounds more like a bug
to
> 
> me.
> 
> Do you can share more details about the problem?
> 
> Cheers,
> 
> Henning
> 
> No virus found in this incoming message.
> Checked by AVG. 
> Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
> 6:49 PM
>  
> 
> No virus found in this outgoing message.
> Checked by AVG. 
> Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
> 6:49 PM
>  
> 
> 
> _______________________________________________
> Users mailing list
> Users at lists.openser.org
> http://lists.openser.org/cgi-bin/mailman/listinfo/users
> 
> No virus found in this incoming message.
> Checked by AVG. 
> Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
> 6:49 PM
>  
> 
> No virus found in this outgoing message.
> Checked by AVG. 
> Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
> 7:27 AM
>  
> 
> 
> _______________________________________________
> Users mailing list
> Users at lists.openser.org
> http://lists.openser.org/cgi-bin/mailman/listinfo/users
> 
> 

No virus found in this incoming message.
Checked by AVG. 
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
 

No virus found in this outgoing message.
Checked by AVG. 
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM