Mik,

Response to your questions below:

-----Original Message----- From: Mik Cheez [mailto:michael_bulk@wildgate.com] Sent: Friday, May 23, 2008 3:34 PM To: Brian Peters Cc: users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

Brian Peters wrote:

I am doing what I thought was a simple volume test that OpenSER should be able to handle, but the results I am getting so far show it is not handling the load I'm sending it.

Here is my configuration:

•      OpenSER 1.3.1
  
  

•      Simple script that uses OpenSER as a proxy to forward 
  

requests to a carrier. All it does is rewrite the host and port of the R-URI and relays to the carrier.

•      children=16
  
  

•      shared memory set to 1 G. (-m 1000)
  
  

•      using transaction module to handle the requests.
  
  

•      Sending 10 INVITE requests per second to OpenSER using
  

Asterisk.

When I do a "tcpdump" trace of the network traffic to/from OpenSER, it is clear that a large percentage of the INVITE requests sent to OpenSER were not handled. Asterisk needed to resend the INVITE requests several times before OpenSER would handle them, and in about 20% of the cases Asterisk terminated the INVITES because it reached the timeout of 30 seconds without any response.

If I go directly from asterisk to the carrier, this test works without any issues. I'm not sure what the limitations are of OpenSER, but this seems like something it should be able to handle.

Any help or information on how to configure OpenSER so it will work would be appreciated.

Thanks,

Brian.

---

---

Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users

You have a config file to share? Are your network settings correct? Are you doing the dump on the OpenSER box, or the Asterisk? How are your system resources on the box (e.g. running 'top')?

10 invites per second is really nothing for OpenSER, so something else must be going on.

---

1. the network settings on the servers are OK. We have used the servers in production environment without OpenSER (asterisk direct to carrier) and at much higher volume than we are testing.

2. The tcpdump is being done from the OpenSER box so that I could see what OpenSER was receiving and sending. The server is clearly receiving the INVITE requests, and OpenSER is not responding.

3. System resources are not an issue. The server is only being used for OpenSER, and CPU usage is barely reaching 1%. Network is only being used for OpenSER.

4. for the config file, here it is (with the IP/hosts blanked out):

## begin openser script ##

# # $Id: openser.cfg 3542 2008-01-16 15:00:34Z miconda $ # # OpenSER basic configuration script # by Anca Vamanu anca@voice-system.ro # # Please refer to the Core CookBook at http://www.openser.org/dokuwiki/doku.php # for a explanation of possible statements, functions and parameters. #

####### Global Parameters #########

debug=3 fork=yes log_stderror=no

listen=udp:xxx.xxx.xxx.xxx:5060 port=5060 # disable_tcp=yes

alias=foo.com

sip_warning=0

check_via=no dns=off rev_dns=off children=8 log_name="openser-test.log" log_facility=LOG_LOCAL0

disable_dns_blacklist=yes disable_dns_failover=yes

####### Modules Section ########

#set module path mpath="/usr/local/openser-1.3.1//lib/openser/modules/"

loadmodule "textops.so" loadmodule "sl.so" loadmodule "maxfwd.so" loadmodule "uri.so" loadmodule "xlog.so"

# ----- mi_fifo ----- loadmodule "mi_fifo.so" modparam("mi_fifo", "fifo_name", "/tmp/openser_1.3.1_fifo")

# ----- rr ----- loadmodule "rr.so" modparam("rr", "append_fromtag", 0) modparam("rr", "enable_full_lr", 1)

# ----- tm ----- loadmodule "tm.so" modparam("tm", "fr_timer", 40) modparam("tm", "fr_inv_timer", 60) modparam("tm", "wt_timer", 20) modparam("tm", "noisy_ctimer", 1)

route{ xlog("New request - M=$rm RURI=$ru F=$fu T=$tu IP=$si ID=$ci\n");

if (!mf_process_maxfwd_header("10")) { sl_send_reply("483","Too Many Hops"); exit; }

if (has_totag()) { # sequential request withing a dialog should # take the path determined by record-routing if (loose_route()) { route(1); } else { if ( is_method("ACK") ) { if ( t_check_trans() ) { route(1); exit; } else { exit; } } sl_send_reply("403","No proper route"); } exit; }

#initial requests

# CANCEL processing if (is_method("CANCEL")) { if (t_check_trans()) route(1); exit; }

t_check_trans();

if (!is_method("INVITE")) { sl_send_reply("403","Not routable"); exit; }

record_route();

# route invites to carrier route(2); }

route[1] { if (!t_relay()) { sl_reply_error(); }; exit; }

route[2] { rewritehostport("yyy.yyy.yyy.yyy:5060");

xlog("routing INVITE request to carrier $ou -> $ru\n");

t_on_failure("1"); route(1); }

failure_route[1] { if (t_was_cancelled() || t_check_status("486|487")) { exit; }

xlog("Routing to carrier failed: $fu => $ru ($(T_reply_code))\n"); }

## end openser script ##

Thanks,

Brian.

Brian,

I just ran your config file, which is minimal feature wise, and had no issues. I checked out the latest SVN version of OpenSER, which you may want to do as well.

Short of that, how about raising the debug level and making it available.

DNS lookups, as Andreas mentions here, can absolutely cause the kind of problem seen here (though I believe it to be more common on Asterisk).

Andreas Heise wrote:

Hi Brian,

did you check if your OpenSER try to do a reserve DNS lookup even if rev_dns=no?

maybe related to your pb... http://openser.org/pipermail/users/2008-May/017397.html

regards, Andreas

Brian Peters schrieb:

...

Mik,

Response to your questions below:

-----Original Message----- From: Mik Cheez [mailto:michael_bulk@wildgate.com] Sent: Friday, May 23, 2008 3:34 PM To: Brian Peters Cc: users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

Brian Peters wrote:

...

I am doing what I thought was a simple volume test that OpenSER should be able to handle, but the results I am getting so far show it is not handling the load I'm sending it.

Here is my configuration:

•      OpenSER 1.3.1
  
  

•      Simple script that uses OpenSER as a proxy to forward 
  

requests to a carrier. All it does is rewrite the host and port of the R-URI and relays to the carrier.

•      children=16
  
  

•      shared memory set to 1 G. (-m 1000)
  
  

•      using transaction module to handle the requests.
  
  

•      Sending 10 INVITE requests per second to OpenSER using
  
  

Asterisk.

...

When I do a "tcpdump" trace of the network traffic to/from OpenSER, it is clear that a large percentage of the INVITE requests sent to OpenSER were not handled. Asterisk needed to resend the INVITE requests several times before OpenSER would handle them, and in about 20% of the cases Asterisk terminated the INVITES because it reached the timeout of 30 seconds without any response.

If I go directly from asterisk to the carrier, this test works without any issues. I'm not sure what the limitations are of OpenSER, but this seems like something it should be able to handle.

Any help or information on how to configure OpenSER so it will work would be appreciated.

Thanks,

Brian.

---

---

Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users

You have a config file to share? Are your network settings correct? Are you doing the dump on the OpenSER box, or the Asterisk? How are your system resources on the box (e.g. running 'top')?

10 invites per second is really nothing for OpenSER, so something else must be going on.

---

1. the network settings on the servers are OK. We have used the

servers in production environment without OpenSER (asterisk direct to carrier) and at much higher volume than we are testing.

2. The tcpdump is being done from the OpenSER box so that I could see

what OpenSER was receiving and sending. The server is clearly receiving the INVITE requests, and OpenSER is not responding.

3. System resources are not an issue. The server is only being used for

OpenSER, and CPU usage is barely reaching 1%. Network is only being used for OpenSER.

4. for the config file, here it is (with the IP/hosts blanked out):

## begin openser script ##

# # $Id: openser.cfg 3542 2008-01-16 15:00:34Z miconda $ # # OpenSER basic configuration script # by Anca Vamanu anca@voice-system.ro # # Please refer to the Core CookBook at http://www.openser.org/dokuwiki/doku.php # for a explanation of possible statements, functions and parameters. #

####### Global Parameters #########

debug=3 fork=yes log_stderror=no

listen=udp:xxx.xxx.xxx.xxx:5060 port=5060 # disable_tcp=yes

alias=foo.com

sip_warning=0

check_via=no dns=off rev_dns=off children=8 log_name="openser-test.log" log_facility=LOG_LOCAL0

disable_dns_blacklist=yes disable_dns_failover=yes

####### Modules Section ########

#set module path mpath="/usr/local/openser-1.3.1//lib/openser/modules/"

loadmodule "textops.so" loadmodule "sl.so" loadmodule "maxfwd.so" loadmodule "uri.so" loadmodule "xlog.so"

# ----- mi_fifo ----- loadmodule "mi_fifo.so" modparam("mi_fifo", "fifo_name", "/tmp/openser_1.3.1_fifo")

# ----- rr ----- loadmodule "rr.so" modparam("rr", "append_fromtag", 0) modparam("rr", "enable_full_lr", 1)

# ----- tm ----- loadmodule "tm.so" modparam("tm", "fr_timer", 40) modparam("tm", "fr_inv_timer", 60) modparam("tm", "wt_timer", 20) modparam("tm", "noisy_ctimer", 1)

route{ xlog("New request - M=$rm RURI=$ru F=$fu T=$tu IP=$si ID=$ci\n");

    if (!mf_process_maxfwd_header("10")) {
            sl_send_reply("483","Too Many Hops");
            exit;
    }

    if (has_totag()) {
            # sequential request withing a dialog should
            # take the path determined by record-routing
            if (loose_route()) {
                    route(1);
            } else {
                    if ( is_method("ACK") ) {
                            if ( t_check_trans() ) {
                                    route(1);
                                    exit;
                            } else {
                                    exit;
                            }
                    }
                    sl_send_reply("403","No proper route");
            }
            exit;
    }

    #initial requests

    # CANCEL processing
    if (is_method("CANCEL"))
    {
            if (t_check_trans())
                    route(1);
            exit;
    }

    t_check_trans();

    if (!is_method("INVITE"))
    {
            sl_send_reply("403","Not routable");
            exit;
    }

    record_route();

    # route invites to carrier
    route(2);

}

route[1] { if (!t_relay()) { sl_reply_error(); }; exit; }

route[2] { rewritehostport("yyy.yyy.yyy.yyy:5060");

    xlog("routing INVITE request to carrier $ou -> $ru\n");

    t_on_failure("1");
    route(1);

}

failure_route[1] { if (t_was_cancelled() || t_check_status("486|487")) { exit; }

    xlog("Routing to carrier failed: $fu => $ru 

($(T_reply_code))\n"); }

## end openser script ##

Thanks,

Brian.

---

Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users

On Friday 23 May 2008, Brian Peters wrote:

[..] 10 invites per second is really nothing for OpenSER, so something else must be going on.

Hi,

i just finished a few load tests with the 1.3 branch. I'm able to complete around 75 calls/second/Proxy with my test setup, and this with a way more complicated configuration then you've used, including many database lookups. This number is similar to previous results with the 1.3 branch. The machines in question are also much much slower then in your case, so this is probably a problem specific to your setup.

Cheers,

Henning

Michael,

Thanks for the suggestion of the UDP buffer size. I pushed it to 8M, but there was no change in the behavior. I am still getting 20% of the calls fail because of OpenSER not processing the messages. Some calls are going through, but there are many that take several re-sends from Asterisk before they get processed by OpenSER.

If there are UDP packet receive failures, would I still see those packets in my tcpdump output? In my tests, I see the INVITE requests in my tcpdump log, but they simply never appear to be handled by OpenSER in a timely manner.

Also, I have been using 1.3.1 for my tests. I did not try to use 1.2.1 yet.

Thanks,

Brian.

From: Michael Young [mailto:myoung@redmonsters.net] Sent: Friday, May 23, 2008 8:25 PM To: users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

Brian -- Have had similar problems recently.

1. Run a netstat -su and see if you are having "packet errors". This means the udp buffer is filling up.

2. Check out the latest from SVN. A patch recently implemented will honor the rev_dns=no.

3. Increase the UDP buffer size (I used this info: http://www.29west.com/docs/THPM/udp-buffer-sizing.html). Mine is set to 1 Mb right now, and is now able to handle 10 invites per second. Before making these changes I was dying at 3 per second.

I posted a message earlier this week, that said that my OpenSER install is not correctly calculating the buffer size as far as I can tell, but have not gotten a response on that yet. Performance is still not as good as previous OpenSER versions with this setup (we have a 1.2.1 server that is running circles around the three 1.3.2 servers). I am using dbaliases to route calls on the 1.3.2 servers, and ENUM on the 1.2.1 server, but that is the only difference, and I would not expect that to make this great of a difference.

Michael

From: users-bounces@lists.openser.org [mailto:users-bounces@lists.openser.org] On Behalf Of Brian Peters Sent: Friday, May 23, 2008 1:15 PM To: users@lists.openser.org Subject: [OpenSER-Users] Openser not processing requests at rate being sent to it

I am doing what I thought was a simple volume test that OpenSER should be able to handle, but the results I am getting so far show it is not handling the load I'm sending it.

Here is my configuration:

- OpenSER 1.3.1

- Simple script that uses OpenSER as a proxy to forward requests to a carrier. All it does is rewrite the host and port of the R-URI and relays to the carrier.

- children=16

- shared memory set to 1 G. (-m 1000)

- using transaction module to handle the requests.

- Sending 10 INVITE requests per second to OpenSER using Asterisk.

When I do a "tcpdump" trace of the network traffic to/from OpenSER, it is clear that a large percentage of the INVITE requests sent to OpenSER were not handled. Asterisk needed to resend the INVITE requests several times before OpenSER would handle them, and in about 20% of the cases Asterisk terminated the INVITES because it reached the timeout of 30 seconds without any response.

If I go directly from asterisk to the carrier, this test works without any issues. I'm not sure what the limitations are of OpenSER, but this seems like something it should be able to handle.

Any help or information on how to configure OpenSER so it will work would be appreciated.

Thanks,

Brian.

No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.0/1462 - Release Date: 5/23/2008 7:20 AM

No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.0/1462 - Release Date: 5/23/2008 7:20 AM

Here is what I know (disclaimer: I am not a coder... as far as I am concerned C is for Cookie, and that's good enough for me):

One of my providers is a company I used to work for. They have a primary OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new INVITES per second. I have their ENUM configured to route half of those calls to my proxy1, and the other half to my proxy2.

At 25 cps (new calls per second), the load on their SER box is .1 to .2 at most. The processor rarely gets to 1% utilized. OpenSER is using between 200 and 400 Mb of RAM.

At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to 1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1 chokes and stops responding to invites. It is not the processor or the RAM that are limiting... there is something else involved that takes the load up to 2.0, at which point it stops responding.

I have stripped down my config, trying to find out where the problem is. At this point I have wild theories, but few cold hard facts.

Their OpenSER box is running ACC, and writing records to a MYSQL cluster that lives on a different box. Mine is not running ACC at all anymore. They use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor, I am using CentOS x86_64. I actually have a Xeon-based server on order, wondering if the problem is only on x86_64 based systems.

Increasing the UDP buffer size is the only thing that has gotten me to 10 cps, before I was dying at 3. My OpenSER proxies route calls to a server farm of ~70 Asterisk boxes. If one of those asterisk servers goes off line, my proxy's load will immediately go up to 2.0, the processor level will go to 100%, and it stops processing many (but not all) new calls. It seems to block some of the processes at this point, if I try to shut OpenSER down I have to manually kill 6 to 8 of the child processes.

Here are the modules I'm loading: loadmodule "mysql.so" loadmodule "sl.so" loadmodule "tm.so" loadmodule "rr.so" loadmodule "maxfwd.so" loadmodule "textops.so" loadmodule "mi_fifo.so" loadmodule "uri_db.so" loadmodule "uri.so" loadmodule "xlog.so" loadmodule "permissions.so" loadmodule "alias_db.so" loadmodule "domain.so"

Maybe this information can help someone else who is running into a similar problem.

Michael

-----Original Message----- From: Henning Westerholt [mailto:henning.westerholt@1und1.de] Sent: Tuesday, May 27, 2008 3:22 AM To: users@lists.openser.org Cc: Michael Young Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

On Saturday 24 May 2008, Michael Young wrote:

I posted a message earlier this week, that said that my OpenSER install is not correctly calculating the buffer size as far as I can tell, but have not gotten a response on that yet. Performance is still not as good as previous OpenSER versions with this setup (we have a 1.2.1 server that is running circles around the three 1.3.2 servers).

Hi Michael,

i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases, probably caused e.g. from the added pseudo-variable stuff, more DNS lookups and further abstractions. But a slowdown like this sounds more like a bug to me.

Do you can share more details about the problem?

Cheers,

Henning

No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM

No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM

PS -- Henning, Bogdan, Daniel, etc -- email me off list and I'll gladly give you access to the x86_64 server if you want to take a look and\or test it with sipp.

Michael

-----Original Message----- From: users-bounces@lists.openser.org [mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young Sent: Thursday, May 29, 2008 6:31 PM To: users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

More information on the performance problems I have been fighting:

I replaced one of my OpenSER production boxes with a new server today. The old one was running x86_64 version of CentOS 5 (two dual core Opteron processors). The new one is running the i386 flavor of CentOS 5 (one quad core Xeon processor). I copied the configs from the old box to the new one. The boxes were built identically using the same process and applications.

The new box is now matching the performance I was seeing on the 1.2.1 box of my provider. This new box is easily handling 50 new calls per second.

So this indicates that something is wrong with the combination of Linux x86_64 and OpenSER 1.3.X.

NOTE: I am NOT blaming OpenSER or saying that the problem lies with the OpenSER code. But something in the combination of those is causing a huge performance problem. For all I know the problem could be in an Ethernet driver in the x86_64 package.

Those of you who have done benchmarking lately -- were your benchmarks run on x86_64? I would be curious to know if others who have reported performance problems lately were on x86_64 (I thought at least one other report indicated they were using that build).

Michael Young

-----Original Message----- From: users-bounces@lists.openser.org [mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young Sent: Tuesday, May 27, 2008 8:16 AM To: 'Henning Westerholt'; users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

Here is what I know (disclaimer: I am not a coder... as far as I am concerned C is for Cookie, and that's good enough for me):

One of my providers is a company I used to work for. They have a primary OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new INVITES per second. I have their ENUM configured to route half of those calls to my proxy1, and the other half to my proxy2.

At 25 cps (new calls per second), the load on their SER box is .1 to .2 at most. The processor rarely gets to 1% utilized. OpenSER is using between 200 and 400 Mb of RAM.

At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to 1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1 chokes and stops responding to invites. It is not the processor or the RAM that are limiting... there is something else involved that takes the load up to 2.0, at which point it stops responding.

I have stripped down my config, trying to find out where the problem is. At this point I have wild theories, but few cold hard facts.

Their OpenSER box is running ACC, and writing records to a MYSQL cluster that lives on a different box. Mine is not running ACC at all anymore. They use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor, I am using CentOS x86_64. I actually have a Xeon-based server on order, wondering if the problem is only on x86_64 based systems.

Increasing the UDP buffer size is the only thing that has gotten me to 10 cps, before I was dying at 3. My OpenSER proxies route calls to a server farm of ~70 Asterisk boxes. If one of those asterisk servers goes off line, my proxy's load will immediately go up to 2.0, the processor level will go to 100%, and it stops processing many (but not all) new calls. It seems to block some of the processes at this point, if I try to shut OpenSER down I have to manually kill 6 to 8 of the child processes.

Here are the modules I'm loading: loadmodule "mysql.so" loadmodule "sl.so" loadmodule "tm.so" loadmodule "rr.so" loadmodule "maxfwd.so" loadmodule "textops.so" loadmodule "mi_fifo.so" loadmodule "uri_db.so" loadmodule "uri.so" loadmodule "xlog.so" loadmodule "permissions.so" loadmodule "alias_db.so" loadmodule "domain.so"

Maybe this information can help someone else who is running into a similar problem.

Michael

-----Original Message----- From: Henning Westerholt [mailto:henning.westerholt@1und1.de] Sent: Tuesday, May 27, 2008 3:22 AM To: users@lists.openser.org Cc: Michael Young Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

On Saturday 24 May 2008, Michael Young wrote:

I posted a message earlier this week, that said that my OpenSER install is not correctly calculating the buffer size as far as I can tell, but have not gotten a response on that yet. Performance is still not as good as previous OpenSER versions with this setup (we have a 1.2.1 server that is running circles around the three 1.3.2 servers).

Hi Michael,

i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases, probably caused e.g. from the added pseudo-variable stuff, more DNS lookups and further abstractions. But a slowdown like this sounds more like a bug to

me.

Do you can share more details about the problem?

Cheers,

Henning

No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM

No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM

_______________________________________________ Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users

No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM

No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM

_______________________________________________ Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users

No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM

No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM

On Friday 30 May 2008, Michael Young wrote:

PS -- Henning, Bogdan, Daniel, etc -- email me off list and I'll gladly give you access to the x86_64 server if you want to take a look and\or test it with sipp.

Hi Michael,

thank you for the offer, in fact we've already a machine available, devel.openser.org runs with x64_86. Its more a matter of insufficient time at the moment than a lack of access from my side.

Cheers,

Henning

I have four different x86_64 boxes which perform identically. I don't think it is a hardware issue... although it could be a driver issue of some kind. All the hardware on those four are similar (Dell rack mount servers, not all the same model but similar).

If I get a chance this weekend I'll load the i386 distro on the Opteron based server I took out of production and see what that does.

Michael

-----Original Message----- From: Mik Cheez [mailto:michael_bulk@wildgate.com] Sent: Thursday, May 29, 2008 7:36 PM To: Michael Young Cc: users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

Wow...that's interesting. Have you tried running i386 on the first box? If that works you may be correct, but if the problem persists you may have to consider a hardware issue. If not that, then perhaps a different 64 bit Linux distribution...

Michael Young wrote:

More information on the performance problems I have been fighting:

I replaced one of my OpenSER production boxes with a new server today. The old one was running x86_64 version of CentOS 5 (two dual core Opteron processors). The new one is running the i386 flavor of CentOS 5 (one quad core Xeon processor). I copied the configs from the old box to the new

one.

The boxes were built identically using the same process and applications.

The new box is now matching the performance I was seeing on the 1.2.1 box

of

my provider. This new box is easily handling 50 new calls per second.

So this indicates that something is wrong with the combination of Linux x86_64 and OpenSER 1.3.X.

NOTE: I am NOT blaming OpenSER or saying that the problem lies with the OpenSER code. But something in the combination of those is causing a huge performance problem. For all I know the problem could be in an Ethernet driver in the x86_64 package.

Those of you who have done benchmarking lately -- were your benchmarks run on x86_64? I would be curious to know if others who have reported performance problems lately were on x86_64 (I thought at least one other report indicated they were using that build).

Michael Young

-----Original Message----- From: users-bounces@lists.openser.org [mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young Sent: Tuesday, May 27, 2008 8:16 AM To: 'Henning Westerholt'; users@lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

Here is what I know (disclaimer: I am not a coder... as far as I am concerned C is for Cookie, and that's good enough for me):

One of my providers is a company I used to work for. They have a primary OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new INVITES per second. I have their ENUM configured to route half of those calls to my proxy1, and the other half to my proxy2.

At 25 cps (new calls per second), the load on their SER box is .1 to .2 at most. The processor rarely gets to 1% utilized. OpenSER is using between

200

and 400 Mb of RAM.

At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets

to

1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1 chokes and stops responding to invites. It is not the processor or the RAM that are limiting... there is something else involved that takes the load

up

to 2.0, at which point it stops responding.

I have stripped down my config, trying to find out where the problem is.

At

this point I have wild theories, but few cold hard facts.

Their OpenSER box is running ACC, and writing records to a MYSQL cluster that lives on a different box. Mine is not running ACC at all anymore.

They

use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor,

I

am using CentOS x86_64. I actually have a Xeon-based server on order, wondering if the problem is only on x86_64 based systems.

Increasing the UDP buffer size is the only thing that has gotten me to 10 cps, before I was dying at 3. My OpenSER proxies route calls to a server farm of ~70 Asterisk boxes. If one of those asterisk servers goes off

line,

my proxy's load will immediately go up to 2.0, the processor level will go to 100%, and it stops processing many (but not all) new calls. It seems to block some of the processes at this point, if I try to shut OpenSER down I have to manually kill 6 to 8 of the child processes.

Here are the modules I'm loading: loadmodule "mysql.so" loadmodule "sl.so" loadmodule "tm.so" loadmodule "rr.so" loadmodule "maxfwd.so" loadmodule "textops.so" loadmodule "mi_fifo.so" loadmodule "uri_db.so" loadmodule "uri.so" loadmodule "xlog.so" loadmodule "permissions.so" loadmodule "alias_db.so" loadmodule "domain.so"

Maybe this information can help someone else who is running into a similar problem.

Michael

-----Original Message----- From: Henning Westerholt [mailto:henning.westerholt@1und1.de] Sent: Tuesday, May 27, 2008 3:22 AM To: users@lists.openser.org Cc: Michael Young Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

On Saturday 24 May 2008, Michael Young wrote:

...

I posted a message earlier this week, that said that my OpenSER install

is

...

not correctly calculating the buffer size as far as I can tell, but have not gotten a response on that yet. Performance is still not as good as previous OpenSER versions with this setup (we have a 1.2.1 server that is running circles around the three 1.3.2 servers).

Hi Michael,

i can confirm a (albeit minor) slowdown from 0.9 branch to actual

releases,

probably caused e.g. from the added pseudo-variable stuff, more DNS

lookups

and further abstractions. But a slowdown like this sounds more like a bug

to

me.

Do you can share more details about the problem?

Cheers,

Henning

No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM

No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM

---

Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users

No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM

No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM

---

Users mailing list Users@lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users

No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM

No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM