23 May
2008
23 May
'08
11:15 p.m.
I am doing what I thought was a simple volume test that OpenSER should be
able to handle, but the results I am getting so far show it is not handling
the load I'm sending it.
Here is my configuration:
- OpenSER 1.3.1
- Simple script that uses OpenSER as a proxy to forward requests to
a carrier. All it does is rewrite the host and port of the R-URI and relays
to the carrier.
- children=16
- shared memory set to 1 G. (-m 1000)
- using transaction module to handle the requests.
- Sending 10 INVITE requests per second to OpenSER using Asterisk.
When I do a "tcpdump" trace of the network traffic to/from OpenSER, it is
clear that a large percentage of the INVITE requests sent to OpenSER were
not handled. Asterisk needed to resend the INVITE requests several times
before OpenSER would handle them, and in about 20% of the cases Asterisk
terminated the INVITES because it reached the timeout of 30 seconds without
any response.
If I go directly from asterisk to the carrier, this test works without any
issues. I'm not sure what the limitations are of OpenSER, but this seems
like something it should be able to handle.
Any help or information on how to configure OpenSER so it will work would be
appreciated.
Thanks,
Brian.
24 May
24 May
12:33 a.m.
Brian Peters wrote:
I am doing what I thought was a simple volume test
that OpenSER should
be able to handle, but the results I am getting so far show it is not
handling the load I’m sending it.
Here is my configuration:
- OpenSER 1.3.1
- Simple script that uses OpenSER as a proxy to forward
requests to a carrier. All it does is rewrite the host and port of the
R-URI and relays to the carrier.
- children=16
- shared memory set to 1 G. (-m 1000)
- using transaction module to handle the requests.
- Sending 10 INVITE requests per second to OpenSER using Asterisk.
When I do a “tcpdump” trace of the network traffic to/from OpenSER, it
is clear that a large percentage of the INVITE requests sent to OpenSER
were not handled. Asterisk needed to resend the INVITE requests several
times before OpenSER would handle them, and in about 20% of the cases
Asterisk terminated the INVITES because it reached the timeout of 30
seconds without any response.
If I go directly from asterisk to the carrier, this test works without
any issues. I’m not sure what the limitations are of OpenSER, but this
seems like something it should be able to handle.
Any help or information on how to configure OpenSER so it will work
would be appreciated.
Thanks,
Brian.
------------------------------------------------------------------------
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
You have a config file to share?
Are your network settings correct?
Are you doing the dump on the OpenSER box, or the Asterisk?
How are your system resources on the box (e.g. running 'top')?
10 invites per second is really nothing for OpenSER, so something else
must be going on.
1:32 a.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
Mik,
Response to your questions below:
-----Original Message-----
From: Mik Cheez [mailto:michael_bulk@wildgate.com]
Sent: Friday, May 23, 2008 3:34 PM
To: Brian Peters
Cc: users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Brian Peters wrote:
I am doing what I thought was a simple volume test
that OpenSER should
be able to handle, but the results I am getting so far show it is not
handling the load I'm sending it.
Here is my configuration:
- OpenSER 1.3.1
- Simple script that uses OpenSER as a proxy to forward
requests to a carrier. All it does is rewrite the host and port of the
R-URI and relays to the carrier.
- children=16
- shared memory set to 1 G. (-m 1000)
- using transaction module to handle the requests.
- Sending 10 INVITE requests per second to OpenSER using
Asterisk.
When I do a "tcpdump" trace of the network traffic to/from OpenSER, it
is clear that a large percentage of the INVITE requests sent to OpenSER
were not handled. Asterisk needed to resend the INVITE requests several
times before OpenSER would handle them, and in about 20% of the cases
Asterisk terminated the INVITES because it reached the timeout of 30
seconds without any response.
If I go directly from asterisk to the carrier, this test works without
any issues. I'm not sure what the limitations are of OpenSER, but this
seems like something it should be able to handle.
Any help or information on how to configure OpenSER so it will work
would be appreciated.
Thanks,
Brian.
------------------------------------------------------------------------
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
You have a config file to share?
Are your network settings correct?
Are you doing the dump on the OpenSER box, or the Asterisk?
How are your system resources on the box (e.g. running 'top')?
10 invites per second is really nothing for OpenSER, so something else
must be going on.
---
1. the network settings on the servers are OK. We have used the servers in
production environment without OpenSER (asterisk direct to carrier) and at
much higher volume than we are testing.
2. The tcpdump is being done from the OpenSER box so that I could see what
OpenSER was receiving and sending. The server is clearly receiving the
INVITE requests, and OpenSER is not responding.
3. System resources are not an issue. The server is only being used for
OpenSER, and CPU usage is barely reaching 1%. Network is only being used for
OpenSER.
4. for the config file, here it is (with the IP/hosts blanked out):
## begin openser script ##
#
# $Id: openser.cfg 3542 2008-01-16 15:00:34Z miconda $
#
# OpenSER basic configuration script
# by Anca Vamanu <anca(a)voice-system.ro>
#
# Please refer to the Core CookBook at
http://www.openser.org/dokuwiki/doku.php
# for a explanation of possible statements, functions and parameters.
#
####### Global Parameters #########
debug=3
fork=yes
log_stderror=no
listen=udp:xxx.xxx.xxx.xxx:5060
port=5060
# disable_tcp=yes
alias=foo.com
sip_warning=0
check_via=no
dns=off
rev_dns=off
children=8
log_name="openser-test.log"
log_facility=LOG_LOCAL0
disable_dns_blacklist=yes
disable_dns_failover=yes
####### Modules Section ########
#set module path
mpath="/usr/local/openser-1.3.1//lib/openser/modules/"
loadmodule "textops.so"
loadmodule "sl.so"
loadmodule "maxfwd.so"
loadmodule "uri.so"
loadmodule "xlog.so"
# ----- mi_fifo -----
loadmodule "mi_fifo.so"
modparam("mi_fifo", "fifo_name", "/tmp/openser_1.3.1_fifo")
# ----- rr -----
loadmodule "rr.so"
modparam("rr", "append_fromtag", 0)
modparam("rr", "enable_full_lr", 1)
# ----- tm -----
loadmodule "tm.so"
modparam("tm", "fr_timer", 40)
modparam("tm", "fr_inv_timer", 60)
modparam("tm", "wt_timer", 20)
modparam("tm", "noisy_ctimer", 1)
route{
xlog("New request - M=$rm RURI=$ru F=$fu T=$tu IP=$si ID=$ci\n");
if (!mf_process_maxfwd_header("10")) {
sl_send_reply("483","Too Many Hops");
exit;
}
if (has_totag()) {
# sequential request withing a dialog should
# take the path determined by record-routing
if (loose_route()) {
route(1);
} else {
if ( is_method("ACK") ) {
if ( t_check_trans() ) {
route(1);
exit;
} else {
exit;
}
}
sl_send_reply("403","No proper route");
}
exit;
}
#initial requests
# CANCEL processing
if (is_method("CANCEL"))
{
if (t_check_trans())
route(1);
exit;
}
t_check_trans();
if (!is_method("INVITE"))
{
sl_send_reply("403","Not routable");
exit;
}
record_route();
# route invites to carrier
route(2);
}
route[1] {
if (!t_relay()) {
sl_reply_error();
};
exit;
}
route[2] {
rewritehostport("yyy.yyy.yyy.yyy:5060");
xlog("routing INVITE request to carrier $ou -> $ru\n");
t_on_failure("1");
route(1);
}
failure_route[1] {
if (t_was_cancelled() || t_check_status("486|487")) {
exit;
}
xlog("Routing to carrier failed: $fu => $ru ($(T_reply_code))\n");
}
## end openser script ##
Thanks,
Brian.
2:32 a.m.
Hi Brian,
did you check if your OpenSER try to do a reserve DNS lookup even if
rev_dns=no?
maybe related to your pb...
http://openser.org/pipermail/users/2008-May/017397.html
regards,
Andreas
Brian Peters schrieb:
Mik,
Response to your questions below:
-----Original Message-----
From: Mik Cheez [mailto:michael_bulk@wildgate.com]
Sent: Friday, May 23, 2008 3:34 PM
To: Brian Peters
Cc: users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Brian Peters wrote:
I am doing what I thought was a simple volume
test that OpenSER should
be able to handle, but the results I am getting so far show it is not
handling the load I'm sending it.
Here is my configuration:
- OpenSER 1.3.1
- Simple script that uses OpenSER as a proxy to forward
requests to a carrier. All it does is rewrite the host and port of the
R-URI and relays to the carrier.
- children=16
- shared memory set to 1 G. (-m 1000)
- using transaction module to handle the requests.
- Sending 10 INVITE requests per second to OpenSER using
Asterisk.
When I do a "tcpdump" trace of the network traffic to/from OpenSER, it
is clear that a large percentage of the INVITE requests sent to OpenSER
were not handled. Asterisk needed to resend the INVITE requests several
times before OpenSER would handle them, and in about 20% of the cases
Asterisk terminated the INVITES because it reached the timeout of 30
seconds without any response.
If I go directly from asterisk to the carrier, this test works without
any issues. I'm not sure what the limitations are of OpenSER, but this
seems like something it should be able to handle.
Any help or information on how to configure OpenSER so it will work
would be appreciated.
Thanks,
Brian.
------------------------------------------------------------------------
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
You have a config file to share?
Are your network settings correct?
Are you doing the dump on the OpenSER box, or the Asterisk?
How are your system resources on the box (e.g. running 'top')?
10 invites per second is really nothing for OpenSER, so something else
must be going on.
---
1. the network settings on the servers are OK. We have used the servers in
production environment without OpenSER (asterisk direct to carrier) and at
much higher volume than we are testing.
2. The tcpdump is being done from the OpenSER box so that I could see what
OpenSER was receiving and sending. The server is clearly receiving the
INVITE requests, and OpenSER is not responding.
3. System resources are not an issue. The server is only being used for
OpenSER, and CPU usage is barely reaching 1%. Network is only being used for
OpenSER.
4. for the config file, here it is (with the IP/hosts blanked out):
## begin openser script ##
#
# $Id: openser.cfg 3542 2008-01-16 15:00:34Z miconda $
#
# OpenSER basic configuration script
# by Anca Vamanu <anca(a)voice-system.ro>
#
# Please refer to the Core CookBook at
http://www.openser.org/dokuwiki/doku.php
# for a explanation of possible statements, functions and parameters.
#
####### Global Parameters #########
debug=3
fork=yes
log_stderror=no
listen=udp:xxx.xxx.xxx.xxx:5060
port=5060
# disable_tcp=yes
alias=foo.com
sip_warning=0
check_via=no
dns=off
rev_dns=off
children=8
log_name="openser-test.log"
log_facility=LOG_LOCAL0
disable_dns_blacklist=yes
disable_dns_failover=yes
####### Modules Section ########
#set module path
mpath="/usr/local/openser-1.3.1//lib/openser/modules/"
loadmodule "textops.so"
loadmodule "sl.so"
loadmodule "maxfwd.so"
loadmodule "uri.so"
loadmodule "xlog.so"
# ----- mi_fifo -----
loadmodule "mi_fifo.so"
modparam("mi_fifo", "fifo_name",
"/tmp/openser_1.3.1_fifo")
# ----- rr -----
loadmodule "rr.so"
modparam("rr", "append_fromtag", 0)
modparam("rr", "enable_full_lr", 1)
# ----- tm -----
loadmodule "tm.so"
modparam("tm", "fr_timer", 40)
modparam("tm", "fr_inv_timer", 60)
modparam("tm", "wt_timer", 20)
modparam("tm", "noisy_ctimer", 1)
route{
xlog("New request - M=$rm RURI=$ru F=$fu T=$tu IP=$si ID=$ci\n");
if (!mf_process_maxfwd_header("10")) {
sl_send_reply("483","Too Many Hops");
exit;
}
if (has_totag()) {
# sequential request withing a dialog should
# take the path determined by record-routing
if (loose_route()) {
route(1);
} else {
if ( is_method("ACK") ) {
if ( t_check_trans() ) {
route(1);
exit;
} else {
exit;
}
}
sl_send_reply("403","No proper route");
}
exit;
}
#initial requests
# CANCEL processing
if (is_method("CANCEL"))
{
if (t_check_trans())
route(1);
exit;
}
t_check_trans();
if (!is_method("INVITE"))
{
sl_send_reply("403","Not routable");
exit;
}
record_route();
# route invites to carrier
route(2);
}
route[1] {
if (!t_relay()) {
sl_reply_error();
};
exit;
}
route[2] {
rewritehostport("yyy.yyy.yyy.yyy:5060");
xlog("routing INVITE request to carrier $ou -> $ru\n");
t_on_failure("1");
route(1);
}
failure_route[1] {
if (t_was_cancelled() || t_check_status("486|487")) {
exit;
}
xlog("Routing to carrier failed: $fu => $ru ($(T_reply_code))\n");
}
## end openser script ##
Thanks,
Brian.
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
2:45 a.m.
Brian,
I just ran your config file, which is minimal feature wise, and had no
issues. I checked out the latest SVN version of OpenSER, which you may
want to do as well.
Short of that, how about raising the debug level and making it available.
DNS lookups, as Andreas mentions here, can absolutely cause the kind of
problem seen here (though I believe it to be more common on Asterisk).
Andreas Heise wrote:
Hi Brian,
did you check if your OpenSER try to do a reserve DNS lookup even if
rev_dns=no?
maybe related to your pb...
http://openser.org/pipermail/users/2008-May/017397.html
regards,
Andreas
Brian Peters schrieb:
Mik,
Response to your questions below:
-----Original Message-----
From: Mik Cheez [mailto:michael_bulk@wildgate.com] Sent: Friday, May
23, 2008 3:34 PM
To: Brian Peters
Cc: users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate
being
sent to it
Brian Peters wrote:
I am doing what I thought was a simple volume
test that OpenSER
should be able to handle, but the results I am getting so far show it
is not handling the load I'm sending it.
Here is my configuration:
- OpenSER 1.3.1
- Simple script that uses OpenSER as a proxy to forward
requests to a carrier. All it does is rewrite the host and port of
the R-URI and relays to the carrier.
- children=16
- shared memory set to 1 G. (-m 1000)
- using transaction module to handle the requests.
- Sending 10 INVITE requests per second to OpenSER using
Asterisk.
When I do a "tcpdump" trace of the network traffic to/from OpenSER,
it is clear that a large percentage of the INVITE requests sent to
OpenSER were not handled. Asterisk needed to resend the INVITE
requests several times before OpenSER would handle them, and in about
20% of the cases Asterisk terminated the INVITES because it reached
the timeout of 30 seconds without any response.
If I go directly from asterisk to the carrier, this test works
without any issues. I'm not sure what the limitations are of OpenSER,
but this seems like something it should be able to handle.
Any help or information on how to configure OpenSER so it will work
would be appreciated.
Thanks,
Brian.
------------------------------------------------------------------------
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
You have a config file to share?
Are your network settings correct?
Are you doing the dump on the OpenSER box, or the Asterisk?
How are your system resources on the box (e.g. running 'top')?
10 invites per second is really nothing for OpenSER, so something else
must be going on.
---
1. the network settings on the servers are OK. We have used the
servers in
production environment without OpenSER (asterisk direct to carrier)
and at
much higher volume than we are testing.
2. The tcpdump is being done from the OpenSER box so that I could see
what
OpenSER was receiving and sending. The server is clearly receiving the
INVITE requests, and OpenSER is not responding.
3. System resources are not an issue. The server is only being used for
OpenSER, and CPU usage is barely reaching 1%. Network is only being
used for
OpenSER.
4. for the config file, here it is (with the IP/hosts blanked out):
## begin openser script ##
#
# $Id: openser.cfg 3542 2008-01-16 15:00:34Z miconda $
#
# OpenSER basic configuration script
# by Anca Vamanu <anca(a)voice-system.ro>
#
# Please refer to the Core CookBook at
http://www.openser.org/dokuwiki/doku.php
# for a explanation of possible statements, functions and parameters.
#
####### Global Parameters #########
debug=3
fork=yes
log_stderror=no
listen=udp:xxx.xxx.xxx.xxx:5060
port=5060
# disable_tcp=yes
alias=foo.com
sip_warning=0
check_via=no
dns=off
rev_dns=off
children=8
log_name="openser-test.log"
log_facility=LOG_LOCAL0
disable_dns_blacklist=yes
disable_dns_failover=yes
####### Modules Section ########
#set module path
mpath="/usr/local/openser-1.3.1//lib/openser/modules/"
loadmodule "textops.so"
loadmodule "sl.so"
loadmodule "maxfwd.so"
loadmodule "uri.so"
loadmodule "xlog.so"
# ----- mi_fifo -----
loadmodule "mi_fifo.so"
modparam("mi_fifo", "fifo_name",
"/tmp/openser_1.3.1_fifo")
# ----- rr -----
loadmodule "rr.so"
modparam("rr", "append_fromtag", 0)
modparam("rr", "enable_full_lr", 1)
# ----- tm -----
loadmodule "tm.so"
modparam("tm", "fr_timer", 40)
modparam("tm", "fr_inv_timer", 60)
modparam("tm", "wt_timer", 20)
modparam("tm", "noisy_ctimer", 1)
route{
xlog("New request - M=$rm RURI=$ru F=$fu T=$tu IP=$si ID=$ci\n");
if (!mf_process_maxfwd_header("10")) {
sl_send_reply("483","Too Many Hops");
exit;
}
if (has_totag()) {
# sequential request withing a dialog should
# take the path determined by record-routing
if (loose_route()) {
route(1);
} else {
if ( is_method("ACK") ) {
if ( t_check_trans() ) {
route(1);
exit;
} else {
exit;
}
}
sl_send_reply("403","No proper route");
}
exit;
}
#initial requests
# CANCEL processing
if (is_method("CANCEL"))
{
if (t_check_trans())
route(1);
exit;
}
t_check_trans();
if (!is_method("INVITE"))
{
sl_send_reply("403","Not routable");
exit;
}
record_route();
# route invites to carrier
route(2);
}
route[1] {
if (!t_relay()) {
sl_reply_error();
};
exit;
}
route[2] {
rewritehostport("yyy.yyy.yyy.yyy:5060");
xlog("routing INVITE request to carrier $ou -> $ru\n");
t_on_failure("1");
route(1);
}
failure_route[1] {
if (t_was_cancelled() || t_check_status("486|487")) {
exit;
}
xlog("Routing to carrier failed: $fu => $ru
($(T_reply_code))\n");
}
## end openser script ##
Thanks,
Brian.
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
26 May
26 May
1:28 p.m.
Hi Andreas,
it is not related to that fix - it was affecting only comparison ops
involving IPs, but it is not the case here as I found no such ops.
Regards,
Bogdan
Andreas Heise wrote:
Hi Brian,
did you check if your OpenSER try to do a reserve DNS lookup even if
rev_dns=no?
maybe related to your pb...
http://openser.org/pipermail/users/2008-May/017397.html
regards,
Andreas
Brian Peters schrieb:
Mik,
Response to your questions below:
-----Original Message-----
From: Mik Cheez [mailto:michael_bulk@wildgate.com]
Sent: Friday, May 23, 2008 3:34 PM
To: Brian Peters
Cc: users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Brian Peters wrote:
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
I am doing what I thought was a simple volume
test that OpenSER should
be able to handle, but the results I am getting so far show it is not
handling the load I'm sending it.
Here is my configuration:
- OpenSER 1.3.1
- Simple script that uses OpenSER as a proxy to forward
requests to a carrier. All it does is rewrite the host and port of the
R-URI and relays to the carrier.
- children=16
- shared memory set to 1 G. (-m 1000)
- using transaction module to handle the requests.
- Sending 10 INVITE requests per second to OpenSER using
Asterisk.
When I do a "tcpdump" trace of the network traffic to/from OpenSER, it
is clear that a large percentage of the INVITE requests sent to OpenSER
were not handled. Asterisk needed to resend the INVITE requests several
times before OpenSER would handle them, and in about 20% of the cases
Asterisk terminated the INVITES because it reached the timeout of 30
seconds without any response.
If I go directly from asterisk to the carrier, this test works without
any issues. I'm not sure what the limitations are of OpenSER, but this
seems like something it should be able to handle.
Any help or information on how to configure OpenSER so it will work
would be appreciated.
Thanks,
Brian.
------------------------------------------------------------------------
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
You have a config file to share?
Are your network settings correct?
Are you doing the dump on the OpenSER box, or the Asterisk?
How are your system resources on the box (e.g. running 'top')?
10 invites per second is really nothing for OpenSER, so something else
must be going on.
---
1. the network settings on the servers are OK. We have used the servers in
production environment without OpenSER (asterisk direct to carrier) and at
much higher volume than we are testing.
2. The tcpdump is being done from the OpenSER box so that I could see what
OpenSER was receiving and sending. The server is clearly receiving the
INVITE requests, and OpenSER is not responding.
3. System resources are not an issue. The server is only being used for
OpenSER, and CPU usage is barely reaching 1%. Network is only being used for
OpenSER.
4. for the config file, here it is (with the IP/hosts blanked out):
## begin openser script ##
#
# $Id: openser.cfg 3542 2008-01-16 15:00:34Z miconda $
#
# OpenSER basic configuration script
# by Anca Vamanu <anca(a)voice-system.ro>
#
# Please refer to the Core CookBook at
http://www.openser.org/dokuwiki/doku.php
# for a explanation of possible statements, functions and parameters.
#
####### Global Parameters #########
debug=3
fork=yes
log_stderror=no
listen=udp:xxx.xxx.xxx.xxx:5060
port=5060
# disable_tcp=yes
alias=foo.com
sip_warning=0
check_via=no
dns=off
rev_dns=off
children=8
log_name="openser-test.log"
log_facility=LOG_LOCAL0
disable_dns_blacklist=yes
disable_dns_failover=yes
####### Modules Section ########
#set module path
mpath="/usr/local/openser-1.3.1//lib/openser/modules/"
loadmodule "textops.so"
loadmodule "sl.so"
loadmodule "maxfwd.so"
loadmodule "uri.so"
loadmodule "xlog.so"
# ----- mi_fifo -----
loadmodule "mi_fifo.so"
modparam("mi_fifo", "fifo_name",
"/tmp/openser_1.3.1_fifo")
# ----- rr -----
loadmodule "rr.so"
modparam("rr", "append_fromtag", 0)
modparam("rr", "enable_full_lr", 1)
# ----- tm -----
loadmodule "tm.so"
modparam("tm", "fr_timer", 40)
modparam("tm", "fr_inv_timer", 60)
modparam("tm", "wt_timer", 20)
modparam("tm", "noisy_ctimer", 1)
route{
xlog("New request - M=$rm RURI=$ru F=$fu T=$tu IP=$si ID=$ci\n");
if (!mf_process_maxfwd_header("10")) {
sl_send_reply("483","Too Many Hops");
exit;
}
if (has_totag()) {
# sequential request withing a dialog should
# take the path determined by record-routing
if (loose_route()) {
route(1);
} else {
if ( is_method("ACK") ) {
if ( t_check_trans() ) {
route(1);
exit;
} else {
exit;
}
}
sl_send_reply("403","No proper route");
}
exit;
}
#initial requests
# CANCEL processing
if (is_method("CANCEL"))
{
if (t_check_trans())
route(1);
exit;
}
t_check_trans();
if (!is_method("INVITE"))
{
sl_send_reply("403","Not routable");
exit;
}
record_route();
# route invites to carrier
route(2);
}
route[1] {
if (!t_relay()) {
sl_reply_error();
};
exit;
}
route[2] {
rewritehostport("yyy.yyy.yyy.yyy:5060");
xlog("routing INVITE request to carrier $ou -> $ru\n");
t_on_failure("1");
route(1);
}
failure_route[1] {
if (t_was_cancelled() || t_check_status("486|487")) {
exit;
}
xlog("Routing to carrier failed: $fu => $ru ($(T_reply_code))\n");
}
## end openser script ##
Thanks,
Brian.
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
10:01 p.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
On Friday 23 May 2008, Brian Peters wrote:
[..]
10 invites per second is really nothing for OpenSER, so something else
must be going on.
Hi,
i just finished a few load tests with the 1.3 branch. I'm able to complete
around 75 calls/second/Proxy with my test setup, and this with a way more
complicated configuration then you've used, including many database lookups.
This number is similar to previous results with the 1.3 branch. The machines
in question are also much much slower then in your case, so this is probably
a problem specific to your setup.
Cheers,
Henning
--
I'm going to Linuxtag.
24 May
24 May
5:24 a.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
Brian -- Have had similar problems recently.
1. Run a netstat –su and see if you are having “packet errors”. This
means the udp buffer is filling up.
2. Check out the latest from SVN. A patch recently implemented will
honor the rev_dns=no.
3. Increase the UDP buffer size (I used this info: HYPERLINK
"http://www.29west.com/docs/THPM/udp-buffer-sizing.html"http://www.29west.co
m/docs/THPM/udp-buffer-sizing.html). Mine is set to 1 Mb right now, and is
now able to handle 10 invites per second. Before making these changes I was
dying at 3 per second.
I posted a message earlier this week, that said that my OpenSER install is
not correctly calculating the buffer size as far as I can tell, but have not
gotten a response on that yet. Performance is still not as good as previous
OpenSER versions with this setup (we have a 1.2.1 server that is running
circles around the three 1.3.2 servers). I am using dbaliases to route calls
on the 1.3.2 servers, and ENUM on the 1.2.1 server, but that is the only
difference, and I would not expect that to make this great of a difference.
Michael
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Brian Peters
Sent: Friday, May 23, 2008 1:15 PM
To: users(a)lists.openser.org
Subject: [OpenSER-Users] Openser not processing requests at rate being sent
to it
I am doing what I thought was a simple volume test that OpenSER should be
able to handle, but the results I am getting so far show it is not handling
the load I’m sending it.
Here is my configuration:
- OpenSER 1.3.1
- Simple script that uses OpenSER as a proxy to forward requests to
a carrier. All it does is rewrite the host and port of the R-URI and relays
to the carrier.
- children=16
- shared memory set to 1 G. (-m 1000)
- using transaction module to handle the requests.
- Sending 10 INVITE requests per second to OpenSER using Asterisk.
When I do a “tcpdump” trace of the network traffic to/from OpenSER, it is
clear that a large percentage of the INVITE requests sent to OpenSER were
not handled. Asterisk needed to resend the INVITE requests several times
before OpenSER would handle them, and in about 20% of the cases Asterisk
terminated the INVITES because it reached the timeout of 30 seconds without
any response.
If I go directly from asterisk to the carrier, this test works without any
issues. I’m not sure what the limitations are of OpenSER, but this seems
like something it should be able to handle.
Any help or information on how to configure OpenSER so it will work would be
appreciated.
Thanks,
Brian.
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.0/1462 - Release Date: 5/23/2008
7:20 AM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.0/1462 - Release Date: 5/23/2008
7:20 AM
26 May
26 May
8:31 p.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
Michael,
Thanks for the suggestion of the UDP buffer size. I pushed it to 8M, but
there was no change in the behavior. I am still getting 20% of the calls
fail because of OpenSER not processing the messages. Some calls are going
through, but there are many that take several re-sends from Asterisk before
they get processed by OpenSER.
If there are UDP packet receive failures, would I still see those packets in
my tcpdump output? In my tests, I see the INVITE requests in my tcpdump
log, but they simply never appear to be handled by OpenSER in a timely
manner.
Also, I have been using 1.3.1 for my tests. I did not try to use 1.2.1 yet.
Thanks,
Brian.
From: Michael Young [mailto:myoung@redmonsters.net]
Sent: Friday, May 23, 2008 8:25 PM
To: users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Brian -- Have had similar problems recently.
1. Run a netstat -su and see if you are having "packet errors". This
means the udp buffer is filling up.
2. Check out the latest from SVN. A patch recently implemented will
honor the rev_dns=no.
3. Increase the UDP buffer size (I used this info:
http://www.29west.com/docs/THPM/udp-buffer-sizing.html). Mine is set to 1 Mb
right now, and is now able to handle 10 invites per second. Before making
these changes I was dying at 3 per second.
I posted a message earlier this week, that said that my OpenSER install is
not correctly calculating the buffer size as far as I can tell, but have not
gotten a response on that yet. Performance is still not as good as previous
OpenSER versions with this setup (we have a 1.2.1 server that is running
circles around the three 1.3.2 servers). I am using dbaliases to route calls
on the 1.3.2 servers, and ENUM on the 1.2.1 server, but that is the only
difference, and I would not expect that to make this great of a difference.
Michael
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Brian Peters
Sent: Friday, May 23, 2008 1:15 PM
To: users(a)lists.openser.org
Subject: [OpenSER-Users] Openser not processing requests at rate being sent
to it
I am doing what I thought was a simple volume test that OpenSER should be
able to handle, but the results I am getting so far show it is not handling
the load I'm sending it.
Here is my configuration:
- OpenSER 1.3.1
- Simple script that uses OpenSER as a proxy to forward requests to
a carrier. All it does is rewrite the host and port of the R-URI and relays
to the carrier.
- children=16
- shared memory set to 1 G. (-m 1000)
- using transaction module to handle the requests.
- Sending 10 INVITE requests per second to OpenSER using Asterisk.
When I do a "tcpdump" trace of the network traffic to/from OpenSER, it is
clear that a large percentage of the INVITE requests sent to OpenSER were
not handled. Asterisk needed to resend the INVITE requests several times
before OpenSER would handle them, and in about 20% of the cases Asterisk
terminated the INVITES because it reached the timeout of 30 seconds without
any response.
If I go directly from asterisk to the carrier, this test works without any
issues. I'm not sure what the limitations are of OpenSER, but this seems
like something it should be able to handle.
Any help or information on how to configure OpenSER so it will work would be
appreciated.
Thanks,
Brian.
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.0/1462 - Release Date: 5/23/2008
7:20 AM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.0/1462 - Release Date: 5/23/2008
7:20 AM
27 May
27 May
1:22 p.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
On Saturday 24 May 2008, Michael Young wrote:
I posted a message earlier this week, that said that
my OpenSER install is
not correctly calculating the buffer size as far as I can tell, but have
not gotten a response on that yet. Performance is still not as good as
previous OpenSER versions with this setup (we have a 1.2.1 server that is
running circles around the three 1.3.2 servers).
Hi Michael,
i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases,
probably caused e.g. from the added pseudo-variable stuff, more DNS lookups
and further abstractions. But a slowdown like this sounds more like a bug to
me.
Do you can share more details about the problem?
Cheers,
Henning
6:16 p.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
Here is what I know (disclaimer: I am not a coder... as far as I am concerned C is for
Cookie, and that's good enough for me):
One of my providers is a company I used to work for. They have a primary OpenSER server
(running 1.2.1) that at peak times receives 20 - 25 new INVITES per second. I have their
ENUM configured to route half of those calls to my proxy1, and the other half to my
proxy2.
At 25 cps (new calls per second), the load on their SER box is .1 to .2 at most. The
processor rarely gets to 1% utilized. OpenSER is using between 200 and 400 Mb of RAM.
At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to 1% utilized,
and the server is using 700 Mb of RAM. At 20 cps, my proxy1 chokes and stops responding to
invites. It is not the processor or the RAM that are limiting... there is something else
involved that takes the load up to 2.0, at which point it stops responding.
I have stripped down my config, trying to find out where the problem is. At this point I
have wild theories, but few cold hard facts.
Their OpenSER box is running ACC, and writing records to a MYSQL cluster that lives on a
different box. Mine is not running ACC at all anymore. They use ENUM, I'm using
dbaliases. They are using Ubuntu on an i386 processor, I am using CentOS x86_64. I
actually have a Xeon-based server on order, wondering if the problem is only on x86_64
based systems.
Increasing the UDP buffer size is the only thing that has gotten me to 10 cps, before I
was dying at 3. My OpenSER proxies route calls to a server farm of ~70 Asterisk boxes. If
one of those asterisk servers goes off line, my proxy's load will immediately go up to
2.0, the processor level will go to 100%, and it stops processing many (but not all) new
calls. It seems to block some of the processes at this point, if I try to shut OpenSER
down I have to manually kill 6 to 8 of the child processes.
Here are the modules I'm loading:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "mi_fifo.so"
loadmodule "uri_db.so"
loadmodule "uri.so"
loadmodule "xlog.so"
loadmodule "permissions.so"
loadmodule "alias_db.so"
loadmodule "domain.so"
Maybe this information can help someone else who is running into a similar problem.
Michael
-----Original Message-----
From: Henning Westerholt [mailto:henning.westerholt@1und1.de]
Sent: Tuesday, May 27, 2008 3:22 AM
To: users(a)lists.openser.org
Cc: Michael Young
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it
On Saturday 24 May 2008, Michael Young wrote:
I posted a message earlier this week, that said that
my OpenSER install is
not correctly calculating the buffer size as far as I can tell, but have
not gotten a response on that yet. Performance is still not as good as
previous OpenSER versions with this setup (we have a 1.2.1 server that is
running circles around the three 1.3.2 servers).
Hi Michael,
i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases,
probably caused e.g. from the added pseudo-variable stuff, more DNS lookups
and further abstractions. But a slowdown like this sounds more like a bug to
me.
Do you can share more details about the problem?
Cheers,
Henning
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM
30 May
30 May
4:31 a.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
More information on the performance problems I have been fighting:
I replaced one of my OpenSER production boxes with a new server today. The
old one was running x86_64 version of CentOS 5 (two dual core Opteron
processors). The new one is running the i386 flavor of CentOS 5 (one quad
core Xeon processor). I copied the configs from the old box to the new one.
The boxes were built identically using the same process and applications.
The new box is now matching the performance I was seeing on the 1.2.1 box of
my provider. This new box is easily handling 50 new calls per second.
So this indicates that something is wrong with the combination of Linux
x86_64 and OpenSER 1.3.X.
NOTE: I am NOT blaming OpenSER or saying that the problem lies with the
OpenSER code. But something in the combination of those is causing a huge
performance problem. For all I know the problem could be in an Ethernet
driver in the x86_64 package.
Those of you who have done benchmarking lately -- were your benchmarks run
on x86_64? I would be curious to know if others who have reported
performance problems lately were on x86_64 (I thought at least one other
report indicated they were using that build).
Michael Young
-----Original Message-----
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young
Sent: Tuesday, May 27, 2008 8:16 AM
To: 'Henning Westerholt'; users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Here is what I know (disclaimer: I am not a coder... as far as I am
concerned C is for Cookie, and that's good enough for me):
One of my providers is a company I used to work for. They have a primary
OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new
INVITES per second. I have their ENUM configured to route half of those
calls to my proxy1, and the other half to my proxy2.
At 25 cps (new calls per second), the load on their SER box is .1 to .2 at
most. The processor rarely gets to 1% utilized. OpenSER is using between 200
and 400 Mb of RAM.
At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to
1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1
chokes and stops responding to invites. It is not the processor or the RAM
that are limiting... there is something else involved that takes the load up
to 2.0, at which point it stops responding.
I have stripped down my config, trying to find out where the problem is. At
this point I have wild theories, but few cold hard facts.
Their OpenSER box is running ACC, and writing records to a MYSQL cluster
that lives on a different box. Mine is not running ACC at all anymore. They
use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor, I
am using CentOS x86_64. I actually have a Xeon-based server on order,
wondering if the problem is only on x86_64 based systems.
Increasing the UDP buffer size is the only thing that has gotten me to 10
cps, before I was dying at 3. My OpenSER proxies route calls to a server
farm of ~70 Asterisk boxes. If one of those asterisk servers goes off line,
my proxy's load will immediately go up to 2.0, the processor level will go
to 100%, and it stops processing many (but not all) new calls. It seems to
block some of the processes at this point, if I try to shut OpenSER down I
have to manually kill 6 to 8 of the child processes.
Here are the modules I'm loading:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "mi_fifo.so"
loadmodule "uri_db.so"
loadmodule "uri.so"
loadmodule "xlog.so"
loadmodule "permissions.so"
loadmodule "alias_db.so"
loadmodule "domain.so"
Maybe this information can help someone else who is running into a similar
problem.
Michael
-----Original Message-----
From: Henning Westerholt [mailto:henning.westerholt@1und1.de]
Sent: Tuesday, May 27, 2008 3:22 AM
To: users(a)lists.openser.org
Cc: Michael Young
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
On Saturday 24 May 2008, Michael Young wrote:
I posted a message earlier this week, that said that
my OpenSER install is
not correctly calculating the buffer size as far as I can tell, but have
not gotten a response on that yet. Performance is still not as good as
previous OpenSER versions with this setup (we have a 1.2.1 server that is
running circles around the three 1.3.2 servers).
Hi Michael,
i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases,
probably caused e.g. from the added pseudo-variable stuff, more DNS lookups
and further abstractions. But a slowdown like this sounds more like a bug to
me.
Do you can share more details about the problem?
Cheers,
Henning
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
4:36 a.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
PS -- Henning, Bogdan, Daniel, etc -- email me off list and I'll gladly give
you access to the x86_64 server if you want to take a look and\or test it
with sipp.
Michael
-----Original Message-----
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young
Sent: Thursday, May 29, 2008 6:31 PM
To: users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
More information on the performance problems I have been fighting:
I replaced one of my OpenSER production boxes with a new server today. The
old one was running x86_64 version of CentOS 5 (two dual core Opteron
processors). The new one is running the i386 flavor of CentOS 5 (one quad
core Xeon processor). I copied the configs from the old box to the new one.
The boxes were built identically using the same process and applications.
The new box is now matching the performance I was seeing on the 1.2.1 box of
my provider. This new box is easily handling 50 new calls per second.
So this indicates that something is wrong with the combination of Linux
x86_64 and OpenSER 1.3.X.
NOTE: I am NOT blaming OpenSER or saying that the problem lies with the
OpenSER code. But something in the combination of those is causing a huge
performance problem. For all I know the problem could be in an Ethernet
driver in the x86_64 package.
Those of you who have done benchmarking lately -- were your benchmarks run
on x86_64? I would be curious to know if others who have reported
performance problems lately were on x86_64 (I thought at least one other
report indicated they were using that build).
Michael Young
-----Original Message-----
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young
Sent: Tuesday, May 27, 2008 8:16 AM
To: 'Henning Westerholt'; users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Here is what I know (disclaimer: I am not a coder... as far as I am
concerned C is for Cookie, and that's good enough for me):
One of my providers is a company I used to work for. They have a primary
OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new
INVITES per second. I have their ENUM configured to route half of those
calls to my proxy1, and the other half to my proxy2.
At 25 cps (new calls per second), the load on their SER box is .1 to .2 at
most. The processor rarely gets to 1% utilized. OpenSER is using between 200
and 400 Mb of RAM.
At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to
1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1
chokes and stops responding to invites. It is not the processor or the RAM
that are limiting... there is something else involved that takes the load up
to 2.0, at which point it stops responding.
I have stripped down my config, trying to find out where the problem is. At
this point I have wild theories, but few cold hard facts.
Their OpenSER box is running ACC, and writing records to a MYSQL cluster
that lives on a different box. Mine is not running ACC at all anymore. They
use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor, I
am using CentOS x86_64. I actually have a Xeon-based server on order,
wondering if the problem is only on x86_64 based systems.
Increasing the UDP buffer size is the only thing that has gotten me to 10
cps, before I was dying at 3. My OpenSER proxies route calls to a server
farm of ~70 Asterisk boxes. If one of those asterisk servers goes off line,
my proxy's load will immediately go up to 2.0, the processor level will go
to 100%, and it stops processing many (but not all) new calls. It seems to
block some of the processes at this point, if I try to shut OpenSER down I
have to manually kill 6 to 8 of the child processes.
Here are the modules I'm loading:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "mi_fifo.so"
loadmodule "uri_db.so"
loadmodule "uri.so"
loadmodule "xlog.so"
loadmodule "permissions.so"
loadmodule "alias_db.so"
loadmodule "domain.so"
Maybe this information can help someone else who is running into a similar
problem.
Michael
-----Original Message-----
From: Henning Westerholt [mailto:henning.westerholt@1und1.de]
Sent: Tuesday, May 27, 2008 3:22 AM
To: users(a)lists.openser.org
Cc: Michael Young
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
On Saturday 24 May 2008, Michael Young wrote:
I posted a message earlier this week, that said that
my OpenSER install is
not correctly calculating the buffer size as far as I can tell, but have
not gotten a response on that yet. Performance is still not as good as
previous OpenSER versions with this setup (we have a 1.2.1 server that is
running circles around the three 1.3.2 servers).
Hi Michael,
i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases,
probably caused e.g. from the added pseudo-variable stuff, more DNS lookups
and further abstractions. But a slowdown like this sounds more like a bug to
me.
Do you can share more details about the problem?
Cheers,
Henning
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
2:20 p.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
Hi Michael,
I will definitely ask it , but once the SVN is frozen for release - than
I would like to debug the issue if possible - at least to see if it has
something (anything) to do with openser.
Thanks and regards,
Bogdan
Michael Young wrote:
PS -- Henning, Bogdan, Daniel, etc -- email me off
list and I'll gladly give
you access to the x86_64 server if you want to take a look and\or test it
with sipp.
Michael
-----Original Message-----
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young
Sent: Thursday, May 29, 2008 6:31 PM
To: users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
More information on the performance problems I have been fighting:
I replaced one of my OpenSER production boxes with a new server today. The
old one was running x86_64 version of CentOS 5 (two dual core Opteron
processors). The new one is running the i386 flavor of CentOS 5 (one quad
core Xeon processor). I copied the configs from the old box to the new one.
The boxes were built identically using the same process and applications.
The new box is now matching the performance I was seeing on the 1.2.1 box of
my provider. This new box is easily handling 50 new calls per second.
So this indicates that something is wrong with the combination of Linux
x86_64 and OpenSER 1.3.X.
NOTE: I am NOT blaming OpenSER or saying that the problem lies with the
OpenSER code. But something in the combination of those is causing a huge
performance problem. For all I know the problem could be in an Ethernet
driver in the x86_64 package.
Those of you who have done benchmarking lately -- were your benchmarks run
on x86_64? I would be curious to know if others who have reported
performance problems lately were on x86_64 (I thought at least one other
report indicated they were using that build).
Michael Young
-----Original Message-----
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young
Sent: Tuesday, May 27, 2008 8:16 AM
To: 'Henning Westerholt'; users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Here is what I know (disclaimer: I am not a coder... as far as I am
concerned C is for Cookie, and that's good enough for me):
One of my providers is a company I used to work for. They have a primary
OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new
INVITES per second. I have their ENUM configured to route half of those
calls to my proxy1, and the other half to my proxy2.
At 25 cps (new calls per second), the load on their SER box is .1 to .2 at
most. The processor rarely gets to 1% utilized. OpenSER is using between 200
and 400 Mb of RAM.
At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to
1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1
chokes and stops responding to invites. It is not the processor or the RAM
that are limiting... there is something else involved that takes the load up
to 2.0, at which point it stops responding.
I have stripped down my config, trying to find out where the problem is. At
this point I have wild theories, but few cold hard facts.
Their OpenSER box is running ACC, and writing records to a MYSQL cluster
that lives on a different box. Mine is not running ACC at all anymore. They
use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor, I
am using CentOS x86_64. I actually have a Xeon-based server on order,
wondering if the problem is only on x86_64 based systems.
Increasing the UDP buffer size is the only thing that has gotten me to 10
cps, before I was dying at 3. My OpenSER proxies route calls to a server
farm of ~70 Asterisk boxes. If one of those asterisk servers goes off line,
my proxy's load will immediately go up to 2.0, the processor level will go
to 100%, and it stops processing many (but not all) new calls. It seems to
block some of the processes at this point, if I try to shut OpenSER down I
have to manually kill 6 to 8 of the child processes.
Here are the modules I'm loading:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "mi_fifo.so"
loadmodule "uri_db.so"
loadmodule "uri.so"
loadmodule "xlog.so"
loadmodule "permissions.so"
loadmodule "alias_db.so"
loadmodule "domain.so"
Maybe this information can help someone else who is running into a similar
problem.
Michael
-----Original Message-----
From: Henning Westerholt [mailto:henning.westerholt@1und1.de]
Sent: Tuesday, May 27, 2008 3:22 AM
To: users(a)lists.openser.org
Cc: Michael Young
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
On Saturday 24 May 2008, Michael Young wrote:
I posted a message earlier this week, that said
that my OpenSER install is
not correctly calculating the buffer size as far as I can tell, but have
not gotten a response on that yet. Performance is still not as good as
previous OpenSER versions with this setup (we have a 1.2.1 server that is
running circles around the three 1.3.2 servers).
Hi Michael,
i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases,
probably caused e.g. from the added pseudo-variable stuff, more DNS lookups
and further abstractions. But a slowdown like this sounds more like a bug to
me.
Do you can share more details about the problem?
Cheers,
Henning
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
4:27 p.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
On Friday 30 May 2008, Michael Young wrote:
PS -- Henning, Bogdan, Daniel, etc -- email me off
list and I'll gladly
give you access to the x86_64 server if you want to take a look and\or test
it with sipp.
Hi Michael,
thank you for the offer, in fact we've already a machine available,
devel.openser.org runs with x64_86. Its more a matter of insufficient time at
the moment than a lack of access from my side.
Cheers,
Henning
5:36 a.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
Wow...that's interesting. Have you tried running i386 on the first box?
If that works you may be correct, but if the problem persists you may
have to consider a hardware issue. If not that, then perhaps a
different 64 bit Linux distribution...
Michael Young wrote:
More information on the performance problems I have
been fighting:
I replaced one of my OpenSER production boxes with a new server today. The
old one was running x86_64 version of CentOS 5 (two dual core Opteron
processors). The new one is running the i386 flavor of CentOS 5 (one quad
core Xeon processor). I copied the configs from the old box to the new one.
The boxes were built identically using the same process and applications.
The new box is now matching the performance I was seeing on the 1.2.1 box of
my provider. This new box is easily handling 50 new calls per second.
So this indicates that something is wrong with the combination of Linux
x86_64 and OpenSER 1.3.X.
NOTE: I am NOT blaming OpenSER or saying that the problem lies with the
OpenSER code. But something in the combination of those is causing a huge
performance problem. For all I know the problem could be in an Ethernet
driver in the x86_64 package.
Those of you who have done benchmarking lately -- were your benchmarks run
on x86_64? I would be curious to know if others who have reported
performance problems lately were on x86_64 (I thought at least one other
report indicated they were using that build).
Michael Young
-----Original Message-----
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young
Sent: Tuesday, May 27, 2008 8:16 AM
To: 'Henning Westerholt'; users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Here is what I know (disclaimer: I am not a coder... as far as I am
concerned C is for Cookie, and that's good enough for me):
One of my providers is a company I used to work for. They have a primary
OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new
INVITES per second. I have their ENUM configured to route half of those
calls to my proxy1, and the other half to my proxy2.
At 25 cps (new calls per second), the load on their SER box is .1 to .2 at
most. The processor rarely gets to 1% utilized. OpenSER is using between 200
and 400 Mb of RAM.
At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to
1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1
chokes and stops responding to invites. It is not the processor or the RAM
that are limiting... there is something else involved that takes the load up
to 2.0, at which point it stops responding.
I have stripped down my config, trying to find out where the problem is. At
this point I have wild theories, but few cold hard facts.
Their OpenSER box is running ACC, and writing records to a MYSQL cluster
that lives on a different box. Mine is not running ACC at all anymore. They
use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor, I
am using CentOS x86_64. I actually have a Xeon-based server on order,
wondering if the problem is only on x86_64 based systems.
Increasing the UDP buffer size is the only thing that has gotten me to 10
cps, before I was dying at 3. My OpenSER proxies route calls to a server
farm of ~70 Asterisk boxes. If one of those asterisk servers goes off line,
my proxy's load will immediately go up to 2.0, the processor level will go
to 100%, and it stops processing many (but not all) new calls. It seems to
block some of the processes at this point, if I try to shut OpenSER down I
have to manually kill 6 to 8 of the child processes.
Here are the modules I'm loading:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "mi_fifo.so"
loadmodule "uri_db.so"
loadmodule "uri.so"
loadmodule "xlog.so"
loadmodule "permissions.so"
loadmodule "alias_db.so"
loadmodule "domain.so"
Maybe this information can help someone else who is running into a similar
problem.
Michael
-----Original Message-----
From: Henning Westerholt [mailto:henning.westerholt@1und1.de]
Sent: Tuesday, May 27, 2008 3:22 AM
To: users(a)lists.openser.org
Cc: Michael Young
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
On Saturday 24 May 2008, Michael Young wrote:
I posted a message earlier this week, that said
that my OpenSER install is
not correctly calculating the buffer size as far as I can tell, but have
not gotten a response on that yet. Performance is still not as good as
previous OpenSER versions with this setup (we have a 1.2.1 server that is
running circles around the three 1.3.2 servers).
Hi Michael,
i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases,
probably caused e.g. from the added pseudo-variable stuff, more DNS lookups
and further abstractions. But a slowdown like this sounds more like a bug to
me.
Do you can share more details about the problem?
Cheers,
Henning
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
6:12 a.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
I have four different x86_64 boxes which perform identically. I don't think
it is a hardware issue... although it could be a driver issue of some kind.
All the hardware on those four are similar (Dell rack mount servers, not all
the same model but similar).
If I get a chance this weekend I'll load the i386 distro on the Opteron
based server I took out of production and see what that does.
Michael
-----Original Message-----
From: Mik Cheez [mailto:michael_bulk@wildgate.com]
Sent: Thursday, May 29, 2008 7:36 PM
To: Michael Young
Cc: users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Wow...that's interesting. Have you tried running i386 on the first box?
If that works you may be correct, but if the problem persists you may
have to consider a hardware issue. If not that, then perhaps a
different 64 bit Linux distribution...
Michael Young wrote:
releases,
More information on the performance problems I have
been fighting:
I replaced one of my OpenSER production boxes with a new server today. The
old one was running x86_64 version of CentOS 5 (two dual core Opteron
processors). The new one is running the i386 flavor of CentOS 5 (one quad
core Xeon processor). I copied the configs from the old box to the new
one.
The boxes were built identically using the same
process and applications.
The new box is now matching the performance I was seeing on the 1.2.1 box
of
my provider. This new box is easily handling 50 new
calls per second.
So this indicates that something is wrong with the combination of Linux
x86_64 and OpenSER 1.3.X.
NOTE: I am NOT blaming OpenSER or saying that the problem lies with the
OpenSER code. But something in the combination of those is causing a huge
performance problem. For all I know the problem could be in an Ethernet
driver in the x86_64 package.
Those of you who have done benchmarking lately -- were your benchmarks run
on x86_64? I would be curious to know if others who have reported
performance problems lately were on x86_64 (I thought at least one other
report indicated they were using that build).
Michael Young
-----Original Message-----
From: users-bounces(a)lists.openser.org
[mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young
Sent: Tuesday, May 27, 2008 8:16 AM
To: 'Henning Westerholt'; users(a)lists.openser.org
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
Here is what I know (disclaimer: I am not a coder... as far as I am
concerned C is for Cookie, and that's good enough for me):
One of my providers is a company I used to work for. They have a primary
OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new
INVITES per second. I have their ENUM configured to route half of those
calls to my proxy1, and the other half to my proxy2.
At 25 cps (new calls per second), the load on their SER box is .1 to .2 at
most. The processor rarely gets to 1% utilized. OpenSER is using between
200
and 400 Mb of RAM.
At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets
to
1% utilized, and the server is using 700 Mb of RAM. At
20 cps, my proxy1
chokes and stops responding to invites. It is not the processor or the RAM
that are limiting... there is something else involved that takes the load
up
to 2.0, at which point it stops responding.
I have stripped down my config, trying to find out where the problem is.
At
this point I have wild theories, but few cold hard
facts.
Their OpenSER box is running ACC, and writing records to a MYSQL cluster
that lives on a different box. Mine is not running ACC at all anymore.
They
use ENUM, I'm using dbaliases. They are using
Ubuntu on an i386 processor,
I
am using CentOS x86_64. I actually have a Xeon-based
server on order,
wondering if the problem is only on x86_64 based systems.
Increasing the UDP buffer size is the only thing that has gotten me to 10
cps, before I was dying at 3. My OpenSER proxies route calls to a server
farm of ~70 Asterisk boxes. If one of those asterisk servers goes off
line,
my proxy's load will immediately go up to 2.0, the
processor level will go
to 100%, and it stops processing many (but not all) new calls. It seems to
block some of the processes at this point, if I try to shut OpenSER down I
have to manually kill 6 to 8 of the child processes.
Here are the modules I'm loading:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "rr.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "mi_fifo.so"
loadmodule "uri_db.so"
loadmodule "uri.so"
loadmodule "xlog.so"
loadmodule "permissions.so"
loadmodule "alias_db.so"
loadmodule "domain.so"
Maybe this information can help someone else who is running into a similar
problem.
Michael
-----Original Message-----
From: Henning Westerholt [mailto:henning.westerholt@1und1.de]
Sent: Tuesday, May 27, 2008 3:22 AM
To: users(a)lists.openser.org
Cc: Michael Young
Subject: Re: [OpenSER-Users] Openser not processing requests at rate being
sent to it
On Saturday 24 May 2008, Michael Young wrote:
> I posted a message earlier this week, that said that my OpenSER install
is
not correctly
calculating the buffer size as far as I can tell, but have
not gotten a response on that yet. Performance is still not as good as
previous OpenSER versions with this setup (we have a 1.2.1 server that is
running circles around the three 1.3.2 servers).
Hi Michael,
i can confirm a (albeit minor) slowdown from 0.9 branch to actual probably caused e.g. from the added pseudo-variable
stuff, more DNS
lookups
and further abstractions. But a slowdown like this
sounds more like a bug
to
me.
Do you can share more details about the problem?
Cheers,
Henning
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008
6:49 PM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
_______________________________________________
Users mailing list
Users(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
No virus found in this outgoing message.
Checked by AVG.
Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008
7:27 AM
4:32 p.m.
New subject: [OpenSER-Users] Openser not processing requests at rate being sent to it
On Friday 30 May 2008, Michael Young wrote:
I replaced one of my OpenSER production boxes with a
new server today. The
old one was running x86_64 version of CentOS 5 (two dual core Opteron
processors). The new one is running the i386 flavor of CentOS 5 (one quad
core Xeon processor). I copied the configs from the old box to the new one.
The boxes were built identically using the same process and applications.
The new box is now matching the performance I was seeing on the 1.2.1 box
of my provider. This new box is easily handling 50 new calls per second.
So this indicates that something is wrong with the combination of Linux
x86_64 and OpenSER 1.3.X.
[..]
Those of you who have done benchmarking lately -- were your benchmarks run
on x86_64? I would be curious to know if others who have reported
performance problems lately were on x86_64 (I thought at least one other
report indicated they were using that build).
Hi Michael,
thank you for the update, this is in fact really interesting. My recent tests
were all done on x86. But i think, as 64 bit machines and distributions are
getting more and more common, that this is not a general problem with openSER
and 64 bit, but with the exact setup and distribution you use. Perhaps this
is also a driver issue, as you mentioned.
Cheers,
Henning
5820
days inactive
5827
days old
17 comments
6 participants
participants (6)
-
Andreas Heise -
Bogdan-Andrei Iancu -
Brian Peters -
Henning Westerholt -
Michael Young -
Mik Cheez