16 Sep
2009
16 Sep
'09
3:33 p.m.
hello
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see that kamailio
is very slow answering to INVITEs from the inbound carriers.
The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is still decent
but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other machines with less
resources, same OS [CentOS 5.2 64 bits] and it is performing very very good. I know that a
high debug level is a enemy but in my case is set to 3 like in the other system, and it is
using the same configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are taken in the same
machine and I see multiples INVITEs hitting it and no response at all until more than 2
seconds
I would appreciate any help or suggestions
cheers
jp
16 Sep
16 Sep
3:40 p.m.
On Mittwoch, 16. September 2009, Juan Perez wrote:
I installed kamailio in a pc with dual quad core @
3.3GHz, 4 GB RAM and see
that kamailio is very slow answering to INVITEs from the inbound carriers.
The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is
still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other
machines with less resources, same OS [CentOS 5.2 64 bits] and it is
performing very very good. I know that a high debug level is a enemy but in
my case is set to 3 like in the other system, and it is using the same
configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are
taken in the same machine and I see multiples INVITEs hitting it and no
response at all until more than 2 seconds
Hi Juan,
i'd suspect that the server could not resolve DNS requests or has some issues
accessing the database. Can you check with tcpdump that DNS resolves are not
blocked somehow, and how the database responds?
Cheers,
Henning
3:45 p.m.
Hello,
On 16.09.2009 17:40 Uhr, Henning Westerholt wrote:
On Mittwoch, 16. September 2009, Juan Perez wrote:
as hint, using
benchmark module is a way to discover which part of the
config file is slow:
http://kamailio.org/docs/modules/1.5.x/benchmark.html
Cheers,
Daniel
--
Daniel-Constantin Mierla
* http://www.asipto.com/
I installed kamailio in a pc with dual quad core
@ 3.3GHz, 4 GB RAM
and see
that kamailio is very slow answering to INVITEs
from the inbound
carriers.
The DEBUG is set to 3 and the idle CPU lowers
only to 87% or 90 %,
which is
still decent but anyways the performance is very
bad.
This is the first time that I see this, I have it installed in other
machines with less resources, same OS [CentOS 5.2 64 bits] and it is
performing very very good. I know that a high debug level is a enemy
but in
my case is set to 3 like in the other system, and
it is using the same
configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are
taken in the same machine and I see multiples INVITEs hitting it and no
response at all until more than 2 seconds
Hi Juan,
i'd suspect that the server could not resolve DNS requests or has some
issues accessing the database. Can you check with tcpdump that DNS
resolves are not blocked somehow, and how the database responds?
8:54 p.m.
Dear Daniel, I misssed your reply, thank you a lot, I will start to work on this
inmediately.
I will try to capture everything to see whether I am missing something else, in my
captures I only filter udp port 5060
regards
jp
----- Original Message ----
From: Daniel-Constantin Mierla <miconda(a)gmail.com>
To: users(a)lists.kamailio.org
Sent: Wednesday, September 16, 2009 11:45:18 AM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
Hello,
On 16.09.2009 17:40 Uhr, Henning Westerholt wrote:
On Mittwoch, 16. September 2009, Juan Perez wrote:
as hint, using benchmark module is a way to
discover which part of the config file is slow:
http://kamailio.org/docs/modules/1.5.x/benchmark.html
Cheers,
Daniel
-- Daniel-Constantin Mierla
* http://www.asipto.com/
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
I installed kamailio in a pc with dual quad core
@ 3.3GHz, 4 GB RAM and see
that kamailio is very slow answering to INVITEs from the inbound carriers.
The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is
still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other
machines with less resources, same OS [CentOS 5.2 64 bits] and it is
performing very very good. I know that a high debug level is a enemy but in
my case is set to 3 like in the other system, and it is using the same
configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are
taken in the same machine and I see multiples INVITEs hitting it and no
response at all until more than 2 seconds
Hi Juan,
i'd suspect that the server could not resolve DNS requests or has some issues
accessing the database. Can you check with tcpdump that DNS resolves are not blocked
somehow, and how the database responds?
3:52 p.m.
thank you Henning
We disabled iptables completely to rule out that posibility and saw the same situation.
On another note, this is my understanding but I may be wrong, once kamailio is started it
loads all the routes [lcr module] and addresses [permissions module] from the DB into
memory. If the DB is not responsive would kamailio be affected?
If no changes are made in the DB that requieres an lcr_reload or address_reload would
Kamailio try to access the DB everytime a transaction is to be processed?
, Also, if this helps, we do not use DNS in our config, only IPs, and we are not handling
Registrations, it is being used as SIP trunking only
regards
jp
________________________________
From: Henning Westerholt <henning.westerholt(a)1und1.de>
To: users(a)lists.kamailio.org
Cc: Juan Perez <jperezsip2008(a)yahoo.com>
Sent: Wednesday, September 16, 2009 11:40:22 AM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
I installed kamailio in a pc with dual quad core @
3.3GHz, 4 GB RAM and see
that kamailio is very slow answering to INVITEs from the inbound carriers.
The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is
still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other
machines with less resources, same OS [CentOS 5.2 64 bits] and it is
performing very very good. I know that a high debug level is a enemy but in
my case is set to 3 like in the other system, and it is using the same
configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are
taken in the same machine and I see multiples INVITEs hitting it and no
response at all until more than 2 seconds
Hi Juan,
i'd suspect that the server could not resolve DNS requests or has some issues
accessing the database. Can you check with tcpdump that DNS resolves are not blocked
somehow, and how the database responds?
Cheers,
Henning
3:54 p.m.
On Mittwoch, 16. September 2009, Juan Perez wrote:
We disabled iptables completely to rule out that
posibility and saw the
same situation.
Hi Juan,
On another note, this is my understanding but I may be
wrong, once kamailio
is started it loads all the routes [lcr module] and addresses [permissions
module] from the DB into memory. If the DB is not responsive would kamailio
be affected? If no changes are made in the DB that requieres an lcr_reload
or address_reload would Kamailio try to access the DB everytime a
transaction is to be processed?
Ok, if you use modules that have a memory cache no database access should be
done during runtime.
, Also, if this helps, we do not use DNS in our
config, only IPs, and we
are not handling Registrations, it is being used as SIP trunking only
Hm, ok.. But the SIP messages that you processing are use DNS names? Can you
check with ngrep or tcpdump anyway if you've any DNS problems?
Another thing could the logging, do you use asynchronous syslog logging?
Henning
4:17 p.m.
synch or asynch ? sorry, but how can I check?
this is the section of the kamailio.conf
debug=3
log_stderror=no
log_facility=LOG_LOCAL0
and this is the section in the syslog.conf
# log messages with LOG_LOCAL0 in /var/log/ser.log
local0.* -/var/log/proxy.log
________________________________
From: Henning Westerholt <henning.westerholt(a)1und1.de>
To: Juan Perez <jperezsip2008(a)yahoo.com>
Cc: users(a)lists.kamailio.org
Sent: Wednesday, September 16, 2009 11:54:38 AM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
We disabled iptables completely to rule out that
posibility and saw the
same situation.
Hi Juan,
On another note, this is my understanding but I may be
wrong, once kamailio
is started it loads all the routes [lcr module] and addresses [permissions
module] from the DB into memory. If the DB is not responsive would kamailio
be affected? If no changes are made in the DB that requieres an lcr_reload
or address_reload would Kamailio try to access the DB everytime a
transaction is to be processed?
Ok, if you use modules that have a memory cache no database access should be done during
runtime.
, Also, if this helps, we do not use DNS in our
config, only IPs, and we
are not handling Registrations, it is being used as SIP trunking only
Hm, ok.. But the SIP messages that you processing are use DNS names? Can you check with
ngrep or tcpdump anyway if you've any DNS problems?
Another thing could the logging, do you use asynchronous syslog logging?
Henning
4:28 p.m.
I think that the "-" in front the log file makes it asynchronous right ?
________________________________
From: Juan Perez <jperezsip2008(a)yahoo.com>
To: Henning Westerholt <henning.westerholt(a)1und1.de>
Cc: users(a)lists.kamailio.org
Sent: Wednesday, September 16, 2009 12:17:41 PM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
synch or asynch ? sorry, but how can I check?
this is the section of the kamailio.conf
debug=3
log_stderror=no
log_facility=LOG_LOCAL0
and this is the section in the syslog.conf
# log messages with LOG_LOCAL0 in /var/log/ser.log
local0.* -/var/log/proxy.log
________________________________
From: Henning Westerholt <henning.westerholt(a)1und1.de>
To: Juan Perez <jperezsip2008(a)yahoo.com>
Cc: users(a)lists.kamailio.org
Sent: Wednesday, September 16, 2009 11:54:38 AM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
We disabled iptables completely to rule out that
posibility and saw the
same situation.
Hi Juan,
On another note, this is my understanding but I may be
wrong, once kamailio
is started it loads all the routes [lcr module] and addresses [permissions
module] from the DB into memory. If the DB is not responsive would kamailio
be affected? If no changes are made in the DB that requieres an lcr_reload
or address_reload would Kamailio try to access the DB everytime a
transaction is to be processed?
Ok, if you use modules that have a memory cache no database access should be done during
runtime.
, Also, if this helps, we do not use DNS in our
config, only IPs, and we
are not handling Registrations, it is being used as SIP trunking only
Hm, ok.. But the SIP messages that you processing are use DNS names? Can you check with
ngrep or tcpdump anyway if you've any DNS problems?
Another thing could the logging, do you use asynchronous syslog logging?
Henning
6:04 p.m.
ok, what else can it be?
If I set debug to 0, or comment out all my xlog lines in the config file[which are set to
L3] then the logging would be ruled out.
Besides, the CPU was 87 % idle, pretty decent, so maybe the CPU was not the culprit.
We ruled out iptables by disabling it, the DB access is not taking place because the
values are in memory
Could it be shared memory , or something similar?
How can I tell?
Regards
jperez
________________________________
From: Henning Westerholt <henning.westerholt(a)1und1.de>
To: Juan Perez <jperezsip2008(a)yahoo.com>
Cc: users(a)lists.kamailio.org
Sent: Wednesday, September 16, 2009 12:43:24 PM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
I think that the "-" in front the log file
makes it asynchronous right ?
Hi Juan,
yes, i think this is the correct setting. So your syslog configuration seems correct.
Regards,
Henning
5:27 p.m.
Juan,
I had similar problems with CentOS 5 x86_64 on a server with multiple
dual-core Opteron processors (a Dell Poweredge server). I was unable to find
the root problem, but I found that using the 32 bit version of CentOS 5 on
an Intel platform resolves the problem. I also tested with various Debian
distros on the Opteron-based server and they performed properly. There is
something about the combination of CentOS 5 64 bit, Dell servers, Opteron
processors, and Kamailio that doesn't get along.
While I suspected DNS-related issues (as did some of the replies on this
list), that wasn't it.
Michael Young
-----Original Message-----
From: users-bounces(a)lists.kamailio.org
[mailto:users-bounces@lists.kamailio.org] On Behalf Of Juan Perez
Sent: Wednesday, September 16, 2009 10:34 AM
To: kamailio
Subject: [Kamailio-Users] kamailio 1.4.3 slow
hello
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see
that kamailio is very slow answering to INVITEs from the inbound carriers.
The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is
still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other
machines with less resources, same OS [CentOS 5.2 64 bits] and it is
performing very very good. I know that a high debug level is a enemy but in
my case is set to 3 like in the other system, and it is using the same
configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are
taken in the same machine and I see multiples INVITEs hitting it and no
response at all until more than 2 seconds
I would appreciate any help or suggestions
cheers
jp
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.101/2376 - Release Date: 09/16/09
11:21:00
7:40 p.m.
Dear Michael
In another scenario in our secondary site I am using another Dell Poweredge 1950, with
dual quad core almost at the same speed and 64 bits , same version of CentOS and it is
working like a charm. A lot of calls and CPU idle almost 100% all the time.
That is we decided to buy a more powerful server to have a more robust platform in our
main site but we haven't been able to move forward with this because of the bad
performance.
Any other ideas guys ?
regards
jp
----- Original Message ----
From: Michael Young <myoung(a)redmonsters.net>
To: kamailio <users(a)lists.kamailio.org>
Sent: Wednesday, September 16, 2009 1:27:46 PM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
Juan,
I had similar problems with CentOS 5 x86_64 on a server with multiple
dual-core Opteron processors (a Dell Poweredge server). I was unable to find
the root problem, but I found that using the 32 bit version of CentOS 5 on
an Intel platform resolves the problem. I also tested with various Debian
distros on the Opteron-based server and they performed properly. There is
something about the combination of CentOS 5 64 bit, Dell servers, Opteron
processors, and Kamailio that doesn't get along.
While I suspected DNS-related issues (as did some of the replies on this
list), that wasn't it.
Michael Young
-----Original Message-----
From: users-bounces(a)lists.kamailio.org
[mailto:users-bounces@lists.kamailio.org] On Behalf Of Juan Perez
Sent: Wednesday, September 16, 2009 10:34 AM
To: kamailio
Subject: [Kamailio-Users] kamailio 1.4.3 slow
hello
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see
that kamailio is very slow answering to INVITEs from the inbound carriers.
The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is
still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other
machines with less resources, same OS [CentOS 5.2 64 bits] and it is
performing very very good. I know that a high debug level is a enemy but in
my case is set to 3 like in the other system, and it is using the same
configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are
taken in the same machine and I see multiples INVITEs hitting it and no
response at all until more than 2 seconds
I would appreciate any help or suggestions
cheers
jp
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.5.409 / Virus Database: 270.13.101/2376 - Release Date: 09/16/09
11:21:00
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
17 Sep
17 Sep
10:05 a.m.
On Mittwoch, 16. September 2009, Juan Perez wrote:
In another scenario in our secondary site I am using
another Dell Poweredge
1950, with dual quad core almost at the same speed and 64 bits , same
version of CentOS and it is working like a charm. A lot of calls and CPU
idle almost 100% all the time. That is we decided to buy a more powerful
server to have a more robust platform in our main site but we haven't been
able to move forward with this because of the bad performance. Any other
ideas guys ?
Hello Juan,
as this looks more like a hardware issue to me i'd try to find issue that
causes this particular server to be that slow. Somewhere it must spend this
time. Some ideas:
- Start kamailio with only one worker children, and then use strace to see
what its doing when its hangs
- enable debugging (only for a few seconds if you've production load on the
system) and then look to the timestamps where its spend the time
- use the benchmark module as Daniel already suggested
- try to replicate the setup in a lab environment and then use a load tester
(e.g. sipp) to give some traffic to the system. then change certain parts of
the configuration and setup to find the culprit
Cheers,
Henning
12:49 p.m.
than you a lot Henning for your suggestions, I will try to do all of that but the lab
stuff because we already have a lab setup but it works superb too, only on the new machine
is when we have found the prob. in all our other scenarios kamailio runs super fast,
processing a lot of calls per seconds and the cpu is 99-100 % idle.
fortunately we can do this in the same machine because it has an amount of traffic
controllable by us so at any given time we can take it out or put more or use sipp, I will
post the results as soon as I have something new, thank you guys a lot again
cheers
jp
________________________________
From: Henning Westerholt <henning.westerholt(a)1und1.de>
To: users(a)lists.kamailio.org
Cc: Juan Perez <jperezsip2008(a)yahoo.com>om>; myoung(a)redmonsters.net
Sent: Thursday, September 17, 2009 6:05:07 AM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
In another scenario in our secondary site I am using
another Dell Poweredge
1950, with dual quad core almost at the same speed and 64 bits , same
version of CentOS and it is working like a charm. A lot of calls and CPU
idle almost 100% all the time. That is we decided to buy a more powerful
server to have a more robust platform in our main site but we haven't been
able to move forward with this because of the bad performance. Any other
ideas guys ?
Hello Juan,
as this looks more like a hardware issue to me i'd try to find issue that causes this
particular server to be that slow. Somewhere it must spend this time. Some ideas:
- Start kamailio with only one worker children, and then use strace to see what its doing
when its hangs
- enable debugging (only for a few seconds if you've production load on the system)
and then look to the timestamps where its spend the time
- use the benchmark module as Daniel already suggested
- try to replicate the setup in a lab environment and then use a load tester (e.g. sipp)
to give some traffic to the system. then change certain parts of the configuration and
setup to find the culprit
Cheers,
Henning
5354
days inactive
5355
days old
13 comments
4 participants
participants (4)
-
Daniel-Constantin Mierla -
Henning Westerholt -
Juan Perez -
Michael Young