hello
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see that kamailio is very slow answering to INVITEs from the inbound carriers. The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other machines with less resources, same OS [CentOS 5.2 64 bits] and it is performing very very good. I know that a high debug level is a enemy but in my case is set to 3 like in the other system, and it is using the same configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are taken in the same machine and I see multiples INVITEs hitting it and no response at all until more than 2 seconds
I would appreciate any help or suggestions
cheers
jp
On Mittwoch, 16. September 2009, Juan Perez wrote:
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see that kamailio is very slow answering to INVITEs from the inbound carriers. The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other machines with less resources, same OS [CentOS 5.2 64 bits] and it is performing very very good. I know that a high debug level is a enemy but in my case is set to 3 like in the other system, and it is using the same configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are taken in the same machine and I see multiples INVITEs hitting it and no response at all until more than 2 seconds
Hi Juan,
i'd suspect that the server could not resolve DNS requests or has some issues accessing the database. Can you check with tcpdump that DNS resolves are not blocked somehow, and how the database responds?
Cheers,
Henning
Hello,
On 16.09.2009 17:40 Uhr, Henning Westerholt wrote:
On Mittwoch, 16. September 2009, Juan Perez wrote:
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM
and see
that kamailio is very slow answering to INVITEs from the inbound
carriers.
The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %,
which is
still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other machines with less resources, same OS [CentOS 5.2 64 bits] and it is performing very very good. I know that a high debug level is a enemy
but in
my case is set to 3 like in the other system, and it is using the same configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are taken in the same machine and I see multiples INVITEs hitting it and no response at all until more than 2 seconds
Hi Juan,
i'd suspect that the server could not resolve DNS requests or has some issues accessing the database. Can you check with tcpdump that DNS resolves are not blocked somehow, and how the database responds?
as hint, using benchmark module is a way to discover which part of the config file is slow: http://kamailio.org/docs/modules/1.5.x/benchmark.html
Cheers, Daniel
Dear Daniel, I misssed your reply, thank you a lot, I will start to work on this inmediately. I will try to capture everything to see whether I am missing something else, in my captures I only filter udp port 5060 regards jp
----- Original Message ---- From: Daniel-Constantin Mierla miconda@gmail.com To: users@lists.kamailio.org Sent: Wednesday, September 16, 2009 11:45:18 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
Hello,
On 16.09.2009 17:40 Uhr, Henning Westerholt wrote:
On Mittwoch, 16. September 2009, Juan Perez wrote:
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see that kamailio is very slow answering to INVITEs from the inbound carriers. The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other machines with less resources, same OS [CentOS 5.2 64 bits] and it is performing very very good. I know that a high debug level is a enemy but in my case is set to 3 like in the other system, and it is using the same configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are taken in the same machine and I see multiples INVITEs hitting it and no response at all until more than 2 seconds
Hi Juan,
i'd suspect that the server could not resolve DNS requests or has some issues accessing the database. Can you check with tcpdump that DNS resolves are not blocked somehow, and how the database responds?
as hint, using benchmark module is a way to discover which part of the config file is slow: http://kamailio.org/docs/modules/1.5.x/benchmark.html
Cheers, Daniel
-- Daniel-Constantin Mierla * http://www.asipto.com/
_______________________________________________ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
thank you Henning
We disabled iptables completely to rule out that posibility and saw the same situation.
On another note, this is my understanding but I may be wrong, once kamailio is started it loads all the routes [lcr module] and addresses [permissions module] from the DB into memory. If the DB is not responsive would kamailio be affected? If no changes are made in the DB that requieres an lcr_reload or address_reload would Kamailio try to access the DB everytime a transaction is to be processed?
, Also, if this helps, we do not use DNS in our config, only IPs, and we are not handling Registrations, it is being used as SIP trunking only
regards
jp
________________________________ From: Henning Westerholt henning.westerholt@1und1.de To: users@lists.kamailio.org Cc: Juan Perez jperezsip2008@yahoo.com Sent: Wednesday, September 16, 2009 11:40:22 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see that kamailio is very slow answering to INVITEs from the inbound carriers. The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other machines with less resources, same OS [CentOS 5.2 64 bits] and it is performing very very good. I know that a high debug level is a enemy but in my case is set to 3 like in the other system, and it is using the same configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are taken in the same machine and I see multiples INVITEs hitting it and no response at all until more than 2 seconds
Hi Juan,
i'd suspect that the server could not resolve DNS requests or has some issues accessing the database. Can you check with tcpdump that DNS resolves are not blocked somehow, and how the database responds?
Cheers,
Henning
On Mittwoch, 16. September 2009, Juan Perez wrote:
We disabled iptables completely to rule out that posibility and saw the same situation.
Hi Juan,
On another note, this is my understanding but I may be wrong, once kamailio is started it loads all the routes [lcr module] and addresses [permissions module] from the DB into memory. If the DB is not responsive would kamailio be affected? If no changes are made in the DB that requieres an lcr_reload or address_reload would Kamailio try to access the DB everytime a transaction is to be processed?
Ok, if you use modules that have a memory cache no database access should be done during runtime.
, Also, if this helps, we do not use DNS in our config, only IPs, and we are not handling Registrations, it is being used as SIP trunking only
Hm, ok.. But the SIP messages that you processing are use DNS names? Can you check with ngrep or tcpdump anyway if you've any DNS problems?
Another thing could the logging, do you use asynchronous syslog logging?
Henning
synch or asynch ? sorry, but how can I check?
this is the section of the kamailio.conf
debug=3 log_stderror=no log_facility=LOG_LOCAL0
and this is the section in the syslog.conf
# log messages with LOG_LOCAL0 in /var/log/ser.log local0.* -/var/log/proxy.log
________________________________ From: Henning Westerholt henning.westerholt@1und1.de To: Juan Perez jperezsip2008@yahoo.com Cc: users@lists.kamailio.org Sent: Wednesday, September 16, 2009 11:54:38 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
We disabled iptables completely to rule out that posibility and saw the same situation.
Hi Juan,
On another note, this is my understanding but I may be wrong, once kamailio is started it loads all the routes [lcr module] and addresses [permissions module] from the DB into memory. If the DB is not responsive would kamailio be affected? If no changes are made in the DB that requieres an lcr_reload or address_reload would Kamailio try to access the DB everytime a transaction is to be processed?
Ok, if you use modules that have a memory cache no database access should be done during runtime.
, Also, if this helps, we do not use DNS in our config, only IPs, and we are not handling Registrations, it is being used as SIP trunking only
Hm, ok.. But the SIP messages that you processing are use DNS names? Can you check with ngrep or tcpdump anyway if you've any DNS problems?
Another thing could the logging, do you use asynchronous syslog logging?
Henning
I think that the "-" in front the log file makes it asynchronous right ?
________________________________ From: Juan Perez jperezsip2008@yahoo.com To: Henning Westerholt henning.westerholt@1und1.de Cc: users@lists.kamailio.org Sent: Wednesday, September 16, 2009 12:17:41 PM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
synch or asynch ? sorry, but how can I check?
this is the section of the kamailio.conf
debug=3 log_stderror=no log_facility=LOG_LOCAL0
and this is the section in the syslog.conf
# log messages with LOG_LOCAL0 in /var/log/ser.log local0.* -/var/log/proxy.log
________________________________ From: Henning Westerholt henning.westerholt@1und1.de To: Juan Perez jperezsip2008@yahoo.com Cc: users@lists.kamailio.org Sent: Wednesday, September 16, 2009 11:54:38 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
We disabled iptables completely to rule out that posibility and saw the same situation.
Hi Juan,
On another note, this is my understanding but I may be wrong, once kamailio is started it loads all the routes [lcr module] and addresses [permissions module] from the DB into memory. If the DB is not responsive would kamailio be affected? If no changes are made in the DB that requieres an lcr_reload or address_reload would Kamailio try to access the DB everytime a transaction is to be processed?
Ok, if you use modules that have a memory cache no database access should be done during runtime.
, Also, if this helps, we do not use DNS in our config, only IPs, and we are not handling Registrations, it is being used as SIP trunking only
Hm, ok.. But the SIP messages that you processing are use DNS names? Can you check with ngrep or tcpdump anyway if you've any DNS problems?
Another thing could the logging, do you use asynchronous syslog logging?
Henning
ok, what else can it be? If I set debug to 0, or comment out all my xlog lines in the config file[which are set to L3] then the logging would be ruled out. Besides, the CPU was 87 % idle, pretty decent, so maybe the CPU was not the culprit. We ruled out iptables by disabling it, the DB access is not taking place because the values are in memory Could it be shared memory , or something similar? How can I tell?
Regards
jperez
________________________________ From: Henning Westerholt henning.westerholt@1und1.de To: Juan Perez jperezsip2008@yahoo.com Cc: users@lists.kamailio.org Sent: Wednesday, September 16, 2009 12:43:24 PM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
I think that the "-" in front the log file makes it asynchronous right ?
Hi Juan,
yes, i think this is the correct setting. So your syslog configuration seems correct.
Regards,
Henning
Juan,
I had similar problems with CentOS 5 x86_64 on a server with multiple dual-core Opteron processors (a Dell Poweredge server). I was unable to find the root problem, but I found that using the 32 bit version of CentOS 5 on an Intel platform resolves the problem. I also tested with various Debian distros on the Opteron-based server and they performed properly. There is something about the combination of CentOS 5 64 bit, Dell servers, Opteron processors, and Kamailio that doesn't get along.
While I suspected DNS-related issues (as did some of the replies on this list), that wasn't it.
Michael Young
-----Original Message----- From: users-bounces@lists.kamailio.org [mailto:users-bounces@lists.kamailio.org] On Behalf Of Juan Perez Sent: Wednesday, September 16, 2009 10:34 AM To: kamailio Subject: [Kamailio-Users] kamailio 1.4.3 slow
hello
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see that kamailio is very slow answering to INVITEs from the inbound carriers. The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other machines with less resources, same OS [CentOS 5.2 64 bits] and it is performing very very good. I know that a high debug level is a enemy but in my case is set to 3 like in the other system, and it is using the same configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are taken in the same machine and I see multiples INVITEs hitting it and no response at all until more than 2 seconds
I would appreciate any help or suggestions
cheers
jp
_______________________________________________ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.409 / Virus Database: 270.13.101/2376 - Release Date: 09/16/09 11:21:00
Dear Michael In another scenario in our secondary site I am using another Dell Poweredge 1950, with dual quad core almost at the same speed and 64 bits , same version of CentOS and it is working like a charm. A lot of calls and CPU idle almost 100% all the time. That is we decided to buy a more powerful server to have a more robust platform in our main site but we haven't been able to move forward with this because of the bad performance. Any other ideas guys ?
regards jp
----- Original Message ---- From: Michael Young myoung@redmonsters.net To: kamailio users@lists.kamailio.org Sent: Wednesday, September 16, 2009 1:27:46 PM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
Juan,
I had similar problems with CentOS 5 x86_64 on a server with multiple dual-core Opteron processors (a Dell Poweredge server). I was unable to find the root problem, but I found that using the 32 bit version of CentOS 5 on an Intel platform resolves the problem. I also tested with various Debian distros on the Opteron-based server and they performed properly. There is something about the combination of CentOS 5 64 bit, Dell servers, Opteron processors, and Kamailio that doesn't get along.
While I suspected DNS-related issues (as did some of the replies on this list), that wasn't it.
Michael Young
-----Original Message----- From: users-bounces@lists.kamailio.org [mailto:users-bounces@lists.kamailio.org] On Behalf Of Juan Perez Sent: Wednesday, September 16, 2009 10:34 AM To: kamailio Subject: [Kamailio-Users] kamailio 1.4.3 slow
hello
I installed kamailio in a pc with dual quad core @ 3.3GHz, 4 GB RAM and see that kamailio is very slow answering to INVITEs from the inbound carriers. The DEBUG is set to 3 and the idle CPU lowers only to 87% or 90 %, which is still decent but anyways the performance is very bad.
This is the first time that I see this, I have it installed in other machines with less resources, same OS [CentOS 5.2 64 bits] and it is performing very very good. I know that a high debug level is a enemy but in my case is set to 3 like in the other system, and it is using the same configuration file. What other thing may affect the performance?
I have made sure that it is not a network issue because the captures are taken in the same machine and I see multiples INVITEs hitting it and no response at all until more than 2 seconds
I would appreciate any help or suggestions
cheers
jp
_______________________________________________ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.409 / Virus Database: 270.13.101/2376 - Release Date: 09/16/09 11:21:00
_______________________________________________ Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
On Mittwoch, 16. September 2009, Juan Perez wrote:
In another scenario in our secondary site I am using another Dell Poweredge 1950, with dual quad core almost at the same speed and 64 bits , same version of CentOS and it is working like a charm. A lot of calls and CPU idle almost 100% all the time. That is we decided to buy a more powerful server to have a more robust platform in our main site but we haven't been able to move forward with this because of the bad performance. Any other ideas guys ?
Hello Juan,
as this looks more like a hardware issue to me i'd try to find issue that causes this particular server to be that slow. Somewhere it must spend this time. Some ideas:
- Start kamailio with only one worker children, and then use strace to see what its doing when its hangs - enable debugging (only for a few seconds if you've production load on the system) and then look to the timestamps where its spend the time - use the benchmark module as Daniel already suggested - try to replicate the setup in a lab environment and then use a load tester (e.g. sipp) to give some traffic to the system. then change certain parts of the configuration and setup to find the culprit
Cheers,
Henning
than you a lot Henning for your suggestions, I will try to do all of that but the lab stuff because we already have a lab setup but it works superb too, only on the new machine is when we have found the prob. in all our other scenarios kamailio runs super fast, processing a lot of calls per seconds and the cpu is 99-100 % idle. fortunately we can do this in the same machine because it has an amount of traffic controllable by us so at any given time we can take it out or put more or use sipp, I will post the results as soon as I have something new, thank you guys a lot again
cheers jp
________________________________ From: Henning Westerholt henning.westerholt@1und1.de To: users@lists.kamailio.org Cc: Juan Perez jperezsip2008@yahoo.com; myoung@redmonsters.net Sent: Thursday, September 17, 2009 6:05:07 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
In another scenario in our secondary site I am using another Dell Poweredge 1950, with dual quad core almost at the same speed and 64 bits , same version of CentOS and it is working like a charm. A lot of calls and CPU idle almost 100% all the time. That is we decided to buy a more powerful server to have a more robust platform in our main site but we haven't been able to move forward with this because of the bad performance. Any other ideas guys ?
Hello Juan,
as this looks more like a hardware issue to me i'd try to find issue that causes this particular server to be that slow. Somewhere it must spend this time. Some ideas:
- Start kamailio with only one worker children, and then use strace to see what its doing when its hangs - enable debugging (only for a few seconds if you've production load on the system) and then look to the timestamps where its spend the time - use the benchmark module as Daniel already suggested - try to replicate the setup in a lab environment and then use a load tester (e.g. sipp) to give some traffic to the system. then change certain parts of the configuration and setup to find the culprit
Cheers,
Henning
-
Daniel-Constantin Mierla -
Henning Westerholt -
Juan Perez -
Michael Young