we finally found what was causing the prob, at least the prob is gone after doing this:

In the syslog config file of the server where kamailio was working fine we found this entry:

local0.none /var/log/messages

we added it to the server with probs and problem solved.

Since we were using the facility 0 to log the messages from kamailio apparently we need to add that entry listed above as well to make the logging work better.

this is how we setup kamailio's logging, using the example from the web:

# log messages with LOG_LOCAL0 in /var/log/ser.log
local0.* -/var/log/proxy.log

I do not know the reason why but it fixed our issue. I wanted to share with you.

cheers and thank you a lot for your assistance

From: Juan Perez <jperezsip2008@yahoo.com>
To: Henning Westerholt <henning.westerholt@1und1.de>; users@lists.kamailio.org
Cc: myoung@redmonsters.net
Sent: Thursday, September 17, 2009 8:49:00 AM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow

than you a lot Henning for your suggestions, I will try to do all of that but the lab stuff because we already have a lab setup but it works superb too, only on the new machine is when we have found the prob. in all our other scenarios kamailio runs super fast, processing a lot of calls per seconds and the cpu is 99-100 % idle.

fortunately we can do this in the same machine because it has an amount of traffic controllable by us so at any given time we can take it out or put more or use sipp, I will post the results as soon as I have something new, thank you guys a lot again

cheers

From: Henning Westerholt <henning.westerholt@1und1.de>
To: users@lists.kamailio.org
Cc: Juan Perez <jperezsip2008@yahoo.com>; myoung@redmonsters.net
Sent: Thursday, September 17, 2009 6:05:07 AM
Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow

On Mittwoch, 16. September 2009, Juan Perez wrote:
> In another scenario in our secondary site I am using another Dell Poweredge
> 1950, with dual quad core almost at the same speed and 64 bits , same
> version of CentOS and it is working like a charm. A lot of calls and CPU
> idle almost 100% all the time. That is we decided to buy a more powerful
> server to have a more robust platform in our main site but we haven't been
> able to move forward with this because of the bad performance. Any other
> ideas guys ?

Hello Juan,

as this looks more like a hardware issue to me i'd try to find issue that causes this particular server to be that slow. Somewhere it must spend this time. Some ideas:

- Start kamailio with only one worker children, and then use strace to see what its doing when its hangs
- enable debugging (only for a few seconds if you've production load on the system) and then look to the timestamps where its spend the time
- use the benchmark module as Daniel already suggested
- try to replicate the setup in a lab environment and then use a load tester (e.g. sipp) to give some traffic to the system. then change certain parts of the configuration and setup to find the culprit

Cheers,

Henning