Hi Guys,
I am running a simple REGISTER load test on:
1) Kamailio sever with 2 cores - mem 5G 2) Kamailio server with 4 cores - mem 16G
Both are EC2 instances.
At -r = 500 i.e. 500 reg/sec sipP test works fine with very few re-trans. But when i increase it to 800 reg/sec it starts retransmissions in bulk.
I don't see server's CPU, load-avg or memory shooting. Running everything by default, even using kamctl to start the instance. So I have not tuned any params yet.
My main concern is how can I make server choke and get its actual capacity, and avoid these retransmissions as apparently kamailio is not even utilizing 2 cores and CPU usage is under 10 always.
Results of both server is same i.e. 500 reg/sec max so I am sure there is some problem that needs a fix, but need some hints here.
Also used multiple SIPp's to rule out if it is SIPp issue, but after 500 I see same problem.
Regards, JSZ
You might be running into semi-invisible/not widely publicised (but definitely very real) PPS limits on given instance sizes.
On March 27, 2017 3:09:56 PM EDT, Jade SZ jitterbuffer@gmail.com wrote:
Hi Guys,
I am running a simple REGISTER load test on:
- Kamailio sever with 2 cores - mem 5G
- Kamailio server with 4 cores - mem 16G
Both are EC2 instances.
At -r = 500 i.e. 500 reg/sec sipP test works fine with very few re-trans. But when i increase it to 800 reg/sec it starts retransmissions in bulk.
I don't see server's CPU, load-avg or memory shooting. Running everything by default, even using kamctl to start the instance. So I have not tuned any params yet.
My main concern is how can I make server choke and get its actual capacity, and avoid these retransmissions as apparently kamailio is not even utilizing 2 cores and CPU usage is under 10 always.
Results of both server is same i.e. 500 reg/sec max so I am sure there is some problem that needs a fix, but need some hints here.
Also used multiple SIPp's to rule out if it is SIPp issue, but after 500 I see same problem.
Regards, JSZ
-- Alex
-- Principal, Evariste Systems LLC (www.evaristesys.com)
Sent from my Google Nexus.
Thanks Alex, that's a great hint. I'll check.
PS: Also thanks a lot for the great blog post on SIP server tuning.
On Tue, Mar 28, 2017 at 12:15 AM, Alex Balashov abalashov@evaristesys.com wrote:
You might be running into semi-invisible/not widely publicised (but definitely very real) PPS limits on given instance sizes.
On March 27, 2017 3:09:56 PM EDT, Jade SZ jitterbuffer@gmail.com wrote:
Hi Guys,
I am running a simple REGISTER load test on:
- Kamailio sever with 2 cores - mem 5G
- Kamailio server with 4 cores - mem 16G
Both are EC2 instances.
At -r = 500 i.e. 500 reg/sec sipP test works fine with very few re-trans. But when i increase it to 800 reg/sec it starts retransmissions in bulk.
I don't see server's CPU, load-avg or memory shooting. Running everything by default, even using kamctl to start the instance. So I have not tuned any params yet.
My main concern is how can I make server choke and get its actual capacity, and avoid these retransmissions as apparently kamailio is not even utilizing 2 cores and CPU usage is under 10 always.
Results of both server is same i.e. 500 reg/sec max so I am sure there is some problem that needs a fix, but need some hints here.
Also used multiple SIPp's to rule out if it is SIPp issue, but after 500 I see same problem.
Regards, JSZ
-- Alex
-- Principal, Evariste Systems LLC (www.evaristesys.com)
Sent from my Google Nexus.
On Tue, Mar 28, 2017 at 12:52:32AM +0500, Jade SZ wrote:
PS: Also thanks a lot for the great blog post on SIP server tuning.
You're very welcome. Glad it proved useful to someone!
On 27/03/2017 21:15, Alex Balashov wrote:
You might be running into semi-invisible/not widely publicised (but definitely very real) PPS limits on given instance sizes.
I assume you refer to AWS instance packets per second limits -- I was not aware of it, although I expected somehow. Has anyone encountered a similar situation and can share some numbers that they faced during past experiences? It would be useful to have in mind when dimensioning sip server deployments...
Cheers, Daniel
On March 27, 2017 3:09:56 PM EDT, Jade SZ jitterbuffer@gmail.com wrote:
Hi Guys,
I am running a simple REGISTER load test on:
- Kamailio sever with 2 cores - mem 5G
- Kamailio server with 4 cores - mem 16G
Both are EC2 instances.
At -r = 500 i.e. 500 reg/sec sipP test works fine with very few re-trans. But when i increase it to 800 reg/sec it starts retransmissions in bulk.
I don't see server's CPU, load-avg or memory shooting. Running everything by default, even using kamctl to start the instance. So I have not tuned any params yet.
My main concern is how can I make server choke and get its actual capacity, and avoid these retransmissions as apparently kamailio is not even utilizing 2 cores and CPU usage is under 10 always.
Results of both server is same i.e. 500 reg/sec max so I am sure there is some problem that needs a fix, but need some hints here.
Also used multiple SIPp's to rule out if it is SIPp issue, but after 500 I see same problem.
Regards, JSZ
-- Alex
-- Principal, Evariste Systems LLC (www.evaristesys.com)
Sent from my Google Nexus.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
besides Alex' suggestion to look at AWS PPS limits, few things to take in consideration in such case:
- if CPU usage is low, then if you use DNS for routing, be sure that DNS server is very responsive - if you do auth with database, be sure that database is very responsive - if you print extensive log messages, be sure syslog is configured asynchronous - check the received queue on sip port with netstat, if it is high value, then kamailio is stuck in some operations (like those above) and doesn't read as fast as the end point transmits - if sipp runs on a low capacity system, I noticed that it cannot handle the responses at high throughput even when they are sent to it and it actually thinks it hasn't received them and do retransmissions
Cheers, Daniel
On 27/03/2017 21:09, Jade SZ wrote:
Hi Guys,
I am running a simple REGISTER load test on:
- Kamailio sever with 2 cores - mem 5G
- Kamailio server with 4 cores - mem 16G
Both are EC2 instances.
At -r = 500 i.e. 500 reg/sec sipP test works fine with very few re-trans. But when i increase it to 800 reg/sec it starts retransmissions in bulk.
I don't see server's CPU, load-avg or memory shooting. Running everything by default, even using kamctl to start the instance. So I have not tuned any params yet.
My main concern is how can I make server choke and get its actual capacity, and avoid these retransmissions as apparently kamailio is not even utilizing 2 cores and CPU usage is under 10 always.
Results of both server is same i.e. 500 reg/sec max so I am sure there is some problem that needs a fix, but need some hints here.
Also used multiple SIPp's to rule out if it is SIPp issue, but after 500 I see same problem.
Regards, JSZ
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi, Interesting to know that AWS has PPS limitation.
Found this article to be useful : http://techblog.cloudperf.net/2016/05/2-million-packets-per-second-on-public...
Regards, Sammy
On Tue, Mar 28, 2017 at 3:37 AM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
Hello,
besides Alex' suggestion to look at AWS PPS limits, few things to take in consideration in such case:
- if CPU usage is low, then if you use DNS for routing, be sure that DNS
server is very responsive
- if you do auth with database, be sure that database is very responsive
- if you print extensive log messages, be sure syslog is configured
asynchronous
- check the received queue on sip port with netstat, if it is high
value, then kamailio is stuck in some operations (like those above) and doesn't read as fast as the end point transmits
- if sipp runs on a low capacity system, I noticed that it cannot handle
the responses at high throughput even when they are sent to it and it actually thinks it hasn't received them and do retransmissions
Cheers, Daniel
On 27/03/2017 21:09, Jade SZ wrote:
Hi Guys,
I am running a simple REGISTER load test on:
- Kamailio sever with 2 cores - mem 5G
- Kamailio server with 4 cores - mem 16G
Both are EC2 instances.
At -r = 500 i.e. 500 reg/sec sipP test works fine with very few re-trans. But when i increase it to 800 reg/sec it starts retransmissions in bulk.
I don't see server's CPU, load-avg or memory shooting. Running everything by default, even using kamctl to start the instance. So I have not tuned any params yet.
My main concern is how can I make server choke and get its actual capacity, and avoid these retransmissions as apparently kamailio is not even utilizing 2 cores and CPU usage is under 10 always.
Results of both server is same i.e. 500 reg/sec max so I am sure there is some problem that needs a fix, but need some hints here.
Also used multiple SIPp's to rule out if it is SIPp issue, but after 500 I see same problem.
Regards, JSZ
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlawww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training - May 22-24 (USA) - www.asipto.com Kamailio World Conference - May 8-10, 2017 - www.kamailioworld.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Alex Balashov -
Daniel-Constantin Mierla -
Jade SZ -
SamyGo