[Devel] Re: [Serdev] OpenSER/SER with OSP performance test results

Tue May 8 09:30:10 CEST 2007

Di-Shi,
First of all, thank you for such a thorough performance test and 
especially for the detailed documentation.  As Jiri points out, we 
encourage performance tests, SER standalone or comparing with 
alternatives, as this both allows developers to get more feedback on how 
their code performs, and users of SER really need performance tests in 
order to assess their own setups, make design decisions, and try to 
identity where bottlenecks may turn up.

We have an iptel.org page for performance: 
http://www.iptel.org/ser/doc/performance
If you allow, I would like to make a link to the test (or you can do it 
yourself; the page can be edited).

Some comments to the test:
- I'm still not sure that I understand the Post-Dial Delay. You state 
that 20% of the calls will complete on the fourth attempt. So, this 
means that three INVITEs will time out on 2,000 ms (as SER 2.0 now has a 
higher timer resolution, see 
http://www.iptel.org/how_the_new_timer_framework_works), which is 6,000 
ms or 6s.  Wouldn't you then expect 20% of the calls to complete in >6s 
?  I may have completely misunderstood this, but calls completing in 
less than 6s would do so due to the impreciseness of the old 0.9 timers?
So, to understand what actually happens, a scatter diagram (instead of 
the groupings) would be better?

- Do you have any idea what happened when CPU > 90% and SER's call 
completion dropped?  Looks strange to me. I also noticed that debugging 
was turned on (which, btw, we have discussed that we probably will turn 
off when we release).

- Kudos to all developers for the near-linear scaling of call per second 
per CPU! (which we knew of course, but I still like to point out :-)

- Allow me to quote Jim Dalton's (TransNexus) blog post about the test 
(http://transnexus.blogspot.com/2007/04/openser-performance-benchmark.html):
"If we had used all four CPU cores we expect the results would have been 
800 calls per second. To be conservative, we would recommend service 
providers to plan on maximum CPU utilization of about 60%. This would 
establish the OpenSER planning gauideline of 500 calls per second on a 
server with two, dual core Xeon CPUs.
If you assume 15% of a service provider's traffic occurs during the busy 
hour, a 50% Answer Seizure Ratio (ASR) and a 3 minute average call 
duration, then 500 calls per second equates to 540 million minutes of 
VoIP traffic per month! We think this is impressive for an open source 
SIP proxy running on a server with a retail price of $2,967."
(of course, when referring to openser here, I assume he really means *SER)

g-)

di-shi at transnexus.com wrote:
> Hi Olaf and Jiri,
>
> Thank you for your comments about the test results.
>
> As we had mentioned, this test was designed to understand the 
> performance of
> OpenSER/SER in product environments. Some of the facts were random. 
> The PDD
> is an indirect measure and not very precise. But there are some 
> details may
> be useful to understand the PDD graph.
> For every call,
> 1. The source is a SIPp client.
> 2. OpenSER/SER receives the INVITE message from the SIPp client and 
> requests
> routing info from a set of OSP servers.
> 3. There are 5 destinations configured on the OSP servers.
>     a. 3 unavailable devices for different reasons. All will cause
> OpenSER/SER fr_timer timeout which we set to 2 sec.
>     b. 1 device rejects the call and replies 404.
>     c. 1 good device. SIPp server.
>     OpenSER/SER gets these 5 destinations in random order. The worst
> condition, OpenSER/SER trys the SIPp server as the last destination. 
> The PPD
> should be 6 sec. For OpenSER 1.1/1.2 test, it is clear that the PDD 
> depends
> on the load. It is reasonable. For SER, it can be explained as the PPD is
> just a little longer than 6 sec for the worst condition. The 6 sec 
> threshold
> is not a good value. It should be set to 6.1 sec. Unforunately, we did not
> realize it until we finished the test for SER.
>
> For the OpenSER/SER configurations we used in the test, you can find them
> under module/osp/etc/sample-osp-openser.cfg and
> module/osp/etc/sample-osp-ser.cfg. We only changed the fr_timer to 2 sec.
> and set the OSP server IPs and the local device IP.
>
> Thanks
>
> Di-Shi Sun.
>
>
> ----- Original Message -----
> From: "Jiri Kuthan" <jiri at iptel.org <mailto:jiri at iptel.org>>
> To: "Olaf Bergmann" <Olaf.Bergmann at freenet-ag.de 
> <mailto:Olaf.Bergmann at freenet-ag.de>>; "Di-Shi Sun"
> <di-shi at transnexus.com <mailto:di-shi at transnexus.com>>
> Cc: <devel at openser.org <mailto:devel at openser.org>>; 
> <serdev at lists.iptel.org <mailto:serdev at lists.iptel.org>>
> Sent: Friday, May 04, 2007 6:29 PM
> Subject: Re: [Serdev] OpenSER/SER with OSP performance test results
>
>
> > At 09:16 04/05/2007, Olaf Bergmann wrote:
> > >Di-Shi Sun wrote:
> > >> Hi All,
> > >>
> > >> We have performed a benchmark test on OpenSER V1.1, V1.2 and SER 
> 2.0 to
> > >> understand and compare the performance of the three releases in a
> > >> simulated production environment.
> > >
> > >Nice, thanks for this interesting piece of work.
> > >
> > >> Summary of the test results:
> > >> ============================
> > >> * The performance of OpenSER V1.2 and SER 2.0 are not materially
> > >> different, however, there are two minor differences.
> > >>   - SER V2.0 requires less memory.
> > >>   - OpenSER V1.2 has less post dial delay.
> > >
> > >Could you please comment on the PDD graph? For my understanding, the
> > >6+ seconds are caused by your failure scenarios? I wonder why the
> > >SER graph seems to be constant while the OpenSER looks like 
> exponential?
> >
> > I have been struggling with the measurement too  (actually I'm even
> > missing PDD definition in the document). In a private conversation with
> > authors I learned that the test scenario is actually about randomized-
> > order forking, with some of the destinations being unavailable. That
> explains why
> > SER is having a constant failure rate but it does not explain why
> > openser is doing better initially (perhaps blacklisting turned on
> > by default in openser?) and going like exponential later.
> >
> > Some few more results would be good in this context too (graph showing
> > the actual delay as opposed to percentage exceeding a threshold -- which
> > is fine for the 'big picture' but hard to disaggregate for tracing
> > what's actually going on).
> >
> > Other thing which came out of a private chat with authors is that ser
> > measurements are in SER's debugging mode (which is set by default
> > on CVS). SER is compiled with PKG_MALLOC, DBG_QM_MALLOC while
> > openser without (F_MALLOC).
> >
> > Otherwise for sake of completenss, I would enjoy attached (open)ser 
> config
> files
> > (among others it shall reveal if black-listing is turned on as that 
> should
> > have dramatic impact on the results), described call-flows, and 
> described
> > scenario (I mean details about this randomized-order forking).
> >
> > Apart from this piece of critique, I think it is a great 
> contribution and
> > with some work along the lines I have suggested it will be an excellent
> > document.
> >
> > -jiri
> >
> >
> >
> > --
> > Jiri Kuthan            http://iptel.org/~jiri/ 
> <http://iptel.org/%7Ejiri/>
> >
> >
> >
> ------------------------------------------------------------------------
>
> _______________________________________________
> Serdev mailing list
> Serdev at lists.iptel.org
> http://lists.iptel.org/mailman/listinfo/serdev
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openser.org/pipermail/devel/attachments/20070508/e2f329b1/attachment.html