[Devel] Re: [Serdev] OpenSER/SER with OSP performance test results

[Jiri Kuthan]

At 09:16 04/05/2007, Olaf Bergmann wrote:
>Di-Shi Sun wrote:
>> Hi All,
>>  
>> We have performed a benchmark test on OpenSER V1.1, V1.2 and SER 2.0 to
>> understand and compare the performance of the three releases in a
>> simulated production environment. 
>
>Nice, thanks for this interesting piece of work.
>
>> Summary of the test results:
>> ============================
>> * The performance of OpenSER V1.2 and SER 2.0 are not materially
>> different, however, there are two minor differences.
>>   - SER V2.0 requires less memory.
>>   - OpenSER V1.2 has less post dial delay.
>
>Could you please comment on the PDD graph? For my understanding, the
>6+ seconds are caused by your failure scenarios? I wonder why the
>SER graph seems to be constant while the OpenSER looks like exponential?

I have been struggling with the measurement too  (actually I'm even 
missing PDD definition in the document). In a private conversation with
authors I learned that the test scenario is actually about randomized-
order forking, with some of the destinations being unavailable. That explains why 
SER is having a constant failure rate but it does not explain why
openser is doing better initially (perhaps blacklisting turned on
by default in openser?) and going like exponential later.

Some few more results would be good in this context too (graph showing
the actual delay as opposed to percentage exceeding a threshold -- which
is fine for the 'big picture' but hard to disaggregate for tracing
what's actually going on).

Other thing which came out of a private chat with authors is that ser
measurements are in SER's debugging mode (which is set by default
on CVS). SER is compiled with PKG_MALLOC, DBG_QM_MALLOC while 
openser without (F_MALLOC).

Otherwise for sake of completenss, I would enjoy attached (open)ser config files
(among others it shall reveal if black-listing is turned on as that should
have dramatic impact on the results), described call-flows, and described 
scenario (I mean details about this randomized-order forking).

Apart from this piece of critique, I think it is a great contribution and
with some work along the lines I have suggested it will be an excellent
document.

-jiri



--
Jiri Kuthan            http://iptel.org/~jiri/