[Devel] Re: [Serdev] OpenSER/SER with OSP performance test results

[di-shi at transnexus.com]

Hi Olaf and Jiri,

Thank you for your comments about the test results.

As we had mentioned, this test was designed to understand the performance of
OpenSER/SER in product environments. Some of the facts were random. The PDD
is an indirect measure and not very precise. But there are some details may
be useful to understand the PDD graph.
For every call,
1. The source is a SIPp client.
2. OpenSER/SER receives the INVITE message from the SIPp client and requests
routing info from a set of OSP servers.
3. There are 5 destinations configured on the OSP servers.
 a. 3 unavailable devices for different reasons. All will cause
OpenSER/SER fr_timer timeout which we set to 2 sec.
 b. 1 device rejects the call and replies 404.
 c. 1 good device. SIPp server.
 OpenSER/SER gets these 5 destinations in random order. The worst
condition, OpenSER/SER trys the SIPp server as the last destination. The PPD
should be 6 sec. For OpenSER 1.1/1.2 test, it is clear that the PDD depends
on the load. It is reasonable. For SER, it can be explained as the PPD is
just a little longer than 6 sec for the worst condition. The 6 sec threshold
is not a good value. It should be set to 6.1 sec. Unforunately, we did not
realize it until we finished the test for SER.

For the OpenSER/SER configurations we used in the test, you can find them
under module/osp/etc/sample-osp-openser.cfg and
module/osp/etc/sample-osp-ser.cfg. We only changed the fr_timer to 2 sec.
and set the OSP server IPs and the local device IP.

Thanks

Di-Shi Sun.


----- Original Message ----- 
From: "Jiri Kuthan" <jiri at iptel.org>
To: "Olaf Bergmann" <Olaf.Bergmann at freenet-ag.de>; "Di-Shi Sun"
<di-shi at transnexus.com>
Cc: <devel at openser.org>; <serdev at lists.iptel.org>
Sent: Friday, May 04, 2007 6:29 PM
Subject: Re: [Serdev] OpenSER/SER with OSP performance test results


> At 09:16 04/05/2007, Olaf Bergmann wrote:
> >Di-Shi Sun wrote:
> >> Hi All,
> >>
> >> We have performed a benchmark test on OpenSER V1.1, V1.2 and SER 2.0 to
> >> understand and compare the performance of the three releases in a
> >> simulated production environment.
> >
> >Nice, thanks for this interesting piece of work.
> >
> >> Summary of the test results:
> >> ============================
> >> * The performance of OpenSER V1.2 and SER 2.0 are not materially
> >> different, however, there are two minor differences.
> >> - SER V2.0 requires less memory.
> >> - OpenSER V1.2 has less post dial delay.
> >
> >Could you please comment on the PDD graph? For my understanding, the
> >6+ seconds are caused by your failure scenarios? I wonder why the
> >SER graph seems to be constant while the OpenSER looks like exponential?
>
> I have been struggling with the measurement too (actually I'm even
> missing PDD definition in the document). In a private conversation with
> authors I learned that the test scenario is actually about randomized-
> order forking, with some of the destinations being unavailable. That
explains why
> SER is having a constant failure rate but it does not explain why
> openser is doing better initially (perhaps blacklisting turned on
> by default in openser?) and going like exponential later.
>
> Some few more results would be good in this context too (graph showing
> the actual delay as opposed to percentage exceeding a threshold -- which
> is fine for the 'big picture' but hard to disaggregate for tracing
> what's actually going on).
>
> Other thing which came out of a private chat with authors is that ser
> measurements are in SER's debugging mode (which is set by default
> on CVS). SER is compiled with PKG_MALLOC, DBG_QM_MALLOC while
> openser without (F_MALLOC).
>
> Otherwise for sake of completenss, I would enjoy attached (open)ser config
files
> (among others it shall reveal if black-listing is turned on as that should
> have dramatic impact on the results), described call-flows, and described
> scenario (I mean details about this randomized-order forking).
>
> Apart from this piece of critique, I think it is a great contribution and
> with some work along the lines I have suggested it will be an excellent
> document.
>
> -jiri
>
>
>
> --
> Jiri Kuthan http://iptel.org/~jiri/
>
>
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://openser.org/pipermail/devel/attachments/20070507/b0724f85/attachment.htm