[SR-Users] Huge TLS performance regression migrating from 3.3.7 to 4.2.6

[Tristan Mahé]

Hi Daniel,

Same config, same server, only kamailio version is different, starting
the timer in the beginning of the config, and ending it just after the
save("location").

Here is the result of 4.2.6:
INFO: benchmark [benchmark.c:351]: _bm_log_timer(): benchmark (timer
CHECKREG [0]): 2480 [ msgs/total/min/max/avg - LR:
100/2585000/1272/190744/25850.000000 | GB:
1900/79666245/1272/335018/41929.602632]

And 3.3:
INFO: benchmark [benchmark.c:351]: benchmark (timer CHECKREG [0]): 17022
[ msgs/total/min/max/avg - LR: 100/2139618/1239/167551/21396.180000 |
GB: 1900/49367288/1217/192815/25982.783158]

Let me try to pinpoint better now what is the difference here.

PS: I sent you a private mail about this, in case you have time :)

On 10/20/2015 11:38 AM, Tristan Mahé wrote:
> Hi Daniel,
>
> I am doing that right now, thank you.
>
> I will keep you posted as told on IRC.
>
> Interesting part to keep everyone in the loop is when I activated the
> "latency" logs options, I could see some messages regarding some core
> functions taking more than 250ms to complete, or sometimes just a
> route(SOMETHING). The other interesting part is that doing some
> optimizations on the tls parameters seems to raise the limit where
> Kamailio becomes completely unresponsive and starts dropping
> connections ( where on the 3.3 there was no need ).
>
> I'll be back very soon with more detailed results.
>
> Best,
>
> Tristan.
>
> On 10/19/2015 07:03 AM, Daniel-Constantin Mierla wrote:
>> Hello,
>>
>> can you use benchamrk module to see if any part of config got slower?
>> if executing of config from beginning of request_route to save() is
>> more or less the same, then setup of connection could be the issue.
>>
>> Since 3.3., there was some work on usrloc module about gruu/outbound
>> and doing unregister if the tcp/tls connection is lost. On tls
>> module, there was some work for SNI. In tcp core, some callbacks were
>> added for websockets. If I haven't missed something relevant, I can't
>> imagine what would create this handling issue between the versions.
>>
>> Cheers,
>> Daniel
>>
>> On 15/10/15 07:29, Tristan Mahé wrote:
>>> Hi Daniel,
>>>
>>> It is on the exact same server, same system configuration, same
>>> configuration trimmed down for fitting 3.3, packages installed from
>>> the kamailio debian wheezy repository.
>>>
>>> I can reproduce it just using sipp and a simple REGISTER loop.
>>>
>>> Do you want me to run some specific tests ? provide you some
>>> specific traces ?
>>>
>>> Best,
>>>
>>> Tristan.
>>>
>>> On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
>>>> Hello,
>>>>
>>>> is the same operating system and server, or are you using some
>>>> other machine for the new version? I can't remember right now any
>>>> big change in tcp/tls side, besides sni, which should not have any
>>>> relevant impact as described here.
>>>>
>>>> Also, have you installed from git branch 4.2 or the
>>>> tarball/packages of 4.2.6?
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 15/10/15 03:36, Tristan Mahé wrote:
>>>>> Hi List,
>>>>>
>>>>> I've been reaching a strange thing lately, trying to upgrade to kamailio
>>>>> 4.2.6 ( identical config ):
>>>>>
>>>>> - 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
>>>>> easily reproduced ).
>>>>> - more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
>>>>> at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
>>>>> - kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
>>>>> accepting new tcp connections, then resume a bit later, then stops
>>>>> again,... ) => starts happening when Kamailio is treating more than 20k
>>>>> tcp connections, with between 500-1k req/s. During those periods, trying
>>>>> to connect to kamailio with an openssl s_client -connect will end up in
>>>>> a timeout or with a long delay before the certificate is received ( 03
>>>>> CONNECTED ).
>>>>> - kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
>>>>> connections: after a while, the number of active connections starts to
>>>>> decrease on kam4, on kam3 it stays constant.
>>>>>
>>>>> I was wondering if I was alone in this situation or if someone else is
>>>>> impacted, has some advices ?
>>>>>
>>>>> It does not seem to be a config issue, as it is the same that was
>>>>> deployed with both versions.
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>>> sr-users at lists.sip-router.org
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>> -- 
>>>> Daniel-Constantin Mierla
>>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>> Book: SIP Routing With Kamailio - http://www.asipto.com
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
>>>> sr-users at lists.sip-router.org
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>
>> -- 
>> Daniel-Constantin Mierla
>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>> Book: SIP Routing With Kamailio - http://www.asipto.com
>
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users at lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151020/de7c1480/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 473 bytes
Desc: OpenPGP digital signature
URL: <http://lists.sip-router.org/pipermail/sr-users/attachments/20151020/de7c1480/attachment.sig>