[SR-Users] Info: initialization of libssl 1.1.+ with PTHREAD_PROCESS_SHARED attribute for pthread mutex and rwlock

Wed Sep 18 16:37:06 CEST 2019

Sorry..forgot bt full output

(gdb) bt full

#0  0x0000562fcc3544c0 in ?? ()

No symbol table info available.

#1  0x0000562fcc354632 in ?? ()

No symbol table info available.

#2  0x0000562fcc354974 in local_timer_run ()

No symbol table info available.

#3  0x0000562fcc3fb256 in ?? ()

No symbol table info available.

#4  0x0000562fcc3fb64c in tcp_receive_loop ()

No symbol table info available.

#5  0x0000562fcc2acf52 in tcp_init_children ()

No symbol table info available.

#6  0x0000562fcc1ad94a in main_loop ()

No symbol table info available.

#7  0x0000562fcc1b4a7b in main ()

No symbol table info available.

(gdb)

On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen <achen at fuze.com> wrote:

> backtrace output:
>
>
> (gdb) bt
>
> #0  0x0000562fcc3544c0 in ?? ()
>
> #1  0x0000562fcc354632 in ?? ()
>
> #2  0x0000562fcc354974 in local_timer_run ()
>
> #3  0x0000562fcc3fb256 in ?? ()
>
> #4  0x0000562fcc3fb64c in tcp_receive_loop ()
>
> #5  0x0000562fcc2acf52 in tcp_init_children ()
>
> #6  0x0000562fcc1ad94a in main_loop ()
>
> #7  0x0000562fcc1b4a7b in main ()
>
> (gdb)
>
>
>
>
> On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen <achen at fuze.com> wrote:
>
>> I'll look for the core.
>>
>> As part of our release process, I don't think we can release master
>> branch of Kamailio.
>>
>> On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> did you get core files? If yes, it would be good to see the gdb
>>> bracktraces.
>>>
>>> Have you used the pre-loaded library workaround?
>>>
>>> The code was not backported to branch 5.2 yet, I wanted to wait a bit
>>> and see if there are reports with good/bad results. If you want to use it,
>>> then you have to install nightly builds from master branch (development
>>> version, should be 5.3.0-pre1 at this moment).
>>>
>>> Cheers,
>>> Daniel
>>> On 18.09.19 15:26, Andrew Chen wrote:
>>>
>>> Hi Daniel,
>>> Which apt repository can we get this workaround?   Today I noticed our
>>> Kamailio 5.2.4 cored during one of our load test sessions:
>>>
>>> Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip
>>> 00007f8501e58585 sp 00007fffd5c84248 error 4 in
>>> libcrypto.so.1.1[7f8501dd4000+29b000]
>>>
>>> [Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip
>>> 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in
>>> libcrypto.so.1.1[7f1a8f90f000+29b000]
>>>
>>> [Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip
>>> 00007f0a85dac128 sp 00007fff6847aad0 error 4 in
>>> libcrypto.so.1.1[7f0a85c40000+29b000]
>>>
>>> [Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip
>>> 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in
>>> libcrypto.so.1.1[7f0a85c40000+29b000]
>>>
>>> [Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip
>>> 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in
>>> libcrypto.so.1.1[7f75e1136000+29b000]
>>>
>>> [Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip
>>> 00007f0f37f31585 sp 00007ffde54daa38 error 4 in
>>> libcrypto.so.1.1[7f0f37ead000+29b000]
>>>
>>> [Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip
>>> 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in
>>> libcrypto.so.1.1[7f8043b0a000+29b000]
>>>
>>> [Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip
>>> 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in
>>> libcrypto.so.1.1[7ff59d250000+29b000]
>>>
>>> [Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip
>>> 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in
>>> libcrypto.so.1.1[7ff59d250000+29b000]
>>>
>>> [Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip
>>> 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in
>>> libcrypto.so.1.1[7f22dd4db000+29b000]
>>>
>>> [Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip
>>> 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in
>>> libcrypto.so.1.1[7f7ccae9b000+29b000]
>>>
>>> [Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip
>>> 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in
>>> libcrypto.so.1.1[7fb309baf000+29b000]
>>>
>>> [Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip
>>> 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in
>>> libcrypto.so.1.1[7fc5908d7000+29b000]
>>>
>>> [Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip
>>> 00007fcc2394a585 sp 00007fffa9c52258 error 4 in
>>> libcrypto.so.1.1[7fcc238c6000+29b000]
>>>
>>> [Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip
>>> 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in
>>> libcrypto.so.1.1[7fcc238c6000+29b000]
>>>
>>> [Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip
>>> 00007f586f812585 sp 00007ffeae1adcd8 error 4 in
>>> libcrypto.so.1.1[7f586f78e000+29b000]
>>>
>>> [Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip
>>> 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in
>>> kamailio[562fcc17c000+4a2000]
>>>
>>> [Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip
>>> 00007f96c3735c3e sp 00007ffe825ca270 error 4 in
>>> dialog.so[7f96c36d6000+be000]
>>>
>>> [Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip
>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in
>>> kamailio[562fcc17c000+4a2000]
>>>
>>> [Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip
>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
>>>
>>> [Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip
>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in
>>> kamailio[562fcc17c000+4a2000]
>>>
>>> [Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip
>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in
>>> kamailio[562fcc17c000+4a2000]
>>>
>>> [Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip
>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
>>>
>>> [Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip
>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
>>>
>>> [Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip
>>> 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in
>>> kamailio[562fcc17c000+4a2000]
>>>
>>>
>>> and I was wondering if this core is related.
>>>
>>> Thanks
>>>
>>>
>>> On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov <abalashov at evaristesys.com>
>>> wrote:
>>>
>>>> Hi Daniel,
>>>>
>>>> Thanks a lot for this work; it's definitely important that Kamailio
>>>> build against OpenSSL v1.1, and I'll be happy to contribute some testing
>>>> in a live environment in the coming weeks. Much appreciated.
>>>>
>>>> Have a good weekend,
>>>>
>>>> -- Alex
>>>>
>>>> On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla
>>>> wrote:
>>>>
>>>> > Hello,
>>>> > after couple of months waiting for feedback about the workaround with
>>>> the
>>>> > preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute
>>>> for
>>>> > pthread mutex/rwlock in order to avoid deadlocks and as it seems to
>>>> be the
>>>> > solution, I just pushed a few commits to move that code in the core of
>>>> > kamailio.
>>>> >
>>>> > It this way the initialization with the PTHREAD_PROCESS_SHARED
>>>> attribute is
>>>> > done by default when libssl v1.1 is detected at compile time, without
>>>> the
>>>> > need of using the approach with a preloaded library.
>>>> >
>>>> > The shared library is still kept in the tls module:
>>>> >   *
>>>> >
>>>> https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared
>>>> >
>>>> > Probably is going to be removed in the near future.
>>>> >
>>>> > I am asking the community members that use Kamailio on systems with
>>>> libssl
>>>> > v1.1+ to test the master branch and give feedback. The master branch
>>>> is
>>>> > anyhow in testing phase for preparing the next major release v5.3.0,
>>>> so it
>>>> > will help for that as well.
>>>> >
>>>> > If no issue is going to be reported soon, the commits will be
>>>> backported to
>>>> > stable branches, to re-enable smooth use of Kamailio with libssl
>>>> v1.1+.
>>>> >
>>>> > When libssl v1.1+ is detected, the core links with pthread, but that
>>>> was
>>>> > already present in the Makefiles.defs for different compile options.
>>>> The
>>>> > pthread more or less a default library on linuxes/unixes, so this
>>>> should
>>>> > not introduce any major dependency to the Kamailio core.
>>>> >
>>>> > Cheers,
>>>> > Daniel
>>>> >
>>>> > --
>>>> > Daniel-Constantin Mierla - http://www.asipto.com
>>>> > http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>
>>>> > _______________________________________________
>>>> > Kamailio (SER) - Users Mailing List
>>>> > sr-users at lists.kamailio.org
>>>> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>> --
>>>> Alex Balashov | Principal | Evariste Systems LLC
>>>>
>>>> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
>>>> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
>>>>
>>>> _______________________________________________
>>>> Kamailio (SER) - Users Mailing List
>>>> sr-users at lists.kamailio.org
>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>
>>>
>>> --
>>> Andy Chen
>>> Sr. Telephony Lead Engineer
>>> 415 516 5535 (M)
>>> achen@ <achen at thinkingphones.com>fuze.com
>>>
>>>
>>> *Confidentiality Notice: The information contained in this e-mail and any
>>> attachments may be confidential. If you are not an intended recipient,
>>> you
>>> are hereby notified that any dissemination, distribution or copying of
>>> this
>>> e-mail is strictly prohibited. If you have received this e-mail in error,
>>> please notify the sender and permanently delete the e-mail and any
>>> attachments immediately. You should not retain, copy or use this e-mail
>>> or
>>> any attachment for any purpose, nor disclose all or any part of the
>>> contents to any other person. Thank you.*
>>>
>>> _______________________________________________
>>> Kamailio (SER) - Users Mailing Listsr-users at lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>> --
>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
>>>
>>>
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> 415 516 5535 (M)
>> achen@ <achen at thinkingphones.com>fuze.com
>>
>>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <achen at thinkingphones.com>fuze.com
>
>

-- 
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <achen at thinkingphones.com>fuze.com

-- 
*Confidentiality Notice: The information contained in this e-mail and any

attachments may be confidential. If you are not an intended recipient, you

are hereby notified that any dissemination, distribution or copying of this

e-mail is strictly prohibited. If you have received this e-mail in error,

please notify the sender and permanently delete the e-mail and any

attachments immediately. You should not retain, copy or use this e-mail or

any attachment for any purpose, nor disclose all or any part of the

contents to any other person. Thank you.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20190918/63eda44e/attachment.html>