Hello, thanks for working on this and providing the workaround solution with preloaded symbols. I would suggest that you push it to master, to make it easier to deploy and test. There is a change I would do: install the openssl_mutex_shared via Makefile from tls module, not from the main Makefile. The ctl module does the same for kamcmd, practically is about setting a var there -- ctl has: MOD_INSTALL_UTILS=../../../utils/kamcmd Another one (I can look into it as well if proves not to be straightforward): place this inside the tls module, like: src/modules/tls/utils/openssl_mutex_shared -- the Makefile of the module needs the path to the folder. If we then get couple of people testing and validating the use of process shared mutex, then we can pursue with openssl devs to add an option for it. Cheers, Daniel On 11.04.19 17:14, Richard Fuchs wrote: > Hi, > > (X-posted to sr-dev as this is getting into the nitty gritty) > > As a short-term workaround for this, I've been playing with the > preloaded library approach to hijack the pthread mutex calls and force > them to provide process-shared mutexes. AFAICT this seems to be > working and only has the minuscule performance impact of using slower > process-shared mutexes in all instances, even when they aren't required. > > The code for the preloaded library itself is very short and simple: > https://gist.github.com/rfuchs/1bb7348b6acbe37e557d94c2f69a1498 > > As a more complete patch that integrates it into the build system > (probably badly): > https://gist.github.com/rfuchs/b240ffe87938a45e6f2a4cf53fe29f17 > > Finally it requires adding it to the startup script, for example in a > systemd service file as: > > Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so' > > > (that's with a hard coded path which isn't optimal of course). > > I don't consider this a proper fix, but only a hacky workaround, but > it might be a solution for the very near future. Throwing it out there > in case other people have been working on similar approaches, and/or > maybe have some comments about this. > > Cheers > > > On 01/04/2019 04.52, Daniel-Constantin Mierla wrote: >> Hello, >> >> an update on this issue -- I spent a bit of time looking at >> libssl/libcrypto library and the problem can be the type of mutexes they >> use now internally starting with v1.1, respectively the pthread mutex. >> They are not process shared and kamailio is a multi-process application, >> working with the same tls connection from multiple processes. >> >> Today I wrote to openssl mailing list, waiting now to see if I get any >> hints from there. >> >> Cheers, >> Daniel >> >> On 01.04.19 10:33, Kristijan Vrban wrote: >>> Hi Andrew, >>> >>> yes, with openssl 1.0.2 Kamailio is now up and running since five >>> days. Looks good so far. >>> >>> Kristijan >>> >>> Am Do., 28. März 2019 um 11:09 Uhr schrieb Andrew Pogrebennyk >>> <apogrebennyk(a)sipwise.com>om>: >>>> On 3/26/19 3:52 PM, Kristijan Vrban wrote: >>>>>> Just curious, did you get to compile with OpenSSL 1.0 and test? >>>>> Just compiled with OpenSSL 1.0 . Gone test now. >>>> Kristijan, >>>> any new occurrences since you have recompiled kamailio with openssl >>>> 1.0? >>>> >>>> Regards, >>>> Andrew >>> _______________________________________________ >>> Kamailio (SER) - Users Mailing List >>> sr-users(a)lists.kamailio.org >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > _______________________________________________ > Kamailio (SER) - Development Mailing List > sr-dev(a)lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Thanks Richard, I pushed a README.md inside openssl_mutex_shared folder with some details about how to use it and a short note to the README of tls module to refer to it for libssl v1.1. I think all Unix-like systems have now pthread library, the issue that can be with some old *BSD-es is the lack of implementation for PTHREAD_PROCESS_SHARED option, but I think it would compile even there. I am going to merge to master, ask the community to do some tests and see the results. Cheers, Daniel On 11.04.19 20:46, Richard Fuchs wrote: > Hi, > > Thanks for the suggestion, that seems to work just fine. I've pushed > it into a private branch `rfuchs/openssl-locking-fix` for now, as I'm > pretty sure that this would break compilation on non-pthread systems. > Ideally it should be built conditionally only for OpenSSL < 1.1. But > anyway, feel free to play with it. > > https://github.com/kamailio/kamailio/tree/rfuchs/openssl-locking-fix > https://github.com/kamailio/kamailio/commit/efdc141ecb5ff72e3224e47deaaa79f… > > > Cheers > > > On 11/04/2019 13.03, Daniel-Constantin Mierla wrote: >> Hello, >> >> thanks for working on this and providing the workaround solution with >> preloaded symbols. >> >> I would suggest that you push it to master, to make it easier to deploy >> and test. There is a change I would do: install the openssl_mutex_shared >> via Makefile from tls module, not from the main Makefile. The ctl module >> does the same for kamcmd, practically is about setting a var there -- >> ctl has: >> >> MOD_INSTALL_UTILS=../../../utils/kamcmd >> >> Another one (I can look into it as well if proves not to be >> straightforward): place this inside the tls module, like: >> src/modules/tls/utils/openssl_mutex_shared -- the Makefile of the module >> needs the path to the folder. >> >> If we then get couple of people testing and validating the use of >> process shared mutex, then we can pursue with openssl devs to add an >> option for it. >> >> Cheers, >> Daniel >> >> On 11.04.19 17:14, Richard Fuchs wrote: >>> Hi, >>> >>> (X-posted to sr-dev as this is getting into the nitty gritty) >>> >>> As a short-term workaround for this, I've been playing with the >>> preloaded library approach to hijack the pthread mutex calls and force >>> them to provide process-shared mutexes. AFAICT this seems to be >>> working and only has the minuscule performance impact of using slower >>> process-shared mutexes in all instances, even when they aren't >>> required. >>> >>> The code for the preloaded library itself is very short and simple: >>> https://gist.github.com/rfuchs/1bb7348b6acbe37e557d94c2f69a1498 >>> >>> As a more complete patch that integrates it into the build system >>> (probably badly): >>> https://gist.github.com/rfuchs/b240ffe87938a45e6f2a4cf53fe29f17 >>> >>> Finally it requires adding it to the startup script, for example in a >>> systemd service file as: >>> >>> Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so' >>> >>> >>> >>> (that's with a hard coded path which isn't optimal of course). >>> >>> I don't consider this a proper fix, but only a hacky workaround, but >>> it might be a solution for the very near future. Throwing it out there >>> in case other people have been working on similar approaches, and/or >>> maybe have some comments about this. >>> >>> Cheers >>> >>> >>> On 01/04/2019 04.52, Daniel-Constantin Mierla wrote: >>>> Hello, >>>> >>>> an update on this issue -- I spent a bit of time looking at >>>> libssl/libcrypto library and the problem can be the type of mutexes >>>> they >>>> use now internally starting with v1.1, respectively the pthread mutex. >>>> They are not process shared and kamailio is a multi-process >>>> application, >>>> working with the same tls connection from multiple processes. >>>> >>>> Today I wrote to openssl mailing list, waiting now to see if I get any >>>> hints from there. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 01.04.19 10:33, Kristijan Vrban wrote: >>>>> Hi Andrew, >>>>> >>>>> yes, with openssl 1.0.2 Kamailio is now up and running since five >>>>> days. Looks good so far. >>>>> >>>>> Kristijan >>>>> >>>>> Am Do., 28. März 2019 um 11:09 Uhr schrieb Andrew Pogrebennyk >>>>> &lt;apogrebennyk(a)sipwise.com&gt;om>: >>>>>> On 3/26/19 3:52 PM, Kristijan Vrban wrote: >>>>>>>> Just curious, did you get to compile with OpenSSL 1.0 and test? >>>>>>> Just compiled with OpenSSL 1.0 . Gone test now. >>>>>> Kristijan, >>>>>> any new occurrences since you have recompiled kamailio with openssl >>>>>> 1.0? >>>>>> >>>>>> Regards, >>>>>> Andrew >>>>> _______________________________________________ >>>>> Kamailio (SER) - Users Mailing List >>>>> sr-users(a)lists.kamailio.org >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>> _______________________________________________ >>> Kamailio (SER) - Development Mailing List >>> sr-dev(a)lists.kamailio.org >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev > _______________________________________________ > Kamailio (SER) - Development Mailing List > sr-dev(a)lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev