Info: initialization of libssl 1.1.+ with PTHREAD_PROCESS_SHARED attribute for pthread mutex and rwlock
Hello, after couple of months waiting for feedback about the workaround with the preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for pthread mutex/rwlock in order to avoid deadlocks and as it seems to be the solution, I just pushed a few commits to move that code in the core of kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED attribute is done by default when libssl v1.1 is detected at compile time, without the need of using the approach with a preloaded library.
The shared library is still kept in the tls module: * https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with libssl v1.1+ to test the master branch and give feedback. The master branch is anyhow in testing phase for preparing the next major release v5.3.0, so it will help for that as well.
If no issue is going to be reported soon, the commits will be backported to stable branches, to re-enable smooth use of Kamailio with libssl v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that was already present in the Makefiles.defs for different compile options. The pthread more or less a default library on linuxes/unixes, so this should not introduce any major dependency to the Kamailio core.
Cheers, Daniel
Hi Daniel,
Glad to hear you integrated this workaround into the master branch! We compiled it for testing and for now it seems to be OK. We do not notice any issue regarding it. We will let you know if so.
Thanks, Julien
De : sr-users sr-users-bounces@lists.kamailio.org au nom de Daniel-Constantin Mierla miconda@gmail.com Répondre à : "Kamailio (SER) - Users Mailing List" sr-users@lists.kamailio.org Date : vendredi 13 septembre 2019 à 13:34 À : "Kamailio (SER) - Users Mailing List" sr-users@lists.kamailio.org, sr-dev sr-dev@lists.kamailio.org Objet : [SR-Users] Info: initialization of libssl 1.1.+ with PTHREAD_PROCESS_SHARED attribute for pthread mutex and rwlock
Hello, after couple of months waiting for feedback about the workaround with the preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for pthread mutex/rwlock in order to avoid deadlocks and as it seems to be the solution, I just pushed a few commits to move that code in the core of kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED attribute is done by default when libssl v1.1 is detected at compile time, without the need of using the approach with a preloaded library.
The shared library is still kept in the tls module: * https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with libssl v1.1+ to test the master branch and give feedback. The master branch is anyhow in testing phase for preparing the next major release v5.3.0, so it will help for that as well.
If no issue is going to be reported soon, the commits will be backported to stable branches, to re-enable smooth use of Kamailio with libssl v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that was already present in the Makefiles.defs for different compile options. The pthread more or less a default library on linuxes/unixes, so this should not introduce any major dependency to the Kamailio core.
Cheers, Daniel
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hi Daniel,
Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated.
Have a good weekend,
-- Alex
On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote:
Hello, after couple of months waiting for feedback about the workaround with the preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for pthread mutex/rwlock in order to avoid deadlocks and as it seems to be the solution, I just pushed a few commits to move that code in the core of kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED attribute is done by default when libssl v1.1 is detected at compile time, without the need of using the approach with a preloaded library.
The shared library is still kept in the tls module:
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with libssl v1.1+ to test the master branch and give feedback. The master branch is anyhow in testing phase for preparing the next major release v5.3.0, so it will help for that as well.
If no issue is going to be reported soon, the commits will be backported to stable branches, to re-enable smooth use of Kamailio with libssl v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that was already present in the Makefiles.defs for different compile options. The pthread more or less a default library on linuxes/unixes, so this should not introduce any major dependency to the Kamailio core.
Cheers, Daniel
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov abalashov@evaristesys.com wrote:
Hi Daniel,
Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated.
Have a good weekend,
-- Alex
On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote:
Hello, after couple of months waiting for feedback about the workaround with the preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for pthread mutex/rwlock in order to avoid deadlocks and as it seems to be
the
solution, I just pushed a few commits to move that code in the core of kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED attribute
is
done by default when libssl v1.1 is detected at compile time, without the need of using the approach with a preloaded library.
The shared library is still kept in the tls module:
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with
libssl
v1.1+ to test the master branch and give feedback. The master branch is anyhow in testing phase for preparing the next major release v5.3.0, so
it
will help for that as well.
If no issue is going to be reported soon, the commits will be backported
to
stable branches, to re-enable smooth use of Kamailio with libssl v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that was already present in the Makefiles.defs for different compile options. The pthread more or less a default library on linuxes/unixes, so this should not introduce any major dependency to the Kamailio core.
Cheers, Daniel
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello,
did you get core files? If yes, it would be good to see the gdb bracktraces.
Have you used the pre-loaded library workaround?
The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment).
Cheers, Daniel
On 18.09.19 15:26, Andrew Chen wrote:
Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov <abalashov@evaristesys.com mailto:abalashov@evaristesys.com> wrote:
Hi Daniel, Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated. Have a good weekend, -- Alex On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote: > Hello, > after couple of months waiting for feedback about the workaround with the > preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for > pthread mutex/rwlock in order to avoid deadlocks and as it seems to be the > solution, I just pushed a few commits to move that code in the core of > kamailio. > > It this way the initialization with the PTHREAD_PROCESS_SHARED attribute is > done by default when libssl v1.1 is detected at compile time, without the > need of using the approach with a preloaded library. > > The shared library is still kept in the tls module: > * > https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared > > Probably is going to be removed in the near future. > > I am asking the community members that use Kamailio on systems with libssl > v1.1+ to test the master branch and give feedback. The master branch is > anyhow in testing phase for preparing the next major release v5.3.0, so it > will help for that as well. > > If no issue is going to be reported soon, the commits will be backported to > stable branches, to re-enable smooth use of Kamailio with libssl v1.1+. > > When libssl v1.1+ is detected, the core links with pthread, but that was > already present in the Makefiles.defs for different compile options. The > pthread more or less a default library on linuxes/unixes, so this should > not introduce any major dependency to the Kamailio core. > > Cheers, > Daniel > > -- > Daniel-Constantin Mierla - http://www.asipto.com > http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ mailto:achen@thinkingphones.comfuze.com http://fuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
I'll look for the core.
As part of our release process, I don't think we can release master branch of Kamailio.
On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
did you get core files? If yes, it would be good to see the gdb bracktraces.
Have you used the pre-loaded library workaround?
The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment).
Cheers, Daniel On 18.09.19 15:26, Andrew Chen wrote:
Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov abalashov@evaristesys.com wrote:
Hi Daniel,
Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated.
Have a good weekend,
-- Alex
On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote:
Hello, after couple of months waiting for feedback about the workaround with
the
preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for pthread mutex/rwlock in order to avoid deadlocks and as it seems to be
the
solution, I just pushed a few commits to move that code in the core of kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED
attribute is
done by default when libssl v1.1 is detected at compile time, without
the
need of using the approach with a preloaded library.
The shared library is still kept in the tls module:
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with
libssl
v1.1+ to test the master branch and give feedback. The master branch is anyhow in testing phase for preparing the next major release v5.3.0, so
it
will help for that as well.
If no issue is going to be reported soon, the commits will be
backported to
stable branches, to re-enable smooth use of Kamailio with libssl v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that was already present in the Makefiles.defs for different compile options. The pthread more or less a default library on linuxes/unixes, so this should not introduce any major dependency to the Kamailio core.
Cheers, Daniel
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
backtrace output:
(gdb) bt
#0 0x0000562fcc3544c0 in ?? ()
#1 0x0000562fcc354632 in ?? ()
#2 0x0000562fcc354974 in local_timer_run ()
#3 0x0000562fcc3fb256 in ?? ()
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
#5 0x0000562fcc2acf52 in tcp_init_children ()
#6 0x0000562fcc1ad94a in main_loop ()
#7 0x0000562fcc1b4a7b in main ()
(gdb)
On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen achen@fuze.com wrote:
I'll look for the core.
As part of our release process, I don't think we can release master branch of Kamailio.
On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
did you get core files? If yes, it would be good to see the gdb bracktraces.
Have you used the pre-loaded library workaround?
The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment).
Cheers, Daniel On 18.09.19 15:26, Andrew Chen wrote:
Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov abalashov@evaristesys.com wrote:
Hi Daniel,
Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated.
Have a good weekend,
-- Alex
On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote:
Hello, after couple of months waiting for feedback about the workaround with
the
preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute
for
pthread mutex/rwlock in order to avoid deadlocks and as it seems to be
the
solution, I just pushed a few commits to move that code in the core of kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED
attribute is
done by default when libssl v1.1 is detected at compile time, without
the
need of using the approach with a preloaded library.
The shared library is still kept in the tls module:
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with
libssl
v1.1+ to test the master branch and give feedback. The master branch is anyhow in testing phase for preparing the next major release v5.3.0,
so it
will help for that as well.
If no issue is going to be reported soon, the commits will be
backported to
stable branches, to re-enable smooth use of Kamailio with libssl v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that
was
already present in the Makefiles.defs for different compile options.
The
pthread more or less a default library on linuxes/unixes, so this
should
not introduce any major dependency to the Kamailio core.
Cheers, Daniel
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
Sorry..forgot bt full output
(gdb) bt full
#0 0x0000562fcc3544c0 in ?? ()
No symbol table info available.
#1 0x0000562fcc354632 in ?? ()
No symbol table info available.
#2 0x0000562fcc354974 in local_timer_run ()
No symbol table info available.
#3 0x0000562fcc3fb256 in ?? ()
No symbol table info available.
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
No symbol table info available.
#5 0x0000562fcc2acf52 in tcp_init_children ()
No symbol table info available.
#6 0x0000562fcc1ad94a in main_loop ()
No symbol table info available.
#7 0x0000562fcc1b4a7b in main ()
No symbol table info available.
(gdb)
On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen achen@fuze.com wrote:
backtrace output:
(gdb) bt
#0 0x0000562fcc3544c0 in ?? ()
#1 0x0000562fcc354632 in ?? ()
#2 0x0000562fcc354974 in local_timer_run ()
#3 0x0000562fcc3fb256 in ?? ()
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
#5 0x0000562fcc2acf52 in tcp_init_children ()
#6 0x0000562fcc1ad94a in main_loop ()
#7 0x0000562fcc1b4a7b in main ()
(gdb)
On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen achen@fuze.com wrote:
I'll look for the core.
As part of our release process, I don't think we can release master branch of Kamailio.
On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
did you get core files? If yes, it would be good to see the gdb bracktraces.
Have you used the pre-loaded library workaround?
The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment).
Cheers, Daniel On 18.09.19 15:26, Andrew Chen wrote:
Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov abalashov@evaristesys.com wrote:
Hi Daniel,
Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated.
Have a good weekend,
-- Alex
On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote:
Hello, after couple of months waiting for feedback about the workaround with
the
preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute
for
pthread mutex/rwlock in order to avoid deadlocks and as it seems to
be the
solution, I just pushed a few commits to move that code in the core of kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED
attribute is
done by default when libssl v1.1 is detected at compile time, without
the
need of using the approach with a preloaded library.
The shared library is still kept in the tls module:
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with
libssl
v1.1+ to test the master branch and give feedback. The master branch
is
anyhow in testing phase for preparing the next major release v5.3.0,
so it
will help for that as well.
If no issue is going to be reported soon, the commits will be
backported to
stable branches, to re-enable smooth use of Kamailio with libssl
v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that
was
already present in the Makefiles.defs for different compile options.
The
pthread more or less a default library on linuxes/unixes, so this
should
not introduce any major dependency to the Kamailio core.
Cheers, Daniel
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
You have to install the debug symbols package, otherwise the backtrace doesn't show any useful information. It just require to install that package and then get again the backtraces. The package name should be kamailio-dbg.
Cheers, Daniel
On 18.09.19 16:37, Andrew Chen wrote:
Sorry..forgot bt full output
(gdb) bt full
#0 0x0000562fcc3544c0 in ?? ()
No symbol table info available.
#1 0x0000562fcc354632 in ?? ()
No symbol table info available.
#2 0x0000562fcc354974 in local_timer_run ()
No symbol table info available.
#3 0x0000562fcc3fb256 in ?? ()
No symbol table info available.
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
No symbol table info available.
#5 0x0000562fcc2acf52 in tcp_init_children ()
No symbol table info available.
#6 0x0000562fcc1ad94a in main_loop ()
No symbol table info available.
#7 0x0000562fcc1b4a7b in main ()
No symbol table info available.
(gdb)
On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen <achen@fuze.com mailto:achen@fuze.com> wrote:
backtrace output: (gdb) bt #0 0x0000562fcc3544c0 in ?? () #1 0x0000562fcc354632 in ?? () #2 0x0000562fcc354974 in local_timer_run () #3 0x0000562fcc3fb256 in ?? () #4 0x0000562fcc3fb64c in tcp_receive_loop () #5 0x0000562fcc2acf52 in tcp_init_children () #6 0x0000562fcc1ad94a in main_loop () #7 0x0000562fcc1b4a7b in main () (gdb) On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen <achen@fuze.com <mailto:achen@fuze.com>> wrote: I'll look for the core. As part of our release process, I don't think we can release master branch of Kamailio. On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote: Hello, did you get core files? If yes, it would be good to see the gdb bracktraces. Have you used the pre-loaded library workaround? The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment). Cheers, Daniel On 18.09.19 15:26, Andrew Chen wrote:Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions: Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000] [Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000] [Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000] [Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000] [Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000] [Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000] [Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000] [Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000] [Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000] [Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000] [Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000] [Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000] [Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000] [Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000] [Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000] [Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000] [Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000] [Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000] [Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000] [Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] [Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000] [Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000] [Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] [Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] [Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000] and I was wondering if this core is related. Thanks On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov <abalashov@evaristesys.com <mailto:abalashov@evaristesys.com>> wrote: Hi Daniel, Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated. Have a good weekend, -- Alex On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote: > Hello, > after couple of months waiting for feedback about the workaround with the > preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for > pthread mutex/rwlock in order to avoid deadlocks and as it seems to be the > solution, I just pushed a few commits to move that code in the core of > kamailio. > > It this way the initialization with the PTHREAD_PROCESS_SHARED attribute is > done by default when libssl v1.1 is detected at compile time, without the > need of using the approach with a preloaded library. > > The shared library is still kept in the tls module: > * > https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared > > Probably is going to be removed in the near future. > > I am asking the community members that use Kamailio on systems with libssl > v1.1+ to test the master branch and give feedback. The master branch is > anyhow in testing phase for preparing the next major release v5.3.0, so it > will help for that as well. > > If no issue is going to be reported soon, the commits will be backported to > stable branches, to re-enable smooth use of Kamailio with libssl v1.1+. > > When libssl v1.1+ is detected, the core links with pthread, but that was > already present in the Makefiles.defs for different compile options. The > pthread more or less a default library on linuxes/unixes, so this should > not introduce any major dependency to the Kamailio core. > > Cheers, > Daniel > > -- > Daniel-Constantin Mierla - http://www.asipto.com > http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com> *Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.* _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat -- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com> -- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com>-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ mailto:achen@thinkingphones.comfuze.com http://fuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
I don't see one for debug:
root@ashintgkama51:~ # apt-cache search kamailio
resource-agents - Cluster Resource Agents
kamailio - very fast, dynamic and configurable SIP server
kamailio-autheph-modules - authentication using ephemeral credentials for the Kamailio SIP server
kamailio-berkeley-bin - Berkeley database module for Kamailio - helper program
kamailio-berkeley-modules - Berkeley database module for the Kamailio SIP server
kamailio-carrierroute-modules - carrierroute module for the Kamailio SIP server
kamailio-cnxcc-modules - cnxcc modules for the Kamailio SIP server
kamailio-cpl-modules - CPL module (CPL interpreter engine) for the Kamailio SIP server
kamailio-erlang-modules - Erlang modules for the Kamailio SIP server
kamailio-geoip-modules - geoip module for the Kamailio SIP Server
kamailio-geoip2-modules - geoip2 module for the Kamailio SIP Server
kamailio-ims-modules - IMS module for the Kamailio SIP server
kamailio-json-modules - Json parser and Json-RPC modules for the Kamailio SIP server
On Wed, Sep 18, 2019 at 11:26 AM Daniel-Constantin Mierla miconda@gmail.com wrote:
You have to install the debug symbols package, otherwise the backtrace doesn't show any useful information. It just require to install that package and then get again the backtraces. The package name should be kamailio-dbg.
Cheers, Daniel On 18.09.19 16:37, Andrew Chen wrote:
Sorry..forgot bt full output
(gdb) bt full
#0 0x0000562fcc3544c0 in ?? ()
No symbol table info available.
#1 0x0000562fcc354632 in ?? ()
No symbol table info available.
#2 0x0000562fcc354974 in local_timer_run ()
No symbol table info available.
#3 0x0000562fcc3fb256 in ?? ()
No symbol table info available.
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
No symbol table info available.
#5 0x0000562fcc2acf52 in tcp_init_children ()
No symbol table info available.
#6 0x0000562fcc1ad94a in main_loop ()
No symbol table info available.
#7 0x0000562fcc1b4a7b in main ()
No symbol table info available.
(gdb)
On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen achen@fuze.com wrote:
backtrace output:
(gdb) bt
#0 0x0000562fcc3544c0 in ?? ()
#1 0x0000562fcc354632 in ?? ()
#2 0x0000562fcc354974 in local_timer_run ()
#3 0x0000562fcc3fb256 in ?? ()
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
#5 0x0000562fcc2acf52 in tcp_init_children ()
#6 0x0000562fcc1ad94a in main_loop ()
#7 0x0000562fcc1b4a7b in main ()
(gdb)
On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen achen@fuze.com wrote:
I'll look for the core.
As part of our release process, I don't think we can release master branch of Kamailio.
On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
did you get core files? If yes, it would be good to see the gdb bracktraces.
Have you used the pre-loaded library workaround?
The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment).
Cheers, Daniel On 18.09.19 15:26, Andrew Chen wrote:
Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov < abalashov@evaristesys.com> wrote:
Hi Daniel,
Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated.
Have a good weekend,
-- Alex
On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote:
Hello, after couple of months waiting for feedback about the workaround
with the
preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute
for
pthread mutex/rwlock in order to avoid deadlocks and as it seems to
be the
solution, I just pushed a few commits to move that code in the core
of
kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED
attribute is
done by default when libssl v1.1 is detected at compile time,
without the
need of using the approach with a preloaded library.
The shared library is still kept in the tls module:
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with
libssl
v1.1+ to test the master branch and give feedback. The master branch
is
anyhow in testing phase for preparing the next major release v5.3.0,
so it
will help for that as well.
If no issue is going to be reported soon, the commits will be
backported to
stable branches, to re-enable smooth use of Kamailio with libssl
v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that
was
already present in the Makefiles.defs for different compile options.
The
pthread more or less a default library on linuxes/unixes, so this
should
not introduce any major dependency to the Kamailio core.
Cheers, Daniel
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
Hello Andrew,
strange - it should be there, as it is available on deb.kamailio.org.
Here e.g. the one for 64bit stretch:
https://deb.kamailio.org/kamailiodev-nightly/pool/main/k/kamailio/kamailio-d...
Cheers,
Henning
Am 18.09.19 um 18:52 schrieb Andrew Chen: I don't see one for debug:
root@ashintgkama51:~ # apt-cache search kamailio
resource-agents - Cluster Resource Agents
kamailio - very fast, dynamic and configurable SIP server
kamailio-autheph-modules - authentication using ephemeral credentials for the Kamailio SIP server
kamailio-berkeley-bin - Berkeley database module for Kamailio - helper program
kamailio-berkeley-modules - Berkeley database module for the Kamailio SIP server
kamailio-carrierroute-modules - carrierroute module for the Kamailio SIP server
kamailio-cnxcc-modules - cnxcc modules for the Kamailio SIP server
kamailio-cpl-modules - CPL module (CPL interpreter engine) for the Kamailio SIP server
kamailio-erlang-modules - Erlang modules for the Kamailio SIP server
kamailio-geoip-modules - geoip module for the Kamailio SIP Server
kamailio-geoip2-modules - geoip2 module for the Kamailio SIP Server
kamailio-ims-modules - IMS module for the Kamailio SIP server
kamailio-json-modules - Json parser and Json-RPC modules for the Kamailio SIP server
On Wed, Sep 18, 2019 at 11:26 AM Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
You have to install the debug symbols package, otherwise the backtrace doesn't show any useful information. It just require to install that package and then get again the backtraces. The package name should be kamailio-dbg.
Cheers, Daniel
On 18.09.19 16:37, Andrew Chen wrote: Sorry..forgot bt full output
(gdb) bt full
#0 0x0000562fcc3544c0 in ?? ()
No symbol table info available.
#1 0x0000562fcc354632 in ?? ()
No symbol table info available.
#2 0x0000562fcc354974 in local_timer_run ()
No symbol table info available.
#3 0x0000562fcc3fb256 in ?? ()
No symbol table info available.
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
No symbol table info available.
#5 0x0000562fcc2acf52 in tcp_init_children ()
No symbol table info available.
#6 0x0000562fcc1ad94a in main_loop ()
No symbol table info available.
#7 0x0000562fcc1b4a7b in main ()
No symbol table info available.
(gdb)
On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen <achen@fuze.commailto:achen@fuze.com> wrote: backtrace output:
(gdb) bt
#0 0x0000562fcc3544c0 in ?? ()
#1 0x0000562fcc354632 in ?? ()
#2 0x0000562fcc354974 in local_timer_run ()
#3 0x0000562fcc3fb256 in ?? ()
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
#5 0x0000562fcc2acf52 in tcp_init_children ()
#6 0x0000562fcc1ad94a in main_loop ()
#7 0x0000562fcc1b4a7b in main ()
(gdb)
On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen <achen@fuze.commailto:achen@fuze.com> wrote: I'll look for the core.
As part of our release process, I don't think we can release master branch of Kamailio.
On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello,
did you get core files? If yes, it would be good to see the gdb bracktraces.
Have you used the pre-loaded library workaround?
The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment).
Cheers, Daniel
On 18.09.19 15:26, Andrew Chen wrote: Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov <abalashov@evaristesys.commailto:abalashov@evaristesys.com> wrote: Hi Daniel,
Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated.
Have a good weekend,
-- Alex
On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote:
Hello, after couple of months waiting for feedback about the workaround with the preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for pthread mutex/rwlock in order to avoid deadlocks and as it seems to be the solution, I just pushed a few commits to move that code in the core of kamailio.
It this way the initialization with the PTHREAD_PROCESS_SHARED attribute is done by default when libssl v1.1 is detected at compile time, without the need of using the approach with a preloaded library.
The shared library is still kept in the tls module:
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Probably is going to be removed in the near future.
I am asking the community members that use Kamailio on systems with libssl v1.1+ to test the master branch and give feedback. The master branch is anyhow in testing phase for preparing the next major release v5.3.0, so it will help for that as well.
If no issue is going to be reported soon, the commits will be backported to stable branches, to re-enable smooth use of Kamailio with libssl v1.1+.
When libssl v1.1+ is detected, the core links with pthread, but that was already present in the Makefiles.defs for different compile options. The pthread more or less a default library on linuxes/unixes, so this should not introduce any major dependency to the Kamailio core.
Cheers, Daniel
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@mailto:achen@thinkingphones.comfuze.comhttp://fuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@mailto:achen@thinkingphones.comfuze.comhttp://fuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@mailto:achen@thinkingphones.comfuze.comhttp://fuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@mailto:achen@thinkingphones.comfuze.comhttp://fuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@mailto:achen@thinkingphones.comfuze.comhttp://fuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Kamailio Merchandising - https://skalatan.de/merchandising/ Kamailio services - https://skalatan.de/services Henning Westerholt - https://skalatan.de/blog/
Thanks Henning
On Wed, Sep 18, 2019 at 2:13 PM Henning Westerholt hw@skalatan.de wrote:
Hello Andrew,
strange - it should be there, as it is available on deb.kamailio.org.
Here e.g. the one for 64bit stretch:
https://deb.kamailio.org/kamailiodev-nightly/pool/main/k/kamailio/kamailio-d...
Cheers,
Henning Am 18.09.19 um 18:52 schrieb Andrew Chen:
I don't see one for debug:
root@ashintgkama51:~ # apt-cache search kamailio
resource-agents - Cluster Resource Agents
kamailio - very fast, dynamic and configurable SIP server
kamailio-autheph-modules - authentication using ephemeral credentials for the Kamailio SIP server
kamailio-berkeley-bin - Berkeley database module for Kamailio - helper program
kamailio-berkeley-modules - Berkeley database module for the Kamailio SIP server
kamailio-carrierroute-modules - carrierroute module for the Kamailio SIP server
kamailio-cnxcc-modules - cnxcc modules for the Kamailio SIP server
kamailio-cpl-modules - CPL module (CPL interpreter engine) for the Kamailio SIP server
kamailio-erlang-modules - Erlang modules for the Kamailio SIP server
kamailio-geoip-modules - geoip module for the Kamailio SIP Server
kamailio-geoip2-modules - geoip2 module for the Kamailio SIP Server
kamailio-ims-modules - IMS module for the Kamailio SIP server
kamailio-json-modules - Json parser and Json-RPC modules for the Kamailio SIP server
On Wed, Sep 18, 2019 at 11:26 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
You have to install the debug symbols package, otherwise the backtrace doesn't show any useful information. It just require to install that package and then get again the backtraces. The package name should be kamailio-dbg.
Cheers, Daniel On 18.09.19 16:37, Andrew Chen wrote:
Sorry..forgot bt full output
(gdb) bt full
#0 0x0000562fcc3544c0 in ?? ()
No symbol table info available.
#1 0x0000562fcc354632 in ?? ()
No symbol table info available.
#2 0x0000562fcc354974 in local_timer_run ()
No symbol table info available.
#3 0x0000562fcc3fb256 in ?? ()
No symbol table info available.
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
No symbol table info available.
#5 0x0000562fcc2acf52 in tcp_init_children ()
No symbol table info available.
#6 0x0000562fcc1ad94a in main_loop ()
No symbol table info available.
#7 0x0000562fcc1b4a7b in main ()
No symbol table info available.
(gdb)
On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen achen@fuze.com wrote:
backtrace output:
(gdb) bt
#0 0x0000562fcc3544c0 in ?? ()
#1 0x0000562fcc354632 in ?? ()
#2 0x0000562fcc354974 in local_timer_run ()
#3 0x0000562fcc3fb256 in ?? ()
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
#5 0x0000562fcc2acf52 in tcp_init_children ()
#6 0x0000562fcc1ad94a in main_loop ()
#7 0x0000562fcc1b4a7b in main ()
(gdb)
On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen achen@fuze.com wrote:
I'll look for the core.
As part of our release process, I don't think we can release master branch of Kamailio.
On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
did you get core files? If yes, it would be good to see the gdb bracktraces.
Have you used the pre-loaded library workaround?
The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment).
Cheers, Daniel On 18.09.19 15:26, Andrew Chen wrote:
Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov < abalashov@evaristesys.com> wrote:
Hi Daniel,
Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated.
Have a good weekend,
-- Alex
On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote:
> Hello, > after couple of months waiting for feedback about the workaround with the > preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for > pthread mutex/rwlock in order to avoid deadlocks and as it seems to be the > solution, I just pushed a few commits to move that code in the core of > kamailio. > > It this way the initialization with the PTHREAD_PROCESS_SHARED attribute is > done by default when libssl v1.1 is detected at compile time, without the > need of using the approach with a preloaded library. > > The shared library is still kept in the tls module: > * > https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens... > > Probably is going to be removed in the near future. > > I am asking the community members that use Kamailio on systems with libssl > v1.1+ to test the master branch and give feedback. The master branch is > anyhow in testing phase for preparing the next major release v5.3.0, so it > will help for that as well. > > If no issue is going to be reported soon, the commits will be backported to > stable branches, to re-enable smooth use of Kamailio with libssl v1.1+. > > When libssl v1.1+ is detected, the core links with pthread, but that was > already present in the Makefiles.defs for different compile options. The > pthread more or less a default library on linuxes/unixes, so this should > not introduce any major dependency to the Kamailio core. > > Cheers, > Daniel > > -- > Daniel-Constantin Mierla - http://www.asipto.com > http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
> _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Kamailio Merchandising - https://skalatan.de/merchandising/ Kamailio services - https://skalatan.de/services Henning Westerholt - https://skalatan.de/blog/
Hi Henning,
I'm running 5.2.4. Will this deb package work?
On Wed, Sep 18, 2019 at 2:39 PM Andrew Chen achen@fuze.com wrote:
Thanks Henning
On Wed, Sep 18, 2019 at 2:13 PM Henning Westerholt hw@skalatan.de wrote:
Hello Andrew,
strange - it should be there, as it is available on deb.kamailio.org.
Here e.g. the one for 64bit stretch:
https://deb.kamailio.org/kamailiodev-nightly/pool/main/k/kamailio/kamailio-d...
Cheers,
Henning Am 18.09.19 um 18:52 schrieb Andrew Chen:
I don't see one for debug:
root@ashintgkama51:~ # apt-cache search kamailio
resource-agents - Cluster Resource Agents
kamailio - very fast, dynamic and configurable SIP server
kamailio-autheph-modules - authentication using ephemeral credentials for the Kamailio SIP server
kamailio-berkeley-bin - Berkeley database module for Kamailio - helper program
kamailio-berkeley-modules - Berkeley database module for the Kamailio SIP server
kamailio-carrierroute-modules - carrierroute module for the Kamailio SIP server
kamailio-cnxcc-modules - cnxcc modules for the Kamailio SIP server
kamailio-cpl-modules - CPL module (CPL interpreter engine) for the Kamailio SIP server
kamailio-erlang-modules - Erlang modules for the Kamailio SIP server
kamailio-geoip-modules - geoip module for the Kamailio SIP Server
kamailio-geoip2-modules - geoip2 module for the Kamailio SIP Server
kamailio-ims-modules - IMS module for the Kamailio SIP server
kamailio-json-modules - Json parser and Json-RPC modules for the Kamailio SIP server
On Wed, Sep 18, 2019 at 11:26 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
You have to install the debug symbols package, otherwise the backtrace doesn't show any useful information. It just require to install that package and then get again the backtraces. The package name should be kamailio-dbg.
Cheers, Daniel On 18.09.19 16:37, Andrew Chen wrote:
Sorry..forgot bt full output
(gdb) bt full
#0 0x0000562fcc3544c0 in ?? ()
No symbol table info available.
#1 0x0000562fcc354632 in ?? ()
No symbol table info available.
#2 0x0000562fcc354974 in local_timer_run ()
No symbol table info available.
#3 0x0000562fcc3fb256 in ?? ()
No symbol table info available.
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
No symbol table info available.
#5 0x0000562fcc2acf52 in tcp_init_children ()
No symbol table info available.
#6 0x0000562fcc1ad94a in main_loop ()
No symbol table info available.
#7 0x0000562fcc1b4a7b in main ()
No symbol table info available.
(gdb)
On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen achen@fuze.com wrote:
backtrace output:
(gdb) bt
#0 0x0000562fcc3544c0 in ?? ()
#1 0x0000562fcc354632 in ?? ()
#2 0x0000562fcc354974 in local_timer_run ()
#3 0x0000562fcc3fb256 in ?? ()
#4 0x0000562fcc3fb64c in tcp_receive_loop ()
#5 0x0000562fcc2acf52 in tcp_init_children ()
#6 0x0000562fcc1ad94a in main_loop ()
#7 0x0000562fcc1b4a7b in main ()
(gdb)
On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen achen@fuze.com wrote:
I'll look for the core.
As part of our release process, I don't think we can release master branch of Kamailio.
On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla < miconda@gmail.com> wrote:
Hello,
did you get core files? If yes, it would be good to see the gdb bracktraces.
Have you used the pre-loaded library workaround?
The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment).
Cheers, Daniel On 18.09.19 15:26, Andrew Chen wrote:
Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions:
Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000]
[Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000]
[Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000]
[Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000]
[Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000]
[Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000]
[Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000]
[Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000]
[Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000]
[Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000]
[Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000]
[Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000]
[Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000]
[Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000]
[Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000]
and I was wondering if this core is related.
Thanks
On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov < abalashov@evaristesys.com> wrote:
> Hi Daniel, > > Thanks a lot for this work; it's definitely important that Kamailio > build against OpenSSL v1.1, and I'll be happy to contribute some > testing > in a live environment in the coming weeks. Much appreciated. > > Have a good weekend, > > -- Alex > > On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla > wrote: > > > Hello, > > after couple of months waiting for feedback about the workaround > with the > > preloaded shared library to set the PTHREAD_PROCESS_SHARED > attribute for > > pthread mutex/rwlock in order to avoid deadlocks and as it seems > to be the > > solution, I just pushed a few commits to move that code in the > core of > > kamailio. > > > > It this way the initialization with the PTHREAD_PROCESS_SHARED > attribute is > > done by default when libssl v1.1 is detected at compile time, > without the > > need of using the approach with a preloaded library. > > > > The shared library is still kept in the tls module: > > * > > > https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens... > > > > Probably is going to be removed in the near future. > > > > I am asking the community members that use Kamailio on systems > with libssl > > v1.1+ to test the master branch and give feedback. The master > branch is > > anyhow in testing phase for preparing the next major release > v5.3.0, so it > > will help for that as well. > > > > If no issue is going to be reported soon, the commits will be > backported to > > stable branches, to re-enable smooth use of Kamailio with libssl > v1.1+. > > > > When libssl v1.1+ is detected, the core links with pthread, but > that was > > already present in the Makefiles.defs for different compile > options. The > > pthread more or less a default library on linuxes/unixes, so this > should > > not introduce any major dependency to the Kamailio core. > > > > Cheers, > > Daniel > > > > -- > > Daniel-Constantin Mierla - http://www.asipto.com > > http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > > > _______________________________________________ > > Kamailio (SER) - Users Mailing List > > sr-users@lists.kamailio.org > > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > > -- > Alex Balashov | Principal | Evariste Systems LLC > > Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) > Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ > > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Kamailio Merchandising - https://skalatan.de/merchandising/ Kamailio services - https://skalatan.de/services Henning Westerholt - https://skalatan.de/blog/
-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ achen@thinkingphones.comfuze.com
I understood it happened on a load test session, which typically is not done in production, but some test/staging system.
Cheers, Daniel
On 18.09.19 16:01, Andrew Chen wrote:
I'll look for the core.
As part of our release process, I don't think we can release master branch of Kamailio.
On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello, did you get core files? If yes, it would be good to see the gdb bracktraces. Have you used the pre-loaded library workaround? The code was not backported to branch 5.2 yet, I wanted to wait a bit and see if there are reports with good/bad results. If you want to use it, then you have to install nightly builds from master branch (development version, should be 5.3.0-pre1 at this moment). Cheers, Daniel On 18.09.19 15:26, Andrew Chen wrote:Hi Daniel, Which apt repository can we get this workaround? Today I noticed our Kamailio 5.2.4 cored during one of our load test sessions: Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in libcrypto.so.1.1[7f8501dd4000+29b000] [Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in libcrypto.so.1.1[7f1a8f90f000+29b000] [Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip 00007f0a85dac128 sp 00007fff6847aad0 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000] [Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in libcrypto.so.1.1[7f0a85c40000+29b000] [Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in libcrypto.so.1.1[7f75e1136000+29b000] [Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in libcrypto.so.1.1[7f0f37ead000+29b000] [Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in libcrypto.so.1.1[7f8043b0a000+29b000] [Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000] [Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in libcrypto.so.1.1[7ff59d250000+29b000] [Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in libcrypto.so.1.1[7f22dd4db000+29b000] [Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in libcrypto.so.1.1[7f7ccae9b000+29b000] [Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in libcrypto.so.1.1[7fb309baf000+29b000] [Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in libcrypto.so.1.1[7fc5908d7000+29b000] [Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000] [Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in libcrypto.so.1.1[7fcc238c6000+29b000] [Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in libcrypto.so.1.1[7f586f78e000+29b000] [Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in kamailio[562fcc17c000+4a2000] [Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip 00007f96c3735c3e sp 00007ffe825ca270 error 4 in dialog.so[7f96c36d6000+be000] [Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000] [Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] [Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000] [Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in kamailio[562fcc17c000+4a2000] [Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] [Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] [Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in kamailio[562fcc17c000+4a2000] and I was wondering if this core is related. Thanks On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov <abalashov@evaristesys.com <mailto:abalashov@evaristesys.com>> wrote: Hi Daniel, Thanks a lot for this work; it's definitely important that Kamailio build against OpenSSL v1.1, and I'll be happy to contribute some testing in a live environment in the coming weeks. Much appreciated. Have a good weekend, -- Alex On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla wrote: > Hello, > after couple of months waiting for feedback about the workaround with the > preloaded shared library to set the PTHREAD_PROCESS_SHARED attribute for > pthread mutex/rwlock in order to avoid deadlocks and as it seems to be the > solution, I just pushed a few commits to move that code in the core of > kamailio. > > It this way the initialization with the PTHREAD_PROCESS_SHARED attribute is > done by default when libssl v1.1 is detected at compile time, without the > need of using the approach with a preloaded library. > > The shared library is still kept in the tls module: > * > https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared > > Probably is going to be removed in the near future. > > I am asking the community members that use Kamailio on systems with libssl > v1.1+ to test the master branch and give feedback. The master branch is > anyhow in testing phase for preparing the next major release v5.3.0, so it > will help for that as well. > > If no issue is going to be reported soon, the commits will be backported to > stable branches, to re-enable smooth use of Kamailio with libssl v1.1+. > > When libssl v1.1+ is detected, the core links with pthread, but that was > already present in the Makefiles.defs for different compile options. The > pthread more or less a default library on linuxes/unixes, so this should > not introduce any major dependency to the Kamailio core. > > Cheers, > Daniel > > -- > Daniel-Constantin Mierla - http://www.asipto.com > http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com> *Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.* _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ mailto:achen@thinkingphones.comfuze.com http://fuze.com
*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*
Hi.
I'm going to jump into this thread with a bit different but related problem. I've tried kamailio master branch with the mentioned workaround and it still crashes almost immediately in my load tests when kamailio is linked to libssl-1.1.1. Haven't followed kamailio recently since had to switch to opensips for my current project but the root of the problem is the same - the multiprocess nature of a proxy and new behavior of libssl that uses pthread functions directly. This workaround with pthread_mutex_init() overriding really helps with deadlocks found with libssl1.1.0, but after switching to libssl1.1.1 it started to crash again. That lib version also uses thread local storage what most probably creates new problems since ssl contex IIRC might be moved between tcp main and workers.
Has anyone tested kamailio master on debian 10 with libssl-1.1.1 and many concurrent tls connections?
I understood it happened on a load test session, which typically is not done in production, but some test/staging system.
Cheers, Daniel
Hello,
can you provide the full backtrace from gdb?
Also, give the output for "kamailio -I" and the exact version for libssl.
Cheers, Daniel
On 20.10.19 22:35, Vitalii Aleksandrov wrote:
Hi.
I'm going to jump into this thread with a bit different but related problem. I've tried kamailio master branch with the mentioned workaround and it still crashes almost immediately in my load tests when kamailio is linked to libssl-1.1.1. Haven't followed kamailio recently since had to switch to opensips for my current project but the root of the problem is the same - the multiprocess nature of a proxy and new behavior of libssl that uses pthread functions directly. This workaround with pthread_mutex_init() overriding really helps with deadlocks found with libssl1.1.0, but after switching to libssl1.1.1 it started to crash again. That lib version also uses thread local storage what most probably creates new problems since ssl contex IIRC might be moved between tcp main and workers.
Has anyone tested kamailio master on debian 10 with libssl-1.1.1 and many concurrent tls connections?
I understood it happened on a load test session, which typically is not done in production, but some test/staging system.
Cheers, Daniel
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Hi,
My bad, I most probably used a container with a bit outdated version of master branch.
Got new sources, tested them and had no problems with TLS. It works fine with libssl 1.1.1d-0+deb10u2. I suppose those fixes with using alternative random number generator or wrapping up the standard one with lock have helped.
Hello,
can you provide the full backtrace from gdb?
Also, give the output for "kamailio -I" and the exact version for libssl.
Cheers, Daniel
On 20.10.19 22:35, Vitalii Aleksandrov wrote:
Hi.
I'm going to jump into this thread with a bit different but related problem. I've tried kamailio master branch with the mentioned workaround and it still crashes almost immediately in my load tests when kamailio is linked to libssl-1.1.1. Haven't followed kamailio recently since had to switch to opensips for my current project but the root of the problem is the same - the multiprocess nature of a proxy and new behavior of libssl that uses pthread functions directly. This workaround with pthread_mutex_init() overriding really helps with deadlocks found with libssl1.1.0, but after switching to libssl1.1.1 it started to crash again. That lib version also uses thread local storage what most probably creates new problems since ssl contex IIRC might be moved between tcp main and workers.
Has anyone tested kamailio master on debian 10 with libssl-1.1.1 and many concurrent tls connections?
I understood it happened on a load test session, which typically is not done in production, but some test/staging system.
Cheers, Daniel
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Hello,
thanks for coming back with the clarifications!
Before the changes related to PRNG, the reporter was doing stress testing and it was crashing occasionally, after the changes, he continued to stress test for several days without any issues -- therefore I thought in your case it could be something else, a matter of a specific use case. But it is better now knowing it was an older version and now no issue was showing up.
Cheers, Daniel
On 23.10.19 14:52, Vitalii Aleksandrov wrote:
Hi,
My bad, I most probably used a container with a bit outdated version of master branch.
Got new sources, tested them and had no problems with TLS. It works fine with libssl 1.1.1d-0+deb10u2. I suppose those fixes with using alternative random number generator or wrapping up the standard one with lock have helped.
Hello,
can you provide the full backtrace from gdb?
Also, give the output for "kamailio -I" and the exact version for libssl.
Cheers, Daniel
On 20.10.19 22:35, Vitalii Aleksandrov wrote:
Hi.
I'm going to jump into this thread with a bit different but related problem. I've tried kamailio master branch with the mentioned workaround and it still crashes almost immediately in my load tests when kamailio is linked to libssl-1.1.1. Haven't followed kamailio recently since had to switch to opensips for my current project but the root of the problem is the same - the multiprocess nature of a proxy and new behavior of libssl that uses pthread functions directly. This workaround with pthread_mutex_init() overriding really helps with deadlocks found with libssl1.1.0, but after switching to libssl1.1.1 it started to crash again. That lib version also uses thread local storage what most probably creates new problems since ssl contex IIRC might be moved between tcp main and workers.
Has anyone tested kamailio master on debian 10 with libssl-1.1.1 and many concurrent tls connections?
I understood it happened on a load test session, which typically is not done in production, but some test/staging system.
Cheers, Daniel
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
-
Alex Balashov -
Andrew Chen -
Daniel-Constantin Mierla -
Henning Westerholt -
Julien Klingenmeyer -
Vitalii Aleksandrov