[SR-Users] Need help analyze crash
Andrew Chen
achen at fuze.com
Thu Jun 3 16:19:50 CEST 2021
Thanks for the suggestion. We have an AMI image with just libssl1.1 and
will test with that.
On Thu, Jun 3, 2021 at 10:12 AM Daniel-Constantin Mierla <miconda at gmail.com>
wrote:
> Well, probably a misunderstanding: the issue was observed on OSes having
> the two libssl 1.0 & 1.1 versions, but it does not mean (it was not proved)
> that the problem was created by having the two version were installed at
> the same time. I would rather think it was more that libssl1.1 was not very
> mature at that moment when OSes shipped both of them.
>
> If I would make a suggestion, that would be to use an OS version that
> comes only with one of them by default. Think like: when libssl1.0 was
> completely removed from an OS, the libssl1.1 was more tested/stable.
>
> Cheers,
> Daniel
> On 03.06.21 15:58, Andrew Chen wrote:
>
> No blaming here. :). Just looking for answers.
>
> After chatting with my peers on options around this, we are going to
> remove libssl1.0 and just have libssl1.1 running.
>
> Thanks again for you input.
>
> On Thu, Jun 3, 2021 at 9:48 AM Daniel-Constantin Mierla <miconda at gmail.com>
> wrote:
>
>> Again, not to be blamed later: I am not sure you face the same kind of
>> issue like reported in the past. I was looking at it from the perspective
>> that db_mysql and carrierroute modules were not touched that much in the
>> 5.4.x series recently.
>>
>> The libssl did a lot of internal/api changes between 1.0 and 1.1, so for
>> a few past years many libs and apps (including kamailio) faced some
>> "strange" crashes/behaviours. Now seems stabilized and, at least on debian
>> 10, there were no related open issues recently. In the future we may need
>> to make --atexit=no the default (maybe needs to be set even on latest
>> ubuntu 20.04), it's the last I remember that surfaced and it appeared after
>> a minor version upgrade of libssl1.1, iirc.
>>
>> Cheers,
>> Daniel
>> On 03.06.21 15:31, Andrew Chen wrote:
>>
>> Thanks Daniel.
>> It sounds like upgrading to a newer Ubuntu seems like the cleanest way of
>> resolving this issue.
>>
>> On Thu, Jun 3, 2021 at 8:58 AM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> it is not related to kamailio tls being compiled with libssl1.1. The
>>> issues were observed on the systems having both libssl1.0 (which was
>>> somehow the default choice) and libssl1.1. I got the chance to test a bit
>>> and it was still crashing even I compiled manually with libssl1.1. I didn't
>>> dig much in the external libraries given that upgrading OS seemed to fix
>>> it. There is a mod param for db_mysql that you can try to play with:
>>> opt_ssl_mode - I don't recall solving much for mysql servers requiring tls.
>>>
>>> Otherwise, on a different direction: if you can get mariadb client
>>> library, you can try to compile kamailio against it (the db_mysql module)
>>> -- if the MySQL server is on a remote system, it should work.
>>>
>>> I can't confirm it is same issue, but if you upgraded in kamailio 5.4.x
>>> series, there were no changes to carrierroute or db_mysql to
>>>
>>> As a side note, for newer operating systems and newer libssl, it may
>>> also require to set --atexit cli parameter for kamailio.
>>>
>>> Cheers,
>>> Daniel
>>> On 03.06.21 14:42, Andrew Chen wrote:
>>>
>>> Hi Daniel,
>>>
>>> I just confirmed mysql is compiled using libssl.so.1.1.
>>>
>>> As for the other kamailio packages, I only see two that are installed
>>> where both are depends on libssl1.1
>>>
>>> kamailio-tls-modules:amd64
>>>
>>> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>= 7.16.2),
>>> libssl1.1 (>= 1.1.0)
>>>
>>>
>>>
>>> kamailio-websocket-modules:amd64
>>>
>>> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1 (>= 1.1.0),
>>> libunistring2 (>= 0.9.7)
>>>
>>> So in my case here there shouldn't be any conflict with mysql client
>>> libssl version?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla <
>>> miconda at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I think the relevant here are the client library. I asked about mysql
>>>> vs mariadb because some distro had (meta-)packages named mysql but
>>>> installing mariadb. Debian at some point switched to something like
>>>> default-mysql-client...
>>>>
>>>> Based on backtrace, the crashes would prevent kamailio to start, as
>>>> carrierroute_db_open() is used in mod_init/child_init callbacks.
>>>>
>>>> A variant you could try is the use of mysql-proxy to bridge from a
>>>> local unix socket to remote mysql, so kamailio connects to mysql server
>>>> via local unix socket and by that won't use libssl.
>>>>
>>>> First you can check with ldd if mysql client libraries are compiled
>>>> against libssl1.1 and the libssl1.0 is also used for other packages.
>>>>
>>>> Cheers,
>>>> Daniel
>>>> On 02.06.21 21:56, Andrew Chen wrote:
>>>>
>>>> Hi Daniel,
>>>> So our DB is in rds and this is what I can gather:
>>>>
>>>> +-------------------------+------------------------------+
>>>>
>>>> | Variable_name | Value |
>>>>
>>>> +-------------------------+------------------------------+
>>>>
>>>> | innodb_version | 5.6.34 |
>>>>
>>>> | protocol_version | 10 |
>>>>
>>>> | slave_type_conversions | |
>>>>
>>>> | version | 5.6.34-log |
>>>>
>>>> | version_comment | MySQL Community Server (GPL) |
>>>>
>>>> | version_compile_machine | x86_64 |
>>>>
>>>> | version_compile_os | Linux |
>>>>
>>>> +-------------------------+------------------------------+
>>>>
>>>>
>>>> The mysql server/client library we received from my apt-cache madison
>>>> output earlier.
>>>>
>>>>
>>>> I don't know if upgrading to 20.04 is an option.
>>>>
>>>>
>>>> Given this said, what are the symptoms of these crashes? I'm guessing
>>>> one of them is a brief disconnect from the DB for
>>>> dialplan/dispatcher/carrierroute lookup?
>>>>
>>>>
>>>> Thanks.
>>>>
>>>> On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla <
>>>> miconda at gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> did you upgrade the Kamailio only or also mysql server/client library?
>>>>> Is it Oracle's MySQL or MariaDB?
>>>>>
>>>>> A while ago, there were a few reports of problems with mysql-client
>>>>> (Oracle) library on Ubuntu 16.04 and 18.04, which typically happened after
>>>>> upgrades of the library, as I understood, the upgrade brought in a
>>>>> dependency on libssl 1.1, while libssl1.0 was used by most of the other
>>>>> packages, resulting in mixed deployments. Upgrading to 20.04 seemed to
>>>>> solve it, as it came with libssl1.1 only.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>> On 02.06.21 20:51, Andrew Chen wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Today I upgraded Kamailio to 5.4.5 and a random crash was just
>>>>> reported in our crash monitoring script. Below is the output of the crash:
>>>>>
>>>>> #3 0x00007fa149ebe58d in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
>>>>>
>>>>> #4 0x00007fa14989e277 in ERR_clear_error () from
>>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>>>
>>>>> #5 0x00007fa149933d81 in X509_STORE_set_default_paths () from
>>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>>>
>>>>> #6 0x00007fa10310d374 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>>
>>>>> #7 0x00007fa10310d9f3 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>>
>>>>> #8 0x00007fa1030e0427 in mysql_real_connect () from
>>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>>
>>>>> #9 0x00007fa10368428c in db_mysql_new_connection () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>>>
>>>>> #10 0x00007fa1501f96dc in db_do_init2 () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>>>
>>>>> #11 0x00007fa1501f83f9 in db_do_init () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>>>
>>>>> #12 0x00007fa103689710 in db_mysql_init () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>>>
>>>>> #13 0x00007fa15062d35f in carrierroute_db_open () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>>>
>>>>> #14 0x00007fa1506719c6 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>>>
>>>>>
>>>>> Is this crash due to libcrypto or mysql client?
>>>>> Currently I have the following mysql client installed on the kamailio
>>>>> instance:
>>>>>
>>>>> ii lib*mysql*client20:amd64 5.7.34-0ubuntu0.18.04.1
>>>>> amd64 MySQL database client library
>>>>>
>>>>> ii *mysql*-client 5.7.34-0ubuntu0.18.04.1
>>>>> all MySQL database client (metapackage depending
>>>>> on the latest version)
>>>>>
>>>>> ii *mysql*-client-5.7 5.7.34-0ubuntu0.18.04.1
>>>>> amd64 MySQL database client binaries
>>>>>
>>>>> ii *mysql*-client-core-5.7 5.7.34-0ubuntu0.18.04.1
>>>>> amd64 MySQL database core client binaries
>>>>>
>>>>> ii *mysql*-common 5.8+1.0.4
>>>>> all MySQL database common files, e.g. /etc/*mysql*
>>>>> /my.cnf
>>>>>
>>>>> root at ashintgtpsg51:/var/lib/cores # apt-cache madison mysql-client
>>>>>
>>>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main
>>>>> amd64 Packages
>>>>>
>>>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>>>> http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
>>>>>
>>>>> mysql-client | 5.7.21-1ubuntu1 |
>>>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64
>>>>> Packages
>>>>>
>>>>> root at ashintgtpsg51:/var/lib/cores #
>>>>>
>>>>>
>>>>> Any suggestions would be greatly appreciated.
>>>>> Thanks in advance.
>>>>>
>>>>> --
>>>>> Andy Chen
>>>>> Sr. Telephony Lead Engineer
>>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>>
>>>>>
>>>>>
>>>>> *Confidentiality Notice: The information contained in this e-mail and
>>>>> any
>>>>> attachments may be confidential. If you are not an intended recipient,
>>>>> you
>>>>> are hereby notified that any dissemination, distribution or copying of
>>>>> this
>>>>> e-mail is strictly prohibited. If you have received this e-mail in
>>>>> error,
>>>>> please notify the sender and permanently delete the e-mail and any
>>>>> attachments immediately. You should not retain, copy or use this
>>>>> e-mail or
>>>>> any attachment for any purpose, nor disclose all or any part of the
>>>>> contents to any other person. Thank you.*
>>>>>
>>>>> __________________________________________________________
>>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>>> * sr-users at lists.kamailio.org
>>>>> Important: keep the mailing list in the recipients, do not reply only to the sender!
>>>>> Edit mailing list options or unsubscribe:
>>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>>>>
>>>>>
>>>>
>>>> --
>>>> Andy Chen
>>>> Sr. Telephony Lead Engineer
>>>> 415 516 5535 (M)
>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>
>>>>
>>>> *Confidentiality Notice: The information contained in this e-mail and
>>>> any
>>>> attachments may be confidential. If you are not an intended recipient,
>>>> you
>>>> are hereby notified that any dissemination, distribution or copying of
>>>> this
>>>> e-mail is strictly prohibited. If you have received this e-mail in
>>>> error,
>>>> please notify the sender and permanently delete the e-mail and any
>>>> attachments immediately. You should not retain, copy or use this e-mail
>>>> or
>>>> any attachment for any purpose, nor disclose all or any part of the
>>>> contents to any other person. Thank you.*
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>>>
>>>>
>>>
>>> --
>>> Andy Chen
>>> Sr. Telephony Lead Engineer
>>> 415 516 5535 (M)
>>> achen@ <achen at thinkingphones.com>fuze.com
>>>
>>>
>>> *Confidentiality Notice: The information contained in this e-mail and any
>>> attachments may be confidential. If you are not an intended recipient,
>>> you
>>> are hereby notified that any dissemination, distribution or copying of
>>> this
>>> e-mail is strictly prohibited. If you have received this e-mail in error,
>>> please notify the sender and permanently delete the e-mail and any
>>> attachments immediately. You should not retain, copy or use this e-mail
>>> or
>>> any attachment for any purpose, nor disclose all or any part of the
>>> contents to any other person. Thank you.*
>>>
>>> --
>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>>
>>>
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> 415 516 5535 (M)
>> achen@ <achen at thinkingphones.com>fuze.com
>>
>>
>> *Confidentiality Notice: The information contained in this e-mail and any
>> attachments may be confidential. If you are not an intended recipient, you
>> are hereby notified that any dissemination, distribution or copying of
>> this
>> e-mail is strictly prohibited. If you have received this e-mail in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>
>>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <achen at thinkingphones.com>fuze.com
>
>
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>
>
--
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <achen at thinkingphones.com>fuze.com
--
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20210603/c24cf79f/attachment.htm>
More information about the sr-users
mailing list