[SR-Users] Need help analyze crash
Andrew Chen
achen at fuze.com
Thu Jun 3 15:58:49 CEST 2021
No blaming here. :). Just looking for answers.
After chatting with my peers on options around this, we are going to remove
libssl1.0 and just have libssl1.1 running.
Thanks again for you input.
On Thu, Jun 3, 2021 at 9:48 AM Daniel-Constantin Mierla <miconda at gmail.com>
wrote:
> Again, not to be blamed later: I am not sure you face the same kind of
> issue like reported in the past. I was looking at it from the perspective
> that db_mysql and carrierroute modules were not touched that much in the
> 5.4.x series recently.
>
> The libssl did a lot of internal/api changes between 1.0 and 1.1, so for a
> few past years many libs and apps (including kamailio) faced some "strange"
> crashes/behaviours. Now seems stabilized and, at least on debian 10, there
> were no related open issues recently. In the future we may need to make
> --atexit=no the default (maybe needs to be set even on latest ubuntu
> 20.04), it's the last I remember that surfaced and it appeared after a
> minor version upgrade of libssl1.1, iirc.
>
> Cheers,
> Daniel
> On 03.06.21 15:31, Andrew Chen wrote:
>
> Thanks Daniel.
> It sounds like upgrading to a newer Ubuntu seems like the cleanest way of
> resolving this issue.
>
> On Thu, Jun 3, 2021 at 8:58 AM Daniel-Constantin Mierla <miconda at gmail.com>
> wrote:
>
>> Hello,
>>
>> it is not related to kamailio tls being compiled with libssl1.1. The
>> issues were observed on the systems having both libssl1.0 (which was
>> somehow the default choice) and libssl1.1. I got the chance to test a bit
>> and it was still crashing even I compiled manually with libssl1.1. I didn't
>> dig much in the external libraries given that upgrading OS seemed to fix
>> it. There is a mod param for db_mysql that you can try to play with:
>> opt_ssl_mode - I don't recall solving much for mysql servers requiring tls.
>>
>> Otherwise, on a different direction: if you can get mariadb client
>> library, you can try to compile kamailio against it (the db_mysql module)
>> -- if the MySQL server is on a remote system, it should work.
>>
>> I can't confirm it is same issue, but if you upgraded in kamailio 5.4.x
>> series, there were no changes to carrierroute or db_mysql to
>>
>> As a side note, for newer operating systems and newer libssl, it may also
>> require to set --atexit cli parameter for kamailio.
>>
>> Cheers,
>> Daniel
>> On 03.06.21 14:42, Andrew Chen wrote:
>>
>> Hi Daniel,
>>
>> I just confirmed mysql is compiled using libssl.so.1.1.
>>
>> As for the other kamailio packages, I only see two that are installed
>> where both are depends on libssl1.1
>>
>> kamailio-tls-modules:amd64
>>
>> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>= 7.16.2),
>> libssl1.1 (>= 1.1.0)
>>
>>
>>
>> kamailio-websocket-modules:amd64
>>
>> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1 (>= 1.1.0),
>> libunistring2 (>= 0.9.7)
>>
>> So in my case here there shouldn't be any conflict with mysql client
>> libssl version?
>>
>> Thanks.
>>
>>
>> On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I think the relevant here are the client library. I asked about mysql vs
>>> mariadb because some distro had (meta-)packages named mysql but installing
>>> mariadb. Debian at some point switched to something like
>>> default-mysql-client...
>>>
>>> Based on backtrace, the crashes would prevent kamailio to start, as
>>> carrierroute_db_open() is used in mod_init/child_init callbacks.
>>>
>>> A variant you could try is the use of mysql-proxy to bridge from a local
>>> unix socket to remote mysql, so kamailio connects to mysql server via
>>> local unix socket and by that won't use libssl.
>>>
>>> First you can check with ldd if mysql client libraries are compiled
>>> against libssl1.1 and the libssl1.0 is also used for other packages.
>>>
>>> Cheers,
>>> Daniel
>>> On 02.06.21 21:56, Andrew Chen wrote:
>>>
>>> Hi Daniel,
>>> So our DB is in rds and this is what I can gather:
>>>
>>> +-------------------------+------------------------------+
>>>
>>> | Variable_name | Value |
>>>
>>> +-------------------------+------------------------------+
>>>
>>> | innodb_version | 5.6.34 |
>>>
>>> | protocol_version | 10 |
>>>
>>> | slave_type_conversions | |
>>>
>>> | version | 5.6.34-log |
>>>
>>> | version_comment | MySQL Community Server (GPL) |
>>>
>>> | version_compile_machine | x86_64 |
>>>
>>> | version_compile_os | Linux |
>>>
>>> +-------------------------+------------------------------+
>>>
>>>
>>> The mysql server/client library we received from my apt-cache madison
>>> output earlier.
>>>
>>>
>>> I don't know if upgrading to 20.04 is an option.
>>>
>>>
>>> Given this said, what are the symptoms of these crashes? I'm guessing
>>> one of them is a brief disconnect from the DB for
>>> dialplan/dispatcher/carrierroute lookup?
>>>
>>>
>>> Thanks.
>>>
>>> On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla <
>>> miconda at gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> did you upgrade the Kamailio only or also mysql server/client library?
>>>> Is it Oracle's MySQL or MariaDB?
>>>>
>>>> A while ago, there were a few reports of problems with mysql-client
>>>> (Oracle) library on Ubuntu 16.04 and 18.04, which typically happened after
>>>> upgrades of the library, as I understood, the upgrade brought in a
>>>> dependency on libssl 1.1, while libssl1.0 was used by most of the other
>>>> packages, resulting in mixed deployments. Upgrading to 20.04 seemed to
>>>> solve it, as it came with libssl1.1 only.
>>>>
>>>> Cheers,
>>>> Daniel
>>>> On 02.06.21 20:51, Andrew Chen wrote:
>>>>
>>>> Hi all,
>>>>
>>>> Today I upgraded Kamailio to 5.4.5 and a random crash was just reported
>>>> in our crash monitoring script. Below is the output of the crash:
>>>>
>>>> #3 0x00007fa149ebe58d in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
>>>>
>>>> #4 0x00007fa14989e277 in ERR_clear_error () from
>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>>
>>>> #5 0x00007fa149933d81 in X509_STORE_set_default_paths () from
>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>>
>>>> #6 0x00007fa10310d374 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>
>>>> #7 0x00007fa10310d9f3 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>
>>>> #8 0x00007fa1030e0427 in mysql_real_connect () from
>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>
>>>> #9 0x00007fa10368428c in db_mysql_new_connection () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>>
>>>> #10 0x00007fa1501f96dc in db_do_init2 () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>>
>>>> #11 0x00007fa1501f83f9 in db_do_init () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>>
>>>> #12 0x00007fa103689710 in db_mysql_init () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>>
>>>> #13 0x00007fa15062d35f in carrierroute_db_open () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>>
>>>> #14 0x00007fa1506719c6 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>>
>>>>
>>>> Is this crash due to libcrypto or mysql client?
>>>> Currently I have the following mysql client installed on the kamailio
>>>> instance:
>>>>
>>>> ii lib*mysql*client20:amd64 5.7.34-0ubuntu0.18.04.1
>>>> amd64 MySQL database client library
>>>>
>>>> ii *mysql*-client 5.7.34-0ubuntu0.18.04.1
>>>> all MySQL database client (metapackage depending on
>>>> the latest version)
>>>>
>>>> ii *mysql*-client-5.7 5.7.34-0ubuntu0.18.04.1
>>>> amd64 MySQL database client binaries
>>>>
>>>> ii *mysql*-client-core-5.7 5.7.34-0ubuntu0.18.04.1
>>>> amd64 MySQL database core client binaries
>>>>
>>>> ii *mysql*-common 5.8+1.0.4
>>>> all MySQL database common files, e.g. /etc/*mysql*
>>>> /my.cnf
>>>>
>>>> root at ashintgtpsg51:/var/lib/cores # apt-cache madison mysql-client
>>>>
>>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main
>>>> amd64 Packages
>>>>
>>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>>> http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
>>>>
>>>> mysql-client | 5.7.21-1ubuntu1 |
>>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64
>>>> Packages
>>>>
>>>> root at ashintgtpsg51:/var/lib/cores #
>>>>
>>>>
>>>> Any suggestions would be greatly appreciated.
>>>> Thanks in advance.
>>>>
>>>> --
>>>> Andy Chen
>>>> Sr. Telephony Lead Engineer
>>>> achen@ <achen at thinkingphones.com>fuze.com
>>>>
>>>>
>>>>
>>>> *Confidentiality Notice: The information contained in this e-mail and
>>>> any
>>>> attachments may be confidential. If you are not an intended recipient,
>>>> you
>>>> are hereby notified that any dissemination, distribution or copying of
>>>> this
>>>> e-mail is strictly prohibited. If you have received this e-mail in
>>>> error,
>>>> please notify the sender and permanently delete the e-mail and any
>>>> attachments immediately. You should not retain, copy or use this e-mail
>>>> or
>>>> any attachment for any purpose, nor disclose all or any part of the
>>>> contents to any other person. Thank you.*
>>>>
>>>> __________________________________________________________
>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>> * sr-users at lists.kamailio.org
>>>> Important: keep the mailing list in the recipients, do not reply only to the sender!
>>>> Edit mailing list options or unsubscribe:
>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>>>
>>>>
>>>
>>> --
>>> Andy Chen
>>> Sr. Telephony Lead Engineer
>>> 415 516 5535 (M)
>>> achen@ <achen at thinkingphones.com>fuze.com
>>>
>>>
>>> *Confidentiality Notice: The information contained in this e-mail and any
>>> attachments may be confidential. If you are not an intended recipient,
>>> you
>>> are hereby notified that any dissemination, distribution or copying of
>>> this
>>> e-mail is strictly prohibited. If you have received this e-mail in error,
>>> please notify the sender and permanently delete the e-mail and any
>>> attachments immediately. You should not retain, copy or use this e-mail
>>> or
>>> any attachment for any purpose, nor disclose all or any part of the
>>> contents to any other person. Thank you.*
>>>
>>> --
>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>>
>>>
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> 415 516 5535 (M)
>> achen@ <achen at thinkingphones.com>fuze.com
>>
>>
>> *Confidentiality Notice: The information contained in this e-mail and any
>> attachments may be confidential. If you are not an intended recipient, you
>> are hereby notified that any dissemination, distribution or copying of
>> this
>> e-mail is strictly prohibited. If you have received this e-mail in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>
>>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <achen at thinkingphones.com>fuze.com
>
>
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>
>
--
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <achen at thinkingphones.com>fuze.com
--
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20210603/d194a5f9/attachment.htm>
More information about the sr-users
mailing list