[SR-Users] Need help analyze crash
Andrew Chen
achen at fuze.com
Thu Jun 3 15:31:22 CEST 2021
Thanks Daniel.
It sounds like upgrading to a newer Ubuntu seems like the cleanest way of
resolving this issue.
On Thu, Jun 3, 2021 at 8:58 AM Daniel-Constantin Mierla <miconda at gmail.com>
wrote:
> Hello,
>
> it is not related to kamailio tls being compiled with libssl1.1. The
> issues were observed on the systems having both libssl1.0 (which was
> somehow the default choice) and libssl1.1. I got the chance to test a bit
> and it was still crashing even I compiled manually with libssl1.1. I didn't
> dig much in the external libraries given that upgrading OS seemed to fix
> it. There is a mod param for db_mysql that you can try to play with:
> opt_ssl_mode - I don't recall solving much for mysql servers requiring tls.
>
> Otherwise, on a different direction: if you can get mariadb client
> library, you can try to compile kamailio against it (the db_mysql module)
> -- if the MySQL server is on a remote system, it should work.
>
> I can't confirm it is same issue, but if you upgraded in kamailio 5.4.x
> series, there were no changes to carrierroute or db_mysql to
>
> As a side note, for newer operating systems and newer libssl, it may also
> require to set --atexit cli parameter for kamailio.
>
> Cheers,
> Daniel
> On 03.06.21 14:42, Andrew Chen wrote:
>
> Hi Daniel,
>
> I just confirmed mysql is compiled using libssl.so.1.1.
>
> As for the other kamailio packages, I only see two that are installed
> where both are depends on libssl1.1
>
> kamailio-tls-modules:amd64
>
> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>= 7.16.2),
> libssl1.1 (>= 1.1.0)
>
>
>
> kamailio-websocket-modules:amd64
>
> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1 (>= 1.1.0),
> libunistring2 (>= 0.9.7)
>
> So in my case here there shouldn't be any conflict with mysql client
> libssl version?
>
> Thanks.
>
>
> On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla <miconda at gmail.com>
> wrote:
>
>> Hello,
>>
>> I think the relevant here are the client library. I asked about mysql vs
>> mariadb because some distro had (meta-)packages named mysql but installing
>> mariadb. Debian at some point switched to something like
>> default-mysql-client...
>>
>> Based on backtrace, the crashes would prevent kamailio to start, as
>> carrierroute_db_open() is used in mod_init/child_init callbacks.
>>
>> A variant you could try is the use of mysql-proxy to bridge from a local
>> unix socket to remote mysql, so kamailio connects to mysql server via
>> local unix socket and by that won't use libssl.
>>
>> First you can check with ldd if mysql client libraries are compiled
>> against libssl1.1 and the libssl1.0 is also used for other packages.
>>
>> Cheers,
>> Daniel
>> On 02.06.21 21:56, Andrew Chen wrote:
>>
>> Hi Daniel,
>> So our DB is in rds and this is what I can gather:
>>
>> +-------------------------+------------------------------+
>>
>> | Variable_name | Value |
>>
>> +-------------------------+------------------------------+
>>
>> | innodb_version | 5.6.34 |
>>
>> | protocol_version | 10 |
>>
>> | slave_type_conversions | |
>>
>> | version | 5.6.34-log |
>>
>> | version_comment | MySQL Community Server (GPL) |
>>
>> | version_compile_machine | x86_64 |
>>
>> | version_compile_os | Linux |
>>
>> +-------------------------+------------------------------+
>>
>>
>> The mysql server/client library we received from my apt-cache madison
>> output earlier.
>>
>>
>> I don't know if upgrading to 20.04 is an option.
>>
>>
>> Given this said, what are the symptoms of these crashes? I'm guessing
>> one of them is a brief disconnect from the DB for
>> dialplan/dispatcher/carrierroute lookup?
>>
>>
>> Thanks.
>>
>> On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>> Hello,
>>>
>>> did you upgrade the Kamailio only or also mysql server/client library?
>>> Is it Oracle's MySQL or MariaDB?
>>>
>>> A while ago, there were a few reports of problems with mysql-client
>>> (Oracle) library on Ubuntu 16.04 and 18.04, which typically happened after
>>> upgrades of the library, as I understood, the upgrade brought in a
>>> dependency on libssl 1.1, while libssl1.0 was used by most of the other
>>> packages, resulting in mixed deployments. Upgrading to 20.04 seemed to
>>> solve it, as it came with libssl1.1 only.
>>>
>>> Cheers,
>>> Daniel
>>> On 02.06.21 20:51, Andrew Chen wrote:
>>>
>>> Hi all,
>>>
>>> Today I upgraded Kamailio to 5.4.5 and a random crash was just reported
>>> in our crash monitoring script. Below is the output of the crash:
>>>
>>> #3 0x00007fa149ebe58d in ?? () from
>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
>>>
>>> #4 0x00007fa14989e277 in ERR_clear_error () from
>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>
>>> #5 0x00007fa149933d81 in X509_STORE_set_default_paths () from
>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>
>>> #6 0x00007fa10310d374 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>
>>> #7 0x00007fa10310d9f3 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>
>>> #8 0x00007fa1030e0427 in mysql_real_connect () from
>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>
>>> #9 0x00007fa10368428c in db_mysql_new_connection () from
>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>
>>> #10 0x00007fa1501f96dc in db_do_init2 () from
>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>
>>> #11 0x00007fa1501f83f9 in db_do_init () from
>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>
>>> #12 0x00007fa103689710 in db_mysql_init () from
>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>
>>> #13 0x00007fa15062d35f in carrierroute_db_open () from
>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>
>>> #14 0x00007fa1506719c6 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>
>>>
>>> Is this crash due to libcrypto or mysql client?
>>> Currently I have the following mysql client installed on the kamailio
>>> instance:
>>>
>>> ii lib*mysql*client20:amd64 5.7.34-0ubuntu0.18.04.1
>>> amd64 MySQL database client library
>>>
>>> ii *mysql*-client 5.7.34-0ubuntu0.18.04.1
>>> all MySQL database client (metapackage depending on
>>> the latest version)
>>>
>>> ii *mysql*-client-5.7 5.7.34-0ubuntu0.18.04.1
>>> amd64 MySQL database client binaries
>>>
>>> ii *mysql*-client-core-5.7 5.7.34-0ubuntu0.18.04.1
>>> amd64 MySQL database core client binaries
>>>
>>> ii *mysql*-common 5.8+1.0.4
>>> all MySQL database common files, e.g. /etc/*mysql*
>>> /my.cnf
>>>
>>> root at ashintgtpsg51:/var/lib/cores # apt-cache madison mysql-client
>>>
>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main
>>> amd64 Packages
>>>
>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>> http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
>>>
>>> mysql-client | 5.7.21-1ubuntu1 |
>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64
>>> Packages
>>>
>>> root at ashintgtpsg51:/var/lib/cores #
>>>
>>>
>>> Any suggestions would be greatly appreciated.
>>> Thanks in advance.
>>>
>>> --
>>> Andy Chen
>>> Sr. Telephony Lead Engineer
>>> achen@ <achen at thinkingphones.com>fuze.com
>>>
>>>
>>>
>>> *Confidentiality Notice: The information contained in this e-mail and any
>>> attachments may be confidential. If you are not an intended recipient,
>>> you
>>> are hereby notified that any dissemination, distribution or copying of
>>> this
>>> e-mail is strictly prohibited. If you have received this e-mail in error,
>>> please notify the sender and permanently delete the e-mail and any
>>> attachments immediately. You should not retain, copy or use this e-mail
>>> or
>>> any attachment for any purpose, nor disclose all or any part of the
>>> contents to any other person. Thank you.*
>>>
>>> __________________________________________________________
>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>> * sr-users at lists.kamailio.org
>>> Important: keep the mailing list in the recipients, do not reply only to the sender!
>>> Edit mailing list options or unsubscribe:
>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>> --
>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>>
>>>
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> 415 516 5535 (M)
>> achen@ <achen at thinkingphones.com>fuze.com
>>
>>
>> *Confidentiality Notice: The information contained in this e-mail and any
>> attachments may be confidential. If you are not an intended recipient, you
>> are hereby notified that any dissemination, distribution or copying of
>> this
>> e-mail is strictly prohibited. If you have received this e-mail in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>
>>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <achen at thinkingphones.com>fuze.com
>
>
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>
>
--
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <achen at thinkingphones.com>fuze.com
--
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20210603/cb241b19/attachment.htm>
More information about the sr-users
mailing list