[SR-Users] Need help analyze crash
Andrew Chen
achen at fuze.com
Thu Jun 3 14:42:40 CEST 2021
Hi Daniel,
I just confirmed mysql is compiled using libssl.so.1.1.
As for the other kamailio packages, I only see two that are installed where
both are depends on libssl1.1
kamailio-tls-modules:amd64
*Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>= 7.16.2),
libssl1.1 (>= 1.1.0)
kamailio-websocket-modules:amd64
*Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1 (>= 1.1.0),
libunistring2 (>= 0.9.7)
So in my case here there shouldn't be any conflict with mysql client libssl
version?
Thanks.
On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla <miconda at gmail.com>
wrote:
> Hello,
>
> I think the relevant here are the client library. I asked about mysql vs
> mariadb because some distro had (meta-)packages named mysql but installing
> mariadb. Debian at some point switched to something like
> default-mysql-client...
>
> Based on backtrace, the crashes would prevent kamailio to start, as
> carrierroute_db_open() is used in mod_init/child_init callbacks.
>
> A variant you could try is the use of mysql-proxy to bridge from a local
> unix socket to remote mysql, so kamailio connects to mysql server via
> local unix socket and by that won't use libssl.
>
> First you can check with ldd if mysql client libraries are compiled
> against libssl1.1 and the libssl1.0 is also used for other packages.
>
> Cheers,
> Daniel
> On 02.06.21 21:56, Andrew Chen wrote:
>
> Hi Daniel,
> So our DB is in rds and this is what I can gather:
>
> +-------------------------+------------------------------+
>
> | Variable_name | Value |
>
> +-------------------------+------------------------------+
>
> | innodb_version | 5.6.34 |
>
> | protocol_version | 10 |
>
> | slave_type_conversions | |
>
> | version | 5.6.34-log |
>
> | version_comment | MySQL Community Server (GPL) |
>
> | version_compile_machine | x86_64 |
>
> | version_compile_os | Linux |
>
> +-------------------------+------------------------------+
>
>
> The mysql server/client library we received from my apt-cache madison
> output earlier.
>
>
> I don't know if upgrading to 20.04 is an option.
>
>
> Given this said, what are the symptoms of these crashes? I'm guessing one
> of them is a brief disconnect from the DB for
> dialplan/dispatcher/carrierroute lookup?
>
>
> Thanks.
>
> On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla <miconda at gmail.com>
> wrote:
>
>> Hello,
>>
>> did you upgrade the Kamailio only or also mysql server/client library? Is
>> it Oracle's MySQL or MariaDB?
>>
>> A while ago, there were a few reports of problems with mysql-client
>> (Oracle) library on Ubuntu 16.04 and 18.04, which typically happened after
>> upgrades of the library, as I understood, the upgrade brought in a
>> dependency on libssl 1.1, while libssl1.0 was used by most of the other
>> packages, resulting in mixed deployments. Upgrading to 20.04 seemed to
>> solve it, as it came with libssl1.1 only.
>>
>> Cheers,
>> Daniel
>> On 02.06.21 20:51, Andrew Chen wrote:
>>
>> Hi all,
>>
>> Today I upgraded Kamailio to 5.4.5 and a random crash was just reported
>> in our crash monitoring script. Below is the output of the crash:
>>
>> #3 0x00007fa149ebe58d in ?? () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
>>
>> #4 0x00007fa14989e277 in ERR_clear_error () from
>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>
>> #5 0x00007fa149933d81 in X509_STORE_set_default_paths () from
>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>
>> #6 0x00007fa10310d374 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>
>> #7 0x00007fa10310d9f3 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>
>> #8 0x00007fa1030e0427 in mysql_real_connect () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>
>> #9 0x00007fa10368428c in db_mysql_new_connection () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>
>> #10 0x00007fa1501f96dc in db_do_init2 () from
>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>
>> #11 0x00007fa1501f83f9 in db_do_init () from
>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>
>> #12 0x00007fa103689710 in db_mysql_init () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>
>> #13 0x00007fa15062d35f in carrierroute_db_open () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>
>> #14 0x00007fa1506719c6 in ?? () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>
>>
>> Is this crash due to libcrypto or mysql client?
>> Currently I have the following mysql client installed on the kamailio
>> instance:
>>
>> ii lib*mysql*client20:amd64 5.7.34-0ubuntu0.18.04.1
>> amd64 MySQL database client library
>>
>> ii *mysql*-client 5.7.34-0ubuntu0.18.04.1
>> all MySQL database client (metapackage depending on the
>> latest version)
>>
>> ii *mysql*-client-5.7 5.7.34-0ubuntu0.18.04.1
>> amd64 MySQL database client binaries
>>
>> ii *mysql*-client-core-5.7 5.7.34-0ubuntu0.18.04.1
>> amd64 MySQL database core client binaries
>>
>> ii *mysql*-common 5.8+1.0.4
>> all MySQL database common files, e.g. /etc/*mysql*/my.cnf
>>
>> root at ashintgtpsg51:/var/lib/cores # apt-cache madison mysql-client
>>
>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic-updates/main amd64
>> Packages
>>
>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>> http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
>>
>> mysql-client | 5.7.21-1ubuntu1 |
>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
>>
>> root at ashintgtpsg51:/var/lib/cores #
>>
>>
>> Any suggestions would be greatly appreciated.
>> Thanks in advance.
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> achen@ <achen at thinkingphones.com>fuze.com
>>
>>
>>
>> *Confidentiality Notice: The information contained in this e-mail and any
>> attachments may be confidential. If you are not an intended recipient, you
>> are hereby notified that any dissemination, distribution or copying of
>> this
>> e-mail is strictly prohibited. If you have received this e-mail in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>>
>> __________________________________________________________
>> Kamailio - Users Mailing List - Non Commercial Discussions
>> * sr-users at lists.kamailio.org
>> Important: keep the mailing list in the recipients, do not reply only to the sender!
>> Edit mailing list options or unsubscribe:
>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>>
>>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <achen at thinkingphones.com>fuze.com
>
>
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
>
> --
> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
> * https://www.asipto.com/sw/kamailio-advanced-training-online/
>
>
--
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <achen at thinkingphones.com>fuze.com
--
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20210603/6ecccb03/attachment.htm>
More information about the sr-users
mailing list