[SR-Users] Need help analyze crash
Daniel-Constantin Mierla
miconda at gmail.com
Thu Jun 3 15:48:55 CEST 2021
Again, not to be blamed later: I am not sure you face the same kind of
issue like reported in the past. I was looking at it from the
perspective that db_mysql and carrierroute modules were not touched that
much in the 5.4.x series recently.
The libssl did a lot of internal/api changes between 1.0 and 1.1, so for
a few past years many libs and apps (including kamailio) faced some
"strange" crashes/behaviours. Now seems stabilized and, at least on
debian 10, there were no related open issues recently. In the future we
may need to make --atexit=no the default (maybe needs to be set even on
latest ubuntu 20.04), it's the last I remember that surfaced and it
appeared after a minor version upgrade of libssl1.1, iirc.
Cheers,
Daniel
On 03.06.21 15:31, Andrew Chen wrote:
> Thanks Daniel.
> It sounds like upgrading to a newer Ubuntu seems like the cleanest way
> of resolving this issue.
>
> On Thu, Jun 3, 2021 at 8:58 AM Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> Hello,
>
> it is not related to kamailio tls being compiled with libssl1.1.
> The issues were observed on the systems having both libssl1.0
> (which was somehow the default choice) and libssl1.1. I got the
> chance to test a bit and it was still crashing even I compiled
> manually with libssl1.1. I didn't dig much in the external
> libraries given that upgrading OS seemed to fix it. There is a mod
> param for db_mysql that you can try to play with: opt_ssl_mode - I
> don't recall solving much for mysql servers requiring tls.
>
> Otherwise, on a different direction: if you can get mariadb client
> library, you can try to compile kamailio against it (the db_mysql
> module) -- if the MySQL server is on a remote system, it should work.
>
> I can't confirm it is same issue, but if you upgraded in kamailio
> 5.4.x series, there were no changes to carrierroute or db_mysql to
>
> As a side note, for newer operating systems and newer libssl, it
> may also require to set --atexit cli parameter for kamailio.
>
> Cheers,
> Daniel
>
> On 03.06.21 14:42, Andrew Chen wrote:
>> Hi Daniel,
>>
>> I just confirmed mysql is compiled using libssl.so.1.1.
>>
>> As for the other kamailio packages, I only see two that are
>> installed where both are depends on libssl1.1
>>
>> kamailio-tls-modules:amd64
>>
>> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>=
>> 7.16.2), libssl1.1 (>= 1.1.0)
>>
>>
>>
>> kamailio-websocket-modules:amd64
>>
>> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1 (>=
>> 1.1.0), libunistring2 (>= 0.9.7)
>>
>>
>> So in my case here there shouldn't be any conflict with mysql
>> client libssl version?
>>
>> Thanks.
>>
>>
>> On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>> I think the relevant here are the client library. I asked
>> about mysql vs mariadb because some distro had
>> (meta-)packages named mysql but installing mariadb. Debian at
>> some point switched to something like default-mysql-client...
>>
>> Based on backtrace, the crashes would prevent kamailio to
>> start, as carrierroute_db_open() is used in
>> mod_init/child_init callbacks.
>>
>> A variant you could try is the use of mysql-proxy to bridge
>> from a local unix socket to remote mysql, so kamailio
>> connects to mysql server via local unix socket and by that
>> won't use libssl.
>>
>> First you can check with ldd if mysql client libraries are
>> compiled against libssl1.1 and the libssl1.0 is also used for
>> other packages.
>>
>> Cheers,
>> Daniel
>>
>> On 02.06.21 21:56, Andrew Chen wrote:
>>> Hi Daniel,
>>> So our DB is in rds and this is what I can gather:
>>>
>>> +-------------------------+------------------------------+
>>>
>>> | Variable_name | Value |
>>>
>>> +-------------------------+------------------------------+
>>>
>>> | innodb_version | 5.6.34 |
>>>
>>> | protocol_version | 10 |
>>>
>>> | slave_type_conversions | |
>>>
>>> | version | 5.6.34-log |
>>>
>>> | version_comment | MySQL Community Server (GPL) |
>>>
>>> | version_compile_machine | x86_64 |
>>>
>>> | version_compile_os | Linux |
>>>
>>> +-------------------------+------------------------------+
>>>
>>>
>>> The mysql server/client library we received from my
>>> apt-cache madison output earlier.
>>>
>>>
>>> I don't know if upgrading to 20.04 is an option.
>>>
>>>
>>> Given this said, what are the symptoms of these crashes?
>>> I'm guessing one of them is a brief disconnect from the DB
>>> for dialplan/dispatcher/carrierroute lookup?
>>>
>>>
>>> Thanks.
>>>
>>>
>>> On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> did you upgrade the Kamailio only or also mysql
>>> server/client library? Is it Oracle's MySQL or MariaDB?
>>>
>>> A while ago, there were a few reports of problems with
>>> mysql-client (Oracle) library on Ubuntu 16.04 and 18.04,
>>> which typically happened after upgrades of the library,
>>> as I understood, the upgrade brought in a dependency on
>>> libssl 1.1, while libssl1.0 was used by most of the
>>> other packages, resulting in mixed deployments.
>>> Upgrading to 20.04 seemed to solve it, as it came with
>>> libssl1.1 only.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 02.06.21 20:51, Andrew Chen wrote:
>>>> Hi all,
>>>>
>>>> Today I upgraded Kamailio to 5.4.5 and a random crash
>>>> was just reported in our crash monitoring script.
>>>> Below is the output of the crash:
>>>>
>>>> #3 0x00007fa149ebe58d in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
>>>>
>>>> #4 0x00007fa14989e277 in ERR_clear_error () from
>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>>
>>>> #5 0x00007fa149933d81 in X509_STORE_set_default_paths
>>>> () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>>
>>>> #6 0x00007fa10310d374 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>
>>>> #7 0x00007fa10310d9f3 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>
>>>> #8 0x00007fa1030e0427 in mysql_real_connect () from
>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>
>>>> #9 0x00007fa10368428c in db_mysql_new_connection ()
>>>> from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>>
>>>> #10 0x00007fa1501f96dc in db_do_init2 () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>>
>>>> #11 0x00007fa1501f83f9 in db_do_init () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>>
>>>> #12 0x00007fa103689710 in db_mysql_init () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>>
>>>> #13 0x00007fa15062d35f in carrierroute_db_open () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>>
>>>> #14 0x00007fa1506719c6 in ?? () from
>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>>
>>>>
>>>> Is this crash due to libcrypto or mysql client?
>>>> Currently I have the following mysql client installed
>>>> on the kamailio instance:
>>>>
>>>> ii lib*mysql*client20:amd64
>>>> 5.7.34-0ubuntu0.18.04.1 amd64 MySQL
>>>> database client library
>>>>
>>>> ii *mysql*-client
>>>> 5.7.34-0ubuntu0.18.04.1 all MySQL
>>>> database client (metapackage depending on the latest
>>>> version)
>>>>
>>>> ii *mysql*-client-5.7
>>>> 5.7.34-0ubuntu0.18.04.1 amd64 MySQL
>>>> database client binaries
>>>>
>>>> ii *mysql*-client-core-5.7
>>>> 5.7.34-0ubuntu0.18.04.1 amd64 MySQL
>>>> database core client binaries
>>>>
>>>> ii *mysql*-common 5.8+1.0.4
>>>> all MySQL database
>>>> common files, e.g. /etc/*mysql*/my.cnf
>>>>
>>>> root at ashintgtpsg51:/var/lib/cores # apt-cache madison
>>>> mysql-client
>>>>
>>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu
>>>> <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
>>>> bionic-updates/main amd64 Packages
>>>>
>>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>>> http://security.ubuntu.com/ubuntu
>>>> <http://security.ubuntu.com/ubuntu>
>>>> bionic-security/main amd64 Packages
>>>>
>>>> mysql-client | 5.7.21-1ubuntu1 |
>>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu
>>>> <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
>>>> bionic/main amd64 Packages
>>>>
>>>> root at ashintgtpsg51:/var/lib/cores #
>>>>
>>>>
>>>> Any suggestions would be greatly appreciated.
>>>> Thanks in advance.
>>>>
>>>> --
>>>> Andy Chen
>>>> Sr. Telephony Lead Engineer
>>>> achen@ <mailto:achen at thinkingphones.com>fuze.com
>>>> <http://fuze.com>
>>>>
>>>>
>>>>
>>>> *Confidentiality Notice: The information contained in
>>>> this e-mail and any
>>>> attachments may be confidential. If you are not an
>>>> intended recipient, you
>>>> are hereby notified that any dissemination,
>>>> distribution or copying of this
>>>> e-mail is strictly prohibited. If you have received
>>>> this e-mail in error,
>>>> please notify the sender and permanently delete the
>>>> e-mail and any
>>>> attachments immediately. You should not retain, copy or
>>>> use this e-mail or
>>>> any attachment for any purpose, nor disclose all or any
>>>> part of the
>>>> contents to any other person. Thank you.*
>>>>
>>>> __________________________________________________________
>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>> * sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>>> Important: keep the mailing list in the recipients, do not reply only to the sender!
>>>> Edit mailing list options or unsubscribe:
>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
>>>
>>> --
>>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>
>>>
>>>
>>>
>>> --
>>> Andy Chen
>>> Sr. Telephony Lead Engineer
>>> 415 516 5535 (M)
>>> achen@ <mailto:achen at thinkingphones.com>fuze.com
>>> <http://fuze.com>
>>>
>>>
>>> *Confidentiality Notice: The information contained in this
>>> e-mail and any
>>> attachments may be confidential. If you are not an intended
>>> recipient, you
>>> are hereby notified that any dissemination, distribution or
>>> copying of this
>>> e-mail is strictly prohibited. If you have received this
>>> e-mail in error,
>>> please notify the sender and permanently delete the e-mail
>>> and any
>>> attachments immediately. You should not retain, copy or use
>>> this e-mail or
>>> any attachment for any purpose, nor disclose all or any part
>>> of the
>>> contents to any other person. Thank you.*
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>> * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>
>>
>>
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> 415 516 5535 (M)
>> achen@ <mailto:achen at thinkingphones.com>fuze.com <http://fuze.com>
>>
>>
>> *Confidentiality Notice: The information contained in this e-mail
>> and any
>> attachments may be confidential. If you are not an intended
>> recipient, you
>> are hereby notified that any dissemination, distribution or
>> copying of this
>> e-mail is strictly prohibited. If you have received this e-mail
>> in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this
>> e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>
> --
> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
> * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>
>
>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <mailto:achen at thinkingphones.com>fuze.com <http://fuze.com>
>
>
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of
> this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20210603/5c4bed97/attachment.htm>
More information about the sr-users
mailing list