[SR-Users] Need help analyze crash
Daniel-Constantin Mierla
miconda at gmail.com
Thu Jun 3 16:12:52 CEST 2021
Well, probably a misunderstanding: the issue was observed on OSes having
the two libssl 1.0 & 1.1 versions, but it does not mean (it was not
proved) that the problem was created by having the two version were
installed at the same time. I would rather think it was more that
libssl1.1 was not very mature at that moment when OSes shipped both of them.
If I would make a suggestion, that would be to use an OS version that
comes only with one of them by default. Think like: when libssl1.0 was
completely removed from an OS, the libssl1.1 was more tested/stable.
Cheers,
Daniel
On 03.06.21 15:58, Andrew Chen wrote:
> No blaming here. :). Just looking for answers.
>
> After chatting with my peers on options around this, we are going to
> remove libssl1.0 and just have libssl1.1 running.
>
> Thanks again for you input.
>
> On Thu, Jun 3, 2021 at 9:48 AM Daniel-Constantin Mierla
> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>
> Again, not to be blamed later: I am not sure you face the same
> kind of issue like reported in the past. I was looking at it from
> the perspective that db_mysql and carrierroute modules were not
> touched that much in the 5.4.x series recently.
>
> The libssl did a lot of internal/api changes between 1.0 and 1.1,
> so for a few past years many libs and apps (including kamailio)
> faced some "strange" crashes/behaviours. Now seems stabilized and,
> at least on debian 10, there were no related open issues recently.
> In the future we may need to make --atexit=no the default (maybe
> needs to be set even on latest ubuntu 20.04), it's the last I
> remember that surfaced and it appeared after a minor version
> upgrade of libssl1.1, iirc.
>
> Cheers,
> Daniel
>
> On 03.06.21 15:31, Andrew Chen wrote:
>> Thanks Daniel.
>> It sounds like upgrading to a newer Ubuntu seems like the
>> cleanest way of resolving this issue.
>>
>> On Thu, Jun 3, 2021 at 8:58 AM Daniel-Constantin Mierla
>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>
>> Hello,
>>
>> it is not related to kamailio tls being compiled with
>> libssl1.1. The issues were observed on the systems having
>> both libssl1.0 (which was somehow the default choice) and
>> libssl1.1. I got the chance to test a bit and it was still
>> crashing even I compiled manually with libssl1.1. I didn't
>> dig much in the external libraries given that upgrading OS
>> seemed to fix it. There is a mod param for db_mysql that you
>> can try to play with: opt_ssl_mode - I don't recall solving
>> much for mysql servers requiring tls.
>>
>> Otherwise, on a different direction: if you can get mariadb
>> client library, you can try to compile kamailio against it
>> (the db_mysql module) -- if the MySQL server is on a remote
>> system, it should work.
>>
>> I can't confirm it is same issue, but if you upgraded in
>> kamailio 5.4.x series, there were no changes to carrierroute
>> or db_mysql to
>>
>> As a side note, for newer operating systems and newer libssl,
>> it may also require to set --atexit cli parameter for kamailio.
>>
>> Cheers,
>> Daniel
>>
>> On 03.06.21 14:42, Andrew Chen wrote:
>>> Hi Daniel,
>>>
>>> I just confirmed mysql is compiled using libssl.so.1.1.
>>>
>>> As for the other kamailio packages, I only see two that are
>>> installed where both are depends on libssl1.1
>>>
>>> kamailio-tls-modules:amd64
>>>
>>> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>=
>>> 7.16.2), libssl1.1 (>= 1.1.0)
>>>
>>>
>>>
>>> kamailio-websocket-modules:amd64
>>>
>>> *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1
>>> (>= 1.1.0), libunistring2 (>= 0.9.7)
>>>
>>>
>>> So in my case here there shouldn't be any conflict with
>>> mysql client libssl version?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla
>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>
>>> Hello,
>>>
>>> I think the relevant here are the client library. I
>>> asked about mysql vs mariadb because some distro had
>>> (meta-)packages named mysql but installing mariadb.
>>> Debian at some point switched to something like
>>> default-mysql-client...
>>>
>>> Based on backtrace, the crashes would prevent kamailio
>>> to start, as carrierroute_db_open() is used in
>>> mod_init/child_init callbacks.
>>>
>>> A variant you could try is the use of mysql-proxy to
>>> bridge from a local unix socket to remote mysql, so
>>> kamailio connects to mysql server via local unix socket
>>> and by that won't use libssl.
>>>
>>> First you can check with ldd if mysql client libraries
>>> are compiled against libssl1.1 and the libssl1.0 is also
>>> used for other packages.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> On 02.06.21 21:56, Andrew Chen wrote:
>>>> Hi Daniel,
>>>> So our DB is in rds and this is what I can gather:
>>>>
>>>> +-------------------------+------------------------------+
>>>>
>>>> | Variable_name | Value |
>>>>
>>>> +-------------------------+------------------------------+
>>>>
>>>> | innodb_version | 5.6.34 |
>>>>
>>>> | protocol_version | 10 |
>>>>
>>>> | slave_type_conversions | |
>>>>
>>>> | version | 5.6.34-log |
>>>>
>>>> | version_comment | MySQL Community Server (GPL) |
>>>>
>>>> | version_compile_machine | x86_64 |
>>>>
>>>> | version_compile_os | Linux |
>>>>
>>>> +-------------------------+------------------------------+
>>>>
>>>>
>>>> The mysql server/client library we received from my
>>>> apt-cache madison output earlier.
>>>>
>>>>
>>>> I don't know if upgrading to 20.04 is an option.
>>>>
>>>>
>>>> Given this said, what are the symptoms of these
>>>> crashes? I'm guessing one of them is a brief
>>>> disconnect from the DB for
>>>> dialplan/dispatcher/carrierroute lookup?
>>>>
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla
>>>> <miconda at gmail.com <mailto:miconda at gmail.com>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> did you upgrade the Kamailio only or also mysql
>>>> server/client library? Is it Oracle's MySQL or MariaDB?
>>>>
>>>> A while ago, there were a few reports of problems
>>>> with mysql-client (Oracle) library on Ubuntu 16.04
>>>> and 18.04, which typically happened after upgrades
>>>> of the library, as I understood, the upgrade
>>>> brought in a dependency on libssl 1.1, while
>>>> libssl1.0 was used by most of the other packages,
>>>> resulting in mixed deployments. Upgrading to 20.04
>>>> seemed to solve it, as it came with libssl1.1 only.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> On 02.06.21 20:51, Andrew Chen wrote:
>>>>> Hi all,
>>>>>
>>>>> Today I upgraded Kamailio to 5.4.5 and a random
>>>>> crash was just reported in our crash monitoring
>>>>> script. Below is the output of the crash:
>>>>>
>>>>> #3 0x00007fa149ebe58d in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
>>>>>
>>>>> #4 0x00007fa14989e277 in ERR_clear_error () from
>>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>>>
>>>>> #5 0x00007fa149933d81 in
>>>>> X509_STORE_set_default_paths () from
>>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
>>>>>
>>>>> #6 0x00007fa10310d374 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>>
>>>>> #7 0x00007fa10310d9f3 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>>
>>>>> #8 0x00007fa1030e0427 in mysql_real_connect ()
>>>>> from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>>>>>
>>>>> #9 0x00007fa10368428c in db_mysql_new_connection
>>>>> () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>>>
>>>>> #10 0x00007fa1501f96dc in db_do_init2 () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>>>
>>>>> #11 0x00007fa1501f83f9 in db_do_init () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>>>>>
>>>>> #12 0x00007fa103689710 in db_mysql_init () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>>>>>
>>>>> #13 0x00007fa15062d35f in carrierroute_db_open ()
>>>>> from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>>>
>>>>> #14 0x00007fa1506719c6 in ?? () from
>>>>> /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so
>>>>>
>>>>>
>>>>> Is this crash due to libcrypto or mysql client?
>>>>> Currently I have the following mysql client
>>>>> installed on the kamailio instance:
>>>>>
>>>>> ii lib*mysql*client20:amd64
>>>>> 5.7.34-0ubuntu0.18.04.1 amd64
>>>>> MySQL database client library
>>>>>
>>>>> ii *mysql*-client
>>>>> 5.7.34-0ubuntu0.18.04.1 all
>>>>> MySQL database client (metapackage depending on
>>>>> the latest version)
>>>>>
>>>>> ii *mysql*-client-5.7
>>>>> 5.7.34-0ubuntu0.18.04.1 amd64
>>>>> MySQL database client binaries
>>>>>
>>>>> ii *mysql*-client-core-5.7
>>>>> 5.7.34-0ubuntu0.18.04.1 amd64
>>>>> MySQL database core client binaries
>>>>>
>>>>> ii *mysql*-common
>>>>> 5.8+1.0.4 all
>>>>> MySQL database common files, e.g. /etc/*mysql*/my.cnf
>>>>>
>>>>> root at ashintgtpsg51:/var/lib/cores # apt-cache
>>>>> madison mysql-client
>>>>>
>>>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu
>>>>> <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
>>>>> bionic-updates/main amd64 Packages
>>>>>
>>>>> mysql-client | 5.7.34-0ubuntu0.18.04.1 |
>>>>> http://security.ubuntu.com/ubuntu
>>>>> <http://security.ubuntu.com/ubuntu>
>>>>> bionic-security/main amd64 Packages
>>>>>
>>>>> mysql-client | 5.7.21-1ubuntu1 |
>>>>> http://us-east-1.ec2.archive.ubuntu.com/ubuntu
>>>>> <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
>>>>> bionic/main amd64 Packages
>>>>>
>>>>> root at ashintgtpsg51:/var/lib/cores #
>>>>>
>>>>>
>>>>> Any suggestions would be greatly appreciated.
>>>>> Thanks in advance.
>>>>>
>>>>> --
>>>>> Andy Chen
>>>>> Sr. Telephony Lead Engineer
>>>>> achen@ <mailto:achen at thinkingphones.com>fuze.com
>>>>> <http://fuze.com>
>>>>>
>>>>>
>>>>>
>>>>> *Confidentiality Notice: The information contained
>>>>> in this e-mail and any
>>>>> attachments may be confidential. If you are not an
>>>>> intended recipient, you
>>>>> are hereby notified that any dissemination,
>>>>> distribution or copying of this
>>>>> e-mail is strictly prohibited. If you have
>>>>> received this e-mail in error,
>>>>> please notify the sender and permanently delete
>>>>> the e-mail and any
>>>>> attachments immediately. You should not retain,
>>>>> copy or use this e-mail or
>>>>> any attachment for any purpose, nor disclose all
>>>>> or any part of the
>>>>> contents to any other person. Thank you.*
>>>>>
>>>>> __________________________________________________________
>>>>> Kamailio - Users Mailing List - Non Commercial Discussions
>>>>> * sr-users at lists.kamailio.org <mailto:sr-users at lists.kamailio.org>
>>>>> Important: keep the mailing list in the recipients, do not reply only to the sender!
>>>>> Edit mailing list options or unsubscribe:
>>>>> * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>
>>>>
>>>>
>>>>
>>>> --
>>>> Andy Chen
>>>> Sr. Telephony Lead Engineer
>>>> 415 516 5535 (M)
>>>> achen@ <mailto:achen at thinkingphones.com>fuze.com
>>>> <http://fuze.com>
>>>>
>>>>
>>>> *Confidentiality Notice: The information contained in
>>>> this e-mail and any
>>>> attachments may be confidential. If you are not an
>>>> intended recipient, you
>>>> are hereby notified that any dissemination,
>>>> distribution or copying of this
>>>> e-mail is strictly prohibited. If you have received
>>>> this e-mail in error,
>>>> please notify the sender and permanently delete the
>>>> e-mail and any
>>>> attachments immediately. You should not retain, copy or
>>>> use this e-mail or
>>>> any attachment for any purpose, nor disclose all or any
>>>> part of the
>>>> contents to any other person. Thank you.*
>>>
>>> --
>>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>>> * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>
>>>
>>>
>>>
>>> --
>>> Andy Chen
>>> Sr. Telephony Lead Engineer
>>> 415 516 5535 (M)
>>> achen@ <mailto:achen at thinkingphones.com>fuze.com
>>> <http://fuze.com>
>>>
>>>
>>> *Confidentiality Notice: The information contained in this
>>> e-mail and any
>>> attachments may be confidential. If you are not an intended
>>> recipient, you
>>> are hereby notified that any dissemination, distribution or
>>> copying of this
>>> e-mail is strictly prohibited. If you have received this
>>> e-mail in error,
>>> please notify the sender and permanently delete the e-mail
>>> and any
>>> attachments immediately. You should not retain, copy or use
>>> this e-mail or
>>> any attachment for any purpose, nor disclose all or any part
>>> of the
>>> contents to any other person. Thank you.*
>>
>> --
>> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
>> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
>> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
>> * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>
>>
>>
>>
>> --
>> Andy Chen
>> Sr. Telephony Lead Engineer
>> 415 516 5535 (M)
>> achen@ <mailto:achen at thinkingphones.com>fuze.com <http://fuze.com>
>>
>>
>> *Confidentiality Notice: The information contained in this e-mail
>> and any
>> attachments may be confidential. If you are not an intended
>> recipient, you
>> are hereby notified that any dissemination, distribution or
>> copying of this
>> e-mail is strictly prohibited. If you have received this e-mail
>> in error,
>> please notify the sender and permanently delete the e-mail and any
>> attachments immediately. You should not retain, copy or use this
>> e-mail or
>> any attachment for any purpose, nor disclose all or any part of the
>> contents to any other person. Thank you.*
>
> --
> Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
> Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
> * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>
>
>
>
> --
> Andy Chen
> Sr. Telephony Lead Engineer
> 415 516 5535 (M)
> achen@ <mailto:achen at thinkingphones.com>fuze.com <http://fuze.com>
>
>
> *Confidentiality Notice: The information contained in this e-mail and any
> attachments may be confidential. If you are not an intended recipient, you
> are hereby notified that any dissemination, distribution or copying of
> this
> e-mail is strictly prohibited. If you have received this e-mail in error,
> please notify the sender and permanently delete the e-mail and any
> attachments immediately. You should not retain, copy or use this e-mail or
> any attachment for any purpose, nor disclose all or any part of the
> contents to any other person. Thank you.*
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
* https://www.asipto.com/sw/kamailio-advanced-training-online/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20210603/ea024972/attachment.htm>
More information about the sr-users
mailing list