<div dir="ltr">No blaming here. :). Just looking for answers.<div><br></div><div>After chatting with my peers on options around this, we are going to remove libssl1.0 and just have libssl1.1 running.</div><div><br></div><div>Thanks again for you input.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 3, 2021 at 9:48 AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Again, not to be blamed later: I am not sure you face the same
kind of issue like reported in the past. I was looking at it from
the perspective that db_mysql and carrierroute modules were not
touched that much in the 5.4.x series recently.</p>
<p>The libssl did a lot of internal/api changes between 1.0 and 1.1,
so for a few past years many libs and apps (including kamailio)
faced some "strange" crashes/behaviours. Now seems stabilized and,
at least on debian 10, there were no related open issues recently.
In the future we may need to make --atexit=no the default (maybe
needs to be set even on latest ubuntu 20.04), it's the last I
remember that surfaced and it appeared after a minor version
upgrade of libssl1.1, iirc.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 03.06.21 15:31, Andrew Chen wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Thanks Daniel.
<div>It sounds like upgrading to a newer Ubuntu seems like the
cleanest way of resolving this issue.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jun 3, 2021 at 8:58 AM
Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>it is not related to kamailio tls being compiled with
libssl1.1. The issues were observed on the systems having
both libssl1.0 (which was somehow the default choice) and
libssl1.1. I got the chance to test a bit and it was still
crashing even I compiled manually with libssl1.1. I didn't
dig much in the external libraries given that upgrading OS
seemed to fix it. There is a mod param for db_mysql that
you can try to play with: opt_ssl_mode - I don't recall
solving much for mysql servers requiring tls.<br>
</p>
<p>Otherwise, on a different direction: if you can get
mariadb client library, you can try to compile kamailio
against it (the db_mysql module) -- if the MySQL server is
on a remote system, it should work.</p>
<p>I can't confirm it is same issue, but if you upgraded in
kamailio 5.4.x series, there were no changes to
carrierroute or db_mysql to <br>
</p>
<p>As a side note, for newer operating systems and newer
libssl, it may also require to set --atexit cli parameter
for kamailio.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 03.06.21 14:42, Andrew Chen wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Daniel,
<div><br>
</div>
<div>I just confirmed mysql is compiled using
libssl.so.1.1.</div>
<div><br>
</div>
<div>As for the other kamailio packages, I only see two
that are installed where both are depends on libssl1.1</div>
<div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px"><br>
</span></div>
<div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">kamailio-tls-modules:amd64</span></div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>Depends</b></span><span style="font-variant-ligatures:no-common-ligatures">:
kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4
(>= 7.16.2), libssl1.1 (>= 1.1.0)</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">kamailio-websocket-modules:amd64</span></p>
<p style="margin:0px;font:11px Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>Depends</b></span><span style="font-variant-ligatures:no-common-ligatures">:
kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1
(>= 1.1.0), libunistring2 (>= 0.9.7)</span></p>
<div><br>
</div>
<div>So in my case here there shouldn't be any conflict
with mysql client libssl version?</div>
<div><br>
</div>
<div>Thanks.</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jun 3, 2021 at
3:10 AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>I think the relevant here are the client library.
I asked about mysql vs mariadb because some distro
had (meta-)packages named mysql but installing
mariadb. Debian at some point switched to
something like default-mysql-client...</p>
<p>Based on backtrace, the crashes would prevent
kamailio to start, as carrierroute_db_open() is
used in mod_init/child_init callbacks.</p>
<p>A variant you could try is the use of mysql-proxy
to bridge from a local unix socket to remote
mysql, so kamailio connects to mysql server via
local unix socket and by that won't use libssl.</p>
<p>First you can check with ldd if mysql client
libraries are compiled against libssl1.1 and the
libssl1.0 is also used for other packages.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 02.06.21 21:56, Andrew Chen wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Daniel,
<div>So our DB is in rds and this is what I
can gather:</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">+-------------------------+------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">| Variable_name <span>
</span>| Value<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">+-------------------------+------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">| innodb_version<span>
</span>| 5.6.34 <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">| protocol_version<span>
</span>| 10 <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
slave_type_conversions<span> </span>|<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">| version <span>
</span>| 5.6.34-log <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">| version_comment <span>
</span>| MySQL Community Server
(GPL) |</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
version_compile_machine | x86_64 <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">| version_compile_os<span>
</span>| Linux<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">+-------------------------+------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">The mysql
server/client library we received from
my apt-cache madison output earlier.</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">I don't know if
upgrading to 20.04 is an option.</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Given this said, what
are the symptoms of these crashes? I'm
guessing one of them is a brief
disconnect from the DB for
dialplan/dispatcher/carrierroute lookup?</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Thanks.</span></p>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jun
2, 2021 at 3:38 PM Daniel-Constantin Mierla
<<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>did you upgrade the Kamailio only or
also mysql server/client library? Is it
Oracle's MySQL or MariaDB?</p>
<p>A while ago, there were a few reports
of problems with mysql-client (Oracle)
library on Ubuntu 16.04 and 18.04, which
typically happened after upgrades of the
library, as I understood, the upgrade
brought in a dependency on libssl 1.1,
while libssl1.0 was used by most of the
other packages, resulting in mixed
deployments. Upgrading to 20.04 seemed
to solve it, as it came with libssl1.1
only.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 02.06.21 20:51, Andrew Chen wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>Today I upgraded Kamailio to
5.4.5 and a random crash was just
reported in our crash monitoring
script. Below is the output of the
crash:</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#3<span> </span>0x00007fa149ebe58d
in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#4<span> </span>0x00007fa14989e277
in ERR_clear_error () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#5<span> </span>0x00007fa149933d81
in X509_STORE_set_default_paths
() from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#6<span> </span>0x00007fa10310d374
in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#7<span> </span>0x00007fa10310d9f3
in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#8<span> </span>0x00007fa1030e0427
in mysql_real_connect () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#9<span> </span>0x00007fa10368428c
in db_mysql_new_connection ()
from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#10
0x00007fa1501f96dc in
db_do_init2 () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#11
0x00007fa1501f83f9 in db_do_init
() from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#12
0x00007fa103689710 in
db_mysql_init () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#13
0x00007fa15062d35f in
carrierroute_db_open () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#14
0x00007fa1506719c6 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
</div>
<div>
<div>Is this crash due to libcrypto
or mysql client?</div>
<div>Currently I have the following
mysql client installed on the
kamailio instance:</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span>lib</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">client20:amd64 <span>
</span>5.7.34-0ubuntu0.18.04.1
<span> </span>amd64<span>
</span>MySQL database
client library</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">-client <span>
</span>5.7.34-0ubuntu0.18.04.1
<span> </span>all<span>
</span>MySQL
database client (metapackage
depending on the latest
version)</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">-client-5.7 <span>
</span>5.7.34-0ubuntu0.18.04.1
<span> </span>amd64<span>
</span>MySQL database
client binaries</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">-client-core-5.7<span>
</span>5.7.34-0ubuntu0.18.04.1
<span> </span>amd64<span>
</span>MySQL database
core client binaries</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">-common <span>
</span>5.8+1.0.4
<span>
</span>all<span>
</span>MySQL database common
files, e.g. /etc/</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">/my.cnf</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(170,171,37)"><span style="font-variant-ligatures:no-common-ligatures">root@ashintgtpsg51:/var/lib/cores
# </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">apt-cache
madison mysql-client</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">mysql-client |
5.7.34-0ubuntu0.18.04.1 | <a href="http://us-east-1.ec2.archive.ubuntu.com/ubuntu" target="_blank">http://us-east-1.ec2.archive.ubuntu.com/ubuntu</a>
bionic-updates/main amd64
Packages</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">mysql-client |
5.7.34-0ubuntu0.18.04.1 | <a href="http://security.ubuntu.com/ubuntu" target="_blank">http://security.ubuntu.com/ubuntu</a>
bionic-security/main amd64
Packages</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">mysql-client |
5.7.21-1ubuntu1 | <a href="http://us-east-1.ec2.archive.ubuntu.com/ubuntu" target="_blank">http://us-east-1.ec2.archive.ubuntu.com/ubuntu</a>
bionic/main amd64 Packages</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(170,171,37)"><span style="font-variant-ligatures:no-common-ligatures">root@ashintgtpsg51:/var/lib/cores
#<span> </span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(170,171,37)"><br>
</p>
</div>
<div>Any suggestions would be
greatly appreciated.</div>
<div>Thanks in advance.</div>
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">Andy
Chen</div>
<div dir="ltr" style="font-size:12.8px">
<div>Sr. Telephony Lead
Engineer</div>
<div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div>
<div><br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality
Notice: The information contained in
this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
may be confidential. If you are not an
intended recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are
hereby notified that any
dissemination, distribution or copying
of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail
is strictly prohibited. If you have
received this e-mail in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please
notify the sender and permanently
delete the e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
immediately. You should not retain,
copy or use this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any
attachment for any purpose, nor
disclose all or any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents
to any other person. Thank you.*</span>
<br>
<fieldset></fieldset>
<pre>__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">Andy
Chen</div>
<div dir="ltr" style="font-size:12.8px">
<div>Sr. Telephony Lead Engineer</div>
<div>415 516 5535 (M)</div>
<div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality
Notice: The information contained in this e-mail
and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
may be confidential. If you are not an intended
recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are
hereby notified that any dissemination,
distribution or copying of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail
is strictly prohibited. If you have received
this e-mail in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please
notify the sender and permanently delete the
e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
immediately. You should not retain, copy or use
this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any
attachment for any purpose, nor disclose all or
any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents
to any other person. Thank you.*</span> </blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">Andy Chen</div>
<div dir="ltr" style="font-size:12.8px">
<div>Sr. Telephony Lead Engineer</div>
<div>415 516 5535 (M)</div>
<div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality
Notice: The information contained in this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
may be confidential. If you are not an intended
recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are
hereby notified that any dissemination, distribution or
copying of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail
is strictly prohibited. If you have received this e-mail
in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please
notify the sender and permanently delete the e-mail and
any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
immediately. You should not retain, copy or use this
e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any
attachment for any purpose, nor disclose all or any part
of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents
to any other person. Thank you.*</span> </blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">Andy Chen</div>
<div dir="ltr" style="font-size:12.8px">
<div>Sr. Telephony Lead Engineer</div>
<div>415 516 5535 (M)</div>
<div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality
Notice: The information contained in this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
may be confidential. If you are not an intended recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are
hereby notified that any dissemination, distribution or copying
of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail
is strictly prohibited. If you have received this e-mail in
error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please
notify the sender and permanently delete the e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
immediately. You should not retain, copy or use this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any
attachment for any purpose, nor disclose all or any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents
to any other person. Thank you.*</span>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr" style="font-size:12.8px">Andy Chen</div><div dir="ltr" style="font-size:12.8px"><div>Sr. Telephony Lead Engineer</div><div>415 516 5535 (M)</div><div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div><div><br></div></div></div></div></div></div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality Notice: The information contained in this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments may be confidential. If you are not an intended recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are hereby notified that any dissemination, distribution or copying of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail is strictly prohibited. If you have received this e-mail in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please notify the sender and permanently delete the e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments immediately. You should not retain, copy or use this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any attachment for any purpose, nor disclose all or any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents to any other person. Thank you.*</span>