<div dir="ltr">Hi Daniel,<div><br></div><div>I just confirmed mysql is compiled using libssl.so.1.1.</div><div><br></div><div>As for the other kamailio packages, I only see two that are installed where both are depends on libssl1.1</div><div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);font-family:Menlo;font-size:11px">kamailio-tls-modules:amd64</span></div>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>Depends</b></span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>= 7.16.2), libssl1.1 (>= 1.1.0)</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font:11px Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">kamailio-websocket-modules:amd64</span></p><p class="gmail-p2" style="margin:0px;font:11px Menlo;color:rgb(0,0,0);min-height:13px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">
</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>Depends</b></span><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1 (>= 1.1.0), libunistring2 (>= 0.9.7)</span></p><div><br></div><div>So in my case here there shouldn't be any conflict with mysql client libssl version?</div><div><br></div><div>Thanks.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>I think the relevant here are the client library. I asked about
mysql vs mariadb because some distro had (meta-)packages named
mysql but installing mariadb. Debian at some point switched to
something like default-mysql-client...</p>
<p>Based on backtrace, the crashes would prevent kamailio to start,
as carrierroute_db_open() is used in mod_init/child_init
callbacks.</p>
<p>A variant you could try is the use of mysql-proxy to bridge from
a local unix socket to remote mysql, so kamailio connects to mysql
server via local unix socket and by that won't use libssl.</p>
<p>First you can check with ldd if mysql client libraries are
compiled against libssl1.1 and the libssl1.0 is also used for
other packages.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 02.06.21 21:56, Andrew Chen wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Daniel,
<div>So our DB is in rds and this is what I can gather:</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">+-------------------------+------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
Variable_name <span>
</span>| Value<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">+-------------------------+------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
innodb_version<span>
</span>| 5.6.34 <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
protocol_version<span> </span>|
10 <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
slave_type_conversions<span> </span>|<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
version <span>
</span>| 5.6.34-log <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
version_comment <span> </span>|
MySQL Community Server (GPL) |</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
version_compile_machine | x86_64 <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">|
version_compile_os<span> </span>|
Linux<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">+-------------------------+------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">The
mysql server/client library we received from my
apt-cache madison output earlier.</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">I
don't know if upgrading to 20.04 is an option.</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Given
this said, what are the symptoms of these crashes? I'm
guessing one of them is a brief disconnect from the DB
for dialplan/dispatcher/carrierroute lookup?</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Thanks.</span></p>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Jun 2, 2021 at 3:38
PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>did you upgrade the Kamailio only or also mysql
server/client library? Is it Oracle's MySQL or MariaDB?</p>
<p>A while ago, there were a few reports of problems with
mysql-client (Oracle) library on Ubuntu 16.04 and 18.04,
which typically happened after upgrades of the library,
as I understood, the upgrade brought in a dependency on
libssl 1.1, while libssl1.0 was used by most of the
other packages, resulting in mixed deployments.
Upgrading to 20.04 seemed to solve it, as it came with
libssl1.1 only.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 02.06.21 20:51, Andrew Chen wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>Today I upgraded Kamailio to 5.4.5 and a random
crash was just reported in our crash monitoring
script. Below is the output of the crash:</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#3<span> </span>0x00007fa149ebe58d
in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#4<span> </span>0x00007fa14989e277
in ERR_clear_error () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#5<span> </span>0x00007fa149933d81
in X509_STORE_set_default_paths () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#6<span> </span>0x00007fa10310d374
in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#7<span> </span>0x00007fa10310d9f3
in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#8<span> </span>0x00007fa1030e0427
in mysql_real_connect () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#9<span> </span>0x00007fa10368428c
in db_mysql_new_connection () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#10
0x00007fa1501f96dc in db_do_init2 () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#11
0x00007fa1501f83f9 in db_do_init () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#12
0x00007fa103689710 in db_mysql_init () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#13
0x00007fa15062d35f in carrierroute_db_open ()
from
/usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">#14
0x00007fa1506719c6 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
</div>
<div>
<div>Is this crash due to libcrypto or mysql client?</div>
<div>Currently I have the following mysql client
installed on the kamailio instance:</div>
<div><br>
</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span>lib</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">client20:amd64 <span>
</span>5.7.34-0ubuntu0.18.04.1
<span> </span>amd64<span> </span>MySQL
database client library</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">-client <span>
</span>5.7.34-0ubuntu0.18.04.1
<span> </span>all<span> </span>MySQL
database client (metapackage depending on the
latest version)</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">-client-5.7 <span>
</span>5.7.34-0ubuntu0.18.04.1
<span> </span>amd64<span> </span>MySQL
database client binaries</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">-client-core-5.7<span>
</span>5.7.34-0ubuntu0.18.04.1
<span> </span>amd64<span> </span>MySQL
database core client binaries</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">ii<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">-common <span>
</span>5.8+1.0.4 <span>
</span>all<span>
</span>MySQL database common files,
e.g. /etc/</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)"><b>mysql</b></span><span style="font-variant-ligatures:no-common-ligatures">/my.cnf</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(170,171,37)"><span style="font-variant-ligatures:no-common-ligatures">root@ashintgtpsg51:/var/lib/cores
# </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">apt-cache
madison mysql-client</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">mysql-client |
5.7.34-0ubuntu0.18.04.1 | <a href="http://us-east-1.ec2.archive.ubuntu.com/ubuntu" target="_blank">http://us-east-1.ec2.archive.ubuntu.com/ubuntu</a>
bionic-updates/main amd64 Packages</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">mysql-client |
5.7.34-0ubuntu0.18.04.1 | <a href="http://security.ubuntu.com/ubuntu" target="_blank">http://security.ubuntu.com/ubuntu</a>
bionic-security/main amd64 Packages</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">mysql-client |
5.7.21-1ubuntu1 | <a href="http://us-east-1.ec2.archive.ubuntu.com/ubuntu" target="_blank">http://us-east-1.ec2.archive.ubuntu.com/ubuntu</a>
bionic/main amd64 Packages</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(170,171,37)"><span style="font-variant-ligatures:no-common-ligatures">root@ashintgtpsg51:/var/lib/cores
#<span> </span></span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(170,171,37)"><br>
</p>
</div>
<div>Any suggestions would be greatly appreciated.</div>
<div>Thanks in advance.</div>
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">Andy
Chen</div>
<div dir="ltr" style="font-size:12.8px">
<div>Sr. Telephony Lead Engineer</div>
<div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div>
<div><br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality
Notice: The information contained in this e-mail and
any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
may be confidential. If you are not an intended
recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are
hereby notified that any dissemination, distribution
or copying of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail
is strictly prohibited. If you have received this
e-mail in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please
notify the sender and permanently delete the e-mail
and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
immediately. You should not retain, copy or use this
e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any
attachment for any purpose, nor disclose all or any
part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents
to any other person. Thank you.*</span> <br>
<fieldset></fieldset>
<pre>__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr" style="font-size:12.8px">Andy Chen</div>
<div dir="ltr" style="font-size:12.8px">
<div>Sr. Telephony Lead Engineer</div>
<div>415 516 5535 (M)</div>
<div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality
Notice: The information contained in this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
may be confidential. If you are not an intended recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are
hereby notified that any dissemination, distribution or copying
of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail
is strictly prohibited. If you have received this e-mail in
error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please
notify the sender and permanently delete the e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments
immediately. You should not retain, copy or use this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any
attachment for any purpose, nor disclose all or any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents
to any other person. Thank you.*</span>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
* <a href="https://www.asipto.com/sw/kamailio-advanced-training-online/" target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a></pre>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr" style="font-size:12.8px">Andy Chen</div><div dir="ltr" style="font-size:12.8px"><div>Sr. Telephony Lead Engineer</div><div>415 516 5535 (M)</div><div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div><div><br></div></div></div></div></div></div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality Notice: The information contained in this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments may be confidential. If you are not an intended recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are hereby notified that any dissemination, distribution or copying of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail is strictly prohibited. If you have received this e-mail in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please notify the sender and permanently delete the e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments immediately. You should not retain, copy or use this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any attachment for any purpose, nor disclose all or any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents to any other person. Thank you.*</span>