<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Vorformatiert Zchn";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLVorformatiertZchn
{mso-style-name:"HTML Vorformatiert Zchn";
mso-style-priority:99;
mso-style-link:"HTML Vorformatiert";
font-family:Consolas;
mso-fareast-language:DE;}
span.E-MailFormatvorlage21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Thanks for letting us know!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">-- <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning Westerholt –
</span><span style="mso-fareast-language:EN-US"><a href="https://skalatan.de/blog/"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a></span><span lang="EN-GB" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Kamailio services –
</span><span style="mso-fareast-language:EN-US"><a href="https://gilawa.com/"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a></span><span style="mso-fareast-language:EN-US">
<span lang="EN-GB"><o:p></o:p></span></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b> Daniel Greenwald <dgreenwald@gmail.com>
<br>
<b>Sent:</b> Tuesday, December 10, 2019 4:44 PM<br>
<b>To:</b> Henning Westerholt <hw@skalatan.de><br>
<b>Cc:</b> Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>; miconda@gmail.com<br>
<b>Subject:</b> Re: [SR-Users] Kamailio 5.2.3 segfaulting<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Here is the bug that hit us in production, just posted to MYSQL bug tracker
<a href="https://bugs.mysql.com/bug.php?id=97935" target="_blank">https://bugs.mysql.com/bug.php?id=97935</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">DO NOT UPGRADE TO MYSQL 5.7.28.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Wed, Nov 27, 2019 at 5:20 PM Daniel Greenwald <<a href="mailto:dgreenwald@gmail.com" target="_blank">dgreenwald@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">So what I can tell you is libmysqlclient-dev was upgraded to 5.7.28-0ubuntu0.16.04.2 from 5.7.26. We were experiencing kamailio memory footprint increasing with load. Aside from the stacktraces posted above there
was this in the logs<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:#1D1C1D;background:#F8F8F8"><core> [core/mem/q_malloc.c:290]: qm_find_free(): qm_find_free(0x7fc19180f000, 5720); Free fragment not found!</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On Wed, Nov 27, 2019 at 3:59 PM Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Thanks for the notice, good that it was sorted out. Could you add the crashing mysql package version here, just to compare in the future if new reports show up..</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Cheers,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Henning</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">-- </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Henning Westerholt – </span><a href="https://skalatan.de/blog/" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<span lang="EN-GB">Kamailio services – </span><a href="https://gilawa.com/" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:35.4pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<b>From:</b> Daniel Greenwald <<a href="mailto:dgreenwald@gmail.com" target="_blank">dgreenwald@gmail.com</a>>
<br>
<b>Sent:</b> Wednesday, November 27, 2019 9:42 PM<br>
<b>To:</b> Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>><br>
<b>Cc:</b> Kamailio (SER) - Users Mailing List <<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>>;
<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a><br>
<b>Subject:</b> Re: [SR-Users] Kamailio 5.2.3 segfaulting<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Thanks we ended up downgrading the mysql packages to 5.7.11-0ubuntu6 and seems to have resolved the issue.Seems the latest package caused craziness. Warning to others..
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
On Wed, Nov 27, 2019 at 2:07 PM Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
Hello,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB">This one package should contain all (packaged) debug info.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB">Cheers,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB">Henning</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB">-- </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB">Henning Westerholt – </span><a href="https://skalatan.de/blog/" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB">Kamailio services – </span><a href="https://gilawa.com/" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<span lang="EN-GB">Kamailio Merchandising – </span><a href="https://skalatan.de/merchandising" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/merchandising</span></a>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:70.8pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<b>From:</b> Daniel Greenwald <<a href="mailto:dgreenwald@gmail.com" target="_blank">dgreenwald@gmail.com</a>>
<br>
<b>Sent:</b> Wednesday, November 27, 2019 7:55 PM<br>
<b>To:</b> Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>><br>
<b>Cc:</b> Kamailio (SER) - Users Mailing List <<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>>;
<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a><br>
<b>Subject:</b> Re: [SR-Users] Kamailio 5.2.3 segfaulting<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
Yeah Yeah but what about the modules? Are there dbg packages for those, I don't see them?<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
On Wed, Nov 27, 2019 at 1:27 PM Henning Westerholt <<a href="mailto:hw@skalatan.de" target="_blank">hw@skalatan.de</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<span lang="EN-GB">With google? </span><span style="font-family:"Segoe UI Emoji",sans-serif">😉</span><span lang="EN-GB"> The package is called kamailio-dbg</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<a href="https://www.google.com/search?source=hp&ei=rL3eXfK2Gqj3qwHg6rGACw&q=kamailio+debian+debug+packages&oq=kamailio+debian+debug+packages" target="_blank"><span lang="EN-GB">https://www.google.com/search?source=hp&ei=rL3eXfK2Gqj3qwHg6rGACw&q=kamailio+debian+debug+packages&oq=kamailio+debian+debug+packages</span></a>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<span lang="EN-GB"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<span lang="EN-GB">-- </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<span lang="EN-GB">Henning Westerholt – </span><a href="https://skalatan.de/blog/" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/blog/</span></a><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<span lang="EN-GB">Kamailio services – </span><a href="https://gilawa.com/" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://gilawa.com</span></a>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<span lang="EN-GB">Kamailio Merchandising – </span><a href="https://skalatan.de/merchandising" target="_blank"><span lang="EN-GB" style="color:#0563C1">https://skalatan.de/merchandising</span></a>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:106.2pt">
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<b>From:</b> sr-users <<a href="mailto:sr-users-bounces@lists.kamailio.org" target="_blank">sr-users-bounces@lists.kamailio.org</a>>
<b>On Behalf Of </b>Daniel Greenwald<br>
<b>Sent:</b> Wednesday, November 27, 2019 6:36 PM<br>
<b>To:</b> Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>><br>
<b>Cc:</b> Kamailio (SER) - Users Mailing List <<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>><br>
<b>Subject:</b> Re: [SR-Users] Kamailio 5.2.3 segfaulting<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
How do I install the symbol packages?<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On Wed, Nov 27, 2019 at 12:23 PM Daniel Greenwald <<a href="mailto:dgreenwald@gmail.com" target="_blank">dgreenwald@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
We noticed mysql did an upgrade to 5.7<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On Wed, Nov 27, 2019 at 12:19 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<p style="margin-left:141.6pt"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On 27.11.19 18:15, Daniel Greenwald wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
yes tls is loaded before db_mysql, is that bad? <o:p></o:p></p>
</div>
</div>
</blockquote>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;margin-left:141.6pt">
Not, tls before any other modules potentially using libssl is the right order.<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
Yes apt updates were run so could have been updates to anything. OS version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we last run a couple of weeks ago.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
Upon further digging I think the tls change was released a couple weeks ago so yes probably unrelated.<o:p></o:p></p>
</div>
</div>
</blockquote>
<p style="margin-left:141.6pt"> <o:p></o:p></p>
<p style="margin-left:141.6pt">Install the packages with debug symbols for kamailio, libmysqlclient and libssl/libcrypto, then grab again the "bt full", it should show more details in the backtrace that may help to identify the issue.<o:p></o:p></p>
<p style="margin-left:141.6pt">Cheers,<br>
Daniel<o:p></o:p></p>
<p style="margin-left:141.6pt"> <o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<p style="margin-left:141.6pt">The issue seems to come from libmysqlclient library, not to kamailio handling sip traffic over tls.<o:p></o:p></p>
<p style="margin-left:141.6pt">Is tls module loaded before the db_mysql module?<o:p></o:p></p>
<p style="margin-left:141.6pt">Have you made other updates to the operating system? Or only the change you mentioned about server requiring and verifying certificates?<o:p></o:p></p>
<p style="margin-left:141.6pt">What version of ubuntu do you run?<o:p></o:p></p>
<p style="margin-left:141.6pt">I see that it uses libssl 1.0, so it is no relation with the change in libssl 1.1 internal api.<o:p></o:p></p>
<p style="margin-left:141.6pt">Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On 27.11.19 17:38, Daniel Greenwald wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt;margin-left:141.6pt">
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGSEGV, Segmentation fault.<br>
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681<br>
681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory.<br>
(gdb) bt<br>
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681<br>
#1 0x00007fc864920b79 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#2 0x00007fc86492117a in lh_retrieve () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#3 0x00007fc864923651 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#4 0x00007fc8649240de in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#5 0x00007fc864924326 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#6 0x00007fc864c8a8e0 in SSL_get_error () from /lib/x86_64-linux-gnu/libssl.so.1.0.0<br>
#7 0x00007fc843831cbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#8 0x00007fc84380d078 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#9 0x00007fc84380d317 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#10 0x00007fc84380e09c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#11 0x00007fc843801fe7 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#12 0x00007fc8438030d8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#13 0x00007fc843804e36 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#14 0x00007fc843db0f76 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so<br>
#15 0x00007fc8433bbb79 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1<br>
#16 0x00007fc8433bf95b in db_do_update () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1<br>
#17 0x00007fc843db5a86 in db_mysql_update () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so<br>
#18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so<br>
#19 0x00007fc83f22aad3 in update_dialog_dbinfo () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so<br>
#20 0x00007fc83f21a065 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so<br>
#21 0x00007fc841017917 in run_rr_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so<br>
#22 0x00007fc84100c0dd in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so<br>
#23 0x00007fc84100c7aa in loose_route () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so<br>
#24 0x00007fc84100ffb5 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so<br>
#25 0x0000000000480a22 in do_action ()<br>
#26 0x000000000048db3f in run_actions ()<br>
#27 0x000000000048e1fa in run_actions_safe ()<br>
#28 0x0000000000450aed in rval_get_int ()<br>
#29 0x00000000004551ed in rval_expr_eval_int ()<br>
#30 0x00000000004804d0 in do_action ()<br>
#31 0x000000000048db3f in run_actions ()<br>
#32 0x000000000047d2a3 in do_action ()<br>
#33 0x000000000048db3f in run_actions ()<br>
#34 0x000000000048e306 in run_top_route ()<br>
#35 0x00000000005b815c in receive_msg ()<br>
#36 0x00000000004ab07f in udp_rcv_loop ()<br>
#37 0x000000000042533c in main_loop ()<br>
---Type <return> to continue, or q <return> to quit--- <br>
#38 0x000000000042cb3a in main ()<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<p style="margin-left:141.6pt">You got a core file, logs show that:<o:p></o:p></p>
<p style="margin-left:141.6pt">2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated<o:p></o:p></p>
<p style="margin-left:141.6pt">Locate it and send the full backtrace with gdb from it.<o:p></o:p></p>
<p style="margin-left:141.6pt">Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On 27.11.19 16:03, Daniel Greenwald wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
The log just shows:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33<br>
2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11<br>
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated<br>
2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD<br>
2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<p style="margin-left:141.6pt">If you use the repo listed at <a href="http://deb.kamailio.org/" target="_blank">
http://deb.kamailio.org/</a> , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.<o:p></o:p></p>
<p style="margin-left:141.6pt">Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On 27.11.19 15:37, Daniel Greenwald wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
I'm using ubuntu packages, I believe 5.2.3 is latest?<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<p style="margin-left:141.6pt">Hello,<o:p></o:p></p>
<p style="margin-left:141.6pt">that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?<o:p></o:p></p>
<p style="margin-left:141.6pt">You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.<o:p></o:p></p>
<p style="margin-left:141.6pt">Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
On 27.11.19 15:22, Daniel Greenwald wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
I'm seeing this in the log and kamailio is crashing every hour or so:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
Any ideas? Should I emergency upgrade to 5.3.1?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:141.6pt">
<o:p></o:p></p>
<pre style="margin-left:141.6pt">_______________________________________________<o:p></o:p></pre>
<pre style="margin-left:141.6pt">Kamailio (SER) - Users Mailing List<o:p></o:p></pre>
<pre style="margin-left:141.6pt"><a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt"><a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></pre>
</blockquote>
<pre style="margin-left:141.6pt">-- <o:p></o:p></pre>
<pre style="margin-left:141.6pt">Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt"><a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt">Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a><o:p></o:p></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre style="margin-left:141.6pt">-- <o:p></o:p></pre>
<pre style="margin-left:141.6pt">Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt"><a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt">Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a><o:p></o:p></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre style="margin-left:141.6pt">-- <o:p></o:p></pre>
<pre style="margin-left:141.6pt">Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt"><a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt">Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a><o:p></o:p></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre style="margin-left:141.6pt">-- <o:p></o:p></pre>
<pre style="margin-left:141.6pt">Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt"><a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt">Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a><o:p></o:p></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre style="margin-left:141.6pt">-- <o:p></o:p></pre>
<pre style="margin-left:141.6pt">Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt"><a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a><o:p></o:p></pre>
<pre style="margin-left:141.6pt">Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a><o:p></o:p></pre>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>