<div dir="ltr">How do I install the symbol packages?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 27, 2019 at 12:23 PM Daniel Greenwald <<a href="mailto:dgreenwald@gmail.com">dgreenwald@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">We noticed mysql did an upgrade to 5.7<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 27, 2019 at 12:19 PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>On 27.11.19 18:15, Daniel Greenwald
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>yes tls is loaded before db_mysql, is that bad? <br>
</div>
</div>
</blockquote>
Not, tls before any other modules potentially using libssl is the
right order.<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Yes apt updates were run so could have been updates to
anything. OS version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws
x86_64) Updates we last run a couple of weeks ago.</div>
<div><br>
</div>
<div>Upon further digging I think the tls change was released a
couple weeks ago so yes probably unrelated.<br>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Install the packages with debug symbols for kamailio,
libmysqlclient and libssl/libcrypto, then grab again the "bt
full", it should show more details in the backtrace that may help
to identify the issue.</p>
<p>Cheers,<br>
Daniel</p>
<p><br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Nov 27, 2019 at 12:03
PM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>The issue seems to come from libmysqlclient library, not
to kamailio handling sip traffic over tls.</p>
<p>Is tls module loaded before the db_mysql module?<br>
</p>
<p>Have you made other updates to the operating system? Or
only the change you mentioned about server requiring and
verifying certificates?</p>
<p>What version of ubuntu do you run?</p>
<p>I see that it uses libssl 1.0, so it is no relation with
the change in libssl 1.1 internal api.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 27.11.19 17:38, Daniel Greenwald wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Core was generated by `/usr/sbin/kamailio
-P /var/run/kamailio/kamailio.pid -f
/etc/kamailio/kamailio.'.<br>
Program terminated with signal SIGSEGV, Segmentation
fault.<br>
#0 __memcmp_sse4_1 () at
../sysdeps/x86_64/multiarch/memcmp-sse4.S:681<br>
681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No
such file or directory.<br>
(gdb) bt<br>
#0 __memcmp_sse4_1 () at
../sysdeps/x86_64/multiarch/memcmp-sse4.S:681<br>
#1 0x00007fc864920b79 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#2 0x00007fc86492117a in lh_retrieve () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#3 0x00007fc864923651 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#4 0x00007fc8649240de in ERR_get_state () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#5 0x00007fc864924326 in ?? () from
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0<br>
#6 0x00007fc864c8a8e0 in SSL_get_error () from
/lib/x86_64-linux-gnu/libssl.so.1.0.0<br>
#7 0x00007fc843831cbc in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#8 0x00007fc84380d078 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#9 0x00007fc84380d317 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#10 0x00007fc84380e09c in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#11 0x00007fc843801fe7 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#12 0x00007fc8438030d8 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#13 0x00007fc843804e36 in mysql_real_query () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20<br>
#14 0x00007fc843db0f76 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so<br>
#15 0x00007fc8433bbb79 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1<br>
#16 0x00007fc8433bf95b in db_do_update () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1<br>
#17 0x00007fc843db5a86 in db_mysql_update () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so<br>
#18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe ()
from
/usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so<br>
#19 0x00007fc83f22aad3 in update_dialog_dbinfo () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so<br>
#20 0x00007fc83f21a065 in dlg_onroute () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so<br>
#21 0x00007fc841017917 in run_rr_callbacks () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so<br>
#22 0x00007fc84100c0dd in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so<br>
#23 0x00007fc84100c7aa in loose_route () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so<br>
#24 0x00007fc84100ffb5 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so<br>
#25 0x0000000000480a22 in do_action ()<br>
#26 0x000000000048db3f in run_actions ()<br>
#27 0x000000000048e1fa in run_actions_safe ()<br>
#28 0x0000000000450aed in rval_get_int ()<br>
#29 0x00000000004551ed in rval_expr_eval_int ()<br>
#30 0x00000000004804d0 in do_action ()<br>
#31 0x000000000048db3f in run_actions ()<br>
#32 0x000000000047d2a3 in do_action ()<br>
#33 0x000000000048db3f in run_actions ()<br>
#34 0x000000000048e306 in run_top_route ()<br>
#35 0x00000000005b815c in receive_msg ()<br>
#36 0x00000000004ab07f in udp_rcv_loop ()<br>
#37 0x000000000042533c in main_loop ()<br>
---Type <return> to continue, or q <return>
to quit--- <br>
#38 0x000000000042cb3a in main ()<br>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Nov 27, 2019
at 11:10 AM Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>You got a core file, logs show that:</p>
<p>2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149:
ALERT: <core> [main.c:764]: handle_sigs():
core was generated</p>
<p>Locate it and send the full backtrace with gdb
from it.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 27.11.19 16:03, Daniel Greenwald wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>I'm sorry you are correct, the version we
are running is 5.2.5. I've just run ulimit -c
unlimited to hopefully get a core dump next
time since I don't see one in
/var/lib/systemd/coredump/. Not sure if there
is anything I need to do to get a coredump. <br>
</div>
<div>The log just shows:</div>
<div><br>
</div>
<div>2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149:
CRITICAL: <core> [core/pass_fd.c:277]:
receive_fd(): EOF on 33<br>
2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core>
[main.c:761]: handle_sigs(): child process
12674 exited by a signal 11<br>
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core>
[main.c:764]: handle_sigs(): core was
generated<br>
2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core>
[main.c:787]: handle_sigs(): terminating due
to SIGCHLD<br>
2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received<br>
2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core>
[main.c:842]: sig_usr(): signal 15 received</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Nov
27, 2019 at 9:49 AM Daniel-Constantin Mierla
<<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>If you use the repo listed at <a href="http://deb.kamailio.org/" target="_blank">http://deb.kamailio.org/</a>
, then it should be 5.2.5, and there is
also option to install 5.2 branch debs
built nightly.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 27.11.19 15:37, Daniel Greenwald
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">I'm using ubuntu packages,
I believe 5.2.3 is latest?<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Wed, Nov 27, 2019 at 9:34 AM
Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>that log message is irrelevant
for actual crash. Did you get core
files? Or other log messages
before the one you pasted?</p>
<p>You should upgrade first to
latest 5.2.x, either 5.2.5 or
latest branch 5.2.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 27.11.19 15:22, Daniel
Greenwald wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>I'm seeing this in the log
and kamailio is crashing every
hour or so:</div>
<div>CRITICAL: <core>
[core/pass_fd.c:277]:
receive_fd(): EOF on
33/var/log/kamailio/kamailio.log</div>
<div><br>
</div>
<div>Any ideas? Should I
emergency upgrade to 5.3.1?<br>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a></pre>
</div>
</blockquote></div>
</blockquote></div>