[SR-Users] Kamailio 5.2.3 segfaulting
Henning Westerholt
hw at skalatan.de
Wed Nov 27 21:59:35 CET 2019
Thanks for the notice, good that it was sorted out. Could you add the crashing mysql package version here, just to compare in the future if new reports show up..
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: Daniel Greenwald <dgreenwald at gmail.com>
Sent: Wednesday, November 27, 2019 9:42 PM
To: Henning Westerholt <hw at skalatan.de>
Cc: Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>; miconda at gmail.com
Subject: Re: [SR-Users] Kamailio 5.2.3 segfaulting
Thanks we ended up downgrading the mysql packages to 5.7.11-0ubuntu6 and seems to have resolved the issue.Seems the latest package caused craziness. Warning to others..
On Wed, Nov 27, 2019 at 2:07 PM Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>> wrote:
Hello,
This one package should contain all (packaged) debug info.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
Kamailio Merchandising – https://skalatan.de/merchandising
From: Daniel Greenwald <dgreenwald at gmail.com<mailto:dgreenwald at gmail.com>>
Sent: Wednesday, November 27, 2019 7:55 PM
To: Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>>
Cc: Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>>; miconda at gmail.com<mailto:miconda at gmail.com>
Subject: Re: [SR-Users] Kamailio 5.2.3 segfaulting
Yeah Yeah but what about the modules? Are there dbg packages for those, I don't see them?
On Wed, Nov 27, 2019 at 1:27 PM Henning Westerholt <hw at skalatan.de<mailto:hw at skalatan.de>> wrote:
With google? 😉 The package is called kamailio-dbg
https://www.google.com/search?source=hp&ei=rL3eXfK2Gqj3qwHg6rGACw&q=kamailio+debian+debug+packages&oq=kamailio+debian+debug+packages
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
Kamailio Merchandising – https://skalatan.de/merchandising
From: sr-users <sr-users-bounces at lists.kamailio.org<mailto:sr-users-bounces at lists.kamailio.org>> On Behalf Of Daniel Greenwald
Sent: Wednesday, November 27, 2019 6:36 PM
To: Daniel-Constantin Mierla <miconda at gmail.com<mailto:miconda at gmail.com>>
Cc: Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>>
Subject: Re: [SR-Users] Kamailio 5.2.3 segfaulting
How do I install the symbol packages?
On Wed, Nov 27, 2019 at 12:23 PM Daniel Greenwald <dgreenwald at gmail.com<mailto:dgreenwald at gmail.com>> wrote:
We noticed mysql did an upgrade to 5.7
On Wed, Nov 27, 2019 at 12:19 PM Daniel-Constantin Mierla <miconda at gmail.com<mailto:miconda at gmail.com>> wrote:
On 27.11.19 18:15, Daniel Greenwald wrote:
yes tls is loaded before db_mysql, is that bad?
Not, tls before any other modules potentially using libssl is the right order.
Yes apt updates were run so could have been updates to anything. OS version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we last run a couple of weeks ago.
Upon further digging I think the tls change was released a couple weeks ago so yes probably unrelated.
Install the packages with debug symbols for kamailio, libmysqlclient and libssl/libcrypto, then grab again the "bt full", it should show more details in the backtrace that may help to identify the issue.
Cheers,
Daniel
On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla <miconda at gmail.com<mailto:miconda at gmail.com>> wrote:
The issue seems to come from libmysqlclient library, not to kamailio handling sip traffic over tls.
Is tls module loaded before the db_mysql module?
Have you made other updates to the operating system? Or only the change you mentioned about server requiring and verifying certificates?
What version of ubuntu do you run?
I see that it uses libssl 1.0, so it is no relation with the change in libssl 1.1 internal api.
Cheers,
Daniel
On 27.11.19 17:38, Daniel Greenwald wrote:
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681
681 ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or directory.
(gdb) bt
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681
#1 0x00007fc864920b79 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#2 0x00007fc86492117a in lh_retrieve () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#3 0x00007fc864923651 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#4 0x00007fc8649240de in ERR_get_state () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#5 0x00007fc864924326 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#6 0x00007fc864c8a8e0 in SSL_get_error () from /lib/x86_64-linux-gnu/libssl.so.1.0.0
#7 0x00007fc843831cbc in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#8 0x00007fc84380d078 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#9 0x00007fc84380d317 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#10 0x00007fc84380e09c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#11 0x00007fc843801fe7 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#12 0x00007fc8438030d8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#13 0x00007fc843804e36 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#14 0x00007fc843db0f76 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
#15 0x00007fc8433bbb79 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
#16 0x00007fc8433bf95b in db_do_update () from /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
#17 0x00007fc843db5a86 in db_mysql_update () from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
#18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
#19 0x00007fc83f22aad3 in update_dialog_dbinfo () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
#20 0x00007fc83f21a065 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
#21 0x00007fc841017917 in run_rr_callbacks () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
#22 0x00007fc84100c0dd in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
#23 0x00007fc84100c7aa in loose_route () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
#24 0x00007fc84100ffb5 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
#25 0x0000000000480a22 in do_action ()
#26 0x000000000048db3f in run_actions ()
#27 0x000000000048e1fa in run_actions_safe ()
#28 0x0000000000450aed in rval_get_int ()
#29 0x00000000004551ed in rval_expr_eval_int ()
#30 0x00000000004804d0 in do_action ()
#31 0x000000000048db3f in run_actions ()
#32 0x000000000047d2a3 in do_action ()
#33 0x000000000048db3f in run_actions ()
#34 0x000000000048e306 in run_top_route ()
#35 0x00000000005b815c in receive_msg ()
#36 0x00000000004ab07f in udp_rcv_loop ()
#37 0x000000000042533c in main_loop ()
---Type <return> to continue, or q <return> to quit---
#38 0x000000000042cb3a in main ()
On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla <miconda at gmail.com<mailto:miconda at gmail.com>> wrote:
You got a core file, logs show that:
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated
Locate it and send the full backtrace with gdb from it.
Cheers,
Daniel
On 27.11.19 16:03, Daniel Greenwald wrote:
I'm sorry you are correct, the version we are running is 5.2.5. I've just run ulimit -c unlimited to hopefully get a core dump next time since I don't see one in /var/lib/systemd/coredump/. Not sure if there is anything I need to do to get a coredump.
The log just shows:
2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33
2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11
2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core> [main.c:764]: handle_sigs(): core was generated
2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core> [main.c:787]: handle_sigs(): terminating due to SIGCHLD
2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core> [main.c:842]: sig_usr(): signal 15 received
On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla <miconda at gmail.com<mailto:miconda at gmail.com>> wrote:
If you use the repo listed at http://deb.kamailio.org/ , then it should be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
Cheers,
Daniel
On 27.11.19 15:37, Daniel Greenwald wrote:
I'm using ubuntu packages, I believe 5.2.3 is latest?
On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla <miconda at gmail.com<mailto:miconda at gmail.com>> wrote:
Hello,
that log message is irrelevant for actual crash. Did you get core files? Or other log messages before the one you pasted?
You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch 5.2.
Cheers,
Daniel
On 27.11.19 15:22, Daniel Greenwald wrote:
I'm seeing this in the log and kamailio is crashing every hour or so:
CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 33/var/log/kamailio/kamailio.log
Any ideas? Should I emergency upgrade to 5.3.1?
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users at lists.kamailio.org<mailto:sr-users at lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com<http://www.asipto.com>
www.twitter.com/miconda<http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla -- www.asipto.com<http://www.asipto.com>
www.twitter.com/miconda<http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla -- www.asipto.com<http://www.asipto.com>
www.twitter.com/miconda<http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla -- www.asipto.com<http://www.asipto.com>
www.twitter.com/miconda<http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com<http://www.kamailioworld.com>
--
Daniel-Constantin Mierla -- www.asipto.com<http://www.asipto.com>
www.twitter.com/miconda<http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com<http://www.kamailioworld.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20191127/435109cd/attachment.html>
More information about the sr-users
mailing list