[SR-Users] Kamailio 5.2.3 segfaulting

Daniel Greenwald dgreenwald at gmail.com
Tue Dec 10 16:43:38 CET 2019


Here is the bug that hit us in production, just posted to MYSQL bug tracker
https://bugs.mysql.com/bug.php?id=97935
DO NOT UPGRADE TO MYSQL 5.7.28.


On Wed, Nov 27, 2019 at 5:20 PM Daniel Greenwald <dgreenwald at gmail.com>
wrote:

> So what I can tell you is libmysqlclient-dev was upgraded to
> 5.7.28-0ubuntu0.16.04.2 from 5.7.26. We were experiencing kamailio memory
> footprint increasing with load. Aside from the stacktraces posted above
> there was this in the logs
> <core> [core/mem/q_malloc.c:290]: qm_find_free():
> qm_find_free(0x7fc19180f000, 5720); Free fragment not found!
>
> On Wed, Nov 27, 2019 at 3:59 PM Henning Westerholt <hw at skalatan.de> wrote:
>
>> Thanks for the notice, good that it was sorted out. Could you add the
>> crashing mysql package version here, just to compare in the future if new
>> reports show up..
>>
>>
>>
>> Cheers,
>>
>>
>>
>> Henning
>>
>>
>>
>> --
>>
>> Henning Westerholt – https://skalatan.de/blog/
>>
>> Kamailio services – https://gilawa.com
>>
>>
>>
>> *From:* Daniel Greenwald <dgreenwald at gmail.com>
>> *Sent:* Wednesday, November 27, 2019 9:42 PM
>> *To:* Henning Westerholt <hw at skalatan.de>
>> *Cc:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>;
>> miconda at gmail.com
>> *Subject:* Re: [SR-Users] Kamailio 5.2.3 segfaulting
>>
>>
>>
>> Thanks we ended up downgrading the mysql packages to 5.7.11-0ubuntu6 and
>> seems to have resolved the issue.Seems the latest package caused craziness.
>> Warning to others..
>>
>>
>>
>> On Wed, Nov 27, 2019 at 2:07 PM Henning Westerholt <hw at skalatan.de>
>> wrote:
>>
>> Hello,
>>
>>
>>
>> This one package should contain all (packaged) debug info.
>>
>>
>>
>> Cheers,
>>
>>
>>
>> Henning
>>
>>
>>
>> --
>>
>> Henning Westerholt – https://skalatan.de/blog/
>>
>> Kamailio services – https://gilawa.com
>>
>> Kamailio Merchandising – https://skalatan.de/merchandising
>>
>>
>>
>> *From:* Daniel Greenwald <dgreenwald at gmail.com>
>> *Sent:* Wednesday, November 27, 2019 7:55 PM
>> *To:* Henning Westerholt <hw at skalatan.de>
>> *Cc:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>;
>> miconda at gmail.com
>> *Subject:* Re: [SR-Users] Kamailio 5.2.3 segfaulting
>>
>>
>>
>> Yeah Yeah but what about the modules? Are there dbg packages for those, I
>> don't see them?
>>
>>
>>
>> On Wed, Nov 27, 2019 at 1:27 PM Henning Westerholt <hw at skalatan.de>
>> wrote:
>>
>> With google? 😉 The package is called kamailio-dbg
>>
>>
>>
>>
>> https://www.google.com/search?source=hp&ei=rL3eXfK2Gqj3qwHg6rGACw&q=kamailio+debian+debug+packages&oq=kamailio+debian+debug+packages
>>
>>
>>
>>
>>
>> --
>>
>> Henning Westerholt – https://skalatan.de/blog/
>>
>> Kamailio services – https://gilawa.com
>>
>> Kamailio Merchandising – https://skalatan.de/merchandising
>>
>>
>>
>> *From:* sr-users <sr-users-bounces at lists.kamailio.org> *On Behalf Of *Daniel
>> Greenwald
>> *Sent:* Wednesday, November 27, 2019 6:36 PM
>> *To:* Daniel-Constantin Mierla <miconda at gmail.com>
>> *Cc:* Kamailio (SER) - Users Mailing List <sr-users at lists.kamailio.org>
>> *Subject:* Re: [SR-Users] Kamailio 5.2.3 segfaulting
>>
>>
>>
>> How do I install the symbol packages?
>>
>>
>>
>> On Wed, Nov 27, 2019 at 12:23 PM Daniel Greenwald <dgreenwald at gmail.com>
>> wrote:
>>
>> We noticed mysql did an upgrade to 5.7
>>
>>
>>
>> On Wed, Nov 27, 2019 at 12:19 PM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>>
>>
>> On 27.11.19 18:15, Daniel Greenwald wrote:
>>
>> yes tls is loaded before db_mysql, is that bad?
>>
>> Not, tls before any other modules potentially using libssl is the right
>> order.
>>
>>
>>
>> Yes apt updates were run so could have been updates to anything. OS
>> version is 16.04.6 LTS (GNU/Linux 4.4.0-1098-aws x86_64) Updates we last
>> run a couple of weeks ago.
>>
>>
>>
>> Upon further digging I think the tls change was released a couple weeks
>> ago so yes probably unrelated.
>>
>>
>>
>> Install the packages with debug symbols for kamailio, libmysqlclient and
>> libssl/libcrypto, then grab again the "bt full", it should show more
>> details in the backtrace that may help to identify the issue.
>>
>> Cheers,
>> Daniel
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Nov 27, 2019 at 12:03 PM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>> The issue seems to come from libmysqlclient library, not to kamailio
>> handling sip traffic over tls.
>>
>> Is tls module loaded before the db_mysql module?
>>
>> Have you made other updates to the operating system? Or only the change
>> you mentioned about server requiring and verifying certificates?
>>
>> What version of ubuntu do you run?
>>
>> I see that it uses libssl 1.0, so it is no relation with the change in
>> libssl 1.1 internal api.
>>
>> Cheers,
>> Daniel
>>
>> On 27.11.19 17:38, Daniel Greenwald wrote:
>>
>> Core was generated by `/usr/sbin/kamailio -P
>> /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0  __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681
>> 681     ../sysdeps/x86_64/multiarch/memcmp-sse4.S: No such file or
>> directory.
>> (gdb) bt
>> #0  __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:681
>> #1  0x00007fc864920b79 in ?? () from
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> #2  0x00007fc86492117a in lh_retrieve () from
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> #3  0x00007fc864923651 in ?? () from
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> #4  0x00007fc8649240de in ERR_get_state () from
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> #5  0x00007fc864924326 in ?? () from
>> /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
>> #6  0x00007fc864c8a8e0 in SSL_get_error () from
>> /lib/x86_64-linux-gnu/libssl.so.1.0.0
>> #7  0x00007fc843831cbc in ?? () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>> #8  0x00007fc84380d078 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>> #9  0x00007fc84380d317 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>> #10 0x00007fc84380e09c in ?? () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>> #11 0x00007fc843801fe7 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>> #12 0x00007fc8438030d8 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>> #13 0x00007fc843804e36 in mysql_real_query () from
>> /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
>> #14 0x00007fc843db0f76 in ?? () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>> #15 0x00007fc8433bbb79 in ?? () from
>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>> #16 0x00007fc8433bf95b in db_do_update () from
>> /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1
>> #17 0x00007fc843db5a86 in db_mysql_update () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so
>> #18 0x00007fc83f22a61b in update_dialog_dbinfo_unsafe () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
>> #19 0x00007fc83f22aad3 in update_dialog_dbinfo () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
>> #20 0x00007fc83f21a065 in dlg_onroute () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
>> #21 0x00007fc841017917 in run_rr_callbacks () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
>> #22 0x00007fc84100c0dd in ?? () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
>> #23 0x00007fc84100c7aa in loose_route () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
>> #24 0x00007fc84100ffb5 in ?? () from
>> /usr/lib/x86_64-linux-gnu/kamailio/modules/rr.so
>> #25 0x0000000000480a22 in do_action ()
>> #26 0x000000000048db3f in run_actions ()
>> #27 0x000000000048e1fa in run_actions_safe ()
>> #28 0x0000000000450aed in rval_get_int ()
>> #29 0x00000000004551ed in rval_expr_eval_int ()
>> #30 0x00000000004804d0 in do_action ()
>> #31 0x000000000048db3f in run_actions ()
>> #32 0x000000000047d2a3 in do_action ()
>> #33 0x000000000048db3f in run_actions ()
>> #34 0x000000000048e306 in run_top_route ()
>> #35 0x00000000005b815c in receive_msg ()
>> #36 0x00000000004ab07f in udp_rcv_loop ()
>> #37 0x000000000042533c in main_loop ()
>> ---Type <return> to continue, or q <return> to quit---
>> #38 0x000000000042cb3a in main ()
>>
>>
>>
>> On Wed, Nov 27, 2019 at 11:10 AM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>> You got a core file, logs show that:
>>
>> 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core>
>> [main.c:764]: handle_sigs(): core was generated
>>
>> Locate it and send the full backtrace with gdb from it.
>>
>> Cheers,
>> Daniel
>>
>> On 27.11.19 16:03, Daniel Greenwald wrote:
>>
>> I'm sorry you are correct, the version we are running is 5.2.5. I've just
>> run ulimit -c unlimited to hopefully get a core dump next time since I
>> don't see one in /var/lib/systemd/coredump/. Not sure if there is anything
>> I need to do to get a coredump.
>>
>> The log just shows:
>>
>>
>>
>> 2019-11-27T14:04:22.298010+00:00,ip-10-0-33-149: CRITICAL: <core>
>> [core/pass_fd.c:277]: receive_fd(): EOF on 33
>> 2019-11-27T14:04:22.299092+00:00,ip-10-0-33-149: ALERT: <core>
>> [main.c:761]: handle_sigs(): child process 12674 exited by a signal 11
>> 2019-11-27T14:04:22.299316+00:00,ip-10-0-33-149: ALERT: <core>
>> [main.c:764]: handle_sigs(): core was generated
>> 2019-11-27T14:04:22.299564+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:787]: handle_sigs(): terminating due to SIGCHLD
>> 2019-11-27T14:04:22.299892+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.300298+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.300569+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.300820+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.301070+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.301315+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.301564+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.301823+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.302056+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.302277+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.302525+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.302773+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>> 2019-11-27T14:04:22.303020+00:00,ip-10-0-33-149: INFO: <core>
>> [main.c:842]: sig_usr(): signal 15 received
>>
>>
>>
>> On Wed, Nov 27, 2019 at 9:49 AM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>> If you use the repo listed at http://deb.kamailio.org/ , then it should
>> be 5.2.5, and there is also option to install 5.2 branch debs built nightly.
>>
>> Cheers,
>> Daniel
>>
>> On 27.11.19 15:37, Daniel Greenwald wrote:
>>
>> I'm using ubuntu packages, I believe 5.2.3 is latest?
>>
>>
>>
>> On Wed, Nov 27, 2019 at 9:34 AM Daniel-Constantin Mierla <
>> miconda at gmail.com> wrote:
>>
>> Hello,
>>
>> that log message is irrelevant for actual crash. Did you get core files?
>> Or other log messages before the one you pasted?
>>
>> You should upgrade first to latest 5.2.x, either 5.2.5 or latest branch
>> 5.2.
>>
>> Cheers,
>> Daniel
>>
>> On 27.11.19 15:22, Daniel Greenwald wrote:
>>
>> I'm seeing this in the log and kamailio is crashing every hour or so:
>>
>> CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on
>> 33/var/log/kamailio/kamailio.log
>>
>>
>>
>> Any ideas? Should I emergency upgrade to 5.3.1?
>>
>>
>>
>> _______________________________________________
>>
>> Kamailio (SER) - Users Mailing List
>>
>> sr-users at lists.kamailio.org
>>
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>> --
>>
>> Daniel-Constantin Mierla -- www.asipto.com
>>
>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>
>> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
>>
>> --
>>
>> Daniel-Constantin Mierla -- www.asipto.com
>>
>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>
>> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
>>
>> --
>>
>> Daniel-Constantin Mierla -- www.asipto.com
>>
>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>
>> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
>>
>> --
>>
>> Daniel-Constantin Mierla -- www.asipto.com
>>
>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>
>> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
>>
>> --
>>
>> Daniel-Constantin Mierla -- www.asipto.com
>>
>> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>>
>> Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20191210/2e42128f/attachment.html>


More information about the sr-users mailing list