Hello,

this looks like a crash during the server shutdown (probably caused from another earlier issue). Can you look if you’ve got another coredump that maybe shows the actual reason?

Cheers,

Henning

From: Marat Gareev <maratkin94@gmail.com>
Sent: Dienstag, 14. Mai 2024 13:47
To: Henning Westerholt <hw@gilawa.com>
Cc: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>
Subject: Re: [SR-Users] TLS module crashes with FIPS OpenSSL

Hello again,

I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64
(gdb) bt
#0  0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6
#1  0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6
#2  0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6
#3  0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6
#4  0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126
#5  0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526
#6  0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls")
    at core/mem/q_malloc.c:1364
#7  0x00007f26baa12ea9 in ?? ()
#8  0x00007f26aa6383c0 in ?? ()
#9  0x000001b3ba70914b in ?? ()
#10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3
#11 0x00007f26aa6383c0 in ?? ()
#12 0x00007f26aa6383c0 in ?? ()
#13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3
#14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#16 0x00007f26baa1a21e in ?? ()
#17 0x00000001000623b0 in ?? ()
#18 0x00007f26aa5276c8 in ?? ()
#19 0x00007ffd66587330 in ?? ()
#20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134
#21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75
#22 0x000000000041f278 in cleanup (show_status=1) at main.c:595
#23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722
#24 0x0000000000421717 in handle_sigs () at main.c:753
#25 0x0000000000430c88 in main_loop () at main.c:1989
#26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213
(gdb)

And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x.

пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:

Hello Henning,

yes, I use this major version

$ openssl version
OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)

Thanks, I'll try updating Kamailio and report the results.

пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:

Hello,

are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.

If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.

Cheers,

Henning

From: Marat Gareev via sr-users <sr-users@lists.kamailio.org>
Sent: Montag, 13. Mai 2024 09:19
To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>
Cc: Marat Gareev <maratkin94@gmail.com>
Subject: [SR-Users] TLS module crashes with FIPS OpenSSL

Hello,

I encountered a problem stopping Kamailio with FIPS OpenSSL:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0  0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1  0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2  0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3  0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4  0x00007ff72954702b in ?? ()
#5  0x0000000100061c08 in ?? ()
#6  0x00007ff7190566c8 in ?? ()
#7  0x00007ffccf196a20 in ?? ()
#8  0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9  0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)

Environment:

Oracle Linux Server 9.3

Kamailio 5.7.3

yum list --installed | grep ssl
openssl.x86_64                        10:3.0.7-24.0.3.el9_fips                  @tools              
openssl-libs.x86_64                   10:3.0.7-24.0.3.el9_fips                  @tools              
openssl-pkcs11.x86_64                 0.4.11-7.el9                              @anaconda           
xmlsec1-openssl.x86_64                1.2.29-9.el9                              @AppStream

What can I do for further investigation?

Thanks