TLS module crashes with FIPS OpenSSL
Hello, I encountered a problem stopping Kamailio with FIPS OpenSSL: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64 (gdb) bt #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 #1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3 #2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #4 0x00007ff72954702b in ?? () #5 0x0000000100061c08 in ?? () #6 0x00007ff7190566c8 in ?? () #7 0x00007ffccf196a20 in ?? () #8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134 #9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75 #10 0x000000000041f278 in cleanup (show_status=1) at main.c:594 #11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721 #12 0x0000000000421717 in handle_sigs () at main.c:752 #13 0x0000000000430c88 in main_loop () at main.c:1988 #14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212 (gdb) Environment: Oracle Linux Server 9.3 Kamailio 5.7.3 yum list --installed | grep ssl openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream What can I do for further investigation? Thanks
Hello, are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version. If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode. Cheers, Henning From: Marat Gareev via sr-users <sr-users@lists.kamailio.org> Sent: Montag, 13. Mai 2024 09:19 To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> Cc: Marat Gareev <maratkin94@gmail.com> Subject: [SR-Users] TLS module crashes with FIPS OpenSSL Hello, I encountered a problem stopping Kamailio with FIPS OpenSSL: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64 (gdb) bt #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 #1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3 #2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #4 0x00007ff72954702b in ?? () #5 0x0000000100061c08 in ?? () #6 0x00007ff7190566c8 in ?? () #7 0x00007ffccf196a20 in ?? () #8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134 #9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75 #10 0x000000000041f278 in cleanup (show_status=1) at main.c:594 #11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721 #12 0x0000000000421717 in handle_sigs () at main.c:752 #13 0x0000000000430c88 in main_loop () at main.c:1988 #14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212 (gdb) Environment: Oracle Linux Server 9.3 Kamailio 5.7.3 yum list --installed | grep ssl openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream What can I do for further investigation? Thanks
Hello Henning, yes, I use this major version $ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022) Thanks, I'll try updating Kamailio and report the results. пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> *Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
Hello again, I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 #1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6 #2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6 #3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6 #4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126 #5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526 #6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:1364 #7 0x00007f26baa12ea9 in ?? () #8 0x00007f26aa6383c0 in ?? () #9 0x000001b3ba70914b in ?? () #10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3 #11 0x00007f26aa6383c0 in ?? () #12 0x00007f26aa6383c0 in ?? () #13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3 #14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #16 0x00007f26baa1a21e in ?? () #17 0x00000001000623b0 in ?? () #18 0x00007f26aa5276c8 in ?? () #19 0x00007ffd66587330 in ?? () #20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134 #21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #22 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #24 0x0000000000421717 in handle_sigs () at main.c:753 #25 0x0000000000430c88 in main_loop () at main.c:1989 #26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213 (gdb) And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x. пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:
Hello Henning,
yes, I use this major version
$ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Thanks, I'll try updating Kamailio and report the results.
пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> *Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
Hello, this looks like a crash during the server shutdown (probably caused from another earlier issue). Can you look if you’ve got another coredump that maybe shows the actual reason? Cheers, Henning From: Marat Gareev <maratkin94@gmail.com> Sent: Dienstag, 14. Mai 2024 13:47 To: Henning Westerholt <hw@gilawa.com> Cc: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> Subject: Re: [SR-Users] TLS module crashes with FIPS OpenSSL Hello again, I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 #1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6 #2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6 #3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6 #4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126 #5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526 #6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:1364 #7 0x00007f26baa12ea9 in ?? () #8 0x00007f26aa6383c0 in ?? () #9 0x000001b3ba70914b in ?? () #10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3 #11 0x00007f26aa6383c0 in ?? () #12 0x00007f26aa6383c0 in ?? () #13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3 #14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #16 0x00007f26baa1a21e in ?? () #17 0x00000001000623b0 in ?? () #18 0x00007f26aa5276c8 in ?? () #19 0x00007ffd66587330 in ?? () #20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134 #21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #22 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #24 0x0000000000421717 in handle_sigs () at main.c:753 #25 0x0000000000430c88 in main_loop () at main.c:1989 #26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213 (gdb) And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x. пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com<mailto:maratkin94@gmail.com>>: Hello Henning, yes, I use this major version $ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022) Thanks, I'll try updating Kamailio and report the results. пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com<mailto:hw@gilawa.com>>: Hello, are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version. If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode. Cheers, Henning From: Marat Gareev via sr-users <sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>> Sent: Montag, 13. Mai 2024 09:19 To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>> Cc: Marat Gareev <maratkin94@gmail.com<mailto:maratkin94@gmail.com>> Subject: [SR-Users] TLS module crashes with FIPS OpenSSL Hello, I encountered a problem stopping Kamailio with FIPS OpenSSL: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64 (gdb) bt #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 #1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3 #2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #4 0x00007ff72954702b in ?? () #5 0x0000000100061c08 in ?? () #6 0x00007ff7190566c8 in ?? () #7 0x00007ffccf196a20 in ?? () #8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134 #9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75 #10 0x000000000041f278 in cleanup (show_status=1) at main.c:594 #11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721 #12 0x0000000000421717 in handle_sigs () at main.c:752 #13 0x0000000000430c88 in main_loop () at main.c:1988 #14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212 (gdb) Environment: Oracle Linux Server 9.3 Kamailio 5.7.3 yum list --installed | grep ssl openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream What can I do for further investigation? Thanks
Can you try with kamailio ... --atexit=no .... On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, < sr-users@lists.kamailio.org> wrote:
Hello again,
I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 #1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6 #2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6 #3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6 #4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126 #5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526 #6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:1364 #7 0x00007f26baa12ea9 in ?? () #8 0x00007f26aa6383c0 in ?? () #9 0x000001b3ba70914b in ?? () #10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3 #11 0x00007f26aa6383c0 in ?? () #12 0x00007f26aa6383c0 in ?? () #13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3 #14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #16 0x00007f26baa1a21e in ?? () #17 0x00000001000623b0 in ?? () #18 0x00007f26aa5276c8 in ?? () #19 0x00007ffd66587330 in ?? () #20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134 #21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #22 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #24 0x0000000000421717 in handle_sigs () at main.c:753 #25 0x0000000000430c88 in main_loop () at main.c:1989 #26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213 (gdb)
And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x.
пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:
Hello Henning,
yes, I use this major version
$ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Thanks, I'll try updating Kamailio and report the results.
пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> *Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Henning, I can't find anything else. But I caught one more segfault in the same scenario (stopping service)... Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3 #1 0x00007f3614e69daa in ossl_provider_free.part () from /lib64/libcrypto.so.3 #2 0x00007f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3 #3 0x00007f3614e68878 in prov_conf_ossl_ctx_free () from /lib64/libcrypto.so.3 #4 0x00007f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3 #5 0x00007f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3 #6 0x00007f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #7 0x00007f36151b921e in ?? () #8 0x000000010000a298 in ?? () #9 0x00007f3604cc66c8 in ?? () #10 0x00007ffead90c920 in ?? () #11 0x000000000071e0a0 in futex_release (lock=0x7f3615b7c930 <syslog>) at core/mem/../mem/../futexlock.h:134 #12 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #13 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #14 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #15 0x0000000000421717 in handle_sigs () at main.c:753 #16 0x0000000000430c88 in main_loop () at main.c:1989 #17 0x0000000000439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213 + unexpected message in log INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in libcrypto.so.3.0.7[7f5a8339d000+25c000] Richard, I start service with the following parameters /usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid -u ser -g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users < sr-users@lists.kamailio.org>:
Can you try with
kamailio ... --atexit=no ....
On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, < sr-users@lists.kamailio.org> wrote:
Hello again,
I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 #1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6 #2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6 #3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6 #4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126 #5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526 #6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:1364 #7 0x00007f26baa12ea9 in ?? () #8 0x00007f26aa6383c0 in ?? () #9 0x000001b3ba70914b in ?? () #10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3 #11 0x00007f26aa6383c0 in ?? () #12 0x00007f26aa6383c0 in ?? () #13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3 #14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #16 0x00007f26baa1a21e in ?? () #17 0x00000001000623b0 in ?? () #18 0x00007f26aa5276c8 in ?? () #19 0x00007ffd66587330 in ?? () #20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134 #21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #22 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #24 0x0000000000421717 in handle_sigs () at main.c:753 #25 0x0000000000430c88 in main_loop () at main.c:1989 #26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213 (gdb)
And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x.
пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:
Hello Henning,
yes, I use this major version
$ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Thanks, I'll try updating Kamailio and report the results.
пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> *Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
The backtrace is from shutdown cleanup as pointed before, so it is not the one that caused the crash. To get more than one core file, so it is not going to have core file overwritten, you have to enable one core file per pid/process (or set core file name pattern), some hints at: - https://www.kamailio.org/wikidocs/tutorials/troubleshooting/coredumpfile/ Searching on the web should reveal other tutorials about it. Then you should get more than one core file on a crash and you should grab the backtrace from all of them. You should also install the debugging symbols for libssl and libcrypto, there could be useful details shown in the backtraces. Cheers, Daniel On 14.05.24 19:38, Marat Gareev via sr-users wrote:
Henning, I can't find anything else. But I caught one more segfault in the same scenario (stopping service)...
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3 #1 0x00007f3614e69daa in ossl_provider_free.part () from /lib64/libcrypto.so.3 #2 0x00007f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3 #3 0x00007f3614e68878 in prov_conf_ossl_ctx_free () from /lib64/libcrypto.so.3 #4 0x00007f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3 #5 0x00007f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3 #6 0x00007f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #7 0x00007f36151b921e in ?? () #8 0x000000010000a298 in ?? () #9 0x00007f3604cc66c8 in ?? () #10 0x00007ffead90c920 in ?? () #11 0x000000000071e0a0 in futex_release (lock=0x7f3615b7c930 <syslog>) at core/mem/../mem/../futexlock.h:134 #12 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #13 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #14 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #15 0x0000000000421717 in handle_sigs () at main.c:753 #16 0x0000000000430c88 in main_loop () at main.c:1989 #17 0x0000000000439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213 + unexpected message in log INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in libcrypto.so.3.0.7[7f5a8339d000+25c000]
Richard, I start service with the following parameters /usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid -u ser -g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local
вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users <sr-users@lists.kamailio.org>:
Can you try with
kamailio ... --atexit=no ....
On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, <sr-users@lists.kamailio.org> wrote:
Hello again,
I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 #1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6 #2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6 #3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6 #4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126 #5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526 #6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:1364 #7 0x00007f26baa12ea9 in ?? () #8 0x00007f26aa6383c0 in ?? () #9 0x000001b3ba70914b in ?? () #10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3 #11 0x00007f26aa6383c0 in ?? () #12 0x00007f26aa6383c0 in ?? () #13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3 #14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #16 0x00007f26baa1a21e in ?? () #17 0x00000001000623b0 in ?? () #18 0x00007f26aa5276c8 in ?? () #19 0x00007ffd66587330 in ?? () #20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134 #21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #22 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #24 0x0000000000421717 in handle_sigs () at main.c:753 #25 0x0000000000430c88 in main_loop () at main.c:1989 #26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213 (gdb)
And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x.
пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:
Hello Henning,
yes, I use this major version
$ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Thanks, I'll try updating Kamailio and report the results.
пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:*Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> *Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
-- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com
Thanks for your reply, Daniel. I'm not sure I understood you and Henning correctly. At the beginning I indicated that I was observing a problem during a stop
I encountered a problem stopping Kamailio with FIPS OpenSSL
I stop service using systemd, so it is expected to see a shutdown_children call in the backtrace, as I understand it. Also I have pid-specific core pattern (kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e) and unlimited core file size. вт, 14 мая 2024 г. в 22:21, Daniel-Constantin Mierla <miconda@gmail.com>:
The backtrace is from shutdown cleanup as pointed before, so it is not the one that caused the crash.
To get more than one core file, so it is not going to have core file overwritten, you have to enable one core file per pid/process (or set core file name pattern), some hints at:
- https://www.kamailio.org/wikidocs/tutorials/troubleshooting/coredumpfile/
Searching on the web should reveal other tutorials about it.
Then you should get more than one core file on a crash and you should grab the backtrace from all of them.
You should also install the debugging symbols for libssl and libcrypto, there could be useful details shown in the backtraces.
Cheers, Daniel
On 14.05.24 19:38, Marat Gareev via sr-users wrote:
Henning, I can't find anything else. But I caught one more segfault in the same scenario (stopping service)...
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3 #1 0x00007f3614e69daa in ossl_provider_free.part () from /lib64/libcrypto.so.3 #2 0x00007f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3 #3 0x00007f3614e68878 in prov_conf_ossl_ctx_free () from /lib64/libcrypto.so.3 #4 0x00007f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3 #5 0x00007f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3 #6 0x00007f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #7 0x00007f36151b921e in ?? () #8 0x000000010000a298 in ?? () #9 0x00007f3604cc66c8 in ?? () #10 0x00007ffead90c920 in ?? () #11 0x000000000071e0a0 in futex_release (lock=0x7f3615b7c930 <syslog>) at core/mem/../mem/../futexlock.h:134 #12 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #13 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #14 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #15 0x0000000000421717 in handle_sigs () at main.c:753 #16 0x0000000000430c88 in main_loop () at main.c:1989 #17 0x0000000000439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213
+ unexpected message in log
INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in libcrypto.so.3.0.7[7f5a8339d000+25c000]
Richard, I start service with the following parameters /usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid -u ser -g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local
вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users < sr-users@lists.kamailio.org>:
Can you try with
kamailio ... --atexit=no ....
On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, < sr-users@lists.kamailio.org> wrote:
Hello again,
I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:
Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 #1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6 #2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6 #3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6 #4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126 #5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526 #6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:1364 #7 0x00007f26baa12ea9 in ?? () #8 0x00007f26aa6383c0 in ?? () #9 0x000001b3ba70914b in ?? () #10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3 #11 0x00007f26aa6383c0 in ?? () #12 0x00007f26aa6383c0 in ?? () #13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3 #14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #16 0x00007f26baa1a21e in ?? () #17 0x00000001000623b0 in ?? () #18 0x00007f26aa5276c8 in ?? () #19 0x00007ffd66587330 in ?? () #20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134 #21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #22 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #24 0x0000000000421717 in handle_sigs () at main.c:753 #25 0x0000000000430c88 in main_loop () at main.c:1989 #26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213 (gdb)
And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x.
пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:
Hello Henning,
yes, I use this major version
$ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Thanks, I'll try updating Kamailio and report the results.
пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org
*Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
-- Daniel-Constantin Mierla (@ asipto.com)twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy, Training and Development Services -- asipto.com
Hello, ok – so we probably had a slight misunderstanding here. I at least thought you are having a problem (crash) in the openssl FIPS part that causes Kamailio to stop. If its only causing a crash at stopping time, this is of course less critical, nevertheless it should be fixed. Maybe you can provide more details on how this problem can be reproduced, or create an github issue with a small cfg to reproduce it. Cheers, Henning From: Marat Gareev via sr-users <sr-users@lists.kamailio.org> Sent: Dienstag, 14. Mai 2024 23:47 To: miconda@gmail.com Cc: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>; Marat Gareev <maratkin94@gmail.com> Subject: [SR-Users] Re: TLS module crashes with FIPS OpenSSL Thanks for your reply, Daniel. I'm not sure I understood you and Henning correctly. At the beginning I indicated that I was observing a problem during a stop
I encountered a problem stopping Kamailio with FIPS OpenSSL
I stop service using systemd, so it is expected to see a shutdown_children call in the backtrace, as I understand it. Also I have pid-specific core pattern (kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e) and unlimited core file size. вт, 14 мая 2024 г. в 22:21, Daniel-Constantin Mierla <miconda@gmail.com<mailto:miconda@gmail.com>>: The backtrace is from shutdown cleanup as pointed before, so it is not the one that caused the crash. To get more than one core file, so it is not going to have core file overwritten, you have to enable one core file per pid/process (or set core file name pattern), some hints at: - https://www.kamailio.org/wikidocs/tutorials/troubleshooting/coredumpfile/ Searching on the web should reveal other tutorials about it. Then you should get more than one core file on a crash and you should grab the backtrace from all of them. You should also install the debugging symbols for libssl and libcrypto, there could be useful details shown in the backtraces. Cheers, Daniel On 14.05.24 19:38, Marat Gareev via sr-users wrote: Henning, I can't find anything else. But I caught one more segfault in the same scenario (stopping service)... Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3 #1 0x00007f3614e69daa in ossl_provider_free.part () from /lib64/libcrypto.so.3 #2 0x00007f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3 #3 0x00007f3614e68878 in prov_conf_ossl_ctx_free () from /lib64/libcrypto.so.3 #4 0x00007f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3 #5 0x00007f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3 #6 0x00007f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #7 0x00007f36151b921e in ?? () #8 0x000000010000a298 in ?? () #9 0x00007f3604cc66c8 in ?? () #10 0x00007ffead90c920 in ?? () #11 0x000000000071e0a0 in futex_release (lock=0x7f3615b7c930 <syslog>) at core/mem/../mem/../futexlock.h:134 #12 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #13 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #14 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #15 0x0000000000421717 in handle_sigs () at main.c:753 #16 0x0000000000430c88 in main_loop () at main.c:1989 #17 0x0000000000439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213 + unexpected message in log INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in libcrypto.so.3.0.7[7f5a8339d000+25c000] Richard, I start service with the following parameters /usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid -u ser -g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users <sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>: Can you try with kamailio ... --atexit=no .... On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, <sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>> wrote: Hello again, I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64 (gdb) bt #0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6 #1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6 #2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6 #3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6 #4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126 #5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526 #6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:1364 #7 0x00007f26baa12ea9 in ?? () #8 0x00007f26aa6383c0 in ?? () #9 0x000001b3ba70914b in ?? () #10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3 #11 0x00007f26aa6383c0 in ?? () #12 0x00007f26aa6383c0 in ?? () #13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3 #14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #16 0x00007f26baa1a21e in ?? () #17 0x00000001000623b0 in ?? () #18 0x00007f26aa5276c8 in ?? () #19 0x00007ffd66587330 in ?? () #20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134 #21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75 #22 0x000000000041f278 in cleanup (show_status=1) at main.c:595 #23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722 #24 0x0000000000421717 in handle_sigs () at main.c:753 #25 0x0000000000430c88 in main_loop () at main.c:1989 #26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213 (gdb) And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x. пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com<mailto:maratkin94@gmail.com>>: Hello Henning, yes, I use this major version $ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022) Thanks, I'll try updating Kamailio and report the results. пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com<mailto:hw@gilawa.com>>: Hello, are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version. If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode. Cheers, Henning From: Marat Gareev via sr-users <sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>> Sent: Montag, 13. Mai 2024 09:19 To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>> Cc: Marat Gareev <maratkin94@gmail.com<mailto:maratkin94@gmail.com>> Subject: [SR-Users] TLS module crashes with FIPS OpenSSL Hello, I encountered a problem stopping Kamailio with FIPS OpenSSL: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64 (gdb) bt #0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3 #1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3 #2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3 #3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3 #4 0x00007ff72954702b in ?? () #5 0x0000000100061c08 in ?? () #6 0x00007ff7190566c8 in ?? () #7 0x00007ffccf196a20 in ?? () #8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134 #9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75 #10 0x000000000041f278 in cleanup (show_status=1) at main.c:594 #11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721 #12 0x0000000000421717 in handle_sigs () at main.c:752 #13 0x0000000000430c88 in main_loop () at main.c:1988 #14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212 (gdb) Environment: Oracle Linux Server 9.3 Kamailio 5.7.3 yum list --installed | grep ssl openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream What can I do for further investigation? Thanks __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org<mailto:sr-users-leave@lists.kamailio.org> Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org<mailto:sr-users-leave@lists.kamailio.org> Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org<mailto:sr-users-leave@lists.kamailio.org> Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: -- Daniel-Constantin Mierla (@ asipto.com<http://asipto.com>) twitter.com/miconda<http://twitter.com/miconda> -- linkedin.com/in/miconda<http://linkedin.com/in/miconda> Kamailio Consultancy, Training and Development Services -- asipto.com<http://asipto.com>
Hello, I have thoughts about the root cause... Sometimes I don't get a segfault, but I see critical errors in the log: 0(987) NOTICE: <core> [main.c:750]: handle_sigs(): Thank you for flying kamailio!!! 7(998) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 3(991) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 4(994) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 8(999) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 1(988) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 9(1000) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac135800), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac164850), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac166878), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac135a80), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac166900), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac126bd0), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac126c70), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac40fd50), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac40fdf0), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac17e920), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac17e898), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:126]: qm_debug_check_frag(): BUG: qm: fragm. 0x7fdaac169690 (address 0x7fdaac1696c8) beginning overwritten (7fdaac165948)! Memory allocator was called from tls: tls_init.c:412. Fragment marked by :140574279598083. Exec from core/mem/q_malloc.c:526. As I understand it, the program does not crash due to luck - the error is ignored due to the memory safety control option. How can I investigate and interpret this error? Thanks. ср, 15 мая 2024 г. в 09:56, Henning Westerholt <hw@gilawa.com>:
Hello,
ok – so we probably had a slight misunderstanding here. I at least thought you are having a problem (crash) in the openssl FIPS part that causes Kamailio to stop.
If its only causing a crash at stopping time, this is of course less critical, nevertheless it should be fixed.
Maybe you can provide more details on how this problem can be reproduced, or create an github issue with a small cfg to reproduce it.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Dienstag, 14. Mai 2024 23:47 *To:* miconda@gmail.com *Cc:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>; Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] Re: TLS module crashes with FIPS OpenSSL
Thanks for your reply, Daniel.
I'm not sure I understood you and Henning correctly.
At the beginning I indicated that I was observing a problem during a stop
I encountered a problem stopping Kamailio with FIPS OpenSSL
I stop service using systemd, so it is expected to see a shutdown_children call in the backtrace, as I understand it.
Also I have pid-specific core pattern (kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e) and unlimited core file size.
вт, 14 мая 2024 г. в 22:21, Daniel-Constantin Mierla <miconda@gmail.com>:
The backtrace is from shutdown cleanup as pointed before, so it is not the one that caused the crash.
To get more than one core file, so it is not going to have core file overwritten, you have to enable one core file per pid/process (or set core file name pattern), some hints at:
- https://www.kamailio.org/wikidocs/tutorials/troubleshooting/coredumpfile/
Searching on the web should reveal other tutorials about it.
Then you should get more than one core file on a crash and you should grab the backtrace from all of them.
You should also install the debugging symbols for libssl and libcrypto, there could be useful details shown in the backtraces.
Cheers, Daniel
On 14.05.24 19:38, Marat Gareev via sr-users wrote:
Henning,
I can't find anything else. But I caught one more segfault in the same scenario (stopping service)...
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64
(gdb) bt
#0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3
#1 0x00007f3614e69daa in ossl_provider_free.part () from /lib64/libcrypto.so.3
#2 0x00007f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3
#3 0x00007f3614e68878 in prov_conf_ossl_ctx_free () from /lib64/libcrypto.so.3
#4 0x00007f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3
#5 0x00007f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3
#6 0x00007f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#7 0x00007f36151b921e in ?? ()
#8 0x000000010000a298 in ?? ()
#9 0x00007f3604cc66c8 in ?? ()
#10 0x00007ffead90c920 in ?? ()
#11 0x000000000071e0a0 in futex_release (lock=0x7f3615b7c930 <syslog>) at core/mem/../mem/../futexlock.h:134
#12 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75
#13 0x000000000041f278 in cleanup (show_status=1) at main.c:595
#14 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722
#15 0x0000000000421717 in handle_sigs () at main.c:753
#16 0x0000000000430c88 in main_loop () at main.c:1989
#17 0x0000000000439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213
+ unexpected message in log
INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in libcrypto.so.3.0.7[7f5a8339d000+25c000]
Richard,
I start service with the following parameters
/usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid -u ser -g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local
вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users < sr-users@lists.kamailio.org>:
Can you try with
kamailio ... --atexit=no ....
On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, < sr-users@lists.kamailio.org> wrote:
Hello again,
I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64
(gdb) bt
#0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6
#1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6
#2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6
#3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6
#4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126
#5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526
#6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls")
at core/mem/q_malloc.c:1364
#7 0x00007f26baa12ea9 in ?? ()
#8 0x00007f26aa6383c0 in ?? ()
#9 0x000001b3ba70914b in ?? ()
#10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3
#11 0x00007f26aa6383c0 in ?? ()
#12 0x00007f26aa6383c0 in ?? ()
#13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3
#14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#16 0x00007f26baa1a21e in ?? ()
#17 0x00000001000623b0 in ?? ()
#18 0x00007f26aa5276c8 in ?? ()
#19 0x00007ffd66587330 in ?? ()
#20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134
#21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75
#22 0x000000000041f278 in cleanup (show_status=1) at main.c:595
#23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722
#24 0x0000000000421717 in handle_sigs () at main.c:753
#25 0x0000000000430c88 in main_loop () at main.c:1989
#26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213
(gdb)
And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x.
пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:
Hello Henning,
yes, I use this major version
$ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Thanks, I'll try updating Kamailio and report the results.
пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> *Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-leave@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
Hi Marat, I don't know the answer to your problem, but would you be able to tell me how you got Kamailio to use the FIPS version of OpenSSL? Is the OpenSSL that comes with Oracle Linux Server FIPS compliant, or did you compile and install a custom version of OpenSSL and set the library path with LD_LIBRARY_PATH? Thank you in advance! On Thu, 16 May 2024 at 06:20, Marat Gareev via sr-users < sr-users@lists.kamailio.org> wrote:
Hello, I have thoughts about the root cause...
Sometimes I don't get a segfault, but I see critical errors in the log:
0(987) NOTICE: <core> [main.c:750]: handle_sigs(): Thank you for flying kamailio!!! 7(998) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 3(991) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 4(994) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 8(999) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 1(988) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 9(1000) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac135800), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac164850), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac166878), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac135a80), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac166900), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac126bd0), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac126c70), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac40fd50), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac40fdf0), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac17e920), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac17e898), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:126]: qm_debug_check_frag(): BUG: qm: fragm. 0x7fdaac169690 (address 0x7fdaac1696c8) beginning overwritten (7fdaac165948)! Memory allocator was called from tls: tls_init.c:412. Fragment marked by :140574279598083. Exec from core/mem/q_malloc.c:526.
As I understand it, the program does not crash due to luck - the error is ignored due to the memory safety control option.
How can I investigate and interpret this error?
Thanks.
ср, 15 мая 2024 г. в 09:56, Henning Westerholt <hw@gilawa.com>:
Hello,
ok – so we probably had a slight misunderstanding here. I at least thought you are having a problem (crash) in the openssl FIPS part that causes Kamailio to stop.
If its only causing a crash at stopping time, this is of course less critical, nevertheless it should be fixed.
Maybe you can provide more details on how this problem can be reproduced, or create an github issue with a small cfg to reproduce it.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Dienstag, 14. Mai 2024 23:47 *To:* miconda@gmail.com *Cc:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>; Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] Re: TLS module crashes with FIPS OpenSSL
Thanks for your reply, Daniel.
I'm not sure I understood you and Henning correctly.
At the beginning I indicated that I was observing a problem during a stop
I encountered a problem stopping Kamailio with FIPS OpenSSL
I stop service using systemd, so it is expected to see a shutdown_children call in the backtrace, as I understand it.
Also I have pid-specific core pattern (kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e) and unlimited core file size.
вт, 14 мая 2024 г. в 22:21, Daniel-Constantin Mierla <miconda@gmail.com>:
The backtrace is from shutdown cleanup as pointed before, so it is not the one that caused the crash.
To get more than one core file, so it is not going to have core file overwritten, you have to enable one core file per pid/process (or set core file name pattern), some hints at:
- https://www.kamailio.org/wikidocs/tutorials/troubleshooting/coredumpfile/
Searching on the web should reveal other tutorials about it.
Then you should get more than one core file on a crash and you should grab the backtrace from all of them.
You should also install the debugging symbols for libssl and libcrypto, there could be useful details shown in the backtraces.
Cheers, Daniel
On 14.05.24 19:38, Marat Gareev via sr-users wrote:
Henning,
I can't find anything else. But I caught one more segfault in the same scenario (stopping service)...
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64
(gdb) bt
#0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3
#1 0x00007f3614e69daa in ossl_provider_free.part () from /lib64/libcrypto.so.3
#2 0x00007f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3
#3 0x00007f3614e68878 in prov_conf_ossl_ctx_free () from /lib64/libcrypto.so.3
#4 0x00007f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3
#5 0x00007f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3
#6 0x00007f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#7 0x00007f36151b921e in ?? ()
#8 0x000000010000a298 in ?? ()
#9 0x00007f3604cc66c8 in ?? ()
#10 0x00007ffead90c920 in ?? ()
#11 0x000000000071e0a0 in futex_release (lock=0x7f3615b7c930 <syslog>) at core/mem/../mem/../futexlock.h:134
#12 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75
#13 0x000000000041f278 in cleanup (show_status=1) at main.c:595
#14 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722
#15 0x0000000000421717 in handle_sigs () at main.c:753
#16 0x0000000000430c88 in main_loop () at main.c:1989
#17 0x0000000000439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213
+ unexpected message in log
INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in libcrypto.so.3.0.7[7f5a8339d000+25c000]
Richard,
I start service with the following parameters
/usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid -u ser -g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local
вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users < sr-users@lists.kamailio.org>:
Can you try with
kamailio ... --atexit=no ....
On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, < sr-users@lists.kamailio.org> wrote:
Hello again,
I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64
(gdb) bt
#0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6
#1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6
#2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6
#3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6
#4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126
#5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526
#6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls")
at core/mem/q_malloc.c:1364
#7 0x00007f26baa12ea9 in ?? ()
#8 0x00007f26aa6383c0 in ?? ()
#9 0x000001b3ba70914b in ?? ()
#10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3
#11 0x00007f26aa6383c0 in ?? ()
#12 0x00007f26aa6383c0 in ?? ()
#13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3
#14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#16 0x00007f26baa1a21e in ?? ()
#17 0x00000001000623b0 in ?? ()
#18 0x00007f26aa5276c8 in ?? ()
#19 0x00007ffd66587330 in ?? ()
#20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134
#21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75
#22 0x000000000041f278 in cleanup (show_status=1) at main.c:595
#23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722
#24 0x0000000000421717 in handle_sigs () at main.c:753
#25 0x0000000000430c88 in main_loop () at main.c:1989
#26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213
(gdb)
And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x.
пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:
Hello Henning,
yes, I use this major version
$ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Thanks, I'll try updating Kamailio and report the results.
пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> *Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-leave@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782
Hi David, sorry for the late reply. I didn't compile OpenSSL manually, I use Oracle Linux Server with FIPS compliant packages. вт, 27 авг. 2024 г. в 04:05, David Cunningham <dcunningham@voisonics.com>:
Hi Marat,
I don't know the answer to your problem, but would you be able to tell me how you got Kamailio to use the FIPS version of OpenSSL? Is the OpenSSL that comes with Oracle Linux Server FIPS compliant, or did you compile and install a custom version of OpenSSL and set the library path with LD_LIBRARY_PATH?
Thank you in advance!
On Thu, 16 May 2024 at 06:20, Marat Gareev via sr-users < sr-users@lists.kamailio.org> wrote:
Hello, I have thoughts about the root cause...
Sometimes I don't get a segfault, but I see critical errors in the log:
0(987) NOTICE: <core> [main.c:750]: handle_sigs(): Thank you for flying kamailio!!! 7(998) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 3(991) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 4(994) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 8(999) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 1(988) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 9(1000) INFO: <core> [main.c:874]: sig_usr(): signal 15 received 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac135800), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac164850), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac166878), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac135a80), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac166900), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac126bd0), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac126c70), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac40fd50), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac40fdf0), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_malloc(367) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac17e920), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:535]: qm_free(): BUG: freeing already freed pointer (0x7fdaac17e898), called from tls: tls_init.c: ser_free(412), first free tls: tls_init.c: ser_free(412) - ignoring 0(987) CRITICAL: <core> [core/mem/q_malloc.c:126]: qm_debug_check_frag(): BUG: qm: fragm. 0x7fdaac169690 (address 0x7fdaac1696c8) beginning overwritten (7fdaac165948)! Memory allocator was called from tls: tls_init.c:412. Fragment marked by :140574279598083. Exec from core/mem/q_malloc.c:526.
As I understand it, the program does not crash due to luck - the error is ignored due to the memory safety control option.
How can I investigate and interpret this error?
Thanks.
ср, 15 мая 2024 г. в 09:56, Henning Westerholt <hw@gilawa.com>:
Hello,
ok – so we probably had a slight misunderstanding here. I at least thought you are having a problem (crash) in the openssl FIPS part that causes Kamailio to stop.
If its only causing a crash at stopping time, this is of course less critical, nevertheless it should be fixed.
Maybe you can provide more details on how this problem can be reproduced, or create an github issue with a small cfg to reproduce it.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Dienstag, 14. Mai 2024 23:47 *To:* miconda@gmail.com *Cc:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>; Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] Re: TLS module crashes with FIPS OpenSSL
Thanks for your reply, Daniel.
I'm not sure I understood you and Henning correctly.
At the beginning I indicated that I was observing a problem during a stop
I encountered a problem stopping Kamailio with FIPS OpenSSL
I stop service using systemd, so it is expected to see a shutdown_children call in the backtrace, as I understand it.
Also I have pid-specific core pattern (kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e) and unlimited core file size.
вт, 14 мая 2024 г. в 22:21, Daniel-Constantin Mierla <miconda@gmail.com
:
The backtrace is from shutdown cleanup as pointed before, so it is not the one that caused the crash.
To get more than one core file, so it is not going to have core file overwritten, you have to enable one core file per pid/process (or set core file name pattern), some hints at:
- https://www.kamailio.org/wikidocs/tutorials/troubleshooting/coredumpfile/
Searching on the web should reveal other tutorials about it.
Then you should get more than one core file on a crash and you should grab the backtrace from all of them.
You should also install the debugging symbols for libssl and libcrypto, there could be useful details shown in the backtraces.
Cheers, Daniel
On 14.05.24 19:38, Marat Gareev via sr-users wrote:
Henning,
I can't find anything else. But I caught one more segfault in the same scenario (stopping service)...
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64
(gdb) bt
#0 0x00007f3614f3c609 in init_thread_deregister.isra () from /lib64/libcrypto.so.3
#1 0x00007f3614e69daa in ossl_provider_free.part () from /lib64/libcrypto.so.3
#2 0x00007f3614ea81a0 in OPENSSL_sk_pop_free () from /lib64/libcrypto.so.3
#3 0x00007f3614e68878 in prov_conf_ossl_ctx_free () from /lib64/libcrypto.so.3
#4 0x00007f3614e5d405 in CRYPTO_free_ex_data () from /lib64/libcrypto.so.3
#5 0x00007f3614e5d59f in context_deinit.part () from /lib64/libcrypto.so.3
#6 0x00007f3614e600b2 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#7 0x00007f36151b921e in ?? ()
#8 0x000000010000a298 in ?? ()
#9 0x00007f3604cc66c8 in ?? ()
#10 0x00007ffead90c920 in ?? ()
#11 0x000000000071e0a0 in futex_release (lock=0x7f3615b7c930 <syslog>) at core/mem/../mem/../futexlock.h:134
#12 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75
#13 0x000000000041f278 in cleanup (show_status=1) at main.c:595
#14 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722
#15 0x0000000000421717 in handle_sigs () at main.c:753
#16 0x0000000000430c88 in main_loop () at main.c:1989
#17 0x0000000000439d13 in main (argc=14, argv=0x7ffead90d2f8) at main.c:3213
+ unexpected message in log
INFO kernel: [26983.427997] traps: kamailio[88753] general protection fault ip:7f5a83585609 sp:7ffc9b2b4400 error:0 in libcrypto.so.3.0.7[7f5a8339d000+25c000]
Richard,
I start service with the following parameters
/usr/local/sbin/kamailio --atexit=no -m 256 -P /var/run/ser/kamailio.pid -u ser -g ser -f /usr/local/etc/kamailio/kamailio.cfg -w /usr/local
вт, 14 мая 2024 г. в 18:57, Richard Chan via sr-users < sr-users@lists.kamailio.org>:
Can you try with
kamailio ... --atexit=no ....
On Tue, 14 May 2024, 13:13 Marat Gareev via sr-users, < sr-users@lists.kamailio.org> wrote:
Hello again,
I've updated Kamailio to 5.7.5, set tls_threads_mode=2 and got another segfault:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.5-4817.x86_64
(gdb) bt
#0 0x00007f26bb352efd in __strlen_avx2 () from /lib64/libc.so.6
#1 0x00007f26bb31a278 in __vfprintf_internal () from /lib64/libc.so.6
#2 0x00007f26bb3dd4ea in __vsyslog_internal () from /lib64/libc.so.6
#3 0x00007f26bb3dd9ca in syslog () from /lib64/libc.so.6
#4 0x000000000071e574 in qm_debug_check_frag (qm=0x7f26aa4ee000, f=0x7f26aa638388, file=0x7f26baa5b0b6 "tls: tls_init.c", line=399, efile=0x8abb39 "core/mem/q_malloc.c", eline=526) at core/mem/q_malloc.c:126
#5 0x00000000007227c3 in qm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls") at core/mem/q_malloc.c:526
#6 0x000000000072d2c9 in qm_shm_free (qmp=0x7f26aa4ee000, p=0x7f26aa6383c0, file=0x7f26baa5b0b6 "tls: tls_init.c", func=0x7f26baa5cdb8 "ser_free", line=399, mname=0x7f26baa5b0b2 "tls")
at core/mem/q_malloc.c:1364
#7 0x00007f26baa12ea9 in ?? ()
#8 0x00007f26aa6383c0 in ?? ()
#9 0x000001b3ba70914b in ?? ()
#10 0x00007f26ba853e4b in ?? () from /lib64/libcrypto.so.3
#11 0x00007f26aa6383c0 in ?? ()
#12 0x00007f26aa6383c0 in ?? ()
#13 0x00007f26ba61cfc5 in conf_modules_finish_int () from /lib64/libcrypto.so.3
#14 0x00007f26ba61d694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#15 0x00007f26ba6c0ff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#16 0x00007f26baa1a21e in ?? ()
#17 0x00000001000623b0 in ?? ()
#18 0x00007f26aa5276c8 in ?? ()
#19 0x00007ffd66587330 in ?? ()
#20 0x000000000071e0a0 in futex_release (lock=0x7f26bb3dd930 <syslog>) at core/mem/../mem/../futexlock.h:134
#21 0x00000000006e993e in destroy_tls () at core/tls_hooks.c:75
#22 0x000000000041f278 in cleanup (show_status=1) at main.c:595
#23 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:722
#24 0x0000000000421717 in handle_sigs () at main.c:753
#25 0x0000000000430c88 in main_loop () at main.c:1989
#26 0x0000000000439d13 in main (argc=14, argv=0x7ffd66587d08) at main.c:3213
(gdb)
And yes, the problem is definitely related to FIPS, because I did not see any errors with regular OpenSSL 3.x.
пн, 13 мая 2024 г. в 13:39, Marat Gareev <maratkin94@gmail.com>:
Hello Henning,
yes, I use this major version
$ openssl version OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Thanks, I'll try updating Kamailio and report the results.
пн, 13 мая 2024 г. в 13:19, Henning Westerholt <hw@gilawa.com>:
Hello,
are you on openssl 3.x by any chance? If yes, please upgrade to kamailio 5.7.5 or 5.8.1 and set tls_thread_mode=2 in the kamailio.cfg, as it fixes certain memory corruption issues on this openssl version.
If you are still getting crashes after the upgrade and setting, please let us know, it might be something related to the FIPS mode.
Cheers,
Henning
*From:* Marat Gareev via sr-users <sr-users@lists.kamailio.org> *Sent:* Montag, 13. Mai 2024 09:19 *To:* Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org> *Cc:* Marat Gareev <maratkin94@gmail.com> *Subject:* [SR-Users] TLS module crashes with FIPS OpenSSL
Hello,
I encountered a problem stopping Kamailio with FIPS OpenSSL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
Missing separate debuginfos, use: dnf debuginfo-install kamailio-5.7.3-4816.x86_64
(gdb) bt
#0 0x00007ff7292380ac in OPENSSL_sk_pop () from /lib64/libcrypto.so.3
#1 0x00007ff72914bf5b in conf_modules_finish_int () from /lib64/libcrypto.so.3
#2 0x00007ff72914c694 in CONF_modules_unload () from /lib64/libcrypto.so.3
#3 0x00007ff7291efff9 in OPENSSL_cleanup () from /lib64/libcrypto.so.3
#4 0x00007ff72954702b in ?? ()
#5 0x0000000100061c08 in ?? ()
#6 0x00007ff7190566c8 in ?? ()
#7 0x00007ffccf196a20 in ?? ()
#8 0x000000000071da8a in futex_release (lock=0x7ff729f08b50 <syslog>) at core/mem/../mem/../futexlock.h:134
#9 0x00000000006e9448 in destroy_tls () at core/tls_hooks.c:75
#10 0x000000000041f278 in cleanup (show_status=1) at main.c:594
#11 0x0000000000420af1 in shutdown_children (sig=15, show_status=1) at main.c:721
#12 0x0000000000421717 in handle_sigs () at main.c:752
#13 0x0000000000430c88 in main_loop () at main.c:1988
#14 0x0000000000439d13 in main (argc=14, argv=0x7ffccf1973f8) at main.c:3212
(gdb)
Environment:
Oracle Linux Server 9.3
Kamailio 5.7.3
yum list --installed | grep ssl
openssl.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-libs.x86_64 10:3.0.7-24.0.3.el9_fips @tools
openssl-pkcs11.x86_64 0.4.11-7.el9 @anaconda
xmlsec1-openssl.x86_64 1.2.29-9.el9 @AppStream
What can I do for further investigation?
Thanks
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
To unsubscribe send an email to sr-users-leave@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
-- David Cunningham, Voisonics Limited http://voisonics.com/ USA: +1 213 221 1092 New Zealand: +64 (0)28 2558 3782
participants (5)
-
Daniel-Constantin Mierla -
David Cunningham -
Henning Westerholt -
Marat Gareev -
Richard Chan