5 Oct
2020
5 Oct
'20
7:11 a.m.
K master has recently crashed several times on Debian 10. Could this have
something to do with openssl libs?
-- Juha
Oct 5 08:32:21 lohi systemd[1]: Stopping LSB: Start/stop OpenSIPg SIP Proxy...
Oct 5 08:32:21 lohi sip-proxy[19922]: Stopping sip-proxy
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30820]: NOTICE: <core> [main.c:735]:
handle_sigs(): Thank you for flying sip-proxy!!!
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30860]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30859]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30858]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30857]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30853]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30856]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30850]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30854]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30834]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30847]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30852]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30849]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30851]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30841]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30830]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30848]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30846]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30836]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30827]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30844]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30845]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30833]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30842]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30839]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30829]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30822]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30840]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30838]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30826]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30837]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30824]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30835]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30823]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30832]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30831]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30828]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30843]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30825]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30821]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30855]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30820]: INFO: <core> [core/sctp_core.c:53]:
sctp_core_destroy(): SCTP API not initialized
Oct 5 08:32:21 lohi kernel: [176646.851504] sip-proxy[30820]: segfault at 7f84196c7730 ip
00007f8421adce26 sp 00007ffde899c0f0 error 4 in libcrypto.so.1.1[7f84219d8000+19e000]
Oct 5 08:32:21 lohi kernel: [176646.851513] Code: 1f 44 00 00 c7 05 9a d9 15 00 01 00 00
00 c7 05 8c d9 15 00 01 00 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 48 85 ff 74 5b 53
<8b> 0f 48 89 fb 85 c9 75 29 8b 53 04 85 d2 75 2e 8b 43 08 85 c0 75
(gdb) where
#0 0x00007f8421adce26 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#1 0x00007f8421add12f in OPENSSL_cleanup () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#2 0x00007f8423a87d8c in __run_exit_handlers (status=0, listp=0x7f8423c09718
<__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at
exit.c:108
#3 0x00007f8423a87eba in __GI_exit (status=<optimized out>) at exit.c:139
#4 0x000055d40be7bb61 in handle_sigs () at main.c:738
#5 0x000055d40be88966 in main_loop () at main.c:1817
#6 0x000055d40be91e32 in main (argc=17, argv=0x7ffde899c948) at main.c:2861
5 Oct
5 Oct
7:18 a.m.
The backtrace is for the main process during shut down, which can be a
side effect of the real issue. You should enable core dump per pid to
see if you get core files from more than one process, then grab
backtraces from all those core files to figure out what was the first
reason of the crash.
Is it the last master version or some older one (kamailio -v should
print the commit id, if you compiled from sources)?
Cheers,
Daniel
On 05.10.20 13:11, Juha Heinanen wrote:
K master has recently crashed several times on Debian
10. Could this have
something to do with openssl libs?
-- Juha
Oct 5 08:32:21 lohi systemd[1]: Stopping LSB: Start/stop OpenSIPg SIP Proxy...
Oct 5 08:32:21 lohi sip-proxy[19922]: Stopping sip-proxy
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30820]: NOTICE: <core> [main.c:735]:
handle_sigs(): Thank you for flying sip-proxy!!!
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30860]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30859]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30858]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30857]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30853]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30856]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30850]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30854]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30834]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30847]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30852]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30849]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30851]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30841]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30830]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30848]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30846]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30836]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30827]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30844]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30845]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30833]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30842]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30839]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30829]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30822]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30840]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30838]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30826]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30837]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30824]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30835]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30823]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30832]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30831]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30828]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30843]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30825]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30821]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30855]: INFO: <core> [main.c:857]:
sig_usr(): signal 15 received
Oct 5 08:32:21 lohi /usr/bin/sip-proxy[30820]: INFO: <core> [core/sctp_core.c:53]:
sctp_core_destroy(): SCTP API not initialized
Oct 5 08:32:21 lohi kernel: [176646.851504] sip-proxy[30820]: segfault at 7f84196c7730
ip 00007f8421adce26 sp 00007ffde899c0f0 error 4 in libcrypto.so.1.1[7f84219d8000+19e000]
Oct 5 08:32:21 lohi kernel: [176646.851513] Code: 1f 44 00 00 c7 05 9a d9 15 00 01 00 00
00 c7 05 8c d9 15 00 01 00 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 48 85 ff 74 5b 53
<8b> 0f 48 89 fb 85 c9 75 29 8b 53 04 85 d2 75 2e 8b 43 08 85 c0 75
(gdb) where
#0 0x00007f8421adce26 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#1 0x00007f8421add12f in OPENSSL_cleanup () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
#2 0x00007f8423a87d8c in __run_exit_handlers (status=0, listp=0x7f8423c09718
<__exit_funcs>,
run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at
exit.c:108
#3 0x00007f8423a87eba in __GI_exit (status=<optimized out>) at exit.c:139
#4 0x000055d40be7bb61 in handle_sigs () at main.c:738
#5 0x000055d40be88966 in main_loop () at main.c:1817
#6 0x000055d40be91e32 in main (argc=17, argv=0x7ffde899c948) at main.c:2861
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla
7:43 a.m.
Daniel-Constantin Mierla writes:
The backtrace is for the main process during shut
down, which can be a
side effect of the real issue. You should enable core dump per pid to
see if you get core files from more than one process, then grab
backtraces from all those core files to figure out what was the first
reason of the crash.
Sorry, but don't know how to enable core dump per pid. Didn't find any
cli option for that.
Is it the last master version or some older one
(kamailio -v should
print the commit id, if you compiled from sources)?
Just built K from current master and it crashes every time with similar
gdb trace when I stop it using 'service sip-proxy stop'.
-- Juha
8:19 a.m.
Juha Heinanen writes:
Just built K from current master and it crashes every
time with similar
gdb trace when I stop it using 'service sip-proxy stop'.
As a test, djJust build K from latest 5.4 branch and it does not crash
at stop with exactly same config setup and environment. So looks like
something is broken in master.
-- Juha
9:04 a.m.
The kamailio wiki has a turorial about troubleshooting with core files
where it gives the details about how to enable core file per pid (I am
on mobile link right now, not easy to give the url).
To understand that it crashes only on shut down anyhow, not a runtime crash?
Cheers,
Daniel
On 05.10.20 13:43, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla
The backtrace is for the main process during shut
down, which can be a
side effect of the real issue. You should enable core dump per pid to
see if you get core files from more than one process, then grab
backtraces from all those core files to figure out what was the first
reason of the crash.
Sorry, but don't know how to enable core dump per pid.
Didn't find any
cli option for that.
Is it the last master version or some older one
(kamailio -v should
print the commit id, if you compiled from sources)?
Just built K from current
master and it crashes every time with similar
gdb trace when I stop it using 'service sip-proxy stop'.
-- Juha
9:13 a.m.
Daniel-Constantin Mierla writes:
The kamailio wiki has a turorial about troubleshooting
with core files
where it gives the details about how to enable core file per pid (I am
on mobile link right now, not easy to give the url).
OK, will try to look.
To understand that it crashes only on shut down
anyhow, not a runtime
crash?
I think that so far only at shutdown.
-- Juha
9:33 a.m.
If it is on shut down, then likely it is only one core file, I thought
was at runtime.
Going through difference between the two branches, I think I found the
issue -- a PR that was targeting to remove the obsoleted libssl API was
removing OPENSSL_cleanup(), which docs recommend not to execute, because
it is done automatically by libssl on exit, but in our case it is needed
to have access to shared memory before being destroyed. The docs don't
say it is obsolete, so it should be kept in Kamailio.
I pushed a commit to fix it, can you try with latest master?
Cheers,
Daniel
On 05.10.20 15:13, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla
The kamailio wiki has a turorial about
troubleshooting with core files
where it gives the details about how to enable core file per pid (I am
on mobile link right now, not easy to give the url).
OK, will try to look.
To understand that it crashes only on shut down
anyhow, not a runtime
crash?
I think that so far only at shutdown.
-- Juha
11:38 a.m.
OK, thanks for testing and reporting back.
Cheers,
Daniel
On 05.10.20 15:51, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla
I pushed a commit to fix it, can you try with
latest master?
Tried with latest master and crash does not happen anymore at
shutdown.
Thanks, Juha
1556
days inactive
1556
days old
8 comments
2 participants
participants (2)
-
Daniel-Constantin Mierla -
Juha Heinanen