<div dir="ltr">I think this may be related.<br><br><a href="https://github.com/kamailio/kamailio/issues/2599">https://github.com/kamailio/kamailio/issues/2599</a><br><br>Kamailio creates the core file when the process exiting.<div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 26, 2021 at 6:13 PM Filippo Graziola <<a href="mailto:filippo.graziola@gmail.com">filippo.graziola@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>thanks for the fast reply, I just tried kamailio (5.4.3) from kamailio repo on debian buster, self-signed certificates, same minimal configuration. No error on start, so it seems specific for ubuntu.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mar 26 gen 2021 alle ore 15:39 Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>would you be able to test on Debian 10 (maybe using docker or
virtual machine/virtualbox) and see if you get the same issue?</p>
<p>I do not have Ubuntu 20.04 at hand and I haven't encountered any
issue lately with tls on Debian 10. In this way we can rule out if
it is specific to Ubuntu version of the libraries or not.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 26.01.21 15:06, Filippo Graziola
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi all,
<div>I have an issue related (my guess) to tls and
http_async_client module that result in a segmentation fault
and a not correct handle of tls connections.</div>
<div><br>
</div>
<div>First with only tls module loaded, not forked:</div>
<div><br>
</div>
<div> 0(1021) INFO: <core> [core/tcp_main.c:4983]:
init_tcp(): using epoll_lt as the io watch method (auto
detected)<br>
0(1021) INFO: rr [../outbound/api.h:52]: ob_load_api():
unable to import bind_ob - maybe module is not loaded<br>
0(1021) INFO: rr [rr_mod.c:185]: mod_init(): outbound module
not available<br>
0(1021) INFO: tls [tls_mod.c:389]: mod_init(): With
ECDH-Support!<br>
0(1021) INFO: tls [tls_mod.c:392]: mod_init(): With Diffie
Hellman<br>
0(1021) WARNING: tls [tls_init.c:784]: tls_h_mod_init_f():
openssl bug #1491 (crash/mem leaks on low memory) workaround
enabled (on low memory tls operations will fail preemptively)
with free memory thresholds 4718592 and 2359296 bytes<br>
0(1021) INFO: <core> [core/cfg/cfg_ctx.c:595]:
cfg_set_now(): tls.low_mem_threshold1 has been changed to
4718592<br>
0(1021) INFO: <core> [core/cfg/cfg_ctx.c:595]:
cfg_set_now(): tls.low_mem_threshold2 has been changed to
2359296<br>
0(1021) INFO: <core> [main.c:2833]: main(): processes
(at least): 9 - shm size: 67108864 - pkg size: 67108864<br>
0(1021) INFO: <core> [core/udp_server.c:154]:
probe_max_receive_buffer(): SO_RCVBUF is initially 212992<br>
0(1021) INFO: <core> [core/udp_server.c:206]:
probe_max_receive_buffer(): SO_RCVBUF is finally 425984<br>
0(1021) INFO: tls [tls_domain.c:305]: ksr_tls_fill_missing():
TLSs<default>: tls_method=12<br>
0(1021) INFO: tls [tls_domain.c:317]: ksr_tls_fill_missing():
TLSs<default>: certificate='/etc/kamailio/fullchain.pem'<br>
0(1021) INFO: tls [tls_domain.c:324]: ksr_tls_fill_missing():
TLSs<default>: ca_list='(null)'<br>
0(1021) INFO: tls [tls_domain.c:331]: ksr_tls_fill_missing():
TLSs<default>: crl='(null)'<br>
0(1021) INFO: tls [tls_domain.c:334]: ksr_tls_fill_missing():
TLSs<default>: require_certificate=0<br>
0(1021) INFO: tls [tls_domain.c:342]: ksr_tls_fill_missing():
TLSs<default>: cipher_list='(null)'<br>
0(1021) INFO: tls [tls_domain.c:349]: ksr_tls_fill_missing():
TLSs<default>: private_key='/etc/kamailio/privkey.pem'<br>
0(1021) INFO: tls [tls_domain.c:352]: ksr_tls_fill_missing():
TLSs<default>: verify_certificate=0<br>
0(1021) INFO: tls [tls_domain.c:356]: ksr_tls_fill_missing():
TLSs<default>: verify_depth=9<br>
0(1021) INFO: tls [tls_domain.c:359]: ksr_tls_fill_missing():
TLSs<default>: verify_client=0<br>
0(1021) NOTICE: tls [tls_domain.c:1105]:
ksr_tls_fix_domain(): registered server_name callback handler
for socket [:0], server_name='<default>' ...<br>
0(1021) INFO: tls [tls_domain.c:711]: set_verification():
TLSs<default>: No client certificate required and no
checks performed<br>
0(1021) INFO: tls [tls_domain.c:305]: ksr_tls_fill_missing():
TLSc<default>: tls_method=20<br>
0(1021) INFO: tls [tls_domain.c:317]: ksr_tls_fill_missing():
TLSc<default>: certificate='(null)'<br>
0(1021) INFO: tls [tls_domain.c:324]: ksr_tls_fill_missing():
TLSc<default>: ca_list='(null)'<br>
0(1021) INFO: tls [tls_domain.c:331]: ksr_tls_fill_missing():
TLSc<default>: crl='(null)'<br>
0(1021) INFO: tls [tls_domain.c:334]: ksr_tls_fill_missing():
TLSc<default>: require_certificate=0<br>
0(1021) INFO: tls [tls_domain.c:342]: ksr_tls_fill_missing():
TLSc<default>: cipher_list='(null)'<br>
0(1021) INFO: tls [tls_domain.c:349]: ksr_tls_fill_missing():
TLSc<default>: private_key='(null)'<br>
0(1021) INFO: tls [tls_domain.c:352]: ksr_tls_fill_missing():
TLSc<default>: verify_certificate=0<br>
0(1021) INFO: tls [tls_domain.c:356]: ksr_tls_fill_missing():
TLSc<default>: verify_depth=9<br>
0(1021) INFO: tls [tls_domain.c:359]: ksr_tls_fill_missing():
TLSc<default>: verify_client=0<br>
0(1021) INFO: tls [tls_domain.c:714]: set_verification():
TLSc<default>: Server MAY present invalid certificate<br>
6(1027) ERROR: tls [tls_server.c:1283]: tls_h_read_f():
protocol level error<br>
6(1027) ERROR: tls [tls_util.h:42]: tls_err_ret(): TLS
accept:error:141FC044:SSL
routines:tls_setup_handshake:internal error<br>
6(1027) ERROR: tls [tls_server.c:1287]: tls_h_read_f():
source IP: XXXXXXXXXXXXXXX<br>
6(1027) ERROR: tls [tls_server.c:1290]: tls_h_read_f():
destination IP: XXXXXXXXXX<br>
6(1027) ERROR: <core> [core/tcp_read.c:1498]:
tcp_read_req(): ERROR: tcp_read_req: error reading - c:
0x7f2cbc1b3948 r: 0x7f2cbc1b3a70 (-1)<br>
</div>
<div><br>
</div>
<div>so no segmentation fault but error in handling.</div>
<div><br>
</div>
<div>Second one also with http_async_client loaded:</div>
<div><br>
</div>
<div> 0(1059) INFO: <core> [core/tcp_main.c:4983]:
init_tcp(): using epoll_lt as the io watch method (auto
detected)<br>
0(1061) INFO: rr [../outbound/api.h:52]: ob_load_api():
unable to import bind_ob - maybe module is not loaded<br>
0(1061) INFO: rr [rr_mod.c:185]: mod_init(): outbound module
not available<br>
0(1061) INFO: tls [tls_mod.c:389]: mod_init(): With
ECDH-Support!<br>
0(1061) INFO: tls [tls_mod.c:392]: mod_init(): With Diffie
Hellman<br>
0(1061) INFO: http_async_client
[http_async_client_mod.c:222]: mod_init(): Initializing Http
Async module<br>
0(1061) WARNING: tls [tls_init.c:784]: tls_h_mod_init_f():
openssl bug #1491 (crash/mem leaks on low memory) workaround
enabled (on low memory tls operations will fail preemptively)
with free memory thresholds 5242880 and 2621440 bytes<br>
0(1061) INFO: <core> [core/cfg/cfg_ctx.c:595]:
cfg_set_now(): tls.low_mem_threshold1 has been changed to
5242880<br>
0(1061) INFO: <core> [core/cfg/cfg_ctx.c:595]:
cfg_set_now(): tls.low_mem_threshold2 has been changed to
2621440<br>
0(1061) INFO: <core> [main.c:2833]: main(): processes
(at least): 10 - shm size: 67108864 - pkg size: 67108864<br>
0(1061) INFO: <core> [core/udp_server.c:154]:
probe_max_receive_buffer(): SO_RCVBUF is initially 212992<br>
0(1061) INFO: <core> [core/udp_server.c:206]:
probe_max_receive_buffer(): SO_RCVBUF is finally 425984<br>
0(1061) INFO: tls [tls_domain.c:305]: ksr_tls_fill_missing():
TLSs<default>: tls_method=12<br>
0(1061) INFO: tls [tls_domain.c:317]: ksr_tls_fill_missing():
TLSs<default>: certificate='/etc/kamailio/fullchain.pem'<br>
0(1061) INFO: tls [tls_domain.c:324]: ksr_tls_fill_missing():
TLSs<default>: ca_list='(null)'<br>
0(1061) INFO: tls [tls_domain.c:331]: ksr_tls_fill_missing():
TLSs<default>: crl='(null)'<br>
0(1061) INFO: tls [tls_domain.c:334]: ksr_tls_fill_missing():
TLSs<default>: require_certificate=0<br>
0(1061) INFO: tls [tls_domain.c:342]: ksr_tls_fill_missing():
TLSs<default>: cipher_list='(null)'<br>
0(1061) INFO: tls [tls_domain.c:349]: ksr_tls_fill_missing():
TLSs<default>: private_key='/etc/kamailio/privkey.pem'<br>
0(1061) INFO: tls [tls_domain.c:352]: ksr_tls_fill_missing():
TLSs<default>: verify_certificate=0<br>
0(1061) INFO: tls [tls_domain.c:356]: ksr_tls_fill_missing():
TLSs<default>: verify_depth=9<br>
0(1061) INFO: tls [tls_domain.c:359]: ksr_tls_fill_missing():
TLSs<default>: verify_client=0<br>
0(1061) NOTICE: tls [tls_domain.c:1105]:
ksr_tls_fix_domain(): registered server_name callback handler
for socket [:0], server_name='<default>' ...<br>
0(1061) INFO: tls [tls_domain.c:711]: set_verification():
TLSs<default>: No client certificate required and no
checks performed<br>
0(1061) INFO: tls [tls_domain.c:305]: ksr_tls_fill_missing():
TLSc<default>: tls_method=20<br>
0(1061) INFO: tls [tls_domain.c:317]: ksr_tls_fill_missing():
TLSc<default>: certificate='(null)'<br>
0(1061) INFO: tls [tls_domain.c:324]: ksr_tls_fill_missing():
TLSc<default>: ca_list='(null)'<br>
0(1061) INFO: tls [tls_domain.c:331]: ksr_tls_fill_missing():
TLSc<default>: crl='(null)'<br>
0(1061) INFO: tls [tls_domain.c:334]: ksr_tls_fill_missing():
TLSc<default>: require_certificate=0<br>
0(1061) INFO: tls [tls_domain.c:342]: ksr_tls_fill_missing():
TLSc<default>: cipher_list='(null)'<br>
0(1061) INFO: tls [tls_domain.c:349]: ksr_tls_fill_missing():
TLSc<default>: private_key='(null)'<br>
0(1061) INFO: tls [tls_domain.c:352]: ksr_tls_fill_missing():
TLSc<default>: verify_certificate=0<br>
0(1061) INFO: tls [tls_domain.c:356]: ksr_tls_fill_missing():
TLSc<default>: verify_depth=9<br>
0(1061) INFO: tls [tls_domain.c:359]: ksr_tls_fill_missing():
TLSc<default>: verify_client=0<br>
0(1061) INFO: tls [tls_domain.c:714]: set_verification():
TLSc<default>: Server MAY present invalid certificate<br>
0(1061) INFO: http_async_client [async_http.c:101]:
async_http_init_sockets(): inter-process event notification
sockets initialized<br>
0(1061) INFO: http_async_client [async_http.c:84]:
async_http_init_worker(): started worker process: 1<br>
0(1059) CRITICAL: <core> [core/mem/q_malloc.c:501]:
qm_free(): BUG: bad pointer 0x1 (out of memory block!) called
from tls: tls_init.c: ser_free(323) - ignoring<br>
Segmentation fault<br>
</div>
<div><br>
</div>
<div>this time, there is a segmentation fault.</div>
<div>The above is a result of this minimal configuration: </div>
<div><br>
</div>
<div>#!KAMAILIO<br>
<br>
####### Global Parameters #########<br>
<br>
/* LOG Levels: 3=DBG, 2=INFO, 1=NOTICE, 0=WARN, -1=ERR, ... */<br>
debug=2<br>
log_stderror=no<br>
memdbg=5<br>
memlog=5<br>
<br>
log_facility=LOG_LOCAL0<br>
log_prefix="{$mt $hdr(CSeq) $ci} "<br>
<br>
children=2<br>
tcp_children=2<br>
auto_aliases=no<br>
alias="XXXXXXXXXXXXX"<br>
<br>
listen=<a>udp:eth0</a><br>
server_signature=no<br>
tcp_connection_lifetime=3605<br>
tcp_max_connections=40960<br>
tcp_accept_no_cl=yes<br>
enable_tls=yes<br>
listen=tls:XXXXXXXXXX:5061 advertise XXXXXXXXXXXX:5061<br>
tls_max_connections=40000<br>
enable_sctp=no<br>
<br>
####### Modules Section ########<br>
<br>
loadmodule "kex.so"<br>
loadmodule "corex.so"<br>
loadmodule "tm.so"<br>
loadmodule "tmx.so"<br>
loadmodule "sl.so"<br>
loadmodule "rr.so"<br>
loadmodule "pv.so"<br>
loadmodule "tls.so"<br>
loadmodule "http_async_client.so"<br>
<br>
#----------------- setting module-specific parameters
---------------<br>
#----- tls params -----<br>
modparam("tls", "config", "/etc/kamailio/tls.cfg")<br>
<br>
#----- http client ----<br>
modparam("http_async_client", "workers", 1)<br>
<br>
####### Routing Logic ########<br>
<br>
request_route {<br>
exit;<br>
}<br>
</div>
<div><br>
</div>
<div>I used the above configuration to take out as much as
possible my mistakes in the configuration, but with my full
kamailio configuration, tls connections give the above errors
but everything else works just fine (also http_async_client
module functions which are used on INVITES) and calls are
going properly (unfortunately tls is required). </div>
<div>I found a couple of issues that are similar <a href="https://github.com/kamailio/kamailio/issues/2560" target="_blank">https://github.com/kamailio/kamailio/issues/2560</a>
and <a href="https://github.com/kamailio/kamailio/issues/2466#" target="_blank">https://github.com/kamailio/kamailio/issues/2466#</a>
but as far as I understood the issue 2466 is closed because
fixes are already included. I tried in any case to compile
from source a few older releases with the same result, changed
also the certificate and private key (letsencrypt), moreover I
have another kamailio (v5.3.4) running on ubuntu 18.04 (same
configuration) without any issues. I saw that there is a
different version of openssl version 1.0.. in ubuntu 18.04,
version 1.1 in ubuntu 20.04, but the segmentation fault seems
to happen after an error on free some memory. </div>
<div>Have you some ideas? tell me if you need more info from
me. </div>
<div><br>
</div>
<div>Thanks</div>
<div>Filippo</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Funding: <a href="https://www.paypal.me/dcmierla" target="_blank">https://www.paypal.me/dcmierla</a></pre>
</div>
</blockquote></div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div>