<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Daniel,</p>
<p>Just openssl compiled with ./config -d option and statically
linked to tlsa module. This module is not installed system-wide,
only used for Kamailio.<br>
</p>
<p>libpthread, I believe, is taken from the system.<br>
</p>
<div class="moz-cite-prefix">Le 02/12/2022 à 11:15,
Daniel-Constantin Mierla a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:e00143da-d648-67bf-bce8-7884f7553144@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Still now showing the debug symbols for libssl/libcrypto:</p>
<ol class="text" style="box-sizing: border-box; margin: 0px
!important; padding: 0px 10px; font-size: 12px; vertical-align:
baseline; color: rgb(51, 51, 51); background: rgb(255, 255,
255); font-family: Consolas, Menlo, Monaco, "Lucida
Console", "Liberation Mono", "DejaVu Sans
Mono", "Bitstream Vera Sans Mono", monospace,
serif; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial;">
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">#0 0x00007fdc7d07754d in __lll_lock_wait () from /lib64/libpthread.so.0
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">No symbol table info available.
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">#1 0x00007fdc7d0743d2 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">No symbol table info available.
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">#2 0x00007fdc737f2c59 in CRYPTO_THREAD_write_lock () from /usr/local/lib64/kamailio/modules/tlsa.so
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">No symbol table info available.
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">#3 0x00007fdc7379e32e in CRYPTO_free_ex_data () from /usr/local/lib64/kamailio/modules/tlsa.so
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">No symbol table info available.
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">#4 0x00007fdc73708ae2 in SSL_free () from /usr/local/lib64/kamailio/modules/tlsa.so
</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;">
<div class="de1" style="box-sizing: border-box; margin: 0px 0px 0px -7px; padding: 0px 8px; font-size: 12px; vertical-align: top; user-select: text; color: rgb(0, 0, 0); border-left: 1px solid rgb(221, 221, 221); position: relative; background: rgb(255, 255, 255); white-space: pre-wrap;">No symbol table info available.</div>
</li>
<li class="li1" style="box-sizing: border-box; margin: 0px 0px
0px -6px; padding: 0px; font-size: 12px; vertical-align:
baseline; list-style-type: decimal; user-select: none;
display: list-item; background: rgb(255, 255, 255); color:
rgb(172, 172, 172); font-family: Consolas, Menlo, Monaco,
"Lucida Console", "Liberation Mono",
"DejaVu Sans Mono", "Bitstream Vera Sans
Mono", monospace, serif;"><br>
</li>
</ol>
<div class="moz-cite-prefix">How are these libs installed?</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Cheers,<br>
Daniel</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 29.11.22 10:38, Ihor Olkhovskyi
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:3ef2d292-d36f-3fd3-806b-8a038ab6185b@gmail.com">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>Daniel,</p>
<p>Thanks for the tip, recompiled and got a new trace</p>
<p><a class="moz-txt-link-freetext"
href="https://pastebin.com/NxCXA1gT" moz-do-not-send="true">https://pastebin.com/NxCXA1gT</a></p>
<p>Not sure what happened, but sometimes running kamctl trap
restores Kamailio functionality to accept connections.<br>
</p>
<p>Thanks in advance!<br>
</p>
<div class="moz-cite-prefix">Le 29/11/2022 à 08:07,
Daniel-Constantin Mierla a écrit :<br>
</div>
<blockquote type="cite"
cite="mid:0d3e7097-0262-7851-6248-b0e0c2213dca@gmail.com">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>Hello,</p>
<p>is libssl/libcrypto compiled/installed with debugging
symbols? They are not shown in the backtrace.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 28.11.22 17:38, Igor Olhovskiy
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJTkRNv8B=Hzt8dKPkDa8OrnU07ftz_Wrut09Mqfa6B5tM4mGg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div dir="ltr">
<div>Hello,</div>
<div><br>
</div>
<div>I'm continuing investigations on Kamailio and stress
test. Got it again in the state where it's not accepting
any new TCP/TLS connections (UDP still works though),
but all looks good from lsof/netnstat part, like system
is not reporting any zombie connections. Restart of
Kamailio process helps<br>
</div>
<div><br>
</div>
<div>This time I got output of kamctl trap<br>
</div>
<div><br>
</div>
<div>Put it here: <a href="https://pastebin.com/iYrNZ8U9"
moz-do-not-send="true" class="moz-txt-link-freetext">https://pastebin.com/iYrNZ8U9</a>
<br>
</div>
<div><br>
</div>
<div>kamailio --version<br>
version: kamailio 5.6.2 (x86_64/linux) 54a9c1<br>
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLOCKLIST, HAVE_RESOLV_RES<br>
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB<br>
poll method support: poll, epoll_lt, epoll_et, sigio_rt,
select.<br>
id: 54a9c1 <br>
compiled on 14:01:01 Oct 18 2022 with gcc 4.8.5</div>
<div><br>
</div>
<div>It's statically linked with tlsa pointing on
openssl-1.1.1q</div>
<div><br>
</div>
<div>Settings related to TLS are:</div>
<div><br>
</div>
<div>fork=yes<br>
children=4<br>
tcp_children=12<br>
enable_tls=yes<br>
<br>
tcp_accept_no_cl=yes<br>
tcp_max_connections=63536<br>
tls_max_connections=63536<br>
tcp_accept_aliases=no<br>
tcp_async=yes<br>
tcp_connect_timeout=10<br>
tcp_conn_wq_max=63536<br>
tcp_crlf_ping=yes<br>
tcp_delayed_ack=yes<br>
tcp_fd_cache=yes<br>
tcp_keepalive=yes<br>
tcp_keepcnt=3<br>
tcp_keepidle=30<br>
tcp_keepintvl=10<br>
tcp_linger2=30<br>
tcp_rd_buf_size=80000<br>
tcp_send_timeout=10<br>
tcp_wq_blk_size=2100<br>
tcp_wq_max=10485760<br>
open_files_limit=63536<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Can you please help to read gdb output and understand
where I missed in config? <br>
</div>
<div><br>
</div>
<div>Thanks in advance!<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>-- <br>
<div dir="ltr" class="gmail_signature"
data-smartmail="gmail_signature">
<div dir="ltr">Best regards,
<div>Ihor (Igor)<br>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
<a class="moz-txt-link-abbreviated moz-txt-link-freetext" href="mailto:sr-users@lists.kamailio.org" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
<a class="moz-txt-link-freetext" href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</blockquote>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a></pre>
</blockquote>
</body>
</html>