<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I pinged Victor to see if he can figure out what happens within
the deb building process that makes the libssl mutex fix not
enabled.</p>
<p>The extra .so preload object should be still installed, try to
see if it is at:</p>
<p>/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 16.12.19 12:09, Aymeric Moizard
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALM7LKN5TRYk+wDiYU3JFVGAHstvASM7_siHuzKmguuocRV5FA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="auto">Good catch!
<div dir="auto"><br>
<div dir="auto">As I said in my first mail, I also add the
issue with latest 5.2.X so I suppose the deb package has the
same issue for 52X.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Is the extra binary to load still there? I
will check that as soon as I'm online...</div>
<div dir="auto"><br>
</div>
<div dir="auto">Tks a lot!</div>
<div dir="auto">Aymeric</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Le lun. 16 déc. 2019 à 11:16,
Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com" moz-do-not-send="true">miconda@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p>Hello,</p>
<p>for some reason the binary doesn't seem to have the
libssl mutex fix, in my system with the libssl 1.1 gives:<br>
</p>
<p># kamailio -I<br>
Print out of kamailio internals<br>
Version: kamailio 5.3.1 (x86_64/linux) f36ac2<br>
Default config:
/tmp/kamailio-5.3/etc/kamailio/kamailio.cfg<br>
Default paths to modules:
/tmp/kamailio-5.3/lib64/kamailio/modules<br>
Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES,
TLS_PTHREAD_MUTEX_SHARED<br>
MAX_RECV_BUFFER_SIZE=262144<br>
MAX_URI_SIZE=1024<br>
BUF_SIZE=65535<br>
DEFAULT PKG_SIZE=8MB<br>
DEFAULT SHM_SIZE=64MB<br>
ADAPTIVE_WAIT_LOOPS=1024<br>
TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt,
select<br>
Source code revision ID: f36ac2 <br>
Compiled with: gcc 9.2.1<br>
Compiled architecture: x86_64<br>
Compiled on: 11:11:20 Dec 16 2019<br>
Thank you for flying kamailio!<br>
</p>
<p>The important part above is the presence of
TLS_PTHREAD_MUTEX_SHARED compile time flag in the output.</p>
<p>Needs to be investigated why the dep packages have the
kamailio binary without the libssl mutex fix enabled.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 16.12.19 09:22, Aymeric Moizard wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">Hi Daniel,
<div><br>
</div>
<div>Tks a lot for lookint at it.</div>
<div><br>
</div>
<div>$ ldd
/usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so<br>
linux-vdso.so.1 (0x00007fff997dd000)<br>
libssl.so.1.1 =>
/usr/lib/x86_64-linux-gnu/libssl.so.1.1
(0x00007fe40b53c000)<br>
libc.so.6 =>
/lib/x86_64-linux-gnu/libc.so.6 (0x00007fe40b19d000)<br>
libcrypto.so.1.1 =>
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
(0x00007fe40ad03000)<br>
libdl.so.2 =>
/lib/x86_64-linux-gnu/libdl.so.2
(0x00007fe40aaff000)<br>
libpthread.so.0 =>
/lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fe40a8e2000)<br>
/lib64/ld-linux-x86-64.so.2
(0x00007fe40ba4a000)<br>
<div><br>
</div>
<div>$ /usr/sbin/kamailio -I<br>
Print out of kamailio internals<br>
Version: kamailio 5.3.1 (x86_64/linux)<br>
Default config: /etc/kamailio/kamailio.cfg<br>
Default paths to modules:
/usr/lib/x86_64-linux-gnu/kamailio/modules<br>
Compile flags: USE_TCP, USE_TLS, USE_SCTP,
TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC,
Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES<br>
MAX_RECV_BUFFER_SIZE=262144<br>
MAX_URI_SIZE=1024<br>
BUF_SIZE=65535<br>
DEFAULT PKG_SIZE=8MB<br>
DEFAULT SHM_SIZE=64MB<br>
ADAPTIVE_WAIT_LOOPS=1024<br>
TCP poll methods: poll, epoll_lt, epoll_et,
sigio_rt, select<br>
Source code revision ID: unknown<br>
Compiled with: gcc 6.3.0<br>
Compiled architecture: x86_64<br>
Compiled on:<br>
Thank you for flying kamailio!<br>
</div>
</div>
</div>
<div><br>
</div>
<div>Additional note:</div>
<div>I have tried to better understand the pike module
and after reading the "end" of the module
documentation,</div>
<div>I do better understand the "Tree of IP" and
settings.</div>
<div><br>
</div>
The pike documentation, for each settins and
description, should refer to the section "Chapter 3.
Developer Guide",
<div>otherwise, the parameters cannot be understood.
Also, it's not possible to understand, according to
me, the real time</div>
<div>for removing an IP from the tree (removing it 100%
or only last node of IP)<br>
<div><br>
</div>
<div>Looking again at my statistics, I feel the first
graph is definitly showing an issue. This graph is
showing</div>
<div>"$stat(location-users)" and
"$stat(location-contacts)". During the 10 hours,
many users are banned, unregistred, etc..<br>
</div>
<div>so it is really not expected that the number of
registred users is maintained. From what I
understand, the fact</div>
<div>that the stats went down when deadlock
dissapeared obviouly means kamailio threads was in a
bad state for the</div>
<div>last 10 hours...</div>
<div><br>
</div>
<div><a
href="https://www.antisip.com/sip-antisip-com-register/status2.htm"
target="_blank" rel="noreferrer"
moz-do-not-send="true">https://www.antisip.com/sip-antisip-com-register/status2.htm</a> <br>
<br>
</div>
<div>If you need more information, let me know...</div>
<div>Regards</div>
<div>Aymeric</div>
<div><br>
</div>
<div>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Le lun. 16 déc.
2019 à 08:22, Daniel-Constantin Mierla <<a
href="mailto:miconda@gmail.com"
target="_blank" rel="noreferrer"
moz-do-not-send="true">miconda@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hello,</p>
<p>can you provide output of ldd for tls.so
and output of "kamailio -I" (that's an
uppercase i)?</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div>On 13.12.19 16:39, Aymeric Moizard wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi List,
<div><br>
</div>
<div>History:</div>
<div>* In the past, I had deadlock which
was, most probably, related to ssl1.1.</div>
<div> We have discussed this issue, and a
fix is supposed to workaround the issue
that was detected.</div>
<div>* With latest 5.2.X, I have
experienced ONCE a similar behavior with
TCP and TLS being mostly stuck. I have
not been using this version much, but
the fix was supposed to be in the core
of kamailio.</div>
<div><br>
</div>
<div>The status of the server this night:</div>
<div>* I'm today running version: kamailio
5.3.1 (x86_64/linux), </div>
<div>* Installed on stretch using <a
href="http://deb.kamailio.org/kamailio53"
target="_blank" rel="noreferrer"
moz-do-not-send="true">http://deb.kamailio.org/kamailio53</a>
repository.</div>
<div>* This versions use libssl1.1</div>
<div>* A user reported that he can't
connect with TCP</div>
<div>* An average of 5000 IPs per 10
minutes are being banned by the pike
module</div>
<div> (could be twice the same)</div>
<div>Yesterday/Today:</div>
<div>* at the end of the outage, I had
2479 IP in my ipban htable. (which is
equivalent to my statistics showing 2
bans/IP every 10 minutes = 5000)</div>
<div>* looking at my logs, it appears that
most (ALL?) ip being banned... are my
regular users.</div>
<div>* looking at my logs, I can't
understand why pike would block them.</div>
<div><br>
</div>
<div>This is a graph for statistics on my
service for the last 24 hours:</div>
<div><a
href="https://www.antisip.com/sip-antisip-com-register/status2.html"
target="_blank" rel="noreferrer"
moz-do-not-send="true">https://www.antisip.com/sip-antisip-com-register/status2.html</a> <br
clear="all">
<div><br>
</div>
<div>Yesterday, at 22:18:39, kamailio
started to BAN some IPs. 52 IPs were
banned in a period of 10 minutes. I
can confirm this from my logs.</div>
<div><br>
</div>
<div>My pike configuration is this one:</div>
<div><br>
</div>
<div>modparam("pike",
"sampling_time_unit", 2)<br>
modparam("pike",
"reqs_density_per_unit", 64)<br>
modparam("pike", "remove_latency", 4)<br>
<br>
</div>
<div>When detecting the issue, this
morning, I typed:</div>
<div><br>
</div>
<div>$> sudo kamctl stats<br>
</div>
<div>$> sudo kamcmd htable.dump ipban<br>
</div>
<div>//FAILURE (answer too large...)</div>
<div> $> sudo kamctl trap<br>
</div>
<div><br>
</div>
<div>Then, I started an agent with TCP
and it worked...???</div>
<div> Then, a few seconds, may be a
minute after:</div>
<div><br>
</div>
<div>$> sudo kamcmd htable.dump ipban<br>
</div>
<div>//SUCCESS and shows 2479 banned ip.</div>
<div><br>
</div>
<div>and... everything is back to normal
in a few minutes.</div>
<div><br>
</div>
<div>I haven't restarted kamailio, and
all statistics are as expected, as
usual.</div>
<div><br>
</div>
<div>Thus, it looks that " sudo kamctl
trap" has triggered something. I
already</div>
<div>experienced a similar behavior
-when testing my ssl1.1 deadlock last
year-.</div>
<div><br>
</div>
<div>2 questions:</div>
<div>1/ I beleive my "pike"
configuration should not ban users. Is
my pike configuration wrong?</div>
<div>As an example, pike has banned an
IP sending one message/second. I
believe my configuration should accept
that?</div>
<div><br>
</div>
<div>2/ Could there still be a TLS issue
with libssl1.1?</div>
<div><br>
</div>
<div>This is the result of the "kamctl
trap":</div>
<div><br>
</div>
<div><a
href="https://sip.antisip.com/kamailio-pike-or-tls-issue-13-12-2019.kamctl-trap"
target="_blank" rel="noreferrer"
moz-do-not-send="true">https://sip.antisip.com/kamailio-pike-or-tls-issue-13-12-2019.kamctl-trap</a><br>
</div>
<div><br>
</div>
<div>Sorry for the long story &
hoping to find a long term solution or
at least a workaround!</div>
<div><br>
</div>
<div>Regards</div>
<div>Aymeric</div>
<div><br>
</div>
-- <br>
<div dir="ltr"><img
src="http://sip.antisip.com/am48.png"
moz-do-not-send="true">Antisip - <a
href="http://www.antisip.com"
target="_blank" rel="noreferrer"
moz-do-not-send="true">http://www.antisip.com</a><br>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.kamailio.org" target="_blank" rel="noreferrer" moz-do-not-send="true">sr-users@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" rel="noreferrer" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" rel="noreferrer" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" rel="noreferrer" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" rel="noreferrer" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank" rel="noreferrer" moz-do-not-send="true">www.kamailioworld.com</a></pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"><img
src="http://sip.antisip.com/am48.png"
moz-do-not-send="true">Antisip - <a
href="http://www.antisip.com" target="_blank"
rel="noreferrer" moz-do-not-send="true">http://www.antisip.com</a><br>
</div>
</div>
</div>
</div>
</blockquote>
<pre cols="72">--
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" rel="noreferrer" moz-do-not-send="true">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" rel="noreferrer" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" rel="noreferrer" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank" rel="noreferrer" moz-do-not-send="true">www.kamailioworld.com</a></pre>
</div>
</blockquote>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a></pre>
</body>
</html>