<div dir="auto">Good catch!<div dir="auto"><br><div dir="auto">As I said in my first mail, I also add the issue with latest 5.2.X so I suppose the deb package has the same issue for 52X.</div><div dir="auto"><br></div><div dir="auto">Is the extra binary to load still there? I will check that as soon as I'm online...</div><div dir="auto"><br></div><div dir="auto">Tks a lot!</div><div dir="auto">Aymeric</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le lun. 16 déc. 2019 à 11:16, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div>
    <p>Hello,</p>
    <p>for some reason the binary doesn't seem to have the libssl mutex
      fix, in my system with the libssl 1.1 gives:<br>
    </p>
    <p># kamailio -I<br>
      Print out of kamailio internals<br>
        Version: kamailio 5.3.1 (x86_64/linux) f36ac2<br>
        Default config: /tmp/kamailio-5.3/etc/kamailio/kamailio.cfg<br>
        Default paths to modules:
      /tmp/kamailio-5.3/lib64/kamailio/modules<br>
        Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
      USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP,
      PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
      USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
      USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES,
      TLS_PTHREAD_MUTEX_SHARED<br>
        MAX_RECV_BUFFER_SIZE=262144<br>
        MAX_URI_SIZE=1024<br>
        BUF_SIZE=65535<br>
        DEFAULT PKG_SIZE=8MB<br>
        DEFAULT SHM_SIZE=64MB<br>
        ADAPTIVE_WAIT_LOOPS=1024<br>
        TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select<br>
        Source code revision ID: f36ac2 <br>
        Compiled with: gcc 9.2.1<br>
        Compiled architecture: x86_64<br>
        Compiled on: 11:11:20 Dec 16 2019<br>
      Thank you for flying kamailio!<br>
    </p>
    <p>The important part above is the presence of
      TLS_PTHREAD_MUTEX_SHARED compile time flag in the output.</p>
    <p>Needs to be investigated why the dep packages have the kamailio
      binary without the libssl mutex fix enabled.</p>
    <p>Cheers,<br>
      Daniel<br>
    </p>
    <div>On 16.12.19 09:22, Aymeric Moizard
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr">Hi Daniel,
          <div><br>
          </div>
          <div>Tks a lot for lookint at it.</div>
          <div><br>
          </div>
          <div>$ ldd /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so<br>
                    linux-vdso.so.1 (0x00007fff997dd000)<br>
                    libssl.so.1.1 =>
            /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fe40b53c000)<br>
                    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
            (0x00007fe40b19d000)<br>
                    libcrypto.so.1.1 =>
            /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
            (0x00007fe40ad03000)<br>
                    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
            (0x00007fe40aaff000)<br>
                    libpthread.so.0 =>
            /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe40a8e2000)<br>
                    /lib64/ld-linux-x86-64.so.2 (0x00007fe40ba4a000)<br>
            <div><br>
            </div>
            <div>$ /usr/sbin/kamailio -I<br>
              Print out of kamailio internals<br>
                Version: kamailio 5.3.1 (x86_64/linux)<br>
                Default config: /etc/kamailio/kamailio.cfg<br>
                Default paths to modules:
              /usr/lib/x86_64-linux-gnu/kamailio/modules<br>
                Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
              USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
              SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
              DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
              USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
              USE_DST_BLACKLIST, HAVE_RESOLV_RES<br>
                MAX_RECV_BUFFER_SIZE=262144<br>
                MAX_URI_SIZE=1024<br>
                BUF_SIZE=65535<br>
                DEFAULT PKG_SIZE=8MB<br>
                DEFAULT SHM_SIZE=64MB<br>
                ADAPTIVE_WAIT_LOOPS=1024<br>
                TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt,
              select<br>
                Source code revision ID: unknown<br>
                Compiled with: gcc 6.3.0<br>
                Compiled architecture: x86_64<br>
                Compiled on:<br>
              Thank you for flying kamailio!<br>
            </div>
          </div>
        </div>
        <div><br>
        </div>
        <div>Additional note:</div>
        <div>I have tried to better understand the pike module and after
          reading the "end" of the module documentation,</div>
        <div>I do better understand the "Tree of IP" and settings.</div>
        <div><br>
        </div>
        The pike documentation, for each settins and description, should
        refer to the section "Chapter 3. Developer Guide",
        <div>otherwise, the parameters cannot be understood. Also, it's
          not possible to understand, according to me, the real time</div>
        <div>for removing an IP from the tree (removing it 100% or only
          last node of IP)<br>
          <div><br>
          </div>
          <div>Looking again at my statistics, I feel the first graph is
            definitly showing an issue.  This graph is showing</div>
          <div>"$stat(location-users)" and "$stat(location-contacts)".
            During the 10 hours, many users are banned, unregistred,
            etc..<br>
          </div>
          <div>so it is really not expected that the number of registred
            users is maintained. From what I understand, the fact</div>
          <div>that the stats went down when deadlock dissapeared
            obviouly means kamailio threads was in a bad state for the</div>
          <div>last 10 hours...</div>
          <div><br>
          </div>
          <div><a href="https://www.antisip.com/sip-antisip-com-register/status2.htm" target="_blank" rel="noreferrer">https://www.antisip.com/sip-antisip-com-register/status2.htm</a>  <br>
            <br>
          </div>
          <div>If you need more information, let me know...</div>
          <div>Regards</div>
          <div>Aymeric</div>
          <div><br>
          </div>
          <div>
            <div class="gmail_quote">
              <div dir="ltr" class="gmail_attr">Le lun. 16 déc. 2019
                à 08:22, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank" rel="noreferrer">miconda@gmail.com</a>>
                a écrit :<br>
              </div>
              <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
                <div>
                  <p>Hello,</p>
                  <p>can you provide output of ldd for tls.so and output
                    of "kamailio -I" (that's an uppercase i)?</p>
                  <p>Cheers,<br>
                    Daniel<br>
                  </p>
                  <div>On 13.12.19 16:39, Aymeric Moizard wrote:<br>
                  </div>
                  <blockquote type="cite">
                    <div dir="ltr">Hi List,
                      <div><br>
                      </div>
                      <div>History:</div>
                      <div>* In the past, I had deadlock which was, most
                        probably, related to ssl1.1.</div>
                      <div>  We have discussed this issue, and a fix is
                        supposed to workaround the issue that was
                        detected.</div>
                      <div>* With latest 5.2.X, I have experienced ONCE
                        a similar behavior with TCP and TLS being mostly
                        stuck. I have not been using this version much,
                        but the fix was supposed to be in the core of
                        kamailio.</div>
                      <div><br>
                      </div>
                      <div>The status of the server this night:</div>
                      <div>* I'm today running version: kamailio 5.3.1
                        (x86_64/linux), </div>
                      <div>* Installed on stretch using <a href="http://deb.kamailio.org/kamailio53" target="_blank" rel="noreferrer">http://deb.kamailio.org/kamailio53</a>
                        repository.</div>
                      <div>* This versions use libssl1.1</div>
                      <div>* A user reported that he can't connect with
                        TCP</div>
                      <div>* An average of 5000 IPs per 10 minutes are
                        being banned by the pike module</div>
                      <div>   (could be twice the same)</div>
                      <div>Yesterday/Today:</div>
                      <div>* at the end of the outage, I had 2479 IP in
                        my ipban htable. (which is equivalent to my
                        statistics showing 2 bans/IP every 10 minutes =
                        5000)</div>
                      <div>* looking at my logs, it appears that most
                        (ALL?) ip being banned... are my regular users.</div>
                      <div>* looking at my logs, I can't understand why
                        pike would block them.</div>
                      <div><br>
                      </div>
                      <div>This is a graph for statistics on my service
                        for the last 24 hours:</div>
                      <div><a href="https://www.antisip.com/sip-antisip-com-register/status2.html" target="_blank" rel="noreferrer">https://www.antisip.com/sip-antisip-com-register/status2.html</a>  <br clear="all">
                        <div><br>
                        </div>
                        <div>Yesterday, at 22:18:39, kamailio started to
                          BAN some IPs. 52 IPs were banned in a period
                          of 10 minutes. I can confirm this from my
                          logs.</div>
                        <div><br>
                        </div>
                        <div>My pike configuration is this one:</div>
                        <div><br>
                        </div>
                        <div>modparam("pike", "sampling_time_unit", 2)<br>
                          modparam("pike", "reqs_density_per_unit", 64)<br>
                          modparam("pike", "remove_latency", 4)<br>
                          <br>
                        </div>
                        <div>When detecting the issue, this morning, I
                          typed:</div>
                        <div><br>
                        </div>
                        <div>$> sudo kamctl stats<br>
                        </div>
                        <div>$> sudo kamcmd htable.dump ipban<br>
                        </div>
                        <div>//FAILURE (answer too large...)</div>
                        <div> $> sudo kamctl trap<br>
                        </div>
                        <div><br>
                        </div>
                        <div>Then, I started an agent with TCP and it
                          worked...???</div>
                        <div> Then, a few seconds, may be a minute
                          after:</div>
                        <div><br>
                        </div>
                        <div>$> sudo kamcmd htable.dump ipban<br>
                        </div>
                        <div>//SUCCESS and shows 2479 banned ip.</div>
                        <div><br>
                        </div>
                        <div>and... everything is back to normal in a
                          few minutes.</div>
                        <div><br>
                        </div>
                        <div>I haven't restarted kamailio, and all
                          statistics are as expected, as usual.</div>
                        <div><br>
                        </div>
                        <div>Thus, it looks that " sudo kamctl trap" has
                          triggered something. I already</div>
                        <div>experienced a similar behavior -when
                          testing my ssl1.1 deadlock last year-.</div>
                        <div><br>
                        </div>
                        <div>2 questions:</div>
                        <div>1/ I beleive my "pike" configuration should
                          not ban users. Is my pike configuration wrong?</div>
                        <div>As an example, pike has banned an IP
                          sending one message/second. I believe my
                          configuration should accept that?</div>
                        <div><br>
                        </div>
                        <div>2/ Could there still be a TLS issue with
                          libssl1.1?</div>
                        <div><br>
                        </div>
                        <div>This is the result of the "kamctl trap":</div>
                        <div><br>
                        </div>
                        <div><a href="https://sip.antisip.com/kamailio-pike-or-tls-issue-13-12-2019.kamctl-trap" target="_blank" rel="noreferrer">https://sip.antisip.com/kamailio-pike-or-tls-issue-13-12-2019.kamctl-trap</a><br>
                        </div>
                        <div><br>
                        </div>
                        <div>Sorry for the long story & hoping to
                          find a long term solution or at least a
                          workaround!</div>
                        <div><br>
                        </div>
                        <div>Regards</div>
                        <div>Aymeric</div>
                        <div><br>
                        </div>
                        -- <br>
                        <div dir="ltr"><img src="http://sip.antisip.com/am48.png">Antisip - <a href="http://www.antisip.com" target="_blank" rel="noreferrer">http://www.antisip.com</a><br>
                        </div>
                      </div>
                    </div>
                    <br>
                    <fieldset></fieldset>
                    <pre>_______________________________________________
Kamailio (SER) - Users Mailing List
<a href="mailto:sr-users@lists.kamailio.org" target="_blank" rel="noreferrer">sr-users@lists.kamailio.org</a>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" target="_blank" rel="noreferrer">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a>
</pre>
                  </blockquote>
                  <pre cols="72">-- 
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" rel="noreferrer">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" rel="noreferrer">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" rel="noreferrer">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank" rel="noreferrer">www.kamailioworld.com</a></pre>
                </div>
              </blockquote>
            </div>
            <br clear="all">
            <div><br>
            </div>
            -- <br>
            <div dir="ltr"><img src="http://sip.antisip.com/am48.png">Antisip - <a href="http://www.antisip.com" target="_blank" rel="noreferrer">http://www.antisip.com</a><br>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <pre cols="72">-- 
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" target="_blank" rel="noreferrer">www.asipto.com</a>
<a href="http://www.twitter.com/miconda" target="_blank" rel="noreferrer">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" target="_blank" rel="noreferrer">www.linkedin.com/in/miconda</a>
Kamailio World Conference - April 27-29, 2020, in Berlin -- <a href="http://www.kamailioworld.com" target="_blank" rel="noreferrer">www.kamailioworld.com</a></pre>
  </div>

</blockquote></div>