<div dir="ltr"><div dir="ltr">To make it easier for you and not have to go through the whole thread, if you want the TL;DR start here: <a href="https://github.com/kamailio/kamailio/issues/1172#issuecomment-312634272">https://github.com/kamailio/kamailio/issues/1172#issuecomment-312634272</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 22, 2019 at 1:19 PM Joel Serrano <<a href="mailto:joel@textplus.com">joel@textplus.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Aymeric, </div><div dir="ltr"><br></div><div>Are you sure the issue is with TCP and not strictly related to TLS? I highly suggest you compile with ssl1.0 and give it a try...</div><div><br></div><div>If you want to read how I got to that conclusion: <a href="https://github.com/kamailio/kamailio/issues/1172" target="_blank">https://github.com/kamailio/kamailio/issues/1172</a><br></div><div><br></div><div>Hope it helps!</div><div>Joel.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 22, 2019 at 11:58 AM Aymeric Moizard <<a href="mailto:amoizard@gmail.com" target="_blank">amoizard@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Daniel,<div><br></div><div>Tks for the tips.</div><div><br></div><div>My traffic does include TLS as well.</div></div><div><br></div>For TCP settings:<div><br></div><div><div>tcp_connection_lifetime=3600</div><div>tcp_async=yes</div><div>tcp_rd_buf_size=16384</div><div>tcp_accept_no_cl=yes</div><div>tcp_max_connections=50000</div><div>tcp_connect_timeout=7</div></div><div><br></div><div>For TLS:</div><div>enable_tls=yes<br></div><div>tls_max_connections=50000<br></div><div><br></div><div>I'm using "set_forward_no_connect();" after lookup(location) since a long time.</div><div><br></div><div>I have added this week "set_reply_no_connect();" in case it will help to avoid the issue.</div><div><br></div><div>If the issue occurs, I will try to get something via "kamctrl trap".</div><div><br></div><div>In order to get a coredump (on restart timeout?) I have added this in my kamailio.service</div><div><br></div><div><div>WorkingDirectory=/var/run/kamailio</div><div>LimitCORE=infinity</div></div><div><br></div><div>I have also DUMP_CORE=yes in /etc/default/kamailio</div><div>and disable_core_dump=no in my kamailio.cfg</div><div><br></div><div>However, I'm not able to see any core dumps when restarting kamailio</div><div>even when I see "sig_alarm_abort(): shutdown timeout triggered, dying"...</div><div><br></div><div>Am I supposed to get a core dump in such case?</div><div><br></div><div>Tks a lot!</div><div>Aymeric</div><div><br></div><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le ven. 22 mars 2019 à 14:19, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF">
    <p>Do you have pure tcp traffic and facing this issue, or there are
      actually tls connections?</p>
    <p>What are the values for core parameters related to tcp connect
      and tcp send timeouts?<br>
    </p>
    <p>As for restart taking long, see exit_timeout parameter:</p>
    <p>  *
      <a class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076moz-txt-link-freetext" href="https://www.kamailio.org/wiki/cookbooks/5.2.x/core#exit_timeout" target="_blank">https://www.kamailio.org/wiki/cookbooks/5.2.x/core#exit_timeout</a></p>
    <p>As for tls with libssl1.1/libcrypto1.1, I think I discover what
      the issue is. With v1.1 they use their own internal locking
      functions, not exposing any api to set them from outside. Before,
      kamailio was initializing the libray telling to use Kamailio
      locks, giving one lock per connection. As i could get from some
      gdb traces I received, with libssl 1.1, the same internal lock is
      used for when attempting to connect to different addresses as well
      as when trying to write to different connections. If one operation
      is slow for what so ever reason, the others are waiting for the
      lock to be lifted by the slow operation. I am digging in the
      source code of libssl1.1 to figure out a solution, it can still
      take a bit because I am travelling for several days with no much
      spare time.</p>
    <p>Among the tunnings would be lower timeouts to connect and send,
      do not attempt to connect unless you are sure the target expects
      new connections (e.g., sending to a gateway/sip server accepting
      traffic via tls, but don't do it even for the requests routed via
      lookup(location) as the registration is using a connection with an
      ephemeral source port and trying to connect back to it will fail).
      If still a major issue for what so ever reason, using a version
      compiled with libssl1.0 would be something to go for it.</p>
    <p>Cheers,<br>
      Daniel<br>
    </p>
    <div class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076moz-cite-prefix">On 21.03.19 19:17, Aymeric Moizard
      wrote:<br>
    </div>
    <blockquote type="cite">
      
      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr">
            <div dir="ltr">
              <div dir="ltr">
                <div dir="ltr">
                  <div dir="ltr">
                    <div dir="ltr">Hi List,
                      <div><br>
                      </div>
                      <div>I want to share that I also met this issue
                        last week with my kamailio 5.2.2.</div>
                      <div><br>
                      </div>
                      <div>As far as I was able to see, SIP application
                        were able to "connect()"</div>
                      <div>with TCP, but my logs wasn't reporting any of
                        the SIP message received</div>
                      <div>with TCP.</div>
                      <div><br>
                      </div>
                      <div>I have an pike right before an xlog showing
                        every incoming request. However</div>
                      <div>I suspect the issue was not related to pike
                        module. The log didn't showed unusual</div>
                      <div>number of blocked traffic.</div>
                      <div><br>
                      </div>
                      <div>I'm almost sure I haven't reached any ulimit
                        restrictions.</div>
                      <div>I have many TCP, UDP childreen...</div>
                      <div>Server was not under high load</div>
                      <div>Nothing unusual.</div>
                      <div><br>
                      </div>
                      <div>I'm running the default build for debian
                        stretch from here:</div>
                      <div>   <a href="http://deb.kamailio.org/kamailio52" target="_blank">http://deb.kamailio.org/kamailio52</a>
                        stretch</div>
                      <div><br>
                      </div>
                      <div>And unfortunatly, I had some tiny pressure to
                        restart the service so I was</div>
                      <div>not able to get deeper into the issue.</div>
                      <div><br>
                      </div>
                      <div>If I'm correct, I will certainly improve much
                        things by using "set_reply_no_connect()".</div>
                      <div>I have added it and restarted!</div>
                      <div>(Tks Daniel for this tip!)</div>
                      <div><br>
                      </div>
                      <div>I have been looking at issue reported here:</div>
                      "Kamailio 5.0.2 suddenly stops processing traffic,
                      then generates a core when restarting."</div>
                    <div dir="ltr"><a href="https://github.com/kamailio/kamailio/issues/1172" target="_blank">https://github.com/kamailio/kamailio/issues/1172</a>
                      <div><br>
                      </div>
                      <div>I have to say that I do have libssl1.1.</div>
                      <div>And I do have crash when I restart my
                        kamailio. (even when I simply restart after a
                        configuration modification)</div>
                      <div><br>
                      </div>
                      <div>
                        <div>Mar 21 18:28:50 sip kamailio[19222]: INFO:
                          <core> [main.c:836]: sig_usr(): signal
                          15 received</div>
                        <div>Mar 21 18:28:50 sip kamailio[19175]: ERROR:
                          ctl [ctl.c:390]: mod_destroy(): ERROR: ctl:
                          could not delete unix socket
                          /var/run/kamailio/kamailio_ctl: Permission
                          denied (13)</div>
                        <div><br>
                        </div>
                        <div>[... one minute without nothing...]</div>
                        <div><br>
                        </div>
                        <div>Mar 21 18:29:42 sip kamailio[19175]: ERROR:
                          jsonrpcs [jsonrpcs_fifo.c:599]:
                          jsonrpc_fifo_destroy(): FIFO stat failed:
                          Permission denied</div>
                        <div>Mar 21 18:29:42 sip kamailio[19175]: ERROR:
                          jsonrpcs [jsonrpcs_sock.c:516]:
                          jsonrpc_dgram_destroy(): socket stat failed:
                          Permission denied</div>
                        <div>Mar 21 18:29:50 sip kamailio[19175]:
                          CRITICAL: <core> [main.c:662]:
                          sig_alarm_abort(): shutdown timeout triggered,
                          dying...</div>
                      </div>
                      <div><br>
                      </div>
                      <div>As the 1172 issue is closed, should I expect
                        kamailio to still have trouble with libssl1.1?</div>
                      <div><br>
                      </div>
                      <div>I just restarted again my service (to see if
                        it restart better after 30 minutes only instead
                        of a week)</div>
                      <div><br>
                      </div>
                      <div>Mar 21 19:07:30 sip kamailio[28737]: INFO:
                        <core> [main.c:836]: sig_usr(): signal 15
                        received<br>
                      </div>
                      <div>
                        <div>Mar 21 19:07:31 sip kamailio[28671]: ERROR:
                          ctl [ctl.c:390]: mod_destroy(): ERROR: ctl:
                          could not delete unix socket
                          /var/run/kamailio/kamailio_ctl: Permission
                          denied (13)</div>
                        <div><br>
                        </div>
                        <div>[... one minute without nothing...]</div>
                        <br class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076gmail-Apple-interchange-newline">
                        <div>Mar 21 19:08:30 sip kamailio[28671]:
                          CRITICAL: <core> [main.c:662]:
                          sig_alarm_abort(): shutdown timeout triggered,
                          dying...</div>
                      </div>
                      <div><br>
                      </div>
                      <div>Still not able to restart in a clean way!<br>
                      </div>
                      <div>Tks!</div>
                      <div>Regards</div>
                      <div>Aymeric</div>
                      <div><br>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
      <br>
      <div class="gmail_quote">
        <div dir="ltr" class="gmail_attr">Le mer. 20 mars 2019 à 15:08,
          Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>
          a écrit :<br>
        </div>
        <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
          <br>
          based on the trap output I think I could figure out what
          happened there.<br>
          <br>
          You have tcp_children to very low value (1 or so), the problem
          is not<br>
          actually that one, but the fact that the connection to
          upstream (the<br>
          device/app sending the request) was closed after receiving the
          request<br>
          and routing of the reply gets stuck in the way of:<br>
          <br>
            - a reply is received and has to be forwarded<br>
            - connection was lost, so Kamailio tries to establish a new
          one, but<br>
          takes time till fails because the upstream is behind nat or so
          based on<br>
          the via header:<br>
          <br>
          Via: SIP/2.0/TLS<br>
10.1.0.4:10002;rport=55229;received=13.94.188.218;branch=z9hG4bK-3336-7f2927bfd703ae907348edff3611bfc9<br>
          <br>
            - the reply is retransmitted and gets to another worker,
          which tries<br>
          to forward it again, but discovers a connection structure for
          that<br>
          destination exists (created by previous reply worker) and now
          waits for<br>
          the connection to be released (or better said, for the mutex
          on writing<br>
          buffer to be unlocked)<br>
          <br>
            - as the second reply waits, there can be other
          retransmissions of the<br>
          reply ending up in other workers stuck on waiting for the
          mutex of the<br>
          connection write buffer<br>
          <br>
          The solution here is to use set_reply_no_connect() -- you can
          put it<br>
          first in request_route block. I think this would be a good
          addition to<br>
          the default configuration file as well, IMO, the sip server
          should not<br>
          connect for sending replies and should do it also for requests
          that go<br>
          behind nat.<br>
          <br>
          Cheers,<br>
          Daniel<br>
          <br>
          On 19.03.19 10:53, Kristijan Vrban wrote:<br>
          > So i had again the situation. But this time, incoming udp
          was<br>
          > affected. Kamailio was sending out OPTIONS (via
          dispatcher module) to<br>
          > a group of asterisk machines<br>
          > but the 200 OK reply to the OPTIONS where not processed,
          so the<br>
          > dispatcher module set all asterisk to inactive, even
          though they<br>
          > replied 200 OK<br>
          ><br>
          > Attached the output of kamctl trap during the situation.
          Hope there is<br>
          > any useful in it. Because after "kamctl trap" it was
          working again<br>
          > without kamailio restart.<br>
          ><br>
          > Best<br>
          > Kristijan<br>
          ><br>
          > Am Mo., 18. März 2019 um 12:27 Uhr schrieb
          Daniel-Constantin Mierla<br>
          > <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>:<br>
          >> Hello,<br>
          >><br>
          >> setting tcp_children=1 is not a god option for
          scallability, practically<br>
          >> you set kamailio to process a single tcp message at
          one time, on high<br>
          >> traffic, that won't work well.<br>
          >><br>
          >> Maybe try to set tcp_children to 2 or 4, that should
          make an eventual<br>
          >> race appear faster.<br>
          >><br>
          >> Regarding the pid, if it is an outgoing connection,
          then it can be<br>
          >> created by any worker process, including a UDP
          worker, if that was the<br>
          >> one receiving the sip message over udp and sends it
          out via tcp.<br>
          >><br>
          >> Cheers,<br>
          >> Daniel<br>
          >><br>
          >> On 18.03.19 10:09, Kristijan Vrban wrote:<br>
          >>> Hi Daniel,<br>
          >>><br>
          >>> for testing, i now had set: "tcp_children=1" and
          so far this issue did not occur<br>
          >>> ever since. So now value to provide for "kamctl
          trap" yet.<br>
          >>><br>
          >>> "kamctl ps" show this two process to handle tcp:<br>
          >>><br>
          >>> ...<br>
          >>>     }, {<br>
          >>>       "IDX":  25,<br>
          >>>       "PID":  71929,<br>
          >>>       "DSC":  "tcp receiver (generic) child=0"<br>
          >>>     }, {<br>
          >>>       "IDX":  26,<br>
          >>>       "PID":  71933,<br>
          >>>       "DSC":  "tcp main process"<br>
          >>>     }<br>
          >>> ...<br>
          >>><br>
          >>><br>
          >>> Ok, but then is was wondering to see a TCP
          connection on a udp receiver child:<br>
          >>><br>
          >>><br>
          >>> netstat -ntp |grep 5061<br>
          >>><br>
          >>> ...<br>
          >>> tcp        0      0 <a href="http://172.17.217.10:5061" rel="noreferrer" target="_blank">172.17.217.10:5061</a> 
              <a href="http://195.70.114.125:18252" rel="noreferrer" target="_blank">195.70.114.125:18252</a><br>
          >>> ESTABLISHED 71895/kamailio<br>
          >>> ...<br>
          >>><br>
          >>> An pid 71895 is:<br>
          >>><br>
          >>> }, {<br>
          >>>       "IDX":  3,<br>
          >>>       "PID":  71895,<br>
          >>>       "DSC":  "udp receiver child=2 sock=<a href="http://127.0.0.1:5060" rel="noreferrer" target="_blank">127.0.0.1:5060</a>"<br>
          >>>     }, {<br>
          >>><br>
          >>><br>
          >>><br>
          >>> And if i look into it via "lsof -p 71895" (the
          udp receiver child)<br>
          >>><br>
          >>> ...<br>
          >>> kamailio 71895 kamailio   14u  sock             
            0,9      0t0<br>
          >>> 8856085 protocol: TCP<br>
          >>> kamailio 71895 kamailio   15u  sock             
            0,9      0t0<br>
          >>> 8886886 protocol: TCP<br>
          >>> kamailio 71895 kamailio   16u  sock             
            0,9      0t0<br>
          >>> 8854886 protocol: TCP<br>
          >>> kamailio 71895 kamailio   17u  sock             
            0,9      0t0<br>
          >>> 8828915 protocol: TCP<br>
          >>> kamailio 71895 kamailio   18u  unix
          0x000000005f73cb91      0t0<br>
          >>> 1680314 type=DGRAM<br>
          >>> kamailio 71895 kamailio   19u  IPv4           
          1846523      0t0<br>
          >>> TCP kamailio-preview:sip-tls->XXX:18252
          (ESTABLISHED)<br>
          >>> kamailio 71895 kamailio   20u  sock             
            0,9      0t0<br>
          >>> 8887192 protocol: TCP<br>
          >>> kamailio 71895 kamailio   21u  sock             
            0,9      0t0<br>
          >>> 8813634 protocol: TCP<br>
          >>> kamailio 71895 kamailio   22u  unix
          0x00000000c19bd102      0t0<br>
          >>> 1681407 type=STREAM<br>
          >>> kamailio 71895 kamailio   23u  sock             
            0,9      0t0<br>
          >>> 8850488 protocol: TCP<br>
          >>> ...<br>
          >>><br>
          >>> Not only the ESTABLISHED TCP session. But also
          this empty sockets<br>
          >>> "protocol: TCP"<br>
          >>> What are they doing there in the udp receiver? Is
          that how it's supposed to be?<br>
          >>><br>
          >>> Kristijan<br>
          >>><br>
          >>> Am Do., 14. März 2019 um 14:48 Uhr schrieb
          Daniel-Constantin Mierla<br>
          >>> <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>:<br>
          >>>> Can you get file written by `kamctl trap`? It
          should have the backtrace<br>
          >>>> for all kamailio processes. You need latest
          kamailio 5.2.<br>
          >>>><br>
          >>>> Also, get the output for: kamctl ps<br>
          >>>><br>
          >>>> Cheers,<br>
          >>>> Daniel<br>
          >>>><br>
          >>>> On 14.03.19 13:52, Kristijan Vrban wrote:<br>
          >>>>> When i attach via gdb to one of the tcp
          worker, i see this:<br>
          >>>>><br>
          >>>>> (gdb) bt<br>
          >>>>> #0  0x00007fdaf4d14470 in futex_wait
          (private=<optimized out>,<br>
          >>>>> expected=1, futex_word=0x7fdaeca92f8c) at<br>
          >>>>>
          ../sysdeps/unix/sysv/linux/futex-internal.h:61<br>
          >>>>> #1  futex_wait_simple
          (private=<optimized out>, expected=1,<br>
          >>>>> futex_word=0x7fdaeca92f8c) at
          ../sysdeps/nptl/futex-internal.h:135<br>
          >>>>> #2  __pthread_rwlock_wrlock_slow
          (rwlock=0x7fdaeca92f80) at<br>
          >>>>> pthread_rwlock_wrlock.c:67<br>
          >>>>> #3  0x00007fdaf0912ee9 in
          CRYPTO_THREAD_write_lock () from<br>
          >>>>>
          /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
          >>>>> #4  0x00007fdaf08e1c08 in ?? () from
          /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
          >>>>> #5  0x00007fdaf08a6f69 in ?? () from
          /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
          >>>>> #6  0x00007fdaf08b36c7 in
          EVP_CIPHER_CTX_ctrl () from<br>
          >>>>>
          /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
          >>>>> #7  0x00007fdaf0c31144 in ?? () from
          /usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
          >>>>> #8  0x00007fdaf0c2bddb in ?? () from
          /usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
          >>>>> #9  0x00007fdaf0c22858 in ?? () from
          /usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
          >>>>> #10 0x00007fdaf0c1af61 in
          SSL_do_handshake () from<br>
          >>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
          >>>>> #11 0x00007fdaf0e8d31b in tls_accept
          (c=0x7fdaed26fa98,<br>
          >>>>> error=0x7ffffe2a2df0) at tls_server.c:422<br>
          >>>>> #12 0x00007fdaf0e96a1b in tls_read_f
          (c=0x7fdaed26fa98,<br>
          >>>>> flags=0x7ffffe2c318c) at
          tls_server.c:1116<br>
          >>>>> #13 0x0000556ead5e7c46 in
          tcp_read_headers (c=0x7fdaed26fa98,<br>
          >>>>> read_flags=0x7ffffe2c318c) at
          core/tcp_read.c:469<br>
          >>>>> #14 0x0000556ead5ef9cb in tcp_read_req
          (con=0x7fdaed26fa98,<br>
          >>>>> bytes_read=0x7ffffe2c3184,
          read_flags=0x7ffffe2c318c) at<br>
          >>>>> core/tcp_read.c:1496<br>
          >>>>> #15 0x0000556ead5f575f in handle_io
          (fm=0x7fdaf597aa98, events=1,<br>
          >>>>> idx=-1) at core/tcp_read.c:1862<br>
          >>>>> #16 0x0000556ead5e2053 in
          io_wait_loop_epoll (h=0x556eadaaeec0 <io_w>,<br>
          >>>>> t=2, repeat=0) at core/io_wait.h:1065<br>
          >>>>> #17 0x0000556ead5f6b35 in
          tcp_receive_loop (unix_sock=49) at<br>
          >>>>> core/tcp_read.c:1974<br>
          >>>>> #18 0x0000556ead4c8e24 in
          tcp_init_children () at core/tcp_main.c:4853<br>
          >>>>> #19 0x0000556ead3c352a in main_loop () at
          main.c:1735<br>
          >>>>> #20 0x0000556ead3ca5f8 in main (argc=13,
          argv=0x7ffffe2c3828) at main.c:2675<br>
          >>>>><br>
          >>>>><br>
          >>>>><br>
          >>>>><br>
          >>>>><br>
          >>>>><br>
          >>>>><br>
          >>>>> Am Do., 14. März 2019 um 13:41 Uhr
          schrieb Kristijan Vrban<br>
          >>>>> <<a href="mailto:vrban.lkml@gmail.com" target="_blank">vrban.lkml@gmail.com</a>>:<br>
          >>>>>> Hi, with full debug is see this in
          log for every incoming TCP SIP request:<br>
          >>>>>><br>
          >>>>>> Mar 14 12:10:15 kamailio-preview
          /usr/sbin/kamailio[17940]: DEBUG:<br>
          >>>>>> <core> [core/tcp_main.c:3871]:
          send2child(): WARNING: no free tcp<br>
          >>>>>> receiver, connection passed to the
          least busy one (105)<br>
          >>>>>> Mar 14 12:10:15 kamailio-preview
          /usr/sbin/kamailio[17940]: DEBUG:<br>
          >>>>>> <core> [core/tcp_main.c:3875]:
          send2child(): selected tcp worker 2<br>
          >>>>>> 27(17937) for activity on [tls:<a href="http://172.17.217.10:5061" rel="noreferrer" target="_blank">172.17.217.10:5061</a>],
          0x7fdaeda8f928<br>
          >>>>>><br>
          >>>>>> So the Kamailio TCP process is
          working, and received TCP traffic. But<br>
          >>>>>> the tcp workers are somehow busy.<br>
          >>>>>><br>
          >>>>>> When i attach via strace to the TCP
          worker, i do not see any activity. Just:<br>
          >>>>>><br>
          >>>>>> futex(0x7fdaeca92f8c,
          FUTEX_WAIT_PRIVATE, 1, NULL<br>
          >>>>>><br>
          >>>>>> and nothing, even when i see the main
          tcp process choose this worker process.<br>
          >>>>>><br>
          >>>>>> Kristijan<br>
          >>>>>><br>
          >>>>>> Am Mi., 27. Feb. 2019 um 15:14 Uhr
          schrieb Kristijan Vrban<br>
          >>>>>> <<a href="mailto:vrban.lkml@gmail.com" target="_blank">vrban.lkml@gmail.com</a>>:<br>
          >>>>>>> first of all thanks for the
          feedback. i prepared our system now to run<br>
          >>>>>>> with debug=3<br>
          >>>>>>> I hope to see more then then.<br>
          >>>>>>><br>
          >>>>>>> Am Mi., 27. Feb. 2019 um 11:53
          Uhr schrieb Kristijan Vrban<br>
          >>>>>>> <<a href="mailto:vrban.lkml@gmail.com" target="_blank">vrban.lkml@gmail.com</a>>:<br>
          >>>>>>>> Hi kamailios,<br>
          >>>>>>>><br>
          >>>>>>>> i have a creepy situation
          with v5.2.1 stable Kamilio. After a day or<br>
          >>>>>>>> so, Kamailio stop to process
          incoming SIP traffic via TCP. The<br>
          >>>>>>>> incoming TCP network packages
          get TCP-ACK from the OS (Debian 9,<br>
          >>>>>>>> 4.18.0-15-generic-Linux) but
          Kamailio does not show any processing for<br>
          >>>>>>>> the SIP-Traffic incoming via
          TCP. No logs, nothing. While traffic via<br>
          >>>>>>>> UDP is working just totally
          fine.<br>
          >>>>>>>><br>
          >>>>>>>> When i look via command
          "netstat -ntp" is see, that the Recv-Q get<br>
          >>>>>>>> bigger and bigger. e.g.:<br>
          >>>>>>>><br>
          >>>>>>>> Proto Recv-Q Send-Q Local
          Address Foreign Address State PID/Program<br>
          >>>>>>>> name tcp 4566 0 <a href="http://172.17.217.12:5060" rel="noreferrer" target="_blank">172.17.217.12:5060</a>
          xxx.xxx.xxx.xxx:57252 ESTABLISHED<br>
          >>>>>>>> 31347/kamailio<br>
          >>>>>>>><br>
          >>>>>>>> After Kamailio restart, all
          is working fine again for a day. We have<br>
          >>>>>>>> maybe 10-20 devices online
          via TCP and low call volume (1-2 call per<br>
          >>>>>>>> minute). The only settings
          for tcp we have is "tcp_delayed_ack=no"<br>
          >>>>>>>><br>
          >>>>>>>> How to could we debug this
          situation? Again, no error, no warings in<br>
          >>>>>>>> the log. Just nothing.<br>
          >>>>>>>><br>
          >>>>>>>> Kristijan<br>
          >>>>>
          _______________________________________________<br>
          >>>>> Kamailio (SER) - Users Mailing List<br>
          >>>>> <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
          >>>>> <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
          >>>> --<br>
          >>>> Daniel-Constantin Mierla -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
          >>>> <a href="http://www.twitter.com/miconda" rel="noreferrer" target="_blank">www.twitter.com/miconda</a>
          -- <a href="http://www.linkedin.com/in/miconda" rel="noreferrer" target="_blank">www.linkedin.com/in/miconda</a><br>
          >>>> Kamailio World Conference - May 6-8, 2019 --
          <a href="http://www.kamailioworld.com" rel="noreferrer" target="_blank">www.kamailioworld.com</a><br>
          >>>> Kamailio Advanced Training - Mar 25-27, 2019,
          in Washington, DC, USA -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
          >>>><br>
          >>> _______________________________________________<br>
          >>> Kamailio (SER) - Users Mailing List<br>
          >>> <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
          >>> <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
          >> --<br>
          >> Daniel-Constantin Mierla -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
          >> <a href="http://www.twitter.com/miconda" rel="noreferrer" target="_blank">www.twitter.com/miconda</a>
          -- <a href="http://www.linkedin.com/in/miconda" rel="noreferrer" target="_blank">www.linkedin.com/in/miconda</a><br>
          >> Kamailio World Conference - May 6-8, 2019 -- <a href="http://www.kamailioworld.com" rel="noreferrer" target="_blank">www.kamailioworld.com</a><br>
          >> Kamailio Advanced Training - Mar 25-27, 2019, in
          Washington, DC, USA -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
          >><br>
          -- <br>
          Daniel-Constantin Mierla -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
          <a href="http://www.twitter.com/miconda" rel="noreferrer" target="_blank">www.twitter.com/miconda</a>
          -- <a href="http://www.linkedin.com/in/miconda" rel="noreferrer" target="_blank">www.linkedin.com/in/miconda</a><br>
          Kamailio World Conference - May 6-8, 2019 -- <a href="http://www.kamailioworld.com" rel="noreferrer" target="_blank">www.kamailioworld.com</a><br>
          Kamailio Advanced Training - Mar 25-27, 2019, in Washington,
          DC, USA -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
          <br>
          <br>
          _______________________________________________<br>
          Kamailio (SER) - Users Mailing List<br>
          <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
          <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
        </blockquote>
      </div>
      <br clear="all">
      <div><br>
      </div>
      -- <br>
      <div dir="ltr" class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076gmail_signature"><img src="http://sip.antisip.com/am48.png">Antisip
        - <a href="http://www.antisip.com" target="_blank">http://www.antisip.com</a><br>
      </div>
    </blockquote>
    <pre class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076moz-signature" cols="72">-- 
Daniel-Constantin Mierla -- <a class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a>
<a class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" target="_blank">www.twitter.com/miconda</a> -- <a class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" target="_blank">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076moz-txt-link-abbreviated" href="http://www.kamailioworld.com" target="_blank">www.kamailioworld.com</a>
Kamailio Advanced Training - Mar 25-27, 2019, in Washington, DC, USA -- <a class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail-m_-2290452773994773076moz-txt-link-abbreviated" href="http://www.asipto.com" target="_blank">www.asipto.com</a></pre>
  </div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-600591059583487172gmail-m_-366874485836619355gmail_signature"><img src="http://sip.antisip.com/am48.png">Antisip - <a href="http://www.antisip.com" target="_blank">http://www.antisip.com</a><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div>
</blockquote></div>