<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello Aymeric,</p>
<p>would you be able to test with tls module compiled against libssl
1.1 and using the pre-loaded shared object workaround?</p>
<p> * <a
href="https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared">https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/openssl_mutex_shared</a></p>
<p>You should be able to use it with any version, no need to test
with kamailio master branch.</p>
<p>Just clone the master branch, then:</p>
<p>cd src/modules/tls/utils/openssl_mutex_shared</p>
<p>make</p>
<p>Either from there or copy openssl_mutex_shared.so to a location
you want, then pre-load it before starting your version of
Kamailio.</p>
<p>The README.md in the folder has some more details.</p>
<p>I would like to have some validation that it works fine before
approaching this topic with libssl project to allow to init the
locks with shared process option.</p>
<p>Thanks,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 26.03.19 16:18, Daniel-Constantin
Mierla wrote:<br>
</div>
<blockquote type="cite"
cite="mid:f1138e7b-0dd6-da7a-0482-dd83f68b67ec@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Hello,</p>
<p>yep, locking there is expected, as listing the tls connections
wait for no other processes to change the content of internal
tls connection structures. So it is a side effect of
libssl/libcrypto getting stuck and the other processing waiting
for it to move one. I have the Kamailio training in USA these
days, so the trip and schedule of the day didn't allow me to
look more at the libsll/libcrypto code in order to find a
solution here. It is a high priority in my list, as I get time
during the next days.</p>
<p>Cheers,<br>
Daniel<br>
</p>
<div class="moz-cite-prefix">On 26.03.19 15:55, Aymeric Moizard
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CALM7LKN6KBeA9+_WdJOm2njr9ff1zA4CcCTX4MiCb9xtWKkOWg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi All,
<div><br>
</div>
<div>I was debugging a TCP issue (most probably, I
may start a thread for this question).</div>
<div><br>
</div>
<div>I was trying to get some info for TCP and
TLS.</div>
<div><br>
</div>
<div>I typed:</div>
<div>$> sudo kamctl rpc tls.list<br>
</div>
<div><br>
</div>
<div>And waited for a while.... until... I
realized that my User-Agent, connected with TCP
was not able to register any more. I think the
rpc command has introduced something wrong.</div>
<div><br>
</div>
<div>The device can successfully "connect", send
the REGISTER over the established TCP
connection. The REGISTER do not appear in the
logs any more, I don't see any traffic for TCP
any more. So the behavior is the same as I had
before: TCP and TLS are both not working and UDP
is still working fine.</div>
<div><br>
</div>
<div>kamctl do not work any more... so kamctl trap
do not work...</div>
<div><br>
</div>
<div>I have been able to type.. manually... for
(all?) kamailio threads:</div>
<div><br>
</div>
<div>gdb /usr/sbin/kamailio 16500 -batch
--eval-command="bt full" >>
kamailio-trap-tcp-down.txt<br>
</div>
<div><br>
</div>
<div>I'm temporarly puting the backtrace I have
here:</div>
<div><a
href="https://sip.antisip.com/kamailio-trap-tcp-down.txt"
moz-do-not-send="true">https://sip.antisip.com/kamailio-trap-tcp-down.txt</a><br>
</div>
<div><br>
</div>
<div>You can see a thread stuck on the json
command line: "<span style="color:rgb(0,0,0);white-space:pre-wrap">tls_list"</span></div>
<div><span style="color:rgb(0,0,0);white-space:pre-wrap">And many other waiting on </span><font
color="#000000"><span style="white-space:pre-wrap">CRYPTO_THREAD_write_lock</span></font></div>
<div><span style="color:rgb(0,0,0);white-space:pre-wrap">
</span></div>
<div><span style="color:rgb(0,0,0);white-space:pre-wrap">? might be related to: </span><font
color="#000000"><span style="white-space:pre-wrap"><a href="https://github.com/openssl/openssl/issues/5376" moz-do-not-send="true">https://github.com/openssl/openssl/issues/5376</a></span></font></div>
<div><font color="#000000"><span style="white-space:pre-wrap">
</span></font></div>
<div>SIDE NOTE:</div>
<div>Right before I was typing the last gdb
command for the last thread, kamailio</div>
<div>has crashed: This was around 5 minutes after
the dead lock started.</div>
<div><br>
</div>
<div>
<div>Mar 26 14:47:11 sip kamailio[16493]: ERROR:
<core> [core/tcp_main.c:2561]:
tcpconn_do_send(): failed to send on
0x7ff8dfc2fdc8 (91.121.30.149:5061-><a
href="http://62.210.97.21:49351"
moz-do-not-send="true">62.210.97.21:49351</a>):
Broken pipe (32)</div>
<div>Mar 26 14:47:11 sip kamailio[16493]: ERROR:
<core> [core/tcp_read.c:1505]:
tcp_read_req(): ERROR: tcp_read_req: error
reading - c: 0x7ff8dfc2fdc8 r: 0x7ff8dfc2fe48
(-1)</div>
<div>Mar 26 14:47:11 sip kamailio[16493]:
WARNING: <core> [core/tcp_read.c:1848]:
handle_io(): F_TCPCONN connection marked as
bad: 0x7ff8dfa6a408 id 846 refcnt 3</div>
<div>Mar 26 14:47:11 sip kamailio[16371]: ALERT:
<core> [main.c:755]: handle_sigs():
child process 16374 exited by a signal 11</div>
<div>Mar 26 14:47:11 sip kamailio[16371]: ALERT:
<core> [main.c:758]: handle_sigs(): core
was not generated</div>
<div>Mar 26 14:47:11 sip kamailio[16371]: INFO:
<core> [main.c:781]: handle_sigs():
terminating due to SIGCHLD</div>
<div>Mar 26 14:47:11 sip kamailio[16493]: INFO:
<core> [main.c:836]: sig_usr(): signal
15 received</div>
<div>Mar 26 14:47:11 sip kamailio[16500]: INFO:
<core> [main.c:836]: sig_usr(): signal
15 received</div>
<div>Mar 26 14:47:11 sip kamailio[16479]: INFO:
<core> [main.c:836]: sig_usr(): signal
15 received</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Unfortunalty, even if I did my best to setup
my service to generate a core on crash, I still
have "core was not generated".... (debian
stretch)</div>
<br class="gmail-Apple-interchange-newline">
<div>Tks for reading!</div>
<div>Regards</div>
<div>Aymeric</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Le mar. 26 mars 2019
à 14:11, Kristijan Vrban <<a
href="mailto:vrban.lkml@gmail.com" moz-do-not-send="true">vrban.lkml@gmail.com</a>>
a écrit :<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">And again one more kamctl
trap file where<br>
<br>
set_reply_no_connect was set.<br>
<br>
Am Di., 26. März 2019 um 08:53 Uhr schrieb Kristijan Vrban<br>
<<a href="mailto:vrban.lkml@gmail.com" target="_blank"
moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
><br>
> Attached also the output of kamctl trap<br>
><br>
> Am Di., 26. März 2019 um 08:42 Uhr schrieb Kristijan
Vrban<br>
> <<a href="mailto:vrban.lkml@gmail.com"
target="_blank" moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
> ><br>
> > > Have you done a test with tools such as sipp,
or was this happening<br>
> > > after a while, with usual phones registering?<br>
> ><br>
> > Usual variety of devices registering via TLS. But
i can not exclude<br>
> > that some devices displaying behavioural problems.<br>
> ><br>
> > > Can you list the tcp connections and see if
they are listed?<br>
> > > kamctl tcp core.tcp_list<br>
> ><br>
> > Need Kex module for that? So i can deliver next
time. But when i do<br>
> > "lsof -u kamailio |grep TCP"<br>
> > i get a long list of more then 2000 lines with:<br>
> ><br>
> > ...<br>
> > kamailio 37561 kamailio 2105u sock
0,9 0t0<br>
> > 27856287 protocol: TCP<br>
> > kamailio 37561 kamailio 2106u sock
0,9 0t0<br>
> > 27856305 protocol: TCP<br>
> > kamailio 37561 kamailio 2107u sock
0,9 0t0<br>
> > 27856306 protocol: TCP<br>
> > kamailio 37561 kamailio 2108u sock
0,9 0t0<br>
> > 27856914 protocol: TCP<br>
> > ...<br>
> ><br>
> > So about the time Kamailio created a lot of socket
in the TCP domain,<br>
> > but which are not bound to any port (eg via
connect(2) or listen(2) or<br>
> > bind(2))<br>
> > Until we get to the maximum number of 2048
connections.<br>
> ><br>
> > Best<br>
> > Kristijan<br>
> ><br>
> > Am Mo., 25. März 2019 um 14:27 Uhr schrieb
Daniel-Constantin Mierla<br>
> > <<a href="mailto:miconda@gmail.com"
target="_blank" moz-do-not-send="true">miconda@gmail.com</a>>:<br>
> > ><br>
> > > Have you done a test with tools such as sipp,
or was this happening<br>
> > > after a while, with usual phones registering?<br>
> > ><br>
> > > Can you list the tcp connections and see if
they are listed?<br>
> > ><br>
> > > kamctl tcp core.tcp_list<br>
> > ><br>
> > > Cheers,<br>
> > > Daniel<br>
> > ><br>
> > > On 25.03.19 08:03, Kristijan Vrban wrote:<br>
> > > >> The solution here is to use
set_reply_no_connect()<br>
> > > > implemented it. Now the issue has
shifted to:<br>
> > > ><br>
> > > > ERROR: <core>
[core/tcp_main.c:3959]: handle_new_connect(): maximum<br>
> > > > number of connections exceeded:
2048/2048<br>
> > > ><br>
> > > > But not a single TCP connection is
active between Kamailio and any<br>
> > > > device. Seems this counter for maximum
number of connections<br>
> > > > now has an issue?<br>
> > > ><br>
> > > > Kristijan<br>
> > > ><br>
> > > > Am Mi., 20. März 2019 um 15:07 Uhr
schrieb Daniel-Constantin Mierla<br>
> > > > <<a href="mailto:miconda@gmail.com"
target="_blank" moz-do-not-send="true">miconda@gmail.com</a>>:<br>
> > > >> Hello,<br>
> > > >><br>
> > > >> based on the trap output I think I
could figure out what happened there.<br>
> > > >><br>
> > > >> You have tcp_children to very low
value (1 or so), the problem is not<br>
> > > >> actually that one, but the fact that
the connection to upstream (the<br>
> > > >> device/app sending the request) was
closed after receiving the request<br>
> > > >> and routing of the reply gets stuck
in the way of:<br>
> > > >><br>
> > > >> - a reply is received and has to
be forwarded<br>
> > > >> - connection was lost, so Kamailio
tries to establish a new one, but<br>
> > > >> takes time till fails because the
upstream is behind nat or so based on<br>
> > > >> the via header:<br>
> > > >><br>
> > > >> Via: SIP/2.0/TLS<br>
> > > >>
10.1.0.4:10002;rport=55229;received=13.94.188.218;branch=z9hG4bK-3336-7f2927bfd703ae907348edff3611bfc9<br>
> > > >><br>
> > > >> - the reply is retransmitted and
gets to another worker, which tries<br>
> > > >> to forward it again, but discovers a
connection structure for that<br>
> > > >> destination exists (created by
previous reply worker) and now waits for<br>
> > > >> the connection to be released (or
better said, for the mutex on writing<br>
> > > >> buffer to be unlocked)<br>
> > > >><br>
> > > >> - as the second reply waits, there
can be other retransmissions of the<br>
> > > >> reply ending up in other workers
stuck on waiting for the mutex of the<br>
> > > >> connection write buffer<br>
> > > >><br>
> > > >> The solution here is to use
set_reply_no_connect() -- you can put it<br>
> > > >> first in request_route block. I
think this would be a good addition to<br>
> > > >> the default configuration file as
well, IMO, the sip server should not<br>
> > > >> connect for sending replies and
should do it also for requests that go<br>
> > > >> behind nat.<br>
> > > >><br>
> > > >> Cheers,<br>
> > > >> Daniel<br>
> > > >><br>
> > > >> On 19.03.19 10:53, Kristijan Vrban
wrote:<br>
> > > >>> So i had again the situation.
But this time, incoming udp was<br>
> > > >>> affected. Kamailio was sending
out OPTIONS (via dispatcher module) to<br>
> > > >>> a group of asterisk machines<br>
> > > >>> but the 200 OK reply to the
OPTIONS where not processed, so the<br>
> > > >>> dispatcher module set all
asterisk to inactive, even though they<br>
> > > >>> replied 200 OK<br>
> > > >>><br>
> > > >>> Attached the output of kamctl
trap during the situation. Hope there is<br>
> > > >>> any useful in it. Because after
"kamctl trap" it was working again<br>
> > > >>> without kamailio restart.<br>
> > > >>><br>
> > > >>> Best<br>
> > > >>> Kristijan<br>
> > > >>><br>
> > > >>> Am Mo., 18. März 2019 um 12:27
Uhr schrieb Daniel-Constantin Mierla<br>
> > > >>> <<a
href="mailto:miconda@gmail.com" target="_blank"
moz-do-not-send="true">miconda@gmail.com</a>>:<br>
> > > >>>> Hello,<br>
> > > >>>><br>
> > > >>>> setting tcp_children=1 is
not a god option for scallability, practically<br>
> > > >>>> you set kamailio to process
a single tcp message at one time, on high<br>
> > > >>>> traffic, that won't work
well.<br>
> > > >>>><br>
> > > >>>> Maybe try to set
tcp_children to 2 or 4, that should make an eventual<br>
> > > >>>> race appear faster.<br>
> > > >>>><br>
> > > >>>> Regarding the pid, if it is
an outgoing connection, then it can be<br>
> > > >>>> created by any worker
process, including a UDP worker, if that was the<br>
> > > >>>> one receiving the sip
message over udp and sends it out via tcp.<br>
> > > >>>><br>
> > > >>>> Cheers,<br>
> > > >>>> Daniel<br>
> > > >>>><br>
> > > >>>> On 18.03.19 10:09, Kristijan
Vrban wrote:<br>
> > > >>>>> Hi Daniel,<br>
> > > >>>>><br>
> > > >>>>> for testing, i now had
set: "tcp_children=1" and so far this issue did not occur<br>
> > > >>>>> ever since. So now value
to provide for "kamctl trap" yet.<br>
> > > >>>>><br>
> > > >>>>> "kamctl ps" show this
two process to handle tcp:<br>
> > > >>>>><br>
> > > >>>>> ...<br>
> > > >>>>> }, {<br>
> > > >>>>> "IDX": 25,<br>
> > > >>>>> "PID": 71929,<br>
> > > >>>>> "DSC": "tcp
receiver (generic) child=0"<br>
> > > >>>>> }, {<br>
> > > >>>>> "IDX": 26,<br>
> > > >>>>> "PID": 71933,<br>
> > > >>>>> "DSC": "tcp main
process"<br>
> > > >>>>> }<br>
> > > >>>>> ...<br>
> > > >>>>><br>
> > > >>>>><br>
> > > >>>>> Ok, but then is was
wondering to see a TCP connection on a udp receiver child:<br>
> > > >>>>><br>
> > > >>>>><br>
> > > >>>>> netstat -ntp |grep 5061<br>
> > > >>>>><br>
> > > >>>>> ...<br>
> > > >>>>> tcp 0 0 <a
href="http://172.17.217.10:5061" rel="noreferrer"
target="_blank" moz-do-not-send="true">172.17.217.10:5061</a>
<a href="http://195.70.114.125:18252" rel="noreferrer"
target="_blank" moz-do-not-send="true">195.70.114.125:18252</a><br>
> > > >>>>> ESTABLISHED
71895/kamailio<br>
> > > >>>>> ...<br>
> > > >>>>><br>
> > > >>>>> An pid 71895 is:<br>
> > > >>>>><br>
> > > >>>>> }, {<br>
> > > >>>>> "IDX": 3,<br>
> > > >>>>> "PID": 71895,<br>
> > > >>>>> "DSC": "udp
receiver child=2 sock=<a href="http://127.0.0.1:5060"
rel="noreferrer" target="_blank" moz-do-not-send="true">127.0.0.1:5060</a>"<br>
> > > >>>>> }, {<br>
> > > >>>>><br>
> > > >>>>><br>
> > > >>>>><br>
> > > >>>>> And if i look into it
via "lsof -p 71895" (the udp receiver child)<br>
> > > >>>>><br>
> > > >>>>> ...<br>
> > > >>>>> kamailio 71895 kamailio
14u sock 0,9 0t0<br>
> > > >>>>> 8856085 protocol: TCP<br>
> > > >>>>> kamailio 71895 kamailio
15u sock 0,9 0t0<br>
> > > >>>>> 8886886 protocol: TCP<br>
> > > >>>>> kamailio 71895 kamailio
16u sock 0,9 0t0<br>
> > > >>>>> 8854886 protocol: TCP<br>
> > > >>>>> kamailio 71895 kamailio
17u sock 0,9 0t0<br>
> > > >>>>> 8828915 protocol: TCP<br>
> > > >>>>> kamailio 71895 kamailio
18u unix 0x000000005f73cb91 0t0<br>
> > > >>>>> 1680314 type=DGRAM<br>
> > > >>>>> kamailio 71895 kamailio
19u IPv4 1846523 0t0<br>
> > > >>>>> TCP
kamailio-preview:sip-tls->XXX:18252 (ESTABLISHED)<br>
> > > >>>>> kamailio 71895 kamailio
20u sock 0,9 0t0<br>
> > > >>>>> 8887192 protocol: TCP<br>
> > > >>>>> kamailio 71895 kamailio
21u sock 0,9 0t0<br>
> > > >>>>> 8813634 protocol: TCP<br>
> > > >>>>> kamailio 71895 kamailio
22u unix 0x00000000c19bd102 0t0<br>
> > > >>>>> 1681407 type=STREAM<br>
> > > >>>>> kamailio 71895 kamailio
23u sock 0,9 0t0<br>
> > > >>>>> 8850488 protocol: TCP<br>
> > > >>>>> ...<br>
> > > >>>>><br>
> > > >>>>> Not only the ESTABLISHED
TCP session. But also this empty sockets<br>
> > > >>>>> "protocol: TCP"<br>
> > > >>>>> What are they doing
there in the udp receiver? Is that how it's supposed to be?<br>
> > > >>>>><br>
> > > >>>>> Kristijan<br>
> > > >>>>><br>
> > > >>>>> Am Do., 14. März 2019 um
14:48 Uhr schrieb Daniel-Constantin Mierla<br>
> > > >>>>> <<a
href="mailto:miconda@gmail.com" target="_blank"
moz-do-not-send="true">miconda@gmail.com</a>>:<br>
> > > >>>>>> Can you get file
written by `kamctl trap`? It should have the backtrace<br>
> > > >>>>>> for all kamailio
processes. You need latest kamailio 5.2.<br>
> > > >>>>>><br>
> > > >>>>>> Also, get the output
for: kamctl ps<br>
> > > >>>>>><br>
> > > >>>>>> Cheers,<br>
> > > >>>>>> Daniel<br>
> > > >>>>>><br>
> > > >>>>>> On 14.03.19 13:52,
Kristijan Vrban wrote:<br>
> > > >>>>>>> When i attach
via gdb to one of the tcp worker, i see this:<br>
> > > >>>>>>><br>
> > > >>>>>>> (gdb) bt<br>
> > > >>>>>>> #0
0x00007fdaf4d14470 in futex_wait (private=<optimized
out>,<br>
> > > >>>>>>> expected=1,
futex_word=0x7fdaeca92f8c) at<br>
> > > >>>>>>>
../sysdeps/unix/sysv/linux/futex-internal.h:61<br>
> > > >>>>>>> #1
futex_wait_simple (private=<optimized out>,
expected=1,<br>
> > > >>>>>>>
futex_word=0x7fdaeca92f8c) at
../sysdeps/nptl/futex-internal.h:135<br>
> > > >>>>>>> #2
__pthread_rwlock_wrlock_slow (rwlock=0x7fdaeca92f80) at<br>
> > > >>>>>>>
pthread_rwlock_wrlock.c:67<br>
> > > >>>>>>> #3
0x00007fdaf0912ee9 in CRYPTO_THREAD_write_lock () from<br>
> > > >>>>>>>
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
> > > >>>>>>> #4
0x00007fdaf08e1c08 in ?? () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
> > > >>>>>>> #5
0x00007fdaf08a6f69 in ?? () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
> > > >>>>>>> #6
0x00007fdaf08b36c7 in EVP_CIPHER_CTX_ctrl () from<br>
> > > >>>>>>>
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
> > > >>>>>>> #7
0x00007fdaf0c31144 in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
> > > >>>>>>> #8
0x00007fdaf0c2bddb in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
> > > >>>>>>> #9
0x00007fdaf0c22858 in ?? () from
/usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
> > > >>>>>>> #10
0x00007fdaf0c1af61 in SSL_do_handshake () from<br>
> > > >>>>>>>
/usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
> > > >>>>>>> #11
0x00007fdaf0e8d31b in tls_accept (c=0x7fdaed26fa98,<br>
> > > >>>>>>>
error=0x7ffffe2a2df0) at tls_server.c:422<br>
> > > >>>>>>> #12
0x00007fdaf0e96a1b in tls_read_f (c=0x7fdaed26fa98,<br>
> > > >>>>>>>
flags=0x7ffffe2c318c) at tls_server.c:1116<br>
> > > >>>>>>> #13
0x0000556ead5e7c46 in tcp_read_headers (c=0x7fdaed26fa98,<br>
> > > >>>>>>>
read_flags=0x7ffffe2c318c) at core/tcp_read.c:469<br>
> > > >>>>>>> #14
0x0000556ead5ef9cb in tcp_read_req (con=0x7fdaed26fa98,<br>
> > > >>>>>>>
bytes_read=0x7ffffe2c3184, read_flags=0x7ffffe2c318c) at<br>
> > > >>>>>>>
core/tcp_read.c:1496<br>
> > > >>>>>>> #15
0x0000556ead5f575f in handle_io (fm=0x7fdaf597aa98,
events=1,<br>
> > > >>>>>>> idx=-1) at
core/tcp_read.c:1862<br>
> > > >>>>>>> #16
0x0000556ead5e2053 in io_wait_loop_epoll (h=0x556eadaaeec0
<io_w>,<br>
> > > >>>>>>> t=2, repeat=0)
at core/io_wait.h:1065<br>
> > > >>>>>>> #17
0x0000556ead5f6b35 in tcp_receive_loop (unix_sock=49) at<br>
> > > >>>>>>>
core/tcp_read.c:1974<br>
> > > >>>>>>> #18
0x0000556ead4c8e24 in tcp_init_children () at
core/tcp_main.c:4853<br>
> > > >>>>>>> #19
0x0000556ead3c352a in main_loop () at main.c:1735<br>
> > > >>>>>>> #20
0x0000556ead3ca5f8 in main (argc=13, argv=0x7ffffe2c3828) at
main.c:2675<br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>><br>
> > > >>>>>>> Am Do., 14. März
2019 um 13:41 Uhr schrieb Kristijan Vrban<br>
> > > >>>>>>> <<a
href="mailto:vrban.lkml@gmail.com" target="_blank"
moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
> > > >>>>>>>> Hi, with
full debug is see this in log for every incoming TCP SIP
request:<br>
> > > >>>>>>>><br>
> > > >>>>>>>> Mar 14
12:10:15 kamailio-preview /usr/sbin/kamailio[17940]: DEBUG:<br>
> > > >>>>>>>> <core>
[core/tcp_main.c:3871]: send2child(): WARNING: no free tcp<br>
> > > >>>>>>>> receiver,
connection passed to the least busy one (105)<br>
> > > >>>>>>>> Mar 14
12:10:15 kamailio-preview /usr/sbin/kamailio[17940]: DEBUG:<br>
> > > >>>>>>>> <core>
[core/tcp_main.c:3875]: send2child(): selected tcp worker 2<br>
> > > >>>>>>>> 27(17937)
for activity on [tls:<a href="http://172.17.217.10:5061"
rel="noreferrer" target="_blank" moz-do-not-send="true">172.17.217.10:5061</a>],
0x7fdaeda8f928<br>
> > > >>>>>>>><br>
> > > >>>>>>>> So the
Kamailio TCP process is working, and received TCP traffic.
But<br>
> > > >>>>>>>> the tcp
workers are somehow busy.<br>
> > > >>>>>>>><br>
> > > >>>>>>>> When i
attach via strace to the TCP worker, i do not see any
activity. Just:<br>
> > > >>>>>>>><br>
> > > >>>>>>>>
futex(0x7fdaeca92f8c, FUTEX_WAIT_PRIVATE, 1, NULL<br>
> > > >>>>>>>><br>
> > > >>>>>>>> and nothing,
even when i see the main tcp process choose this worker
process.<br>
> > > >>>>>>>><br>
> > > >>>>>>>> Kristijan<br>
> > > >>>>>>>><br>
> > > >>>>>>>> Am Mi., 27.
Feb. 2019 um 15:14 Uhr schrieb Kristijan Vrban<br>
> > > >>>>>>>> <<a
href="mailto:vrban.lkml@gmail.com" target="_blank"
moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
> > > >>>>>>>>> first of
all thanks for the feedback. i prepared our system now to
run<br>
> > > >>>>>>>>> with
debug=3<br>
> > > >>>>>>>>> I hope
to see more then then.<br>
> > > >>>>>>>>><br>
> > > >>>>>>>>> Am Mi.,
27. Feb. 2019 um 11:53 Uhr schrieb Kristijan Vrban<br>
> > > >>>>>>>>> <<a
href="mailto:vrban.lkml@gmail.com" target="_blank"
moz-do-not-send="true">vrban.lkml@gmail.com</a>>:<br>
> > > >>>>>>>>>> Hi
kamailios,<br>
> > > >>>>>>>>>><br>
> > > >>>>>>>>>> i
have a creepy situation with v5.2.1 stable Kamilio. After a
day or<br>
> > > >>>>>>>>>> so,
Kamailio stop to process incoming SIP traffic via TCP. The<br>
> > > >>>>>>>>>>
incoming TCP network packages get TCP-ACK from the OS
(Debian 9,<br>
> > > >>>>>>>>>>
4.18.0-15-generic-Linux) but Kamailio does not show any
processing for<br>
> > > >>>>>>>>>> the
SIP-Traffic incoming via TCP. No logs, nothing. While
traffic via<br>
> > > >>>>>>>>>> UDP
is working just totally fine.<br>
> > > >>>>>>>>>><br>
> > > >>>>>>>>>> When
i look via command "netstat -ntp" is see, that the Recv-Q
get<br>
> > > >>>>>>>>>>
bigger and bigger. e.g.:<br>
> > > >>>>>>>>>><br>
> > > >>>>>>>>>>
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program<br>
> > > >>>>>>>>>> name
tcp 4566 0 <a href="http://172.17.217.12:5060"
rel="noreferrer" target="_blank" moz-do-not-send="true">172.17.217.12:5060</a>
xxx.xxx.xxx.xxx:57252 ESTABLISHED<br>
> > > >>>>>>>>>>
31347/kamailio<br>
> > > >>>>>>>>>><br>
> > > >>>>>>>>>>
After Kamailio restart, all is working fine again for a day.
We have<br>
> > > >>>>>>>>>>
maybe 10-20 devices online via TCP and low call volume (1-2
call per<br>
> > > >>>>>>>>>>
minute). The only settings for tcp we have is
"tcp_delayed_ack=no"<br>
> > > >>>>>>>>>><br>
> > > >>>>>>>>>> How
to could we debug this situation? Again, no error, no
warings in<br>
> > > >>>>>>>>>> the
log. Just nothing.<br>
> > > >>>>>>>>>><br>
> > > >>>>>>>>>>
Kristijan<br>
> > > >>>>>>>
_______________________________________________<br>
> > > >>>>>>> Kamailio (SER) -
Users Mailing List<br>
> > > >>>>>>> <a
href="mailto:sr-users@lists.kamailio.org" target="_blank"
moz-do-not-send="true">sr-users@lists.kamailio.org</a><br>
> > > >>>>>>> <a
href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
> > > >>>>>> --<br>
> > > >>>>>> Daniel-Constantin
Mierla -- <a href="http://www.asipto.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.asipto.com</a><br>
> > > >>>>>> <a
href="http://www.twitter.com/miconda" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a>
-- <a href="http://www.linkedin.com/in/miconda"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a><br>
> > > >>>>>> Kamailio World
Conference - May 6-8, 2019 -- <a
href="http://www.kamailioworld.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.kamailioworld.com</a><br>
> > > >>>>>> Kamailio Advanced
Training - Mar 25-27, 2019, in Washington, DC, USA -- <a
href="http://www.asipto.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.asipto.com</a><br>
> > > >>>>>><br>
> > > >>>>>
_______________________________________________<br>
> > > >>>>> Kamailio (SER) - Users
Mailing List<br>
> > > >>>>> <a
href="mailto:sr-users@lists.kamailio.org" target="_blank"
moz-do-not-send="true">sr-users@lists.kamailio.org</a><br>
> > > >>>>> <a
href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
> > > >>>> --<br>
> > > >>>> Daniel-Constantin Mierla --
<a href="http://www.asipto.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.asipto.com</a><br>
> > > >>>> <a
href="http://www.twitter.com/miconda" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a>
-- <a href="http://www.linkedin.com/in/miconda"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a><br>
> > > >>>> Kamailio World Conference -
May 6-8, 2019 -- <a href="http://www.kamailioworld.com"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.kamailioworld.com</a><br>
> > > >>>> Kamailio Advanced Training -
Mar 25-27, 2019, in Washington, DC, USA -- <a
href="http://www.asipto.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.asipto.com</a><br>
> > > >>>><br>
> > > >> --<br>
> > > >> Daniel-Constantin Mierla -- <a
href="http://www.asipto.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.asipto.com</a><br>
> > > >> <a
href="http://www.twitter.com/miconda" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a>
-- <a href="http://www.linkedin.com/in/miconda"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a><br>
> > > >> Kamailio World Conference - May 6-8,
2019 -- <a href="http://www.kamailioworld.com"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.kamailioworld.com</a><br>
> > > >> Kamailio Advanced Training - Mar
25-27, 2019, in Washington, DC, USA -- <a
href="http://www.asipto.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.asipto.com</a><br>
> > > >><br>
> > > --<br>
> > > Daniel-Constantin Mierla -- <a
href="http://www.asipto.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.asipto.com</a><br>
> > > <a href="http://www.twitter.com/miconda"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.twitter.com/miconda</a>
-- <a href="http://www.linkedin.com/in/miconda"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.linkedin.com/in/miconda</a><br>
> > > Kamailio World Conference - May 6-8, 2019 --
<a href="http://www.kamailioworld.com" rel="noreferrer"
target="_blank" moz-do-not-send="true">www.kamailioworld.com</a><br>
> > > Kamailio Advanced Training - Mar 25-27, 2019,
in Washington, DC, USA -- <a href="http://www.asipto.com"
rel="noreferrer" target="_blank" moz-do-not-send="true">www.asipto.com</a><br>
> > ><br>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank"
moz-do-not-send="true">sr-users@lists.kamailio.org</a><br>
<a
href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature"><img
src="http://sip.antisip.com/am48.png" moz-do-not-send="true">Antisip
- <a href="http://www.antisip.com" target="_blank"
moz-do-not-send="true">http://www.antisip.com</a><br>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com" moz-do-not-send="true">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda" moz-do-not-send="true">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda" moz-do-not-send="true">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com" moz-do-not-send="true">www.kamailioworld.com</a>
Kamailio Advanced Training - Mar 25-27, 2019, in Washington, DC, USA -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com" moz-do-not-send="true">www.asipto.com</a></pre>
</blockquote>
<pre class="moz-signature" cols="72">--
Daniel-Constantin Mierla -- <a class="moz-txt-link-abbreviated" href="http://www.asipto.com">www.asipto.com</a>
<a class="moz-txt-link-abbreviated" href="http://www.twitter.com/miconda">www.twitter.com/miconda</a> -- <a class="moz-txt-link-abbreviated" href="http://www.linkedin.com/in/miconda">www.linkedin.com/in/miconda</a>
Kamailio World Conference - May 6-8, 2019 -- <a class="moz-txt-link-abbreviated" href="http://www.kamailioworld.com">www.kamailioworld.com</a></pre>
</body>
</html>