<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi List,<div><br></div><div>I want to share that I also met this issue last week with my kamailio 5.2.2.</div><div><br></div><div>As far as I was able to see, SIP application were able to "connect()"</div><div>with TCP, but my logs wasn't reporting any of the SIP message received</div><div>with TCP.</div><div><br></div><div>I have an pike right before an xlog showing every incoming request. However</div><div>I suspect the issue was not related to pike module. The log didn't showed unusual</div><div>number of blocked traffic.</div><div><br></div><div>I'm almost sure I haven't reached any ulimit restrictions.</div><div>I have many TCP, UDP childreen...</div><div>Server was not under high load</div><div>Nothing unusual.</div><div><br></div><div>I'm running the default build for debian stretch from here:</div><div> <a href="http://deb.kamailio.org/kamailio52">http://deb.kamailio.org/kamailio52</a> stretch</div><div><br></div><div>And unfortunatly, I had some tiny pressure to restart the service so I was</div><div>not able to get deeper into the issue.</div><div><br></div><div>If I'm correct, I will certainly improve much things by using "set_reply_no_connect()".</div><div>I have added it and restarted!</div><div>(Tks Daniel for this tip!)</div><div><br></div><div>I have been looking at issue reported here:</div>"Kamailio 5.0.2 suddenly stops processing traffic, then generates a core when restarting."</div><div dir="ltr"><a href="https://github.com/kamailio/kamailio/issues/1172">https://github.com/kamailio/kamailio/issues/1172</a><div><br></div><div>I have to say that I do have libssl1.1.</div><div>And I do have crash when I restart my kamailio. (even when I simply restart after a configuration modification)</div><div><br></div><div><div>Mar 21 18:28:50 sip kamailio[19222]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received</div><div>Mar 21 18:28:50 sip kamailio[19175]: ERROR: ctl [ctl.c:390]: mod_destroy(): ERROR: ctl: could not delete unix socket /var/run/kamailio/kamailio_ctl: Permission denied (13)</div><div><br></div><div>[... one minute without nothing...]</div><div><br></div><div>Mar 21 18:29:42 sip kamailio[19175]: ERROR: jsonrpcs [jsonrpcs_fifo.c:599]: jsonrpc_fifo_destroy(): FIFO stat failed: Permission denied</div><div>Mar 21 18:29:42 sip kamailio[19175]: ERROR: jsonrpcs [jsonrpcs_sock.c:516]: jsonrpc_dgram_destroy(): socket stat failed: Permission denied</div><div>Mar 21 18:29:50 sip kamailio[19175]: CRITICAL: <core> [main.c:662]: sig_alarm_abort(): shutdown timeout triggered, dying...</div></div><div><br></div><div>As the 1172 issue is closed, should I expect kamailio to still have trouble with libssl1.1?</div><div><br></div><div>I just restarted again my service (to see if it restart better after 30 minutes only instead of a week)</div><div><br></div><div>Mar 21 19:07:30 sip kamailio[28737]: INFO: <core> [main.c:836]: sig_usr(): signal 15 received<br></div><div><div>Mar 21 19:07:31 sip kamailio[28671]: ERROR: ctl [ctl.c:390]: mod_destroy(): ERROR: ctl: could not delete unix socket /var/run/kamailio/kamailio_ctl: Permission denied (13)</div><div><br></div><div>[... one minute without nothing...]</div><br class="gmail-Apple-interchange-newline"><div>Mar 21 19:08:30 sip kamailio[28671]: CRITICAL: <core> [main.c:662]: sig_alarm_abort(): shutdown timeout triggered, dying...</div></div><div><br></div><div>Still not able to restart in a clean way!<br></div><div>Tks!</div><div>Regards</div><div>Aymeric</div><div><br></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le mer. 20 mars 2019 à 15:08, Daniel-Constantin Mierla <<a href="mailto:miconda@gmail.com">miconda@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
based on the trap output I think I could figure out what happened there.<br>
<br>
You have tcp_children to very low value (1 or so), the problem is not<br>
actually that one, but the fact that the connection to upstream (the<br>
device/app sending the request) was closed after receiving the request<br>
and routing of the reply gets stuck in the way of:<br>
<br>
- a reply is received and has to be forwarded<br>
- connection was lost, so Kamailio tries to establish a new one, but<br>
takes time till fails because the upstream is behind nat or so based on<br>
the via header:<br>
<br>
Via: SIP/2.0/TLS<br>
10.1.0.4:10002;rport=55229;received=13.94.188.218;branch=z9hG4bK-3336-7f2927bfd703ae907348edff3611bfc9<br>
<br>
- the reply is retransmitted and gets to another worker, which tries<br>
to forward it again, but discovers a connection structure for that<br>
destination exists (created by previous reply worker) and now waits for<br>
the connection to be released (or better said, for the mutex on writing<br>
buffer to be unlocked)<br>
<br>
- as the second reply waits, there can be other retransmissions of the<br>
reply ending up in other workers stuck on waiting for the mutex of the<br>
connection write buffer<br>
<br>
The solution here is to use set_reply_no_connect() -- you can put it<br>
first in request_route block. I think this would be a good addition to<br>
the default configuration file as well, IMO, the sip server should not<br>
connect for sending replies and should do it also for requests that go<br>
behind nat.<br>
<br>
Cheers,<br>
Daniel<br>
<br>
On 19.03.19 10:53, Kristijan Vrban wrote:<br>
> So i had again the situation. But this time, incoming udp was<br>
> affected. Kamailio was sending out OPTIONS (via dispatcher module) to<br>
> a group of asterisk machines<br>
> but the 200 OK reply to the OPTIONS where not processed, so the<br>
> dispatcher module set all asterisk to inactive, even though they<br>
> replied 200 OK<br>
><br>
> Attached the output of kamctl trap during the situation. Hope there is<br>
> any useful in it. Because after "kamctl trap" it was working again<br>
> without kamailio restart.<br>
><br>
> Best<br>
> Kristijan<br>
><br>
> Am Mo., 18. März 2019 um 12:27 Uhr schrieb Daniel-Constantin Mierla<br>
> <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>:<br>
>> Hello,<br>
>><br>
>> setting tcp_children=1 is not a god option for scallability, practically<br>
>> you set kamailio to process a single tcp message at one time, on high<br>
>> traffic, that won't work well.<br>
>><br>
>> Maybe try to set tcp_children to 2 or 4, that should make an eventual<br>
>> race appear faster.<br>
>><br>
>> Regarding the pid, if it is an outgoing connection, then it can be<br>
>> created by any worker process, including a UDP worker, if that was the<br>
>> one receiving the sip message over udp and sends it out via tcp.<br>
>><br>
>> Cheers,<br>
>> Daniel<br>
>><br>
>> On 18.03.19 10:09, Kristijan Vrban wrote:<br>
>>> Hi Daniel,<br>
>>><br>
>>> for testing, i now had set: "tcp_children=1" and so far this issue did not occur<br>
>>> ever since. So now value to provide for "kamctl trap" yet.<br>
>>><br>
>>> "kamctl ps" show this two process to handle tcp:<br>
>>><br>
>>> ...<br>
>>> }, {<br>
>>> "IDX": 25,<br>
>>> "PID": 71929,<br>
>>> "DSC": "tcp receiver (generic) child=0"<br>
>>> }, {<br>
>>> "IDX": 26,<br>
>>> "PID": 71933,<br>
>>> "DSC": "tcp main process"<br>
>>> }<br>
>>> ...<br>
>>><br>
>>><br>
>>> Ok, but then is was wondering to see a TCP connection on a udp receiver child:<br>
>>><br>
>>><br>
>>> netstat -ntp |grep 5061<br>
>>><br>
>>> ...<br>
>>> tcp 0 0 <a href="http://172.17.217.10:5061" rel="noreferrer" target="_blank">172.17.217.10:5061</a> <a href="http://195.70.114.125:18252" rel="noreferrer" target="_blank">195.70.114.125:18252</a><br>
>>> ESTABLISHED 71895/kamailio<br>
>>> ...<br>
>>><br>
>>> An pid 71895 is:<br>
>>><br>
>>> }, {<br>
>>> "IDX": 3,<br>
>>> "PID": 71895,<br>
>>> "DSC": "udp receiver child=2 sock=<a href="http://127.0.0.1:5060" rel="noreferrer" target="_blank">127.0.0.1:5060</a>"<br>
>>> }, {<br>
>>><br>
>>><br>
>>><br>
>>> And if i look into it via "lsof -p 71895" (the udp receiver child)<br>
>>><br>
>>> ...<br>
>>> kamailio 71895 kamailio 14u sock 0,9 0t0<br>
>>> 8856085 protocol: TCP<br>
>>> kamailio 71895 kamailio 15u sock 0,9 0t0<br>
>>> 8886886 protocol: TCP<br>
>>> kamailio 71895 kamailio 16u sock 0,9 0t0<br>
>>> 8854886 protocol: TCP<br>
>>> kamailio 71895 kamailio 17u sock 0,9 0t0<br>
>>> 8828915 protocol: TCP<br>
>>> kamailio 71895 kamailio 18u unix 0x000000005f73cb91 0t0<br>
>>> 1680314 type=DGRAM<br>
>>> kamailio 71895 kamailio 19u IPv4 1846523 0t0<br>
>>> TCP kamailio-preview:sip-tls->XXX:18252 (ESTABLISHED)<br>
>>> kamailio 71895 kamailio 20u sock 0,9 0t0<br>
>>> 8887192 protocol: TCP<br>
>>> kamailio 71895 kamailio 21u sock 0,9 0t0<br>
>>> 8813634 protocol: TCP<br>
>>> kamailio 71895 kamailio 22u unix 0x00000000c19bd102 0t0<br>
>>> 1681407 type=STREAM<br>
>>> kamailio 71895 kamailio 23u sock 0,9 0t0<br>
>>> 8850488 protocol: TCP<br>
>>> ...<br>
>>><br>
>>> Not only the ESTABLISHED TCP session. But also this empty sockets<br>
>>> "protocol: TCP"<br>
>>> What are they doing there in the udp receiver? Is that how it's supposed to be?<br>
>>><br>
>>> Kristijan<br>
>>><br>
>>> Am Do., 14. März 2019 um 14:48 Uhr schrieb Daniel-Constantin Mierla<br>
>>> <<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>>:<br>
>>>> Can you get file written by `kamctl trap`? It should have the backtrace<br>
>>>> for all kamailio processes. You need latest kamailio 5.2.<br>
>>>><br>
>>>> Also, get the output for: kamctl ps<br>
>>>><br>
>>>> Cheers,<br>
>>>> Daniel<br>
>>>><br>
>>>> On 14.03.19 13:52, Kristijan Vrban wrote:<br>
>>>>> When i attach via gdb to one of the tcp worker, i see this:<br>
>>>>><br>
>>>>> (gdb) bt<br>
>>>>> #0 0x00007fdaf4d14470 in futex_wait (private=<optimized out>,<br>
>>>>> expected=1, futex_word=0x7fdaeca92f8c) at<br>
>>>>> ../sysdeps/unix/sysv/linux/futex-internal.h:61<br>
>>>>> #1 futex_wait_simple (private=<optimized out>, expected=1,<br>
>>>>> futex_word=0x7fdaeca92f8c) at ../sysdeps/nptl/futex-internal.h:135<br>
>>>>> #2 __pthread_rwlock_wrlock_slow (rwlock=0x7fdaeca92f80) at<br>
>>>>> pthread_rwlock_wrlock.c:67<br>
>>>>> #3 0x00007fdaf0912ee9 in CRYPTO_THREAD_write_lock () from<br>
>>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
>>>>> #4 0x00007fdaf08e1c08 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
>>>>> #5 0x00007fdaf08a6f69 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
>>>>> #6 0x00007fdaf08b36c7 in EVP_CIPHER_CTX_ctrl () from<br>
>>>>> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1<br>
>>>>> #7 0x00007fdaf0c31144 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
>>>>> #8 0x00007fdaf0c2bddb in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
>>>>> #9 0x00007fdaf0c22858 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
>>>>> #10 0x00007fdaf0c1af61 in SSL_do_handshake () from<br>
>>>>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1<br>
>>>>> #11 0x00007fdaf0e8d31b in tls_accept (c=0x7fdaed26fa98,<br>
>>>>> error=0x7ffffe2a2df0) at tls_server.c:422<br>
>>>>> #12 0x00007fdaf0e96a1b in tls_read_f (c=0x7fdaed26fa98,<br>
>>>>> flags=0x7ffffe2c318c) at tls_server.c:1116<br>
>>>>> #13 0x0000556ead5e7c46 in tcp_read_headers (c=0x7fdaed26fa98,<br>
>>>>> read_flags=0x7ffffe2c318c) at core/tcp_read.c:469<br>
>>>>> #14 0x0000556ead5ef9cb in tcp_read_req (con=0x7fdaed26fa98,<br>
>>>>> bytes_read=0x7ffffe2c3184, read_flags=0x7ffffe2c318c) at<br>
>>>>> core/tcp_read.c:1496<br>
>>>>> #15 0x0000556ead5f575f in handle_io (fm=0x7fdaf597aa98, events=1,<br>
>>>>> idx=-1) at core/tcp_read.c:1862<br>
>>>>> #16 0x0000556ead5e2053 in io_wait_loop_epoll (h=0x556eadaaeec0 <io_w>,<br>
>>>>> t=2, repeat=0) at core/io_wait.h:1065<br>
>>>>> #17 0x0000556ead5f6b35 in tcp_receive_loop (unix_sock=49) at<br>
>>>>> core/tcp_read.c:1974<br>
>>>>> #18 0x0000556ead4c8e24 in tcp_init_children () at core/tcp_main.c:4853<br>
>>>>> #19 0x0000556ead3c352a in main_loop () at main.c:1735<br>
>>>>> #20 0x0000556ead3ca5f8 in main (argc=13, argv=0x7ffffe2c3828) at main.c:2675<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> Am Do., 14. März 2019 um 13:41 Uhr schrieb Kristijan Vrban<br>
>>>>> <<a href="mailto:vrban.lkml@gmail.com" target="_blank">vrban.lkml@gmail.com</a>>:<br>
>>>>>> Hi, with full debug is see this in log for every incoming TCP SIP request:<br>
>>>>>><br>
>>>>>> Mar 14 12:10:15 kamailio-preview /usr/sbin/kamailio[17940]: DEBUG:<br>
>>>>>> <core> [core/tcp_main.c:3871]: send2child(): WARNING: no free tcp<br>
>>>>>> receiver, connection passed to the least busy one (105)<br>
>>>>>> Mar 14 12:10:15 kamailio-preview /usr/sbin/kamailio[17940]: DEBUG:<br>
>>>>>> <core> [core/tcp_main.c:3875]: send2child(): selected tcp worker 2<br>
>>>>>> 27(17937) for activity on [tls:<a href="http://172.17.217.10:5061" rel="noreferrer" target="_blank">172.17.217.10:5061</a>], 0x7fdaeda8f928<br>
>>>>>><br>
>>>>>> So the Kamailio TCP process is working, and received TCP traffic. But<br>
>>>>>> the tcp workers are somehow busy.<br>
>>>>>><br>
>>>>>> When i attach via strace to the TCP worker, i do not see any activity. Just:<br>
>>>>>><br>
>>>>>> futex(0x7fdaeca92f8c, FUTEX_WAIT_PRIVATE, 1, NULL<br>
>>>>>><br>
>>>>>> and nothing, even when i see the main tcp process choose this worker process.<br>
>>>>>><br>
>>>>>> Kristijan<br>
>>>>>><br>
>>>>>> Am Mi., 27. Feb. 2019 um 15:14 Uhr schrieb Kristijan Vrban<br>
>>>>>> <<a href="mailto:vrban.lkml@gmail.com" target="_blank">vrban.lkml@gmail.com</a>>:<br>
>>>>>>> first of all thanks for the feedback. i prepared our system now to run<br>
>>>>>>> with debug=3<br>
>>>>>>> I hope to see more then then.<br>
>>>>>>><br>
>>>>>>> Am Mi., 27. Feb. 2019 um 11:53 Uhr schrieb Kristijan Vrban<br>
>>>>>>> <<a href="mailto:vrban.lkml@gmail.com" target="_blank">vrban.lkml@gmail.com</a>>:<br>
>>>>>>>> Hi kamailios,<br>
>>>>>>>><br>
>>>>>>>> i have a creepy situation with v5.2.1 stable Kamilio. After a day or<br>
>>>>>>>> so, Kamailio stop to process incoming SIP traffic via TCP. The<br>
>>>>>>>> incoming TCP network packages get TCP-ACK from the OS (Debian 9,<br>
>>>>>>>> 4.18.0-15-generic-Linux) but Kamailio does not show any processing for<br>
>>>>>>>> the SIP-Traffic incoming via TCP. No logs, nothing. While traffic via<br>
>>>>>>>> UDP is working just totally fine.<br>
>>>>>>>><br>
>>>>>>>> When i look via command "netstat -ntp" is see, that the Recv-Q get<br>
>>>>>>>> bigger and bigger. e.g.:<br>
>>>>>>>><br>
>>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program<br>
>>>>>>>> name tcp 4566 0 <a href="http://172.17.217.12:5060" rel="noreferrer" target="_blank">172.17.217.12:5060</a> xxx.xxx.xxx.xxx:57252 ESTABLISHED<br>
>>>>>>>> 31347/kamailio<br>
>>>>>>>><br>
>>>>>>>> After Kamailio restart, all is working fine again for a day. We have<br>
>>>>>>>> maybe 10-20 devices online via TCP and low call volume (1-2 call per<br>
>>>>>>>> minute). The only settings for tcp we have is "tcp_delayed_ack=no"<br>
>>>>>>>><br>
>>>>>>>> How to could we debug this situation? Again, no error, no warings in<br>
>>>>>>>> the log. Just nothing.<br>
>>>>>>>><br>
>>>>>>>> Kristijan<br>
>>>>> _______________________________________________<br>
>>>>> Kamailio (SER) - Users Mailing List<br>
>>>>> <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
>>>>> <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
>>>> --<br>
>>>> Daniel-Constantin Mierla -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
>>>> <a href="http://www.twitter.com/miconda" rel="noreferrer" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" rel="noreferrer" target="_blank">www.linkedin.com/in/miconda</a><br>
>>>> Kamailio World Conference - May 6-8, 2019 -- <a href="http://www.kamailioworld.com" rel="noreferrer" target="_blank">www.kamailioworld.com</a><br>
>>>> Kamailio Advanced Training - Mar 25-27, 2019, in Washington, DC, USA -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
>>>><br>
>>> _______________________________________________<br>
>>> Kamailio (SER) - Users Mailing List<br>
>>> <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
>>> <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
>> --<br>
>> Daniel-Constantin Mierla -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
>> <a href="http://www.twitter.com/miconda" rel="noreferrer" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" rel="noreferrer" target="_blank">www.linkedin.com/in/miconda</a><br>
>> Kamailio World Conference - May 6-8, 2019 -- <a href="http://www.kamailioworld.com" rel="noreferrer" target="_blank">www.kamailioworld.com</a><br>
>> Kamailio Advanced Training - Mar 25-27, 2019, in Washington, DC, USA -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
>><br>
-- <br>
Daniel-Constantin Mierla -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
<a href="http://www.twitter.com/miconda" rel="noreferrer" target="_blank">www.twitter.com/miconda</a> -- <a href="http://www.linkedin.com/in/miconda" rel="noreferrer" target="_blank">www.linkedin.com/in/miconda</a><br>
Kamailio World Conference - May 6-8, 2019 -- <a href="http://www.kamailioworld.com" rel="noreferrer" target="_blank">www.kamailioworld.com</a><br>
Kamailio Advanced Training - Mar 25-27, 2019, in Washington, DC, USA -- <a href="http://www.asipto.com" rel="noreferrer" target="_blank">www.asipto.com</a><br>
<br>
<br>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><img src="http://sip.antisip.com/am48.png">Antisip - <a href="http://www.antisip.com" target="_blank">http://www.antisip.com</a><br></div>