<div dir="ltr">Thanks Jose for your great suggestions and input.</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Sep 27, 2018 at 12:52 PM José Seabra <<a href="mailto:joseseabra4@gmail.com">joseseabra4@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>It is hard to me tell you if the issue is or not in kamailio, kamailio is like a sip framework and the way how it works depends a lot how do you had implemented it.</div><div><br></div><div>That's why i was giving you some hints in order to help you move forward on troubleshooting.</div><div><br></div><div>Anyway, if do you see that the package left the kamailio box more or less at same time that it was registered on the kamailio logs, i would say that the issue is not in kamailio box.</div><div><br></div><div>But if do you notice that the 30 seconds is in between the time registered on the kamailio logs and when the package left the kamailio box(wireshark), so probably it is something between kamailio and the OS and here is necessary investigate why kamailio is delaying the package.</div><div><br></div><div>Best Regards</div></div></div><br><div class="gmail_quote"><div dir="ltr">Andrew Chen <<a href="mailto:achen@fuze.com" target="_blank">achen@fuze.com</a>> escreveu no dia quinta, 27/09/2018 à(s) 17:01:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">The system does tcpdump real time on the system. We do see the delay going out the box to the destination client side by the 30 seconds. There is also suspicion about the network quality as well given we see retransmits of ACK packets during the time the UPDATE was to be delivered. I just want to make sure there is nothing on the Kamailio side would be causing this delay, which is what our engineering team is suspecting.</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Sep 27, 2018 at 11:23 AM José Seabra <<a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,</div><div dir="ltr">So, Kamailio receives the UPDATE msg from your application server and relays it to the client and the delay is on UPDATE msg relay between kamailio and the client.</div><div dir="ltr"> <br></div><div dir="ltr"><div><div>As the wireshark capture the outgoing packages just after the OS processing them, then if you take a network trace, trigger the issue and try to identify the package sent by kamailio(UPDATE) in the network trace, there you can see if there is any delay between the application(Kamailio) and network level, comparing the time registered between the logs and the pcap.</div><div><br></div><div>If the 30 seconds is not between the kamailio and network level, i would say that the issue is not in your kamailio box.</div></div><div><br></div><div>Other way is trying to check out the OS sending queue status:</div><div><ul><li style="margin-left:15px">netstat -altp</li></ul><div>Regards<br></div></div><div><br></div><div>José</div></div></div></div><br><div class="gmail_quote"><div dir="ltr">Andrew Chen <<a href="mailto:achen@fuze.com" target="_blank">achen@fuze.com</a>> escreveu no dia quinta, 27/09/2018 à(s) 14:17:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">So I log when the server side sends an UPDATE request to the Kamailio and then I log when the t_relay is completed to the client application. Those were on the same second. It's when the message left the box and then arrived at the client side is where the 30 sec difference came from. So our engineering is questioning if the t_relay really did relay it properly out of the system as it was logged. They are suspecting there could be some issue with the t_relay and the OS level tcp queue.</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Sep 27, 2018 at 6:23 AM José Seabra <<a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>update to my last email.</div><div><br></div><div>If do you see that the SIP transaction(UPDATE) arrives at your client fast enough, it isn't a problem with TCP send Queue (ignore my last email).</div><div>Do you can see the reply to the SIP transaction(UPDATE) sent from your client arriving to the host where kamailio is running? If not, seems that it isn't a problem in your Kamailio.</div><div><br></div><div>Regards</div><div>José</div></div></div></div><br><div class="gmail_quote"><div dir="ltr">José Seabra <<a href="mailto:joseseabra4@gmail.com" target="_blank">joseseabra4@gmail.com</a>> escreveu no dia quinta, 27/09/2018 à(s) 10:47:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px">Hi,</span></font><div><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px"><br></span></font></div><div><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px">Kamailio has some configuration options that logs a msg to the syslog when some config script action takes more time that is expected, the options are:</span></font></div><div><font color="#333333" face="Arial, sans-serif"><div><ul><li style="font-size:14px">latency_limit_action=500<br></li><ul><li><span style="font-size:14px"><a href="https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_action" target="_blank">https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_action</a></span></li></ul><li style="font-size:14px">latency_limit_db=500<br></li><ul><li><span style="font-size:14px"><a href="https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_db" target="_blank">https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_limit_db</a></span></li></ul><li style="font-size:14px">latency_log=2<br></li><ul><li><span style="font-size:14px"><a href="https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_log" target="_blank">https://www.kamailio.org/wiki/cookbooks/4.1.x/core#latency_log</a> </span></li></ul></ul></div></font></div><div><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px">The issue could also be related with the TCP write queue, maybe you have set low values on the following parameters:</span></font></div><ul><li>tcp_conn_wq_max </li><ul><li><a href="https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_conn_wq_max" target="_blank">https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_conn_wq_max</a></li></ul><li>tcp_wq_max </li><ul><li><a href="https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_wq_max" target="_blank">https://www.kamailio.org/wiki/cookbooks/4.1.x/core#tcp_wq_max</a></li></ul></ul><div>Other hint is to look for the OS send Queue, the package could be delayed because the other end is not reading it fast enough:<br><ul><li>netstat -altp<br></li></ul><div><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px">it also will be important try to isolate the issue in terms of SIP, the transaction delayed(UPDATE) is always the same in all situation that the issue occurs? </span></font></div></div><div><br></div><div><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px">Hope that the information above helped you.</span></font></div><div><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px"><br></span></font></div><div><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px">Regards</span></font></div><div><font color="#333333" face="Arial, sans-serif"><span style="font-size:14px">José</span></font></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-5673522953952005898m_5294008757195145862m_-8267464669571804625m_-2870227974380445624m_-2471828778899708019gmail_signature" data-smartmail="gmail_signature">Cumprimentos<div>José Seabra</div></div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-5673522953952005898m_5294008757195145862m_-8267464669571804625m_-2870227974380445624gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr" style="font-size:12.8px">Andy Chen</div><div dir="ltr" style="font-size:12.8px"><div>Sr. Telephony Lead Engineer</div><div>415 516 5535 (M)</div><div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div><div><br></div></div></div></div></div></div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality Notice: The information contained in this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments may be confidential. If you are not an intended recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are hereby notified that any dissemination, distribution or copying of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail is strictly prohibited. If you have received this e-mail in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please notify the sender and permanently delete the e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments immediately. You should not retain, copy or use this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any attachment for any purpose, nor disclose all or any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents to any other person. Thank you.*</span>_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-5673522953952005898m_5294008757195145862m_-8267464669571804625gmail_signature" data-smartmail="gmail_signature">Cumprimentos<div>José Seabra</div></div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-5673522953952005898m_5294008757195145862gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr" style="font-size:12.8px">Andy Chen</div><div dir="ltr" style="font-size:12.8px"><div>Sr. Telephony Lead Engineer</div><div>415 516 5535 (M)</div><div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div><div><br></div></div></div></div></div></div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality Notice: The information contained in this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments may be confidential. If you are not an intended recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are hereby notified that any dissemination, distribution or copying of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail is strictly prohibited. If you have received this e-mail in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please notify the sender and permanently delete the e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments immediately. You should not retain, copy or use this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any attachment for any purpose, nor disclose all or any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents to any other person. Thank you.*</span>_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-5673522953952005898gmail_signature" data-smartmail="gmail_signature">Cumprimentos<div>José Seabra</div></div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr" style="font-size:12.8px">Andy Chen</div><div dir="ltr" style="font-size:12.8px"><div>Sr. Telephony Lead Engineer</div><div>415 516 5535 (M)</div><div><a href="mailto:achen@thinkingphones.com" target="_blank">achen@</a><a href="http://fuze.com" target="_blank">fuze.com</a></div><div><br></div></div></div></div></div></div>
<br>
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">*Confidentiality Notice: The information contained in this e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments may be confidential. If you are not an intended recipient, you</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">are hereby notified that any dissemination, distribution or copying of this</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">e-mail is strictly prohibited. If you have received this e-mail in error,</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">please notify the sender and permanently delete the e-mail and any</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">attachments immediately. You should not retain, copy or use this e-mail or</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">any attachment for any purpose, nor disclose all or any part of the</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;background-color:rgb(255,255,255)">contents to any other person. Thank you.*</span>