Hi! We meet such an issue with some kinds of client internet connections (e.g. reproduces always on local wifi, but doesn't repr. on 3g inet). We don't know what are exact network characteristics, but what happens from perspective of traffic on SIP server is the following:
1. the SIP client registers successfully to Kamailio (which is recent git master HEAD) with TCP transport for SIP; 2. another client calls this user; 3. Kamailio relays INVITE, it gets transmitted fully and ACK-ed on TCP level; 4. then, within 0.1-0.2 second, Kamailio fires first retransmit of that INVITE, which doesn't get fully transmitted (it is shown trunkated in sniffer output, no TCP ACK replies for it); 5. Kamailio tries to retransmit INVITE again, with the same result as in #5. 6. The caller gets SIP/2.0 408 Request Timeout.
A piece of traffic from "ngrep -t -e -d any -W byline port 5060": https://gist.githubusercontent.com/krieger-od/219f9975e5efb980ff5b/raw/096a6...
The called side app is based on mobile Linphone app. Switching to UDP is not an option (in some networks SIP messages get delivered trunkated which breaks calls), we will check how it works with TLS transport a bit later when there's technical possibility.
I have two questions: 1. How would I completely disable retransmissions to TCP connections? 2. Any ideas what can be the reason for this issue? Retransmissions by themselves?
Hello,
On 19/06/15 11:29, Andrey Utkin wrote:
Hi! We meet such an issue with some kinds of client internet connections (e.g. reproduces always on local wifi, but doesn't repr. on 3g inet). We don't know what are exact network characteristics, but what happens from perspective of traffic on SIP server is the following:
- the SIP client registers successfully to Kamailio (which is recent
git master HEAD) with TCP transport for SIP; 2. another client calls this user; 3. Kamailio relays INVITE, it gets transmitted fully and ACK-ed on TCP level; 4. then, within 0.1-0.2 second, Kamailio fires first retransmit of that INVITE, which doesn't get fully transmitted (it is shown trunkated in sniffer output, no TCP ACK replies for it); 5. Kamailio tries to retransmit INVITE again, with the same result as in #5. 6. The caller gets SIP/2.0 408 Request Timeout.
A piece of traffic from "ngrep -t -e -d any -W byline port 5060": https://gist.githubusercontent.com/krieger-od/219f9975e5efb980ff5b/raw/096a6...
The called side app is based on mobile Linphone app. Switching to UDP is not an option (in some networks SIP messages get delivered trunkated which breaks calls), we will check how it works with TLS transport a bit later when there's technical possibility.
I have two questions:
- How would I completely disable retransmissions to TCP connections?
- Any ideas what can be the reason for this issue? Retransmissions by
themselves?
there should be no retransmission on TCP done by Kamailio.
I a look at time stamps, it is different than standard SIP retransmission intervals. Have you changed them in tm parameters?
As you say it is over wireless, might be the network driver doing some re-tranmissions.
Also, ngrep is not always good at capturing big packets, so just seeing partial sip packets is ok as long as the receiving party doesn't complain of broken/incomplete sip packet.
Cheers, Daniel
2015-06-19 12:45 GMT+03:00 Daniel-Constantin Mierla miconda@gmail.com:
Hello,
there should be no retransmission on TCP done by Kamailio.
I a look at time stamps, it is different than standard SIP retransmission intervals. Have you changed them in tm parameters?
We have these values:
root@ip-10-45-159-161:~# grep '"tm"' /usr/local/etc/kamailio/modules.cfg modparam("tm", "failure_reply_mode", 3) modparam("tm", "fr_timer", 30000) modparam("tm", "fr_inv_timer", 120000) root@ip-10-45-159-161:~#
As you say it is over wireless, might be the network driver doing some re-tranmissions.
No, that's not wireless at the server's end (which I guess you meant). It is wireless at callee client's LAN.
Also, ngrep is not always good at capturing big packets, so just seeing partial sip packets is ok as long as the receiving party doesn't complain of broken/incomplete sip packet.
I accept the chance that I misinterpret the output of ngrep. So I need to collect more info, like the traffic from client network.
Thank you very much for commenting.
P. S. Kamailio is built from git revision d7a05349537a80c177c3c169776aa57e058314f3
2015-06-19 12:45 GMT+03:00 Daniel-Constantin Mierla miconda@gmail.com:
Also, ngrep is not always good at capturing big packets, so just seeing partial sip packets is ok as long as the receiving party doesn't complain of broken/incomplete sip packet.
Thank you Daniel for answering. We still struggle from this weird issue. Now I can reproduce it in my location with recent Kamailio from git, with all SIP over TCP clients - Linphone, Sipdroid, Jitsi (desktop Java-based app). Here are ngrep logs from both server and client sides, and server syslog containing Kamailio output (filtered by substring "tcp" case-insensitively). https://gist.github.com/krieger-od/55427f2b3923b910bacb https://gist.github.com/krieger-od/c9fe6ea4bb64fac82cda https://gist.github.com/krieger-od/96ef40ca15ef2407b5f4
Here you can see that only a "tail" of INVITE message gets transmitted to client side, which is weird. This is Amazon server; it have MTU 9001 by default, but setting it to 900 or 1100 haven't made any difference. Also I have reproduced this issue on DigitalOcean VPS, so this mustn't be Amazon-specific issue. Because I have tried different SIP useragents supporting TCP, I'm afraid this can be considered Kamailio issue (honestly, I still don't quite believe as I percept Kamailio as robust and stable software). Any review and comment helps.
Have you grabbed the sip trace on client side to see what it is receiving? Are the clients reporting errors?
If you have a snom phone, you can easily see the received sip packets via web interface. Perhaps the desktop phones will have also some logs printing what is happening that can be accessed easily.
Eventually you can try to run a kamailio locally, near the client, using it as an intermediate proxy between the phone and the main sip server.
The timestamps I checked in previous traces were not following the sip retransmissions intervals (0.5sec, 1sec, 2sec, ...), a clear indication that it is not kamailio transaction layer doing retransmissions.
As I said before, ngrep is not a source to trust when dealing with large packets. Also, it can happen that it prints the same packet twice.
Daniel
On 23/06/15 17:32, Andrey Utkin wrote:
2015-06-19 12:45 GMT+03:00 Daniel-Constantin Mierla miconda@gmail.com:
Also, ngrep is not always good at capturing big packets, so just seeing partial sip packets is ok as long as the receiving party doesn't complain of broken/incomplete sip packet.
Thank you Daniel for answering. We still struggle from this weird issue. Now I can reproduce it in my location with recent Kamailio from git, with all SIP over TCP clients
- Linphone, Sipdroid, Jitsi (desktop Java-based app).
Here are ngrep logs from both server and client sides, and server syslog containing Kamailio output (filtered by substring "tcp" case-insensitively). https://gist.github.com/krieger-od/55427f2b3923b910bacb https://gist.github.com/krieger-od/c9fe6ea4bb64fac82cda https://gist.github.com/krieger-od/96ef40ca15ef2407b5f4
Here you can see that only a "tail" of INVITE message gets transmitted to client side, which is weird. This is Amazon server; it have MTU 9001 by default, but setting it to 900 or 1100 haven't made any difference. Also I have reproduced this issue on DigitalOcean VPS, so this mustn't be Amazon-specific issue. Because I have tried different SIP useragents supporting TCP, I'm afraid this can be considered Kamailio issue (honestly, I still don't quite believe as I percept Kamailio as robust and stable software). Any review and comment helps.
2015-06-23 18:49 GMT+03:00 Daniel-Constantin Mierla miconda@gmail.com:
Have you grabbed the sip trace on client side to see what it is receiving? Are the clients reporting errors?
Yes, see this https://gist.github.com/krieger-od/c9fe6ea4bb64fac82cda this is taken on Linux box running Jitsi desktop app. It doesn't report anything, or I haven't seen the log, ut just doesn't show the incoming call.
If you have a snom phone, you can easily see the received sip packets via web interface. Perhaps the desktop phones will have also some logs printing what is happening that can be accessed easily.
Eventually you can try to run a kamailio locally, near the client, using it as an intermediate proxy between the phone and the main sip server.
The timestamps I checked in previous traces were not following the sip retransmissions intervals (0.5sec, 1sec, 2sec, ...), a clear indication that it is not kamailio transaction layer doing retransmissions.
As I said before, ngrep is not a source to trust when dealing with large packets. Also, it can happen that it prints the same packet twice.
But what sniffer should I try instead of ngrep to have more details and confidence?
Also I guess you mean this to be an issue of Linux kernel on any side, or possibly of routing hardware somewhere in the route?
Andrey Utkin wrote
2015-06-23 18:49 GMT+03:00 Daniel-Constantin Mierla <
miconda@
>:
But what sniffer should I try instead of ngrep to have more details and confidence?
Hi!
Try to use tcpdump, with following review in Wireshark. This should show you complete picture.
Cheers
-- View this message in context: http://sip-router.1086192.n5.nabble.com/How-to-disable-retransmits-via-TCP-c... Sent from the Users mailing list archive at Nabble.com.
Hi,
I would also add that if you see partial packets you can try to remove any transport protocol (udp/tcp) and port filters. It will help if you are dealing with IP fragmentation. Otherwise sniffer won't catch IP fragments since they don't have transport level headers.
Best Regards, Vitaliy
On 23/06/15 19:06, Andrey Utkin wrote:
2015-06-23 18:49 GMT+03:00 Daniel-Constantin Mierla miconda@gmail.com:
Have you grabbed the sip trace on client side to see what it is receiving? Are the clients reporting errors?
Yes, see this https://gist.github.com/krieger-od/c9fe6ea4bb64fac82cda this is taken on Linux box running Jitsi desktop app. It doesn't report anything, or I haven't seen the log, ut just doesn't show the incoming call.
Jisti has options to enable logging -- search on the web how to enable and where is located the log file.
It would be interesting to see if it receives any packet and prints any error message there.
Have you tried with tls? That will rule out eventual packet mangling on the way done by providers/routers ALGs.
Cheers, Daniel
If you have a snom phone, you can easily see the received sip packets via web interface. Perhaps the desktop phones will have also some logs printing what is happening that can be accessed easily.
Eventually you can try to run a kamailio locally, near the client, using it as an intermediate proxy between the phone and the main sip server.
The timestamps I checked in previous traces were not following the sip retransmissions intervals (0.5sec, 1sec, 2sec, ...), a clear indication that it is not kamailio transaction layer doing retransmissions.
As I said before, ngrep is not a source to trust when dealing with large packets. Also, it can happen that it prints the same packet twice.
But what sniffer should I try instead of ngrep to have more details and confidence?
Also I guess you mean this to be an issue of Linux kernel on any side, or possibly of routing hardware somewhere in the route?
Thanks for all the suggestions. Have taken pcap dumps with commands tcpdump -i any -w /tmp/sip_tcp.pcap host <counterpart host>
http://whdd.org/sip_tcp_server.pcap http://whdd.org/sip_tcp_client.pcap
The only obviously bad thing I see is "[TCP Previous segment not captured]" mark on the packet client side dump which is the ending of INVITE request body. Googling on this message, I've found some advice at StackOverflow to tune TCP socket TCP_MAXSEG option.
http://whdd.org/sip_tcp_mtu800_server.pcap http://whdd.org/sip_tcp_mtu800_client.pcap
For this test I have set MTU to 800 on the server's only actual net interface, and restarted Kamailio. The situation stays the same, despite the packets show differently in wireshark. On client side, there's still "[TCP Previous segment not captured]", and on server side there's a mark "[TCP segment of a reassembled PDU]".
Any review is greatly appreciated!
Hi, Don't want to sound like a captain obvious, but what I can say after a quick look at your traces is that both client and server are behind NAT and either some entity drops big packets silently or somebody drops ICMP (fragmentation needed) which doesn't allow TCP to adjust max MSS. The other strange thing is that you said that you set server's MTU to 800 bytes, but server has sent a huge TCP segment with a total length of 1548 bytes and "Don't fragment" bit set.
On 24/06/15 17:30, Andrey Utkin wrote:
Thanks for all the suggestions. Have taken pcap dumps with commands tcpdump -i any -w /tmp/sip_tcp.pcap host <counterpart host>
http://whdd.org/sip_tcp_server.pcap http://whdd.org/sip_tcp_client.pcap
The only obviously bad thing I see is "[TCP Previous segment not captured]" mark on the packet client side dump which is the ending of INVITE request body. Googling on this message, I've found some advice at StackOverflow to tune TCP socket TCP_MAXSEG option.
http://whdd.org/sip_tcp_mtu800_server.pcap http://whdd.org/sip_tcp_mtu800_client.pcap
For this test I have set MTU to 800 on the server's only actual net interface, and restarted Kamailio. The situation stays the same, despite the packets show differently in wireshark. On client side, there's still "[TCP Previous segment not captured]", and on server side there's a mark "[TCP segment of a reassembled PDU]".
Any review is greatly appreciated!
Hi,
But the fact that packet sent from a server to a client looks differently captured from both points says that somebody tries to "help" you (read SIP ALG). So I would join to Daniel's suggestion to try with TLS.
On 24/06/15 17:30, Andrey Utkin wrote:
Thanks for all the suggestions. Have taken pcap dumps with commands tcpdump -i any -w /tmp/sip_tcp.pcap host <counterpart host>
http://whdd.org/sip_tcp_server.pcap http://whdd.org/sip_tcp_client.pcap
The only obviously bad thing I see is "[TCP Previous segment not captured]" mark on the packet client side dump which is the ending of INVITE request body. Googling on this message, I've found some advice at StackOverflow to tune TCP socket TCP_MAXSEG option.
http://whdd.org/sip_tcp_mtu800_server.pcap http://whdd.org/sip_tcp_mtu800_client.pcap
For this test I have set MTU to 800 on the server's only actual net interface, and restarted Kamailio. The situation stays the same, despite the packets show differently in wireshark. On client side, there's still "[TCP Previous segment not captured]", and on server side there's a mark "[TCP segment of a reassembled PDU]".
Any review is greatly appreciated!
-
Andrey Utkin -
Daniel-Constantin Mierla -
Vasiliy Ganchev -
Vitaliy Aleksandrov