Hi folk!
Have a strange issue, and cannot understand what is wrong. Test scheme UA(sip) -> INVITE -> Kamailio The transport protocol used is TCP. The issue is reproduced randomly, in case of wrong INVITE, Kamailio does not parse Method from R-URI and answer "400 CSeq method does not match request method". The log with debug level 3 for such INVITE: https://gist.github.com/vance-od/c4e1c783adba02d80c58
my first confuse is why in line 2 different number of bytes: "read= 1025 bytes, parsed=1037" the second is in line 12: Method value is empty after this everithing brokes and kamailio answers 400.
Just another call: the same setup (same device, kamailio instance etc), same conditions, just another call. Everything is OK, similar part for good call: https://gist.github.com/vance-od/01b5dff2d81f0878cff4
All my attempts to find the diff between to messages goes to nothing, the 2 INVITEs looks similar, the wireshark analyse of TCP level - also had no result (all is similar)
The issue happens only with one of our devices and only in case of TCP/TLS (UDP constantly ok, this is why I am assured the problem is in device, but I need proofs)
version: kamailio 4.2.5 (i386/linux) flags: STATS: Off, EXTRA_DEBUG, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
will appreciate any help or idea ,with such a mess!
Cheers!
-- View this message in context: http://sip-router.1086192.n5.nabble.com/random-incorrect-method-parsing-tp14... Sent from the Users mailing list archive at Nabble.com.
Hello,
can you change the sources and replace:
DBG(" method: <%.*s>\n",fl->u.request.method.len, ZSW(fl->u.request.method.s));
with:
DBG(" method: <%.*s> (%d)\n",fl->u.request.method.len, ZSW(fl->u.request.method.s), fl->u.request.method.len);
inside parser/msg_parser.c +625
Then recompile, reinstall and wait for same case again.
Maybe there is a '\0', although the logs are not showing it. Or the length is not properly set.
Cheers, Daniel
On 08/12/15 16:21, Vasiliy Ganchev wrote:
Hi folk!
Have a strange issue, and cannot understand what is wrong. Test scheme UA(sip) -> INVITE -> Kamailio The transport protocol used is TCP. The issue is reproduced randomly, in case of wrong INVITE, Kamailio does not parse Method from R-URI and answer "400 CSeq method does not match request method". The log with debug level 3 for such INVITE: https://gist.github.com/vance-od/c4e1c783adba02d80c58
my first confuse is why in line 2 different number of bytes: "read= 1025 bytes, parsed=1037" the second is in line 12: Method value is empty after this everithing brokes and kamailio answers 400.
Just another call: the same setup (same device, kamailio instance etc), same conditions, just another call. Everything is OK, similar part for good call: https://gist.github.com/vance-od/01b5dff2d81f0878cff4
All my attempts to find the diff between to messages goes to nothing, the 2 INVITEs looks similar, the wireshark analyse of TCP level - also had no result (all is similar)
The issue happens only with one of our devices and only in case of TCP/TLS (UDP constantly ok, this is why I am assured the problem is in device, but I need proofs)
version: kamailio 4.2.5 (i386/linux) flags: STATS: Off, EXTRA_DEBUG, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
will appreciate any help or idea ,with such a mess!
Cheers!
-- View this message in context: http://sip-router.1086192.n5.nabble.com/random-incorrect-method-parsing-tp14... Sent from the Users mailing list archive at Nabble.com.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel!
Thank you for the suggestion, after the deep investigation, looks like the root of the problem found:
when device send TCP ACK (before INVITE), in this packet is incapsulated "VSS-Monitoring ethernet trailer, Source Port: 0", example in file in attachment. Looks like kamailio parse it, and add the received /0 to the beginning of following SIP request (INVITE), as a result - incorrect Method parsing and 400 error.
The way to solve current problem, is to modify the check below: --- parser/msg_parser.c (revision xxxxx) +++ parser/msg_parser.c (working copy) @@ -607,7 +607,7 @@ hdr_flags_t flags;
/* eat crlf from the beginning */ - for (tmp=buf; (*tmp=='\n' || *tmp=='\r')&& + for (tmp=buf; (*tmp=='\n' || *tmp=='\r' || *tmp=='\0')&& tmp-buf < len ; tmp++);
Do we need such change in kamailio?
Cheers! issued_packet_with_extra_0.pcap http://sip-router.1086192.n5.nabble.com/file/n143771/issued_packet_with_extra_0.pcap
-- View this message in context: http://sip-router.1086192.n5.nabble.com/random-incorrect-method-parsing-tp14... Sent from the Users mailing list archive at Nabble.com.
Hello,
good it was caught. I think it is good/safe to skip the '\0' together with the white spaces at the beginning for the request. I will analyze the impact and push the patch for it.
Cheers, Daniel
On 09/12/15 12:05, Vasiliy Ganchev wrote:
Hi Daniel!
Thank you for the suggestion, after the deep investigation, looks like the root of the problem found:
when device send TCP ACK (before INVITE), in this packet is incapsulated "VSS-Monitoring ethernet trailer, Source Port: 0", example in file in attachment. Looks like kamailio parse it, and add the received /0 to the beginning of following SIP request (INVITE), as a result - incorrect Method parsing and 400 error.
The way to solve current problem, is to modify the check below: --- parser/msg_parser.c (revision xxxxx) +++ parser/msg_parser.c (working copy) @@ -607,7 +607,7 @@ hdr_flags_t flags;
/* eat crlf from the beginning */
- for (tmp=buf; (*tmp=='\n' || *tmp=='\r')&&
- for (tmp=buf; (*tmp=='\n' || *tmp=='\r' || *tmp=='\0')&& tmp-buf < len ; tmp++);
Do we need such change in kamailio?
Cheers! issued_packet_with_extra_0.pcap http://sip-router.1086192.n5.nabble.com/file/n143771/issued_packet_with_extra_0.pcap
-- View this message in context: http://sip-router.1086192.n5.nabble.com/random-incorrect-method-parsing-tp14... Sent from the Users mailing list archive at Nabble.com.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
update on this -- I pushed last evening to master a slightly different patch, to skip other white space characters before the method is started to be parsed.
Not having much time, I didn't dig further in the code, but the position of the patch suggests that these leading whitespaces are still kept in the message buffer, which may result in sending them out as well, unless before sending, its another trim of whitespaces.
My opinion is that the tcp read code should trim these whitespaces and provide msg buffer with the start of the method for sip requests. So I will analyze more and then see if a different fix would be better, before considering to backport to stable branches.
Cheers, Daniel
On 09/12/15 16:41, Daniel-Constantin Mierla wrote:
Hello,
good it was caught. I think it is good/safe to skip the '\0' together with the white spaces at the beginning for the request. I will analyze the impact and push the patch for it.
Cheers, Daniel
On 09/12/15 12:05, Vasiliy Ganchev wrote:
Hi Daniel!
Thank you for the suggestion, after the deep investigation, looks like the root of the problem found:
when device send TCP ACK (before INVITE), in this packet is incapsulated "VSS-Monitoring ethernet trailer, Source Port: 0", example in file in attachment. Looks like kamailio parse it, and add the received /0 to the beginning of following SIP request (INVITE), as a result - incorrect Method parsing and 400 error.
The way to solve current problem, is to modify the check below: --- parser/msg_parser.c (revision xxxxx) +++ parser/msg_parser.c (working copy) @@ -607,7 +607,7 @@ hdr_flags_t flags;
/* eat crlf from the beginning */
- for (tmp=buf; (*tmp=='\n' || *tmp=='\r')&&
- for (tmp=buf; (*tmp=='\n' || *tmp=='\r' || *tmp=='\0')&& tmp-buf < len ; tmp++);
Do we need such change in kamailio?
Cheers! issued_packet_with_extra_0.pcap http://sip-router.1086192.n5.nabble.com/file/n143771/issued_packet_with_extra_0.pcap
-- View this message in context: http://sip-router.1086192.n5.nabble.com/random-incorrect-method-parsing-tp14... Sent from the Users mailing list archive at Nabble.com.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel!
just additional info about the nature of that "extra zero". Vendor use them as a padding bytes RFC http://www.ietf.org/rfc/rfc0894.txt says: "If necessary, the data field should be padded (with octets of zero) to meet the Ethernet minimum frame size. This padding is not part of the IP packet and is not included in the total length field of the IP header."
Looks like these padding bytes are not removed/ignored correctly be NIC, that leads to the issue in kamailio.
Cheers!
-- View this message in context: http://sip-router.1086192.n5.nabble.com/random-incorrect-method-parsing-tp14... Sent from the Users mailing list archive at Nabble.com.
-
Daniel-Constantin Mierla -
Vasiliy Ganchev