<div dir="ltr">Hello,<div><br></div><div>I added to the config file:</div><div><br></div><div>tcp_no_connect=yes<br></div><div><br></div><div>And with that param, the same test results in a different behavior, but still not working:</div><div><br></div><div>"message":" DEBUG: {1 12583750 BYE RVXZVMHKop} <core> [core\/msg_translator.c:161]: check_via_address(): (198.1.54.228, 198.1.54.228, 0)"}<br><font color="#ff0000">"message":" ERROR: {1 12583750 BYE RVXZVMHKop} tm [..\/..\/core\/forward.h:292]: msg_send_buffer(): tcp_send failed"}<br>"message":" DEBUG: {1 12583750 BYE RVXZVMHKop} tm [t_fwd.c:1537]: t_send_branch(): send to <a href="http://35.191.9.20:56470">35.191.9.20:56470</a> (3) failed"}</font><br>"message":" WARNING: {1 12583750 BYE RVXZVMHKop} tm [t_fwd.c:1557]: t_send_branch(): sending request on branch 0 failed"}<br>"message":" DEBUG: {1 12583750 BYE RVXZVMHKop} tm [t_funcs.c:336]: t_relay_to(): t_forward_nonack returned error -1 (-477)"}<br>"message":" DEBUG: {1 12583750 BYE RVXZVMHKop} tm [t_funcs.c:354]: t_relay_to(): -477 error reply generation delayed "}<br></div><div><br></div><div>Although, netstats says the connection is active:</div><div><br></div><div>root@sbc-gslb-test-1:~# netstat -putan | grep 56470<br>tcp 0 0 <a href="http://10.116.15.237:443">10.116.15.237:443</a> <a href="http://35.191.9.20:56470">35.191.9.20:56470</a> ESTABLISHED 3920/kamailio<br>root@sbc-gslb-test-1:~#<br></div><div><br></div><div><br></div><div>Anyone?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 20, 2019 at 9:26 AM Joel Serrano <<a href="mailto:joel@textplus.com">joel@textplus.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Bumping this thread up!<div><br></div><div>I did some more tests trying to narrow down the problem and this is what I found...:</div><div><br></div><div>On the INVITE, I add the TCP connection information I want to save (for later reuse). Snippets:</div><div><br></div><div>...(found this in the misc/examples/pkg/sip-router-oob.cfg, but I haven't noticed any changes to the headers or anything)...</div><div><br></div><div> # Force response to received connection<br> force_rport();<br> if (proto==TCP || proto == TLS) {<br> force_tcp_alias();<br> xlog("L_NOTICE", "force_tcp_alias() done");<br> }<br></div><div>...</div><div><br></div><div>...(I also have this)...</div><div><br></div><div> if (is_first_hop()) {<br> xlog("L_NOTICE", "Adding LB info to contact - M=$rm ID=$ci\n");<br> add_contact_alias("$tcp(c_si)", "$tcp(c_sp)", "tls");<br> }<br></div><div>...</div><div><br></div><div>Which effectively makes the contact look like:</div><div><br></div><div><sip:linphone@104.175.176.242:50312;alias=35.191.9.21~50705~3;transport=tls><br></div><div><br></div><div>..180..</div><div>..200 OK..</div><div>..ACK..</div><div><br></div><div>Then, callee ends the call (so the BYE comes from callee to caller), when I run handle_ruri_alias() I see in the logs that the everything is handled correctly:</div><div><br></div><div>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} nathelper [nathelper.c:1144]: handle_ruri_alias(): setting dst_uri to <sip:35.191.9.21:50705;transport=tls>"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} nathelper [nathelper.c:1166]: handle_ruri_alias(): rewriting r-uri to <sip:linphone@104.175.176.242:50312;transport=tls>"}</div><div><br></div><div>But then, Kamalio won't reuse the existing TCP connection and tries to create a new one:</div><div><br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} tm [t_lookup.c:1328]: t_newtran(): msg (0x7f85883b14c8) id=27\/1974 global id=25\/1974 T start=0xffffffffffffffff"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} tm [t_lookup.c:497]: t_lookup_request(): start searching: hash=63128, isACK=0"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} tm [t_lookup.c:455]: matching_3261(): RFC3261 transaction matching failed - via branch [z9hG4bK896f.dc04734743b0f0997f39c4fff07c0fbb.0]"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} tm [t_lookup.c:675]: t_lookup_request(): no transaction found"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} tm [t_hooks.c:336]: run_reqin_callbacks_internal(): trans=0x7f8583b17208, callback type 1, id 0 entered"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} <core> [core\/crypto\/md5utils.c:67]: MD5StringArray(): MD5 calculated: 71c229aff3c0b4f6e9e77c4990b74e5e"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} siputils [checks.c:123]: has_totag(): totag found"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} rr [loose.c:1095]: check_route_param(): route params checking against [;r2=on;lr=on;ftag=tAsjXhyIX;did=7d1.e6a2;nat=yes] (orig: [r2=on;lr=on;ftag=tAsjXhyIX;did=7d1.e6a2;nat=yes])"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} rr [loose.c:1101]: check_route_param(): params are <;r2=on;lr=on;ftag=tAsjXhyIX;did=7d1.e6a2;nat=yes>"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} siputils [checks.c:123]: has_totag(): totag found"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} <core> [core\/msg_translator.c:161]: check_via_address(): (198.1.54.228, 198.1.54.228, 0)"}<br><font color="#ff0000">"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} <core> [core\/tcp_main.c:2060]: tcp_send(): no open tcp connection found, opening new one"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} <core> [core\/ip_addr.c:229]: print_ip(): tcpconn_new: new tcp connection: 35.191.9.21"}<br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} <core> [core\/tcp_main.c:1242]: tcpconn_new(): on port 50705, type 3"}</font><br>"message":" DEBUG: {1 12543376 BYE QQy-qagkcB} <core> [core\/tcp_main.c:1561]: tcpconn_add(): hashes: 337:3545:0, 3"}<br></div><div><br></div><div>Am I still missing anything?</div><div><br></div><div>Is this a bug and I should open a GH issue?</div><div><br></div><div>Any suggestions/comments/ideas are very welcome!</div><div><br></div><div>Thanks, </div><div>Joel.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 1, 2019 at 11:46 AM Joel Serrano <<a href="mailto:joel@textplus.com" target="_blank">joel@textplus.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Yuriy, <div><br></div><div>Thanks for your suggestion, I've tried tcp_accept_aliases=yes in config and I added force_tcp_alias() in the request route, but I haven't seen any changes.</div><div><br></div><div>All the VIA headers look exactly the same, and I still get this in the logs:</div><div><br></div><div>"message":" DEBUG: {1 11762916 BYE d2T9-YOxYk} <core> [core\/tcp_main.c:2060]: tcp_send(): no open tcp connection found, opening new one"}<br></div><div><br></div><div><br></div><div>The docs say: </div><div><br></div><div>"f<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px">orce_tcp_alias(port)</span></div><p style="margin:0px 0px 1.4em;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px">adds a tcp port alias for the current connection (if tcp). Useful if you want to send all the trafic to port_alias through the same connection this request came from [it could help for firewall or nat traversal]. With no parameters adds the port from the message via as the alias. When the “aliased” connection is closed (e.g. it's idle for too much time), all the port aliases are removed."</p><p style="margin:0px 0px 1.4em;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px">I tried also using force_tcp_alias(5353) as an example, just to see if I find "5353" added to any headers, but no luck, it wasn't added anywhere..</p><p style="margin:0px 0px 1.4em;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px"><br></p><p style="margin:0px 0px 1.4em;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px">Any other suggestions? Am I missing something?</p><p style="margin:0px 0px 1.4em;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px"><br></p><p style="margin:0px 0px 1.4em;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px">Thanks, </p><p style="margin:0px 0px 1.4em;padding:0px;color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px">Joel.</p></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 31, 2019 at 10:53 PM Yuriy Gorlichenko <<a href="mailto:ovoshlook@gmail.com" target="_blank">ovoshlook@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">You have to use<div dir="auto">tcp_accept_aliases=yes</div><div dir="auto">But this is not enough as this param will be triggered by function</div><div dir="auto">force_tcp_alias() you need to use in the route for request ( for example record_route or subroutes) </div><div dir="auto">It will add param paramname=<portnum> (I Don't remember specific name) </div><div dir="auto">to Via header that will be used for all dialog requests belongs one being affected</div><div dir="auto"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 1 Nov 2019, 00:52 Joel Serrano, <<a href="mailto:joel@textplus.com" rel="noreferrer" target="_blank">joel@textplus.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello, <br><div><br></div><div>I'm setting up a Kamailio instance behind a TCP load balancer (with proxy protocol and NAT routing: meaning Kam stays in the flow all the time).</div><div><br></div><div>I've managed to get working almost everything we need for our service, except for one thing, and that is for Kam to use existing connections for subsequent transactions:</div><div><br></div><div>Following this example:</div><div><br></div><div><div><br></div><div><div><img src="https://mail.google.com/mail/?ui=2&ik=4cb8059f98&attid=0.1&th=16e257fd109e4e49&view=fimg&rm=16e257fd109e4e49&sz=w1600-h1000&attbid=ANGjdJ-91VHFzKV4hniupoRVBXwam-q5S4OzAdN4gy3xbdsMj4mq4jMHzqZAiNWCoW81L4r2mg7JSPiju8xWuIJzzKXfQ4TM-pAK-XcezfHICxM9RjHRbirzKBr1mdM&disp=emb&realattid=ii_k2fbh1u00&zw" alt="image.png" width="542" height="270"></div></div></div><div><br></div><div><br></div><div>EXT & INT represent the external and internal interface of a LB between the UAC and Kamailio, using TLS on both legs and proxy protocol.</div><div><br></div><div>Transaction 1: INVITE, 100, 180, 183, 200 OK</div><div><br></div><div>UAC <a href="http://1.1.1.1:1111" rel="noreferrer noreferrer" target="_blank">1.1.1.1:1111</a> -> <a href="http://2.2.2.2:443" rel="noreferrer noreferrer" target="_blank">2.2.2.2:443</a> (EXT) <a href="http://3.3.3.3:3333" rel="noreferrer noreferrer" target="_blank">3.3.3.3:3333</a> (INT) -> <a href="http://7.7.7.7:5060" rel="noreferrer noreferrer" target="_blank">7.7.7.7:5060</a> (Kamailio) </div><div><br></div><div>Transaction 2: ACK</div><div><br></div><div><div>UAC <a href="http://1.1.1.1:1112" rel="noreferrer noreferrer" target="_blank">1.1.1.1:1112</a> -> <a href="http://2.2.2.2:443" rel="noreferrer noreferrer" target="_blank">2.2.2.2:443</a> (EXT) <a href="http://4.4.4.4:4444" rel="noreferrer noreferrer" target="_blank">4.4.4.4:4444</a> (INT) -> <a href="http://7.7.7.7:5060" rel="noreferrer noreferrer" target="_blank">7.7.7.7:5060</a> (Kamailio) </div><div></div></div><div><br></div><div>Transaction 3: BYE</div><div><br></div><div>Kam <a href="http://7.7.7.7:5060" rel="noreferrer noreferrer" target="_blank">7.7.7.7:5060</a> -> <a href="http://3.3.3.3:3333" rel="noreferrer noreferrer" target="_blank">3.3.3.3:3333</a> (INT) <a href="http://2.2.2.2:443" rel="noreferrer noreferrer" target="_blank">2.2.2.2:443</a> (EXT) -> <a href="http://1.1.1.1:1111" rel="noreferrer noreferrer" target="_blank">1.1.1.1:1111</a> UAC</div><div><br></div><div><div><br></div></div><div><br></div><div>My problem is with Transaction 3. In this case the BYE is originated by the callee, and Kam has to send it to the caller. As the TCP load balancer is between Kam and the UAC, Kam has to send it to the LB so then the LB can forward it back to the UAC. This works well for msgs that belong to the same transaction (INVITE, 100, 180, 183, 200 OK) but it fails when they don't belong to the same transaction.</div><div><br></div><div>Thanks to the newly added $tcp(c_si) and $tcp(c_sp) pseudovars, I can save the internal IP:Port of the LB, so I can send stuff later to it, my problem is that Kam doesn't seem to allow this?</div><div><br></div><div>On the original INVITE, I use the following to save where I have to reach the UAC:</div><div><br></div><div>add_contact_alias("$tcp(c_si)", "$tcp(c_sp)", "tls");<br></div><div><br></div><div>Then, handle_ruri_alias() will take care of setting $du to the correct (internal LB) IP:Port so I can reach the UAC, this works.</div><div><br></div><div>My problem is that Kamailio doesn't identify that there is a valid existing TLS connection still up (from the INVITE), and tries to create a new one (and this obviously doesn't gives all sorts of problems).</div><div><br></div><div>So when I run handle_ruri_alias(), and $du is set to <a href="http://3.3.3.3:3333" rel="noreferrer noreferrer" target="_blank">3.3.3.3:3333</a> (from the example above), instead of using the existing connection, Kamailio tries to create a new one.</div><div><br></div><div>I have a log statement right before with the result of tcp_conid_state(1) (the connid is 1 for this connection) and the $rc is 1 (Connection is OK), but when I tell Kamailio it has to use it I get this in the logs:</div><div><br></div><div>DEBUG: {1 11726467 BYE gqR1qqNK8B} <core> [core\/tcp_main.c:2060]: tcp_send(): no open tcp connection found, opening new one"}<br></div><div><br></div><div>And then the problems begin...</div><div><br></div><div><br></div><div>I have tried playing around with:</div><div><br></div><div>tcp_reuse_port<br></div><div>tcp_connection_match</div><div><br></div><div>But no luck..!</div><div><br></div><div>I also thought it could be a problem of the connection being created on one worker, and a different worker handling BYE transaction, so tested with children=1 and tcp_children=1, but still same problem.</div><div><br></div><div>A more detailed log:</div><div><br></div><div>In blue my log statement checking for the status of conid "1", in red Kam not being able to find it, although it exists (as validated by tcp_conid_state(), and even in netstat I can see the connection established). In this log, <a href="http://35.191.0.66:60271" rel="noreferrer noreferrer" target="_blank">35.191.0.66:60271</a> would be the equivalent of <a href="http://3.3.3.3:3333" rel="noreferrer noreferrer" target="_blank">3.3.3.3:3333</a> and <a href="http://104.175.176.242:28157" rel="noreferrer noreferrer" target="_blank">104.175.176.242:28157</a> would be <a href="http://1.1.1.1:1111" rel="noreferrer noreferrer" target="_blank">1.1.1.1:1111</a> from the example above.</div><div><br></div><div>...<br><font color="#0000ff">"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} <core> [core\/tcp_main.c:1657]: _tcpconn_find(): found connection by id: 1"}<br>"message":" NOTICE: {1 11727734 BYE 5-LX4GdI9X} <script>: JOEL TEST New request - M=BYE TCP STATUS:1 ID=5-LX4GdI9X"}</font><br>...<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} nathelper [nathelper.c:1144]: handle_ruri_alias(): setting dst_uri to <sip:35.191.0.66:60271;transport=tls>"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} nathelper [nathelper.c:1166]: handle_ruri_alias(): rewriting r-uri to <sip:linphone@104.175.176.242:28157;transport=tls>"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} tm [t_lookup.c:1328]: t_newtran(): msg (0x7f3c884259d0) id=534\/18664 global id=532\/18664 T start=0xffffffffffffffff"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} tm [t_lookup.c:497]: t_lookup_request(): start searching: hash=63496, isACK=0"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} tm [t_lookup.c:455]: matching_3261(): RFC3261 transaction matching failed - via branch [z9hG4bK808f.eee2444f92a02cb33e1b7a21f20bc6bb.0]"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} tm [t_lookup.c:675]: t_lookup_request(): no transaction found"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} tm [t_hooks.c:336]: run_reqin_callbacks_internal(): trans=0x7f3c83b8c598, callback type 1, id 0 entered"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} <core> [core\/crypto\/md5utils.c:67]: MD5StringArray(): MD5 calculated: 3071029feb05962b26b53a9664a14210"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} siputils [checks.c:123]: has_totag(): totag found"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} rr [loose.c:1095]: check_route_param(): route params checking against [;r2=on;lr=on;ftag=Eb~TbdfTA;did=cab.01e2;nat=yes] (orig: [r2=on;lr=on;ftag=Eb~TbdfTA;did=cab.01e2;nat=yes])"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} rr [loose.c:1101]: check_route_param(): params are <;r2=on;lr=on;ftag=Eb~TbdfTA;did=cab.01e2;nat=yes>"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} siputils [checks.c:123]: has_totag(): totag found"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} <core> [core\/msg_translator.c:161]: check_via_address(): (198.1.54.228, 198.1.54.228, 0)"}<br><font color="#ff0000">"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} <core> [core\/tcp_main.c:2060]: tcp_send(): no open tcp connection found, opening new one"}<br></font>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} <core> [core\/ip_addr.c:229]: print_ip(): tcpconn_new: new tcp connection: 35.191.0.66"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} <core> [core\/tcp_main.c:1242]: tcpconn_new(): on port 60271, type 3"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} <core> [core\/tcp_main.c:1561]: tcpconn_add(): hashes: 1446:2350:0, 5"}<br>"message":" DEBUG: {1 11727734 BYE 5-LX4GdI9X} tls [tls_server.c:199]: tls_complete_init(): completing tls connection initialization"}<br>...<br></div><div><br></div><div><br></div><div>So time to seek help from the community, any suggestions/ideas/comments? Sorry if all this sounds confusing, I've tried my best to put in text the whole scenario in and "understandable" way... </div><div><br></div><div>Is this even doable?</div><div><br></div><div>Thanks, </div><div>Joel.</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" rel="noreferrer noreferrer" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div>
_______________________________________________<br>
Kamailio (SER) - Users Mailing List<br>
<a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
<a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div>