Hello Daniel,
The traces and the simplified config file can be downloaded by using the link below:

http://updates.xorcom.com/~xorcom/2019-jul-21.tar.gz

Regarding to an IP route from the local IP appearing in Record-Route to the target IP. I don't have a special route for it. But the net.ipv4.ip_forward is enabled in the system.
Anyway, I'm sending you IP addresses and the current routing table:
 
# ip a
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:ad:af:e9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.9.103/20 brd 192.168.15.255 scope global dynamic ens32
       valid_lft 116999sec preferred_lft 116999sec
    inet6 fe80::20c:29ff:fead:afe9/64 scope link
       valid_lft forever preferred_lft forever
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:d8:26:e7:21:dc brd ff:ff:ff:ff:ff:ff
    inet 10.28.80.1/24 scope global lxdbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::501d:dbff:fe72:876e/64 scope link
       valid_lft forever preferred_lft forever
5: vethHT3VX5@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
    link/ether fe:d8:26:e7:21:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fcd8:26ff:fee7:21dc/64 scope link
       valid_lft forever preferred_lft forever

# ip r
default via 192.168.0.1 dev ens32 proto dhcp src 192.168.9.103 metric 100
10.28.80.0/24 dev lxdbr0 proto kernel scope link src 10.28.80.1
192.168.0.0/20 dev ens32 proto kernel scope link src 192.168.9.103
192.168.0.1 dev ens32 proto dhcp scope link src 192.168.9.103 metric 100 

Thank you,
Leonid


On Sun, Jul 21, 2019 at 9:35 AM Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Can you load debugger module and enable cfgtrace to see the actions executed in config file for such INVITE - reproduce and send the log messages. Might be an effect of something done there.

Did I asked if you have IP route from the local IP appearing in Record-Route to the target IP address? Kamailio has the rule of trying to reuse first the local ip where the request was received also for sending out.

On Thu, Jul 18, 2019 at 2:09 PM Leonid Fainshtein <leonid.fainshtein@xorcom.com> wrote:
Yes, I have.

Best regards,
Leonid Fainshtein
Xorcom Ltd


On Thu, Jul 18, 2019 at 9:31 AM Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,
traveling, so not much time available ...
I looked over the logs when expected socket is not used, I couldn't spot any message about selecting the socket, so there is a route from them "wrong" socket to the destination.

Do you have "mhomed=1" in kamailio.cfg?

Cheers,
Daniel

On Wed, Jul 17, 2019 at 5:27 PM Leonid Fainshtein <leonid.fainshtein@xorcom.com> wrote:
Dear Daniel,
Did you have a chance to check the traces?

Best regards,
Leonid

On Wed, Jul 10, 2019 at 9:15 AM Leonid Fainshtein <leonid.fainshtein@xorcom.com> wrote:
Hello Daniel,
The requested traces can be downloaded by using the link below:

http://updates.xorcom.com/~xorcom/kam-tcp-problem.tar.gz

I don't use the force send socket option and doesn't route out via
dispatcher in this particular call flow.
I found that the problem happens only when the "listen" parameters are
defined in the Kamailio configuration.
Thus the server where I made the tests have the following IPs configured:

2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel
state UP group default qlen 1000
   link/ether 00:0c:29:ad:af:e9 brd ff:ff:ff:ff:ff:ff
   inet 192.168.9.103/20 brd 192.168.15.255 scope global dynamic ens32
3: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP group default qlen 1000
   link/ether fe:d8:26:e7:21:dc brd ff:ff:ff:ff:ff:ff
   inet 10.28.80.1/24 scope global lxdbr0

The request is accepted on 10.28.80.1 and forwarded from 192.168.9.103

If I define:
listen=udp:10.28.80.1:5060
listen=tcp:10.28.80.1:5060
listen=udp:192.168.9.103:5060
listen=tcp:192.168.9.103:5060

Then the problem occurs. Ref. files syslog-bad.log and bad.cap.
If I remove all of the 'listen' parameters then the forwarded INVITE
request is built properly. Ref. files syslog-good.log and good.cap

Best regards,
Leonid Fainshtein
Xorcom Ltd

Best regards,
Leonid Fainshtein
Xorcom Ltd


On Tue, Jul 9, 2019 at 9:53 AM Daniel-Constantin Mierla
<miconda@gmail.com> wrote:
>
> Hello,
>
> set debug = 3 in kamailio cfg and reproduce this case. Send here all the
> log messages printed by kamailio from the moment it receives the request
> till it sends it out.
>
> Some further questions:
>
>   - do you use any force send socket option?
>   - do you route out via dispatcher? If yes, is the socket attribute set?
>
> Cheers,
> Daniel
>
> On 08.07.19 21:23, Leonid Fainshtein wrote:
> > Hello,
> > The source address is correct: 192.168.0.31. I see it in tcpdump and
> > also in sngrep.
> >
> > Thank you,
> > Leonid
> >
> > On Mon, Jul 8, 2019 at 9:02 PM Daniel-Constantin Mierla
> > <miconda@gmail.com> wrote:
> >> Hello,
> >>
> >> when you look at the network traffic 9e.g., with ngrep, sngrep, ...)
> >> what is shown as source address for outbound leg?
> >>
> >> Cheers,
> >> Daniel
> >>
> >> On 08.07.19 19:21, Leonid Fainshtein wrote:
> >>> I just found Daniel's response to a similar question (ref.:
> >>> https://lists.kamailio.org/pipermail/sr-users/2019-February/104853.html):
> >>>
> >>> "check the routing rules/table of the operating systems, there should be
> >>> some differences between the two servers.
> >>> If you mhomed=1 and an unexpected interface is used for routing out the
> >>> traffic, it means that the operating system has internal routing rules that
> >>> allow going from that interface to the target address."
> >>>
> >>> Don't see anything suspicious in my server routing table:
> >>>
> >>> default via 192.168.0.1 dev eno1 proto static
> >>> 10.159.65.0/24 dev lxdbr0 proto kernel scope link src 10.159.65.1
> >>> 172.200.4.0/24 dev eno1 proto kernel scope link src 172.200.4.1
> >>> 192.168.0.0/20 dev eno1 proto kernel scope link src 192.168.0.31
> >>>
> >>> The request is received on the lxdbr0 interface (10.159.65.1) and sent
> >>> out from the eno1 interface (192.168.0.31).
> >>> I even tried to delete the default route but nothing helped. The
> >>> request is sent out with 10.159.65.1 in the via and Record-Route
> >>> fields...
> >>>
> >>> Best regards,
> >>> Leonid
> >>>
> >>> On Thu, Jul 4, 2019 at 6:20 PM Leonid Fainshtein
> >>> <leonid.fainshtein@xorcom.com> wrote:
> >>>> Hi,
> >>>> Kamailio server has two legs that are connected to different networks.
> >>>> I'm using Kamailio v.5.2.3 and the "enable_double_rr" is implicitly set to "1".
> >>>> The leg "A" IP address is 10.159.65.1
> >>>> The leg "B" IP address is 192.168.0.31
> >>>> The call is initiated from 10.159.65.18
> >>>>
> >>>> According to the "rr" module documentation, function record_route()
> >>>> should insert two "Record_Route" header fields when a request is
> >>>> accepted on one leg is should go out via the second leg. This works as
> >>>> expected in case of UDP protocol:
> >>>>
> >>>> INVITE sip:2000@192.168.6.106:5460;transport=UDP SIP/2.0
> >>>> Record-Route: <sip:192.168.0.31;r2=on;lr;did=e2c.a191>
> >>>> Record-Route: <sip:10.159.65.1;r2=on;lr;did=e2c.a191>
> >>>> Via: SIP/2.0/UDP
> >>>> 192.168.0.31;branch=z9hG4bKcfa5.d64ecbd87d5315b5993c4ccf16f86537.0
> >>>> Via: SIP/2.0/UDP 10.159.65.18:5060;rport=5060;branch=z9hG4bK3a9e9a4d
> >>>>
> >>>> But when the TCP protocol is used then the outbound message looks like this:
> >>>>
> >>>> INVITE sip:2005@192.168.0.178:35058;transport=tcp SIP/2.0
> >>>> Record-Route: <sip:10.159.65.1;transport=tcp;lr;did=bb6.7dc1>
> >>>> Via: SIP/2.0/TCP
> >>>> 10.159.65.1;branch=z9hG4bKc85a.14afc3867dd3220826f9b9940f78168f.0;i=3
> >>>> Via: SIP/2.0/TCP 10.159.65.18:5060;rport=58616;branch=z9hG4bK1469331f
> >>>>
> >>>> There are two problems there:
> >>>> a) only one Record-Route with leg is inserted
> >>>> b) the added "Via" header field contains the leg "A" IP address
> >>>> instead of expected leg "B" IP address (192.168.0.31). In the LAN
> >>>> trace I see that in reality the message was sent from leg "B".
> >>>>
> >>>> Is it a bug?
> >>>>
> >>>> Best regards,
> >>>> Leonid Fainshtein
> >>> _______________________________________________
> >>> Kamailio (SER) - Users Mailing List
> >>> sr-users@lists.kamailio.org
> >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >> --
> >> Daniel-Constantin Mierla -- www.asipto.com
> >> www.twitter.com/miconda -- www.linkedin.com/in/miconda
> >>
> --
> Daniel-Constantin Mierla -- www.asipto.com
> www.twitter.com/miconda -- www.linkedin.com/in/miconda
>


--


--