Hi,

We're attempting to implement a mid-registrar for websocket connections but are running into some issues where the tcpconn-id and socket attributes are not stored correctly in memory (usrloc).
REGISTER Flow: UA1 -> P1 (Mid-Registrar) ~> P2 (Core Proxy/Registrar)
INVITE Flow: UA2 -> P2 -> P1 -> UA1

UA1: baresip cli connecting to P1 using websockets
UA2: softphone

Everything is running locally on a dev machine, and we're using Docker to launch P1 and P2.

The below code snippet is what we use on P1, and has been simplified.
The general idea is as following:
1) UA1 sets up the websocket connection and sends a REGISTER request to P1
2) P1 checks if this request has already been sent to P2
    2.a) If not, then relay the request to P2. Upon receiving a 200 OK from P2 we save the registration in usrloc
    2.b) If so, then absorb the request (save) and generate a reply
3) UA2 tries to call UA1 via P1 and P2

The problem occurs at 2.a since the REGISTER request is forwarded using UDP and the 200 OK is also received using UDP.
When the registration is stored in reply route, the Socket (udp:x.x.x.x:xxxx) and Tcpconn-Id (-1) are not set to what we would prefer (see Output 1)
TCP was also tried as the transport between P1 and P2. Upon receiving the reply from P2 the Tcpconn-Id and Socket of the connection between P1 and P2 was saved in the binding.
When 2.b occurs the Socket and Tcpconn-Id are set to the correct values (see Output 2)

Making a call when the binding state is at 2.a we receive the following error: "TCP/TLS connection (id: 0) for WebSocket could not be found"
Making a call when the binding state is at 2.b works perfectly fine since the correct Tcpconn-Id is set.

We did find a solution, or rather a workaround to this issue:
1) Always call the save function in request_route
2) If P2 responds with an error (non-200 OK) use the unregister function to remove the binding

We also tried the tcpops tcp_set_otcpid and tcp_set_otcpid_flag functions after lookup, but this didn't do anything.

KEMI Script
def ksr_request_route(self, msg):
    if kemi.is_REGISTER():
        aor = "aor"

        if result := kemi.htable.sht_is_null("registrations", aor) == 1:
            kemi.tm.t_on_reply("ksr_register_reply_route")
            kemi.dispatcher.ds_select_dst(1, 4)
            kemi.tm.t_relay()
            return 1
       
        kemi.registrar.save("registrations", 1)

        return 1

    if kemi.is_INVITE():
        aor = "aor"
        if result := kemi.registrar.lookup_uri("registrations", aor):
            kemi.tm.t_relay()
            return 1

def ksr_register_reply_route(self, msg):
    if kemi.tm.t_check_status("200") == 1:
        kemi.registrar.save("registrations", 1)

        aor = "aor"

        kemi.htable.sht_sets("registrations", aor, "")

        return 1

Output 1: Binding state when request is being forwarded
kamcmd ul.dump
{
Domains: {
Domain: {
Domain: registrations
Size: 1024
AoRs: {
Info: {
AoR: grant@127.0.0.1
HashID: 409622189
Contacts: {
Contact: {
Address: sip:grant-0x7fe915a79138@172.16.99.12:9;transport=ws
Expires: 13
Q: 1.000000
Call-ID: 7cdfbf93f23bb54f
CSeq: 34818
User-Agent: n/a
Received: sip:192.168.80.1:62184;transport=ws
Path: [not set]
State: CS_NEW
Flags: 1
CFlags: 20
Socket: udp:10.2.0.15:5060
Methods: -1
Ruid: uloc-626a80d3-46-1
Instance: <urn:uuid:c67df38d-d5e0-e834-1d98-a8df2d3f291b>
Reg-Id: 0
Server-Id: 0
Tcpconn-Id: -1
Keepalive: 0
Last-Keepalive: 1651146965
KA-Roundtrip: 0
Last-Modified: 1651146965
}
}
}
}
Stats: {
Records: 1
Max-Slots: 1
}
}
}
}
kamcmd core.tcp_list
{
id: 1
type: WS
state: CONN_OK
timeout: 2
lifetime: 120
ref_count: 2
src_ip: 192.168.80.1
src_port: 62184
dst_ip: 192.168.80.15
dst_port: 80
}
kamcmd ws.dump
{
connections: {
1: ws:192.168.80.1:62184 -> ws:192.168.80.15:80 (state: OPEN,  last used 3s ago, sub-protocol: sip)
}
info: {
wscounter: 1
truncated: no
}
}

Output 2: Binding state when request is being absorbed
kamcmd ul.dump
{
Domains: {
Domain: {
Domain: registrations
Size: 1024
AoRs: {
Info: {
AoR: grant@127.0.0.1
HashID: 409622189
Contacts: {
Contact: {
Address: sip:grant-0x7fe915a79138@172.16.99.12:9;transport=ws
Expires: 9
Q: 1.000000
Call-ID: 7cdfbf93f23bb54f
CSeq: 34819
User-Agent: baresip v2.0.2 (x86_64/darwin)
Received: sip:192.168.80.1:62184;transport=ws
Path: [not set]
State: CS_NEW
Flags: 1
CFlags: 20
Socket: tcp:192.168.80.15:80
Methods: 5087
Ruid: uloc-626a80d3-46-1
Instance: <urn:uuid:c67df38d-d5e0-e834-1d98-a8df2d3f291b>
Reg-Id: 0
Server-Id: 0
Tcpconn-Id: 1
Keepalive: 0
Last-Keepalive: 1651146979
KA-Roundtrip: 0
Last-Modified: 1651146979
}
}
}
}
Stats: {
Records: 1
Max-Slots: 1
}
}
}
}
kamcmd core.tcp_list
{
id: 1
type: WS
state: CONN_OK
timeout: 3
lifetime: 120
ref_count: 2
src_ip: 192.168.80.1
src_port: 62184
dst_ip: 192.168.80.15
dst_port: 80
}
kamcmd ws.dump
{
connections: {
1: ws:192.168.80.1:62184 -> ws:192.168.80.15:80 (state: OPEN,  last used 1s ago, sub-protocol: sip)
}
info: {
wscounter: 1
truncated: no
}
}