16 May
2018
16 May
'18
midnight
### Description
Kamailio is using the wrong source port when relaying INVITE packets with t_relay().
Another interesting thing is that while doing this, Kamailio reports to Homer via HEPv2
that it sent the packet from port 4242, but a tcpdump on the server shows this to be false
so maybe homer was just looking at the R-URI port?
There is a somewhat random element to this, so it is very difficult to reproduce. I will
provide the scenario which led to this problem.
#### Reproduction
This is the code block which is responsible for relaying INVITEs to phone from our PBX
servers, it is also the only instance of port 5062 being inserted into a SIP message.
```
if(is_method("INVITE"))
{
if(lookup("location", "sip:$rU@location"))
{
remove_hf("Contact");
append_hf("Contact: <sip:$sel(cfg_get.global.publicIP):5062>\r\n"); //A
vendor's SIPIS server doesn't work on port 4242, does changing this port break SIP
compliance?
rtpproxy_manage("cow");
t_relay();
exit;
}
}
```
- Kamailio listens for UDP+TCP on ports 5060,5062, and 4242.
- Phone A registers to tcp:KamailioIP:4242 from tcp:CustomerIP:10042
- Phone B registers to tcp:KamailioIP:4242 from tcp:CustomerIP:11042
- These phones have been re-registering from and to the same ports for a long time without
problems with the same REGISTER callID (same transaction so nothing about the connection
has changed).
- At this point we have TCP connections `tcp:CustomerIP:10042 -> tcp:KamailioIP:4242`
and `tcp:CustomerIP:11042 -> tcp:KamailioIP:4242`
- Kamailio sends INVITEs from tcp:KamailioIP:4242 to the address of the registered phone
with a contact header that tells the phones to send responses to tcp:KamailioIP:5062 (as
seen in the above code block). This causes a new TCP connection to be made from the
customer's router to Kamailio, just for this transaction.
- All is well until phone A sends a response and the customer's router, by pure
chance, decides to use port 11042 for the new TCP connection. `tcp:CustomerIP:11042 ->
tcp:KamailioIP:5062`
- From this point on Kamailio begins to route INVITEs, that were meant for phone B, to
phone A using this TCP connection `tcp:CustomerIP:11042 -> tcp:KamailioIP:5062` (source
port 5062!) rather than the connection that phone B is actually registered to
`tcp:CustomerIP:11042 -> tcp:KamailioIP:4242`.
- The customer's router sends packets from this TCP socket to phone A because the
connection identified by the Kamailio source port had been created by phone A.
- Kamailio is still routing other types of packets to phone B from port 4242 like NOTIFY
in response to a subscription and OPTIONs keep alives, but INVITEs (which are an entirely
new SIP transaction) are being sent from the wrong source port.
### Possible ~Solutions~ Problems
I have the default tcp_accept_aliases=0, so ;alias in a Via header is not a problem. We
are also not using force_send_socket.
I've tried to figure out how Kamailio works, and I am wondering if this function might
be adding the new TCP connection from phone A to port 5062 as an alias of the already
existing TCP connection from phone B to port 4242.
https://github.com/kamailio/kamailio/blob/cb7810c939da9c8f4385b530539487528…
I also found these comments interesting.
https://github.com/kamailio/kamailio/blob/52111974b4571e0562e8e731df80f48db…
https://github.com/kamailio/kamailio/blob/52111974b4571e0562e8e731df80f48db…
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.0.5 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled on 04:38:13 Feb 2 2018 with gcc 4.8.5
```
* **Operating System**:
```
Linux dallas-sip1-int 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532
20 Jun
20 Jun
2:01 p.m.
Closed #1532.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#event-1690947455
2:01 p.m.
If you refer that the local port of a tcp connection initiated by kamailio is sort of
random number, not the port on which kamailio listens, then this is how the TCP stack
works. Homer likely takes the internal socket used to receive the packet or the one
specific for the interface to send out.
In tcp, it is not possible (or better said, rather impossible) to force local socket for a
tcp connection, you can search more on google about, for example a discussion at:
*
https://superuser.com/questions/1118735/how-are-source-ports-determined-and…
I am going to close this issue -- if you want to discuss more on this topic, the right
place is sr-users(a)lists.kamailio.org mailing list.
If I misunderstood what you reported and you still think it is an issue of kamailio,
reopen and try to describe from another angle, like what it is happening for a specific
case and what you expected to happen.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#issuecomment-398724607
2:58 p.m.
First of all what this article:
- https://superuser.com/questions/1118735/how-are-
source-ports-determined-and-how-can-i-force-it-to-use-a-specific-port
<https://superuser.com/questions/1118735/how-are-source-ports-determined-and-how-can-i-force-it-to-use-a-specific-port>
Fails to mention very simple way to force local port number on outgoing tcp
connections: the bind(...) call
Second why don't kamalio do not re-use an existing connection to a given
destination is also interesting question
2018-06-20 14:01 GMT+02:00 Daniel-Constantin Mierla <
notifications(a)github.com>gt;:
If you refer that the local port of a tcp connection
initiated by kamailio
is sort of random number, not the port on which kamailio listens, then this
is how the TCP stack works. Homer likely takes the internal socket used to
receive the packet or the one specific for the interface to send out.
In tcp, it is not possible (or better said, rather impossible) to force
local socket for a tcp connection, you can search more on google about, for
example a discussion at:
- https://superuser.com/questions/1118735/how-are-
source-ports-determined-and-how-can-i-force-it-to-use-a-specific-port
<https://superuser.com/questions/1118735/how-are-source-ports-determined-and-how-can-i-force-it-to-use-a-specific-port>
I am going to close this issue -- if you want to discuss more on this
topic, the right place is sr-users(a)lists.kamailio.org mailing list.
If I misunderstood what you reported and you still think it is an issue of
kamailio, reopen and try to describe from another angle, like what it is
happening for a specific case and what you expected to happen.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/kamailio/kamailio/issues/1532#issuecomment-398724607>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AF36ZTnPeEFpbsX3WC7CisFNUaLYTXLCks5t-jmzgaJpZM4UAV3q>
.
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
3:19 p.m.
By default, Kamailio is reusing the tcp connection when possible.
Again, about the local port for tcp connections, read more about tcp protocol and how a
connection is identified (some references about ephemeral ports are also there) --
practically it is the tcp stack implementation/operating system kernel that does the
selection of the local port, impossible to ensure it from user space.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#issuecomment-398745383
21 Jun
21 Jun
8:47 p.m.
The ephemeral ports you mention are used ONLY if you do not do BIND call
on socket for outgoing connection.
If before doing CONNECT you do BIND, the local port number will be one
specified in BIND.
Following small program demonstrates it.
#!/usr/bin/python
import socket
import threading
READER_ADDR = ("127.0.0.1", 23008)
WRITER1_ADDR = ("127.0.0.1", 23001)
WRITER2_ADDR = ("127.0.0.1", 23002)
LISTEN_SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
LISTEN_SOCK.bind(READER_ADDR)
LISTEN_SOCK.listen(3)
DONE = False
class wait_conn(threading.Thread):
def run(self):
while not DONE:
s, a = LISTEN_SOCK.accept()
print("Incomming connection from: ", s.getpeername())
s.close()
t = wait_conn()
t.start()
w1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
w1.bind(WRITER1_ADDR)
w1.connect(READER_ADDR)
w1.close()
w2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
w2.bind(WRITER2_ADDR)
w2.connect(READER_ADDR)
w2.close()
DONE = True
t.join(1.0)
LISTEN_SOCK.close()
2018-06-20 15:19 GMT+02:00 Daniel-Constantin Mierla <
notifications(a)github.com>gt;:
By default, Kamailio is reusing the tcp connection
when possible.
Again, about the local port for tcp connections, read more about tcp
protocol and how a connection is identified (some references about
ephemeral ports are also there) -- practically it is the tcp stack
implementation/operating system kernel that does the selection of the local
port, impossible to ensure it from user space.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/kamailio/kamailio/issues/1532#issuecomment-398745383>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AF36ZSFtK7ZUdmBxJE8DCetWIJGGXa_Lks5t-kvggaJpZM4UAV3q>
.
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
20 Jun
20 Jun
5:40 p.m.
Let me rephrase, Kamailio is using the wrong local TCP socket as the source.
It is my understanding that a TCP connection is defined by 4 things: source IP, source
port, destination IP, and destination port. A device which has registered to Kamailio over
TCP establishes a connection which should be used for all requests from the server. The
'location' table used by the registrar module has a 'socket' column which
shows the local socket that the remote device registered to. This is the socket
transport:address:port which I would expect Kamailio to use for all INVITEs to the remote
device.
The problem we were having was that Kamailio started sending INVITEs from the wrong local
socket, then the customer's router couldn't figure out to translate NAT for that
socket. This was recreated when the remote device received an INVITE which had a contact
header which differed from the sending socket, and caused the device to create a new TCP
connection for it's response. For some reason, Kamailio saw this new connection and
started sending calls (initial INVITEs in a completely new transaction) to it instead of
the actual TCP connection that was used for registration. Another possibly important
factor is that the "new" socket created by the remote router for NAT was similar
to another device's registration socket. It had the same remote source, so maybe
Kamailio matched this with an existing registration for a different device?
Device A: tcp:CustomerIP:10042 -> tcp:KamailioIP:4242
Device B: tcp:CustomerIP:11042 -> tcp:KamailioIP:4242
Device A new connection for a 200 OK response: tcp:CustomerIP:11042 ->
tcp:KamailioIP:5062
I am not using any functions which update the registration for the device with this new
TCP connection, and interestingly other kinds of packets like NOTIFY in response to a
subscription and OPTIONs keep alives continue to use the registration socket. I think it
is somehow internal to Kamailio.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#issuecomment-398796849
7:03 p.m.
Maybe you can attach a pcap with the traffic for this case, from the REGISTER sent to
Kamailio to the INVITE exposing the issue.
There are some aspects related to SIP that you confuse or maybe you didn't explain it
properly above. The Contact header in the INVITE is not the socket of Kamailio because
Kamailio is not generating the INVITE. If the device is behind the NAT, the sip server
cannot connect to it, unless the nat router is a port forwarding firewall.
Kamailio is reusing the tcp connection whenever the destination address is matching an
active one. The pcap with the sip traffic should clarify what happens.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#issuecomment-398824413
21 Jun
21 Jun
9:27 p.m.
Obviously the difference is that kamailio does bind() and listen() on the socket (ip:port)
first -- maybe that should have said to be clear about what case we talk here. So try with
socket(), bind(), listen() and then connect().
Once there is listen() on a socket, read events for it mean accept() should be done. The
kernel won't create an outgoing connection from a listen socket, but use ephemeral
ports.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#issuecomment-399216138
22 Jun
22 Jun
10:33 a.m.
Well i don't want to be nit picking however, i believe you're mistaken
here. You can reuse port on which you are listening for outgoing
connections too.
You need to use* setsockopt(... SO_REUSEPORT ).*..
here goes the demo:
#!/usr/bin/python3
import socket
import threading
READER1_ADDR = ("127.0.0.1", 23008)
READER2_ADDR = ("127.0.0.1", 23009)
class wait_conn(threading.Thread):
def __init__(self, addr, name):
super(wait_conn,self).__init__()
self.name = name
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
self.sock.bind(addr)
self.sock.listen(3)
def run(self):
s, a = self.sock.accept()
print(self.name, ": Incomming connection from: ", s.getpeername())
s.close()
self.sock.close()
t1 = wait_conn(READER1_ADDR, "r1")
t1.start()
t2 = wait_conn(READER2_ADDR, "r2")
t2.start()
w1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
w1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
w1.bind(READER1_ADDR)
w1.connect(READER2_ADDR)
w1.close()
t1.join(1.0)
t2.join(1.0)
2018-06-21 21:27 GMT+02:00 Daniel-Constantin Mierla <
notifications(a)github.com>gt;:
Obviously the difference is that kamailio does bind()
and listen() on the
socket (ip:port) first -- maybe that should have said to be clear about
what case we talk here. So try with socket(), bind(), listen() and then
connect().
Once there is listen() on a socket, read events for it mean accept()
should be done. The kernel won't create an outgoing connection from a
listen socket, but use ephemeral ports.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/kamailio/kamailio/issues/1532#issuecomment-399216138>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AF36ZceX6KXDz8U4aUKDqzbkS-kEvCLiks5t-_OKgaJpZM4UAV3q>
.
_______________________________________________
Kamailio (SER) - Development Mailing List
sr-dev(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
21 Jun
21 Jun
11:50 p.m.
Yikes, I don't have pcaps of this issue anymore and it is difficult to reproduce
because it involves a random collision of port numbers on the customer's network with
NAT involved.
On top of that I seem to be having trouble getting my issue across.
In this case I have two connections created by phones with the same destination but
different sources, then a third which has a the same source but different destination.
Device A: tcp:CustomerIP:10042 -> `tcp:KamailioIP:4242`
Device B: `tcp:CustomerIP:11042` -> `tcp:KamailioIP:4242`
Device A new connection for a 200 OK response: `tcp:CustomerIP:11042` ->
tcp:KamailioIP:5062
Kamailio sees this as:
Device A: `tcp:KamailioIP:4242` -> tcp:CustomerIP:10042
Device B: `tcp:KamailioIP:4242` -> `tcp:CustomerIP:11042`
Device A receives a SIP message with a contact telling them to respond to Kamailio on
5062, which causes it to create a new TCP connection which in turn confuses Kamailio.
Device A tcp:KamailioIP:5062 -> `tcp:CustomerIP:11042`
Kamailio is reusing the tcp connection whenever the
destination address is matching an active one.
As seen above, Kamailio thinks Device
A has two connections with the same destination.
This invite is not in response to any packet, it is being relayed from my PBX as the first
packet of a new transaction and I am using lookup() and then t_relay() to set the
Request-URI and then relay the packet to that destination.
If t_relay uses the R-URI destination to find the TCP socket it should send from, how does
it choose between these two sockets?
The problem I am seeing is that it incorrectly chooses to send the INVITE from the
`socket' that has a source port of 5062 because the newest connection it has become
aware of for that destination is the socket with 5062.
I hope I've been able to clearly describe what, I think, is happening here. If not, we
currently have a mitigation strategy to avoid sending a packet with a contact that makes
phones want to create another connection to Kamailio.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#issuecomment-399254671
22 Jun
22 Jun
12:05 a.m.
It is very difficult to recreate because it involved the random element of a
customer's router using the same originating TCP port for two connections, which is
technically correct for TCP. I am seeing an INVITE meant for Device A go to Device B
because there are TCP connections for each which share the same tcp:IP:port destination. I
would expect that Kamailio should try to use the TCP socket (defined by the 4 numbers
srcIP, srcPort, dstIP, destPort) which was used by device A for it's registration, not
the new one which had been created by device.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#issuecomment-399258248
11:29 a.m.
Vadim Lebedev - you are chasing the wrong direction here. If you look at the docs, there
is a global parameter tcp_reuse_port for this purpose, setting SO_REUSEPORT if available
in the OS. However, the discussion here is in the context of a collision of ports as
mentioned again by @Forty-Tw0 in the comment:
* https://github.com/kamailio/kamailio/issues/1532#issuecomment-399254671
So to conclude:
* kamailio is reusing the connection from REGISTER to send back traffic to the device,
it is no other way if the device is behind a non-port forwarding NAT (e.g., symmetric
NAT)
* if it doesn't, then the pcap should help to see if the INVITE created some other
connection or uses a different Contact address so the REGISTER connection is not matching
anymore
The reason I closed here is because this seems like a specific use case issue that can be
discussed on sr-users once we see pcaps, rather than an issue in the code. Based on
conclusion there, we can chase futher the code or config options. We do not use issue
tracker as a discussion forum to decide if it is a bug or not, the issue here have to be
opened when it is clear a bug in the code (or a feature request).
@Forty-Tw0 - as suggested, let's discuss this use case scenario on sr-users mailing
list. Again, there are many aspects that might not use a clear terminology, like `Device A
receives a SIP message with a contact telling them to respond to Kamailio on 5062, which
causes it to create a new TCP connection` -- what to respond on 5062? The SIP response or
follow up requests? These are either Via or Record-Route or Contact headers ... so a lot
to clarify and can be just config things that needs to be tuned, sr-users is the right
place to discuss and get to a conclusion of how it can be fixed.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1532#issuecomment-399382420
2158
days inactive
2196
days old
12 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Jared Hull
-
Vadim Lebedev