We are developing an app with a pjsip client.
I see this error when trying to send an invite from kamailio.
Mar 17 00:29:46 raspberrypi /usr/sbin/kamailio[18720]: ERROR: <core> [tcp_main.c:4258]: tcpconn_main_timeout(): connect 62.44.134.85:43829 failed (timeout)
A few seconds before 62.44.134.85:43829 did a register.
I can understand that either the TCP connection was close by the other end or no one is listening.
How much can I see in a pcap from this scenario? I cannot see the invite, but somehow there must be some TCP communication to show what is failing.
The subject is of interest to me as well.
Topology is Kamailio acting as an edge proxy to a farm of Asterisk boxes, proxy handles far-end NAT traversal, transport conversion from UDP/TCP/TLS on the WAN to UDP only on the LAN towards Asterisk. Keep-alive for endpoints on the Internet is achieved using Asterisk's quality OPTIONS messages.
The TCP error shows up rarely in the logs, at which time Asterisk's quality OPTIONS messages stop being delivered to the far endpoint, shortly after the peer goes OFFLINE. What's weird, say there's 60 IP desk phones under the same far NAT, only occasionally (1-3 times a day) some will experience the problem.
What can be done to mitigate this problem? Tuning OS/Kamailio TCP settings, lower UAC register times?
Current TCP settings: tcp_connection_lifetime=3605 tcp_send_timeout=3 tcp_connect_timeout=5 tcp_max_connections=4096
Thanks.
On Sat, Mar 17, 2018 at 5:45 AM, Kjeld Flarup kjeld.flarup@liberalismen.dk wrote:
We are developing an app with a pjsip client.
I see this error when trying to send an invite from kamailio.
Mar 17 00:29:46 raspberrypi /usr/sbin/kamailio[18720]: ERROR: <core> [tcp_main.c:4258]: tcpconn_main_timeout(): connect 62.44.134.85:43829 failed (timeout)
A few seconds before 62.44.134.85:43829 did a register.
I can understand that either the TCP connection was close by the other end or no one is listening.
How much can I see in a pcap from this scenario? I cannot see the invite, but somehow there must be some TCP communication to show what is failing.
-- -------------------- Med Liberalistiske Hilsner ---------------------- Civilingeniør, Kjeld Flarup - Mit sind er mere åbent end min tegnebog Sofienlundvej 6B https://maps.google.com/?q=Sofienlundvej+6B&entry=gmail&source=g, 7560 Hjerm, Tlf: 40 29 41 49 Den ikke akademiske hjemmeside for liberalismen - www.liberalismen.dk
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
With netstat I can see that there is an active tcp connection, while I get the tcpconn_main_timeout
netstat | grep 192.168.2.9:5070 tcp 0 0 192.168.2.9:5070 0x3e2c8655.mobile:34355 ESTABLISHED
Does this mean that Kamailio doesn't reuse this connection, and instead tries to establish a new one.
And is it possible to have two concurrent connections, especially when it is trying to send an invite to a client behind NAT.
-------------------- Med Liberalistiske Hilsner ---------------------- Civilingeniør, Kjeld Flarup - Mit sind er mere åbent end min tegnebog Sofienlundvej 6B, 7560 Hjerm, Tlf: 40 29 41 49 Den ikke akademiske hjemmeside for liberalismen - www.liberalismen.dk
On 03/18/2018 04:13 AM, Sergiu Pojoga wrote:
The subject is of interest to me as well.
Topology is Kamailio acting as an edge proxy to a farm of Asterisk boxes, proxy handles far-end NAT traversal, transport conversion from UDP/TCP/TLS on the WAN to UDP only on the LAN towards Asterisk. Keep-alive for endpoints on the Internet is achieved using Asterisk's quality OPTIONS messages.
The TCP error shows up rarely in the logs, at which time Asterisk's quality OPTIONS messages stop being delivered to the far endpoint, shortly after the peer goes OFFLINE. What's weird, say there's 60 IP desk phones under the same far NAT, only occasionally (1-3 times a day) some will experience the problem.
What can be done to mitigate this problem? Tuning OS/Kamailio TCP settings, lower UAC register times?
Current TCP settings: tcp_connection_lifetime=3605 tcp_send_timeout=3 tcp_connect_timeout=5 tcp_max_connections=4096
Thanks.
On Sat, Mar 17, 2018 at 5:45 AM, Kjeld Flarup <kjeld.flarup@liberalismen.dk mailto:kjeld.flarup@liberalismen.dk> wrote:
We are developing an app with a pjsip client. I see this error when trying to send an invite from kamailio. Mar 17 00:29:46 raspberrypi /usr/sbin/kamailio[18720]: ERROR: <core> [tcp_main.c:4258]: tcpconn_main_timeout(): connect 62.44.134.85:43829 <http://62.44.134.85:43829> failed (timeout) A few seconds before 62.44.134.85:43829 <http://62.44.134.85:43829> did a register. I can understand that either the TCP connection was close by the other end or no one is listening. How much can I see in a pcap from this scenario? I cannot see the invite, but somehow there must be some TCP communication to show what is failing. -- -------------------- Med Liberalistiske Hilsner ---------------------- Civilingeniør, Kjeld Flarup - Mit sind er mere åbent end min tegnebog Sofienlundvej 6B <https://maps.google.com/?q=Sofienlundvej+6B&entry=gmail&source=g>, 7560 Hjerm, Tlf: 40 29 41 49 Den ikke akademiske hjemmeside for liberalismen - www.liberalismen.dk <http://www.liberalismen.dk> _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On Sun, Mar 18, 2018 at 12:59:28PM +0100, Kjeld Flarup wrote:
Does this mean that Kamailio doesn't reuse this connection, and instead tries to establish a new one.
And is it possible to have two concurrent connections, especially when it is trying to send an invite to a client behind NAT.
It is impossible for a listener to reuse/reconnect a TCP connection even without NAT (a listener can only reuse the ip/port it was listening on). This is the main drawback of SIP over TCP with a REGISTERing endpoint IMHO.
Interesting
Just to be sure that I understand You correctly. When a Register is done, then an Invite, must create a new TCP connection.
I agree, that a perfect implementetion would be to keep the TCP stream up while the client is connected, and use that connection for all communication between the two stacks.
How about reregisters can they reuse the connection? Or should the connection be closed once the packet is consumed?
2018-03-19 13:47 GMT+01:00 Daniel Tryba d.tryba@pocos.nl:
On Sun, Mar 18, 2018 at 12:59:28PM +0100, Kjeld Flarup wrote:
Does this mean that Kamailio doesn't reuse this connection, and instead tries to establish a new one.
And is it possible to have two concurrent connections, especially when
it is
trying to send an invite to a client behind NAT.
It is impossible for a listener to reuse/reconnect a TCP connection even without NAT (a listener can only reuse the ip/port it was listening on). This is the main drawback of SIP over TCP with a REGISTERing endpoint IMHO.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On Mon, Mar 19, 2018 at 04:23:01PM +0100, Kjeld Flarup wrote:
Interesting
Just to be sure that I understand You correctly. When a Register is done, then an Invite, must create a new TCP connection.
That is not what I tried to say. All I wanted to say was:
uac ipA:portX -> syn -> server ipB:5060 uac ipA:portX <- syn-ack <- server ipB:5060 uac ipA:portX -> ack -> server ipB:5060 uac ipA:portX <-> server ipB:5060
Whether the uac is behind NAT or not, it is impossible to recreate the connection between ipA:portX and ipB:5060 if there is ever a network interruption.
Atleast that used to be true before SO_REUSEPORT support in kernels so maybe the uac may be capable to reconnect with the same port, but it is definitly impossible for the server to do this since there is simply no listener to connect to.
I agree, that a perfect implementetion would be to keep the TCP stream up while the client is connected, and use that connection for all communication between the two stacks.
How about reregisters can they reuse the connection? Or should the connection be closed once the packet is consumed?
This is all up to the endpoint, but having a DNS based loadbalancing setup (SRV records, round robin A records) may have an influence.
Most clients I have seen sofar will simply reuse an existing TCP connection for both INVITEs and re-REGISTERs. Clients with decend NAPTR/SRV implementation will probably query DNS and iterate over the available servers (with hopefuly 1 per server).
Kamailio will just reuse an existing connection as far as I have seen.
I got a bit wiser on my problem.
It seems that the sequence of events matters.
I have a PBX which should send the call to the App. If the App does not respond within three seconds, the call should be forwarded to a GSM number.
I have two scenarios, this one works:
1. PBX gets call 2. App registers 3. PBX sends invite 4. App rings
This fails
1. PBX gets call 2. PBX sends a push 3. App registers 4. PBX generates tcpconn_main_timeout or handle_tcpconn_ev on connect to App
I'm using a simple loop to wait for the registration (I got plenty of ressources!)
The obvious difference is that in the failing scenario, the call is in progress when the register arrives.
I do not use the technique with t_suspend. Would that make a difference?
We are using TCP. We have tried with UDP, and then the Invite is send to the App.
-------------------- Med Liberalistiske Hilsner ---------------------- Civilingeniør, Kjeld Flarup - Mit sind er mere åbent end min tegnebog Sofienlundvej 6B, 7560 Hjerm, Tlf: 40 29 41 49 Den ikke akademiske hjemmeside for liberalismen - www.liberalismen.dk
On 03/19/2018 05:57 PM, Daniel Tryba wrote:
On Mon, Mar 19, 2018 at 04:23:01PM +0100, Kjeld Flarup wrote:
Interesting
Just to be sure that I understand You correctly. When a Register is done, then an Invite, must create a new TCP connection.
That is not what I tried to say. All I wanted to say was:
uac ipA:portX -> syn -> server ipB:5060 uac ipA:portX <- syn-ack <- server ipB:5060 uac ipA:portX -> ack -> server ipB:5060 uac ipA:portX <-> server ipB:5060
Whether the uac is behind NAT or not, it is impossible to recreate the connection between ipA:portX and ipB:5060 if there is ever a network interruption.
Atleast that used to be true before SO_REUSEPORT support in kernels so maybe the uac may be capable to reconnect with the same port, but it is definitly impossible for the server to do this since there is simply no listener to connect to.
I agree, that a perfect implementetion would be to keep the TCP stream up while the client is connected, and use that connection for all communication between the two stacks.
How about reregisters can they reuse the connection? Or should the connection be closed once the packet is consumed?
This is all up to the endpoint, but having a DNS based loadbalancing setup (SRV records, round robin A records) may have an influence.
Most clients I have seen sofar will simply reuse an existing TCP connection for both INVITEs and re-REGISTERs. Clients with decend NAPTR/SRV implementation will probably query DNS and iterate over the available servers (with hopefuly 1 per server).
Kamailio will just reuse an existing connection as far as I have seen.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
My own fault
Because I needed to be able to forward the call to multiple GSM numbers at the same voip provider, I split the call onto several instances of Kamalio to be able to create new call id's
As a result, the registers were made on one instance, and the invites on another. That apparently worked fine most of the time, because Kamailio is a proxy.
-------------------- Med Liberalistiske Hilsner ---------------------- Civilingeniør, Kjeld Flarup - Mit sind er mere åbent end min tegnebog Sofienlundvej 6B, 7560 Hjerm, Tlf: 40 29 41 49 Den ikke akademiske hjemmeside for liberalismen - www.liberalismen.dk
On 03/19/2018 10:09 PM, Kjeld Flarup wrote:
I got a bit wiser on my problem.
It seems that the sequence of events matters.
I have a PBX which should send the call to the App. If the App does not respond within three seconds, the call should be forwarded to a GSM number.
I have two scenarios, this one works:
- PBX gets call
- App registers
- PBX sends invite
- App rings
This fails
- PBX gets call
- PBX sends a push
- App registers
- PBX generates tcpconn_main_timeout or handle_tcpconn_ev on connect to App
I'm using a simple loop to wait for the registration (I got plenty of ressources!)
The obvious difference is that in the failing scenario, the call is in progress when the register arrives.
I do not use the technique with t_suspend. Would that make a difference?
We are using TCP. We have tried with UDP, and then the Invite is send to the App.
-------------------- Med Liberalistiske Hilsner ---------------------- Civilingeniør, Kjeld Flarup - Mit sind er mere åbent end min tegnebog Sofienlundvej 6B, 7560 Hjerm, Tlf: 40 29 41 49 Den ikke akademiske hjemmeside for liberalismen -www.liberalismen.dk
On 03/19/2018 05:57 PM, Daniel Tryba wrote:
On Mon, Mar 19, 2018 at 04:23:01PM +0100, Kjeld Flarup wrote:
Interesting
Just to be sure that I understand You correctly. When a Register is done, then an Invite, must create a new TCP connection.
That is not what I tried to say. All I wanted to say was:
uac ipA:portX -> syn -> server ipB:5060 uac ipA:portX <- syn-ack <- server ipB:5060 uac ipA:portX -> ack -> server ipB:5060 uac ipA:portX <-> server ipB:5060
Whether the uac is behind NAT or not, it is impossible to recreate the connection between ipA:portX and ipB:5060 if there is ever a network interruption.
Atleast that used to be true before SO_REUSEPORT support in kernels so maybe the uac may be capable to reconnect with the same port, but it is definitly impossible for the server to do this since there is simply no listener to connect to.
I agree, that a perfect implementetion would be to keep the TCP stream up while the client is connected, and use that connection for all communication between the two stacks.
How about reregisters can they reuse the connection? Or should the connection be closed once the packet is consumed?
This is all up to the endpoint, but having a DNS based loadbalancing setup (SRV records, round robin A records) may have an influence.
Most clients I have seen sofar will simply reuse an existing TCP connection for both INVITEs and re-REGISTERs. Clients with decend NAPTR/SRV implementation will probably query DNS and iterate over the available servers (with hopefuly 1 per server).
Kamailio will just reuse an existing connection as far as I have seen.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On Wed, Mar 21, 2018 at 07:44:50PM +0100, Kjeld Flarup wrote:
My own fault
Because I needed to be able to forward the call to multiple GSM numbers at the same voip provider, I split the call onto several instances of Kamalio to be able to create new call id's
As a result, the registers were made on one instance, and the invites on another. That apparently worked fine most of the time, because Kamailio is a proxy.
You maybe could fix this by using the Path module by storing the proxy/registrar that received the call. If the endpoint registers directly you might have to create your on path headers before save() (atleast that was my conclusion for 4.3).
Thanks for Your suggestion. I'm however not sure that it will solve the problem which is a TCP connection problem.
I solved the issue by sending the call back to the kamailio which handles the registrations, and just let the other kamailio's do the topoh module.
-------------------- Med Liberalistiske Hilsner ---------------------- Civilingeniør, Kjeld Flarup - Mit sind er mere åbent end min tegnebog Sofienlundvej 6B, 7560 Hjerm, Tlf: 40 29 41 49 Den ikke akademiske hjemmeside for liberalismen - www.liberalismen.dk
On 03/22/2018 10:24 AM, Daniel Tryba wrote:
On Wed, Mar 21, 2018 at 07:44:50PM +0100, Kjeld Flarup wrote:
My own fault
Because I needed to be able to forward the call to multiple GSM numbers at the same voip provider, I split the call onto several instances of Kamalio to be able to create new call id's
As a result, the registers were made on one instance, and the invites on another. That apparently worked fine most of the time, because Kamailio is a proxy.
You maybe could fix this by using the Path module by storing the proxy/registrar that received the call. If the endpoint registers directly you might have to create your on path headers before save() (atleast that was my conclusion for 4.3).
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On Thu, Mar 22, 2018 at 09:26:02PM +0100, Kjeld Flarup wrote:
Thanks for Your suggestion. I'm however not sure that it will solve the problem which is a TCP connection problem.
I solved the issue by sending the call back to the kamailio which handles the registrations, and just let the other kamailio's do the topoh module.
That is exactly what Path does. My setup is a little different having 2 edge proxies/loadbalancers infront of 3 registrars, but the edge adds a Path header with source/transport (path_add_received), this gets stored in the location table. Now any backend knows which edge proxy to use, that proxy uses the already established TCP connection to the endpoint.
-
Daniel Tryba -
Kjeld Flarup -
Sergiu Pojoga