Here is the pcap format if it can help to find out the problem...
________________________________
From: sr-users <sr-users-bounces(a)lists.kamailio.org> on behalf of Cedric Thiebault
<cedric.thiebault(a)sensefly.com>
Sent: Monday, October 18, 2021 10:07 AM
To: Henning Westerholt <hw(a)skalatan.de>de>; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Kamailio on AWS Kubernetes with NLB
Here is sipgrep output:
root@ip-172-31-11-79:/# sipgrep
interface: eth0 (172.31.0.0/255.255.240.0)
filter: (ip or ip6) and ( portrange 5060-5061) or (udp and ip[6:2] & 0x3fff != 0)
T 2021/10/18 08:04:53.496957 194.209.160.181:41733 -> 172.31.11.79:5060
[AP]
REGISTER sip:test-sip.sensefly.io:5060;transport=TCP SIP/2.0.
Via: SIP/2.0/TCP 194.209.160.181:36703;branch=z9hG4bK-524287-1---0bc0be77eac080a1;rport.
Max-Forwards: 70.
Contact:
<sip:cedric@194.209.160.181:36703;rinstance=010fcab4c692c129;transport=tcp>.
To: <sip:cedric@test-sip.sensefly.io:5060;transport=TCP>.
From: <sip:cedric@test-sip.sensefly.io:5060;transport=TCP>;tag=f8166d78.
Call-ID: 9PbwQuHQ1PQQsE0-ZIrNEQ...
CSeq: 1 REGISTER.
Expires: 70.
Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE.
Supported: replaces, norefersub, extended-refer, timer, outbound, path,
X-cisco-serviceuri.
User-Agent: Z 5.5.6 v2.10.15.1-141-g4fc603c55.
Allow-Events: presence, kpml, talk.
Content-Length: 0.
.
It seems right no?
________________________________
From: sr-users <sr-users-bounces(a)lists.kamailio.org> on behalf of Cedric Thiebault
<cedric.thiebault(a)sensefly.com>
Sent: Monday, October 18, 2021 9:28 AM
To: Henning Westerholt <hw(a)skalatan.de>de>; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Kamailio on AWS Kubernetes with NLB
Hello,
Thanks for your help :-)
Here are my logs with debug loglevel:
43(48) DEBUG: <core> [core/tcp_read.c:1536]: tcp_read_req(): content-length=0
43(48) INFO: <core> [core/parser/parse_fline.c:147]: parse_first_line(): method not
followed by SP
43(48) ERROR: <core> [core/parser/parse_fline.c:271]: parse_first_line():
parse_first_line: bad message (offset: 0)
43(48) DEBUG: <core> [core/parser/msg_parser.c:673]: parse_msg(): invalid message
43(48) ERROR: <core> [core/parser/msg_parser.c:747]: parse_msg(): ERROR: parse_msg:
message=<>
43(48) ERROR: <core> [core/receive.c:374]: receive_msg(): core parsing of SIP
message failed (194.209.160.181:51603/2)
43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list
(nil)
43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list
(nil)
43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list
(nil)
43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list
(nil)
43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list
(nil)
43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list
(nil)
43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list
(nil)
43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list
(nil)
43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list
(nil)
43(48) DEBUG: <core> [core/tcp_read.c:1656]: release_tcpconn(): releasing con
0x7f7bcb670558, state -2, fd=7, id=68 ([194.209.160.181]:51603 ->
[194.209.160.181]:5060)
43(48) DEBUG: <core> [core/tcp_read.c:1660]: release_tcpconn(): extra_data (nil)
48(53) DEBUG: <core> [core/tcp_main.c:3559]: handle_tcp_child(): reader response=
7f7bcb670558, -2 from 3
48(53) DEBUG: <core> [core/ip_addr.c:571]: print_ip(): tcpconn_new: new tcp
connection: 172.31.4.8
48(53) DEBUG: <core> [core/tcp_main.c:1175]: tcpconn_new(): on port 25099, type 2,
socket 71
48(53) DEBUG: <core> [core/tcp_main.c:1494]: tcpconn_add(): hashes: 342:3855:3308,
69
48(53) DEBUG: <core> [core/io_wait.h:374]: io_watch_add(): DBG:
io_watch_add(0x559718b4f840, 71, 2, 0x7f7bcb670558), fd_no=59
48(53) DEBUG: <core> [core/io_wait.h:599]: io_watch_del(): DBG: io_watch_del
(0x559718b4f840, 71, -1, 0x0) fd_no=60 called
48(53) DEBUG: <core> [core/tcp_main.c:4457]: handle_tcpconn_ev(): sending to child,
events 2001
48(53) DEBUG: <core> [core/tcp_main.c:4127]: send2child(): selected tcp worker idx:4
proc:44 pid:49 for activity on [tcp:172.31.11.79:5060], 0x7f7bcb670558
44(49) DEBUG: <core> [core/tcp_read.c:1737]: handle_io(): received n=8
con=0x7f7bcb670558, fd=7
44(49) DEBUG: <core> [core/tcp_read.c:300]: tcp_read_data(): EOF on connection
0x7f7bcb670558 (state: 3, flags: 4118) - FD 7, bytes 23, rd-flags 10000
([172.31.4.8]:25099 -> [172.31.4.8]:5060)
44(49) DEBUG: <core> [core/tcp_read.c:184]: tcp_emit_closed_event(): TCP closed
event creation triggered (reason: 0)
44(49) DEBUG: <core> [core/tcp_read.c:192]: tcp_emit_closed_event(): no callback
registering for handling TCP closed event
44(49) DEBUG: <core> [core/tcp_read.c:1536]: tcp_read_req(): content-length=0
44(49) DEBUG: <core> [core/tcp_main.c:1590]: _tcpconn_find(): found connection by
id: 69
44(49) DEBUG: <core> [core/tcp_main.c:2471]: tcpconn_send_put(): send from reader
(49 (44)), reusing fd
44(49) DEBUG: <core> [core/tcp_main.c:2706]: tcpconn_do_send(): sending...
44(49) DEBUG: <core> [core/tcp_main.c:2739]: tcpconn_do_send(): after real write: c=
0x7f7bcb670558 n=2 fd=7
44(49) DEBUG: <core> [core/tcp_main.c:2740]: tcpconn_do_send(): buf=
I'm going to investigate SIP requests with ngrep now...
Cedric
________________________________
From: Henning Westerholt <hw(a)skalatan.de>
Sent: Sunday, October 17, 2021 1:20 PM
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Cc: Cedric Thiebault <cedric.thiebault(a)sensefly.com>
Subject: RE: Kamailio on AWS Kubernetes with NLB
Hello,
I would try to connect to the kamailio container and increase debug level to see what is
happening on a lower level. You can also use tools like ngrep/sngrep to see if the SIP
request is routed to the instance.
Cheers,
Henning
--
Henning Westerholt –
https://skalatan.de/blog/
Kamailio services –
https://gilawa.com<https://gilawa.com/>
From: sr-users <sr-users-bounces(a)lists.kamailio.org> On Behalf Of Cedric Thiebault
Sent: Friday, October 15, 2021 3:37 PM
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Subject: Re: [SR-Users] Kamailio on AWS Kubernetes with NLB
Adding a log in kamailio.cfg in request_route method:
xlog("L_INFO", "INFO - request_route - $rm ($fu ($si:$sp) to $tu,
$ci)\n");
Using docker-compose I get:
24(30) INFO: {1 1 REGISTER Hotel7dIcm7DCPtmR3jACw..} <script>: INFO - request_route
- REGISTER (sip:cedric@localhost:5060;transport=TCP (172.19.0.1:43696) to
sip:cedric@localhost:5060;transport=TCP, Hotel7dIcm7DCPtmR3jACw..)
But I don't see any log message on AWS :-(
Cedric
________________________________
From: sr-users
<sr-users-bounces@lists.kamailio.org<mailto:sr-users-bounces@lists.kamailio.org>>
on behalf of Cedric Thiebault
<cedric.thiebault@sensefly.com<mailto:cedric.thiebault@sensefly.com>>
Sent: Friday, October 15, 2021 11:52 AM
To: sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
<sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>
Subject: [SR-Users] Kamailio on AWS Kubernetes with NLB
Hello,
I'm trying to deploy Kamailio on Kubernetes (AWS EKS with Network Load Balancer) but
I'm not able to register using TCP.
I'm using official Docker image 5.5.0-buster.
I did some tests with Docker Compose and it worked fine, but on K8S it does not :-(
Here are my logs:
Listening on
udp: 127.0.0.1:5060
udp: 172.31.4.29:5060
udp: 172.31.0.221:5060
tcp: 127.0.0.1:5060
tcp: 172.31.4.29:5060
tcp: 172.31.0.221:5060
Aliases:
tcp: ip-172-31-0-221.eu-central-1.compute.internal:5060
tcp:
172-31-4-29.kube-prometheus-stack-prometheus-node-exporter.infra-prometheus.svc.cluster.local:5060
tcp: localhost4.localdomain4:5060
tcp: localhost4:5060
tcp: localhost.localdomain:5060
tcp: localhost:5060
udp: ip-172-31-0-221.eu-central-1.compute.internal:5060
udp:
172-31-4-29.kube-prometheus-stack-kubelet.kube-system.svc.cluster.local:5060
udp: localhost4.localdomain4:5060
udp: localhost4:5060
udp: localhost.localdomain:5060
udp: localhost:5060
0(1) INFO: <core> [core/tcp_main.c:4983]: init_tcp(): using epoll_lt as the io
watch method (auto detected)
0(1) INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe
module is not loaded
0(1) INFO: rr [rr_mod.c:188]: mod_init(): outbound module not available
0(1) INFO: <core> [main.c:3030]: main(): processes (at least): 41 - shm size:
67108864 - pkg size: 8388608
0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is
initially 212992
0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is
finally 425984
0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is
initially 212992
0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is
finally 425984
0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is
initially 212992
0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is
finally 425984
29(35) INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/35
31(37) INFO: ctl [io_listener.c:213]: io_listen_loop(): io_listen_loop: using epoll_lt io
watch method (config)
35(41) INFO: <core> [core/parser/parse_fline.c:147]: parse_first_line(): method not
followed by SP
35(41) ERROR: <core> [core/parser/parse_fline.c:271]: parse_first_line():
parse_first_line: bad message (offset: 0)
35(41) ERROR: <core> [core/parser/msg_parser.c:747]: parse_msg(): ERROR: parse_msg:
message=<>
35(41) ERROR: <core> [core/receive.c:374]: receive_msg(): core parsing of SIP
message failed (194.230.146.66:9055/2)
And my service:
apiVersion: v1
kind: Service
metadata:
name: kamailio-tcp
annotations:
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-target-group-attributes:
preserve_client_ip.enabled=true
service.beta.kubernetes.io/aws-load-balancer-type: external
labels:
app.kubernetes.io/instance: test-sip
app.kubernetes.io/name: kamailio-tcp
app.kubernetes.io/version: 5.5.0-buster
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- name: sip-tcp
nodePort: 30567
port: 5060
protocol: TCP
targetPort: 5060
selector:
app.kubernetes.io/instance: test-sip
app.kubernetes.io/name: kamailio
I have a Netwok Load Balancer create with a listener for TCP 5060 but when I try to
connect with Zoiper or Linphone, it does not work :-(
I used tcpdump on my pod and it received well some SIP commands.
Any ideas of what could be wrong here?
Thanks in advance!
Cedric