Re: [SR-Users] Kamailio on AWS Kubernetes with NLB

Here is the pcap format if it can help to find out the problem... ________________________________ From: sr-users &lt;sr-users-bounces(a)lists.kamailio.org&gt; on behalf of Cedric Thiebault &lt;cedric.thiebault(a)sensefly.com&gt; Sent: Monday, October 18, 2021 10:07 AM To: Henning Westerholt &lt;hw(a)skalatan.de&gt;de>; Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Re: [SR-Users] Kamailio on AWS Kubernetes with NLB Here is sipgrep output: root@ip-172-31-11-79:/# sipgrep interface: eth0 (172.31.0.0/255.255.240.0) filter: (ip or ip6) and ( portrange 5060-5061) or (udp and ip[6:2] & 0x3fff != 0) T 2021/10/18 08:04:53.496957 194.209.160.181:41733 -> 172.31.11.79:5060 [AP] REGISTER sip:test-sip.sensefly.io:5060;transport=TCP SIP/2.0. Via: SIP/2.0/TCP 194.209.160.181:36703;branch=z9hG4bK-524287-1---0bc0be77eac080a1;rport. Max-Forwards: 70. Contact: <sip:cedric@194.209.160.181:36703;rinstance=010fcab4c692c129;transport=tcp>. To: <sip:cedric@test-sip.sensefly.io:5060;transport=TCP>. From: <sip:cedric@test-sip.sensefly.io:5060;transport=TCP>;tag=f8166d78. Call-ID: 9PbwQuHQ1PQQsE0-ZIrNEQ... CSeq: 1 REGISTER. Expires: 70. Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE. Supported: replaces, norefersub, extended-refer, timer, outbound, path, X-cisco-serviceuri. User-Agent: Z 5.5.6 v2.10.15.1-141-g4fc603c55. Allow-Events: presence, kpml, talk. Content-Length: 0. . It seems right no? ________________________________ From: sr-users &lt;sr-users-bounces(a)lists.kamailio.org&gt; on behalf of Cedric Thiebault &lt;cedric.thiebault(a)sensefly.com&gt; Sent: Monday, October 18, 2021 9:28 AM To: Henning Westerholt &lt;hw(a)skalatan.de&gt;de>; Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Re: [SR-Users] Kamailio on AWS Kubernetes with NLB Hello, Thanks for your help :-) Here are my logs with debug loglevel: 43(48) DEBUG: <core> [core/tcp_read.c:1536]: tcp_read_req(): content-length=0 43(48) INFO: <core> [core/parser/parse_fline.c:147]: parse_first_line(): method not followed by SP 43(48) ERROR: <core> [core/parser/parse_fline.c:271]: parse_first_line(): parse_first_line: bad message (offset: 0) 43(48) DEBUG: <core> [core/parser/msg_parser.c:673]: parse_msg(): invalid message 43(48) ERROR: <core> [core/parser/msg_parser.c:747]: parse_msg(): ERROR: parse_msg: message=<> 43(48) ERROR: <core> [core/receive.c:374]: receive_msg(): core parsing of SIP message failed (194.209.160.181:51603/2) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list (nil) 43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list (nil) 43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list (nil) 43(48) DEBUG: <core> [core/tcp_read.c:1656]: release_tcpconn(): releasing con 0x7f7bcb670558, state -2, fd=7, id=68 ([194.209.160.181]:51603 -> [194.209.160.181]:5060) 43(48) DEBUG: <core> [core/tcp_read.c:1660]: release_tcpconn(): extra_data (nil) 48(53) DEBUG: <core> [core/tcp_main.c:3559]: handle_tcp_child(): reader response= 7f7bcb670558, -2 from 3 48(53) DEBUG: <core> [core/ip_addr.c:571]: print_ip(): tcpconn_new: new tcp connection: 172.31.4.8 48(53) DEBUG: <core> [core/tcp_main.c:1175]: tcpconn_new(): on port 25099, type 2, socket 71 48(53) DEBUG: <core> [core/tcp_main.c:1494]: tcpconn_add(): hashes: 342:3855:3308, 69 48(53) DEBUG: <core> [core/io_wait.h:374]: io_watch_add(): DBG: io_watch_add(0x559718b4f840, 71, 2, 0x7f7bcb670558), fd_no=59 48(53) DEBUG: <core> [core/io_wait.h:599]: io_watch_del(): DBG: io_watch_del (0x559718b4f840, 71, -1, 0x0) fd_no=60 called 48(53) DEBUG: <core> [core/tcp_main.c:4457]: handle_tcpconn_ev(): sending to child, events 2001 48(53) DEBUG: <core> [core/tcp_main.c:4127]: send2child(): selected tcp worker idx:4 proc:44 pid:49 for activity on [tcp:172.31.11.79:5060], 0x7f7bcb670558 44(49) DEBUG: <core> [core/tcp_read.c:1737]: handle_io(): received n=8 con=0x7f7bcb670558, fd=7 44(49) DEBUG: <core> [core/tcp_read.c:300]: tcp_read_data(): EOF on connection 0x7f7bcb670558 (state: 3, flags: 4118) - FD 7, bytes 23, rd-flags 10000 ([172.31.4.8]:25099 -> [172.31.4.8]:5060) 44(49) DEBUG: <core> [core/tcp_read.c:184]: tcp_emit_closed_event(): TCP closed event creation triggered (reason: 0) 44(49) DEBUG: <core> [core/tcp_read.c:192]: tcp_emit_closed_event(): no callback registering for handling TCP closed event 44(49) DEBUG: <core> [core/tcp_read.c:1536]: tcp_read_req(): content-length=0 44(49) DEBUG: <core> [core/tcp_main.c:1590]: _tcpconn_find(): found connection by id: 69 44(49) DEBUG: <core> [core/tcp_main.c:2471]: tcpconn_send_put(): send from reader (49 (44)), reusing fd 44(49) DEBUG: <core> [core/tcp_main.c:2706]: tcpconn_do_send(): sending... 44(49) DEBUG: <core> [core/tcp_main.c:2739]: tcpconn_do_send(): after real write: c= 0x7f7bcb670558 n=2 fd=7 44(49) DEBUG: <core> [core/tcp_main.c:2740]: tcpconn_do_send(): buf= I'm going to investigate SIP requests with ngrep now... Cedric ________________________________ From: Henning Westerholt &lt;hw(a)skalatan.de&gt; Sent: Sunday, October 17, 2021 1:20 PM To: Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Cc: Cedric Thiebault &lt;cedric.thiebault(a)sensefly.com&gt; Subject: RE: Kamailio on AWS Kubernetes with NLB Hello, I would try to connect to the kamailio container and increase debug level to see what is happening on a lower level. You can also use tools like ngrep/sngrep to see if the SIP request is routed to the instance. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com<https://gilawa.com/> From: sr-users &lt;sr-users-bounces(a)lists.kamailio.org&gt; On Behalf Of Cedric Thiebault Sent: Friday, October 15, 2021 3:37 PM To: Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Re: [SR-Users] Kamailio on AWS Kubernetes with NLB Adding a log in kamailio.cfg in request_route method: xlog("L_INFO", "INFO - request_route - $rm ($fu ($si:$sp) to $tu, $ci)\n"); Using docker-compose I get: 24(30) INFO: {1 1 REGISTER Hotel7dIcm7DCPtmR3jACw..} <script>: INFO - request_route - REGISTER (sip:cedric@localhost:5060;transport=TCP (172.19.0.1:43696) to sip:cedric@localhost:5060;transport=TCP, Hotel7dIcm7DCPtmR3jACw..) But I don't see any log message on AWS :-( Cedric ________________________________ From: sr-users <sr-users-bounces@lists.kamailio.org<mailto:sr-users-bounces@lists.kamailio.org>> on behalf of Cedric Thiebault <cedric.thiebault@sensefly.com<mailto:cedric.thiebault@sensefly.com>> Sent: Friday, October 15, 2021 11:52 AM To: sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org> <sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>> Subject: [SR-Users] Kamailio on AWS Kubernetes with NLB Hello, I'm trying to deploy Kamailio on Kubernetes (AWS EKS with Network Load Balancer) but I'm not able to register using TCP. I'm using official Docker image 5.5.0-buster. I did some tests with Docker Compose and it worked fine, but on K8S it does not :-( Here are my logs: Listening on udp: 127.0.0.1:5060 udp: 172.31.4.29:5060 udp: 172.31.0.221:5060 tcp: 127.0.0.1:5060 tcp: 172.31.4.29:5060 tcp: 172.31.0.221:5060 Aliases: tcp: ip-172-31-0-221.eu-central-1.compute.internal:5060 tcp: 172-31-4-29.kube-prometheus-stack-prometheus-node-exporter.infra-prometheus.svc.cluster.local:5060 tcp: localhost4.localdomain4:5060 tcp: localhost4:5060 tcp: localhost.localdomain:5060 tcp: localhost:5060 udp: ip-172-31-0-221.eu-central-1.compute.internal:5060 udp: 172-31-4-29.kube-prometheus-stack-kubelet.kube-system.svc.cluster.local:5060 udp: localhost4.localdomain4:5060 udp: localhost4:5060 udp: localhost.localdomain:5060 udp: localhost:5060 0(1) INFO: <core> [core/tcp_main.c:4983]: init_tcp(): using epoll_lt as the io watch method (auto detected) 0(1) INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded 0(1) INFO: rr [rr_mod.c:188]: mod_init(): outbound module not available 0(1) INFO: <core> [main.c:3030]: main(): processes (at least): 41 - shm size: 67108864 - pkg size: 8388608 0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 29(35) INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/35 31(37) INFO: ctl [io_listener.c:213]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) 35(41) INFO: <core> [core/parser/parse_fline.c:147]: parse_first_line(): method not followed by SP 35(41) ERROR: <core> [core/parser/parse_fline.c:271]: parse_first_line(): parse_first_line: bad message (offset: 0) 35(41) ERROR: <core> [core/parser/msg_parser.c:747]: parse_msg(): ERROR: parse_msg: message=<> 35(41) ERROR: <core> [core/receive.c:374]: receive_msg(): core parsing of SIP message failed (194.230.146.66:9055/2) And my service: apiVersion: v1 kind: Service metadata: name: kamailio-tcp annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true service.beta.kubernetes.io/aws-load-balancer-type: external labels: app.kubernetes.io/instance: test-sip app.kubernetes.io/name: kamailio-tcp app.kubernetes.io/version: 5.5.0-buster spec: type: LoadBalancer externalTrafficPolicy: Local ports: - name: sip-tcp nodePort: 30567 port: 5060 protocol: TCP targetPort: 5060 selector: app.kubernetes.io/instance: test-sip app.kubernetes.io/name: kamailio I have a Netwok Load Balancer create with a listener for TCP 5060 but when I try to connect with Zoiper or Linphone, it does not work :-( I used tcpdump on my pod and it received well some SIP commands. Any ideas of what could be wrong here? Thanks in advance! Cedric