Kamailio on AWS Kubernetes with NLB

List overview All Threads
Download

newer

older

Use constants in attrs column of...

Routing dialogs (INVITE) from Lua

Cedric Thiebault

15 Oct 2021 15 Oct '21

9:52 a.m.

Hello,

I'm trying to deploy Kamailio on Kubernetes (AWS EKS with Network Load Balancer) but I'm not able to register using TCP. I'm using official Docker image 5.5.0-buster. I did some tests with Docker Compose and it worked fine, but on K8S it does not :-(

Here are my logs:

Listening on udp: 127.0.0.1:5060 udp: 172.31.4.29:5060 udp: 172.31.0.221:5060 tcp: 127.0.0.1:5060 tcp: 172.31.4.29:5060 tcp: 172.31.0.221:5060 Aliases: tcp: ip-172-31-0-221.eu-central-1.compute.internal:5060 tcp: 172-31-4-29.kube-prometheus-stack-prometheus-node-exporter.infra-prometheus.svc.cluster.local:5060 tcp: localhost4.localdomain4:5060 tcp: localhost4:5060 tcp: localhost.localdomain:5060 tcp: localhost:5060 udp: ip-172-31-0-221.eu-central-1.compute.internal:5060 udp: 172-31-4-29.kube-prometheus-stack-kubelet.kube-system.svc.cluster.local:5060 udp: localhost4.localdomain4:5060 udp: localhost4:5060 udp: localhost.localdomain:5060 udp: localhost:5060

0(1) INFO: <core> [core/tcp_main.c:4983]: init_tcp(): using epoll_lt as the io watch method (auto detected) 0(1) INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded 0(1) INFO: rr [rr_mod.c:188]: mod_init(): outbound module not available 0(1) INFO: <core> [main.c:3030]: main(): processes (at least): 41 - shm size: 67108864 - pkg size: 8388608 0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992 0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984 29(35) INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/35 31(37) INFO: ctl [io_listener.c:213]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) 35(41) INFO: <core> [core/parser/parse_fline.c:147]: parse_first_line(): method not followed by SP 35(41) ERROR: <core> [core/parser/parse_fline.c:271]: parse_first_line(): parse_first_line: bad message (offset: 0) 35(41) ERROR: <core> [core/parser/msg_parser.c:747]: parse_msg(): ERROR: parse_msg: message=<> 35(41) ERROR: <core> [core/receive.c:374]: receive_msg(): core parsing of SIP message failed (194.230.146.66:9055/2)

And my service:

apiVersion: v1 kind: Service metadata: name: kamailio-tcp annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*' service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true service.beta.kubernetes.io/aws-load-balancer-type: external labels: app.kubernetes.io/instance: test-sip app.kubernetes.io/name: kamailio-tcp app.kubernetes.io/version: 5.5.0-buster spec: type: LoadBalancer externalTrafficPolicy: Local ports: - name: sip-tcp nodePort: 30567 port: 5060 protocol: TCP targetPort: 5060 selector: app.kubernetes.io/instance: test-sip app.kubernetes.io/name: kamailio

I have a Netwok Load Balancer create with a listener for TCP 5060 but when I try to connect with Zoiper or Linphone, it does not work :-(

I used tcpdump on my pod and it received well some SIP commands. Any ideas of what could be wrong here?

Thanks in advance!

Cedric

Attachments:

attachment.html (text/html — 14.6 KB)

Show replies by date

Cedric Thiebault

15 Oct 15 Oct

1:37 p.m.

Adding a log in kamailio.cfg in request_route method: xlog("L_INFO", "INFO - request_route - $rm ($fu ($si:$sp) to $tu, $ci)\n");

Using docker-compose I get: 24(30) INFO: {1 1 REGISTER Hotel7dIcm7DCPtmR3jACw..} <script>: INFO - request_route - REGISTER (sip:cedric@localhost:5060;transport=TCP (172.19.0.1:43696) to sip:cedric@localhost:5060;transport=TCP, Hotel7dIcm7DCPtmR3jACw..)

But I don't see any log message on AWS :-(

Cedric ________________________________ From: sr-users sr-users-bounces@lists.kamailio.org on behalf of Cedric Thiebault cedric.thiebault@sensefly.com Sent: Friday, October 15, 2021 11:52 AM To: sr-users@lists.kamailio.org sr-users@lists.kamailio.org Subject: [SR-Users] Kamailio on AWS Kubernetes with NLB

Hello,

Here are my logs:

And my service:

I have a Netwok Load Balancer create with a listener for TCP 5060 but when I try to connect with Zoiper or Linphone, it does not work :-(

I used tcpdump on my pod and it received well some SIP commands. Any ideas of what could be wrong here?

Thanks in advance!

Cedric

Henning Westerholt

17 Oct 17 Oct

11:20 a.m.

Hello,

I would try to connect to the kamailio container and increase debug level to see what is happening on a lower level. You can also use tools like ngrep/sngrep to see if the SIP request is routed to the instance.

Cheers,

Henning -- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.com https://gilawa.com/

From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Cedric Thiebault Sent: Friday, October 15, 2021 3:37 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Kamailio on AWS Kubernetes with NLB

Adding a log in kamailio.cfg in request_route method: xlog("L_INFO", "INFO - request_route - $rm ($fu ($si:$sp) to $tu, $ci)\n");

But I don't see any log message on AWS :-(

Cedric ________________________________ From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> on behalf of Cedric Thiebault <cedric.thiebault@sensefly.commailto:cedric.thiebault@sensefly.com> Sent: Friday, October 15, 2021 11:52 AM To: sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: [SR-Users] Kamailio on AWS Kubernetes with NLB

Hello,

Here are my logs:

And my service:

I have a Netwok Load Balancer create with a listener for TCP 5060 but when I try to connect with Zoiper or Linphone, it does not work :-(

I used tcpdump on my pod and it received well some SIP commands. Any ideas of what could be wrong here?

Thanks in advance!

Cedric

Cedric Thiebault

18 Oct 18 Oct

7:28 a.m.

Hello,

Thanks for your help :-)

Here are my logs with debug loglevel:

43(48) DEBUG: <core> [core/tcp_read.c:1536]: tcp_read_req(): content-length=0 43(48) INFO: <core> [core/parser/parse_fline.c:147]: parse_first_line(): method not followed by SP 43(48) ERROR: <core> [core/parser/parse_fline.c:271]: parse_first_line(): parse_first_line: bad message (offset: 0) 43(48) DEBUG: <core> [core/parser/msg_parser.c:673]: parse_msg(): invalid message 43(48) ERROR: <core> [core/parser/msg_parser.c:747]: parse_msg(): ERROR: parse_msg: message=<> 43(48) ERROR: <core> [core/receive.c:374]: receive_msg(): core parsing of SIP message failed (194.209.160.181:51603/2) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/usr_avp.c:637]: destroy_avp_list(): destroying list (nil) 43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list (nil) 43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list (nil) 43(48) DEBUG: <core> [core/xavp.c:541]: xavp_destroy_list(): destroying xavp list (nil) 43(48) DEBUG: <core> [core/tcp_read.c:1656]: release_tcpconn(): releasing con 0x7f7bcb670558, state -2, fd=7, id=68 ([194.209.160.181]:51603 -> [194.209.160.181]:5060) 43(48) DEBUG: <core> [core/tcp_read.c:1660]: release_tcpconn(): extra_data (nil) 48(53) DEBUG: <core> [core/tcp_main.c:3559]: handle_tcp_child(): reader response= 7f7bcb670558, -2 from 3 48(53) DEBUG: <core> [core/ip_addr.c:571]: print_ip(): tcpconn_new: new tcp connection: 172.31.4.8 48(53) DEBUG: <core> [core/tcp_main.c:1175]: tcpconn_new(): on port 25099, type 2, socket 71 48(53) DEBUG: <core> [core/tcp_main.c:1494]: tcpconn_add(): hashes: 342:3855:3308, 69 48(53) DEBUG: <core> [core/io_wait.h:374]: io_watch_add(): DBG: io_watch_add(0x559718b4f840, 71, 2, 0x7f7bcb670558), fd_no=59 48(53) DEBUG: <core> [core/io_wait.h:599]: io_watch_del(): DBG: io_watch_del (0x559718b4f840, 71, -1, 0x0) fd_no=60 called 48(53) DEBUG: <core> [core/tcp_main.c:4457]: handle_tcpconn_ev(): sending to child, events 2001 48(53) DEBUG: <core> [core/tcp_main.c:4127]: send2child(): selected tcp worker idx:4 proc:44 pid:49 for activity on [tcp:172.31.11.79:5060], 0x7f7bcb670558 44(49) DEBUG: <core> [core/tcp_read.c:1737]: handle_io(): received n=8 con=0x7f7bcb670558, fd=7 44(49) DEBUG: <core> [core/tcp_read.c:300]: tcp_read_data(): EOF on connection 0x7f7bcb670558 (state: 3, flags: 4118) - FD 7, bytes 23, rd-flags 10000 ([172.31.4.8]:25099 -> [172.31.4.8]:5060) 44(49) DEBUG: <core> [core/tcp_read.c:184]: tcp_emit_closed_event(): TCP closed event creation triggered (reason: 0) 44(49) DEBUG: <core> [core/tcp_read.c:192]: tcp_emit_closed_event(): no callback registering for handling TCP closed event 44(49) DEBUG: <core> [core/tcp_read.c:1536]: tcp_read_req(): content-length=0 44(49) DEBUG: <core> [core/tcp_main.c:1590]: _tcpconn_find(): found connection by id: 69 44(49) DEBUG: <core> [core/tcp_main.c:2471]: tcpconn_send_put(): send from reader (49 (44)), reusing fd 44(49) DEBUG: <core> [core/tcp_main.c:2706]: tcpconn_do_send(): sending... 44(49) DEBUG: <core> [core/tcp_main.c:2739]: tcpconn_do_send(): after real write: c= 0x7f7bcb670558 n=2 fd=7 44(49) DEBUG: <core> [core/tcp_main.c:2740]: tcpconn_do_send(): buf=

I'm going to investigate SIP requests with ngrep now...

Cedric

________________________________ From: Henning Westerholt hw@skalatan.de Sent: Sunday, October 17, 2021 1:20 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Cc: Cedric Thiebault cedric.thiebault@sensefly.com Subject: RE: Kamailio on AWS Kubernetes with NLB

Hello,

Cheers,

Henning

Henning Westerholt – https://skalatan.de/blog/

Kamailio services – https://gilawa.com https://gilawa.com/

Adding a log in kamailio.cfg in request_route method:

xlog("L_INFO", "INFO - request_route - $rm ($fu ($si:$sp) to $tu, $ci)\n");

Using docker-compose I get:

24(30) INFO: {1 1 REGISTER Hotel7dIcm7DCPtmR3jACw..} <script>: INFO - request_route - REGISTER (sip:cedric@localhost:5060;transport=TCP (172.19.0.1:43696) to sip:cedric@localhost:5060;transport=TCP, Hotel7dIcm7DCPtmR3jACw..)

But I don't see any log message on AWS :-(

Cedric

________________________________

From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> on behalf of Cedric Thiebault <cedric.thiebault@sensefly.commailto:cedric.thiebault@sensefly.com> Sent: Friday, October 15, 2021 11:52 AM To: sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: [SR-Users] Kamailio on AWS Kubernetes with NLB

Hello,

I'm trying to deploy Kamailio on Kubernetes (AWS EKS with Network Load Balancer) but I'm not able to register using TCP.

I'm using official Docker image 5.5.0-buster.

I did some tests with Docker Compose and it worked fine, but on K8S it does not :-(

Here are my logs:

Listening on

udp: 127.0.0.1:5060

udp: 172.31.4.29:5060

udp: 172.31.0.221:5060

tcp: 127.0.0.1:5060

tcp: 172.31.4.29:5060

tcp: 172.31.0.221:5060

Aliases:

tcp: ip-172-31-0-221.eu-central-1.compute.internal:5060

tcp: 172-31-4-29.kube-prometheus-stack-prometheus-node-exporter.infra-prometheus.svc.cluster.local:5060

tcp: localhost4.localdomain4:5060

tcp: localhost4:5060

tcp: localhost.localdomain:5060

tcp: localhost:5060

udp: ip-172-31-0-221.eu-central-1.compute.internal:5060

udp: 172-31-4-29.kube-prometheus-stack-kubelet.kube-system.svc.cluster.local:5060

udp: localhost4.localdomain4:5060

udp: localhost4:5060

udp: localhost.localdomain:5060

udp: localhost:5060

0(1) INFO: <core> [core/tcp_main.c:4983]: init_tcp(): using epoll_lt as the io watch method (auto detected)

0(1) INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded

0(1) INFO: rr [rr_mod.c:188]: mod_init(): outbound module not available

0(1) INFO: <core> [main.c:3030]: main(): processes (at least): 41 - shm size: 67108864 - pkg size: 8388608

0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992

0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984

0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992

0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984

0(1) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992

0(1) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984

29(35) INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/35

31(37) INFO: ctl [io_listener.c:213]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config)

35(41) INFO: <core> [core/parser/parse_fline.c:147]: parse_first_line(): method not followed by SP

35(41) ERROR: <core> [core/parser/parse_fline.c:271]: parse_first_line(): parse_first_line: bad message (offset: 0)

35(41) ERROR: <core> [core/parser/msg_parser.c:747]: parse_msg(): ERROR: parse_msg: message=<>

35(41) ERROR: <core> [core/receive.c:374]: receive_msg(): core parsing of SIP message failed (194.230.146.66:9055/2)

And my service:

apiVersion: v1

kind: Service

metadata:

annotations:

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'

service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing

service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true

service.beta.kubernetes.io/aws-load-balancer-type: external

labels:

app.kubernetes.io/instance: test-sip

app.kubernetes.io/name: kamailio-tcp

app.kubernetes.io/version: 5.5.0-buster

spec:

type: LoadBalancer

externalTrafficPolicy: Local

ports:

- name: sip-tcp

nodePort: 30567

port: 5060

protocol: TCP

targetPort: 5060

selector:

app.kubernetes.io/instance: test-sip

app.kubernetes.io/name: kamailio

I have a Netwok Load Balancer create with a listener for TCP 5060 but when I try to connect with Zoiper or Linphone, it does not work :-(

I used tcpdump on my pod and it received well some SIP commands.

Any ideas of what could be wrong here?

Thanks in advance!

Cedric

Cedric Thiebault

8:07 a.m.

Here is sipgrep output:

root@ip-172-31-11-79:/# sipgrep interface: eth0 (172.31.0.0/255.255.240.0) filter: (ip or ip6) and ( portrange 5060-5061) or (udp and ip[6:2] & 0x3fff != 0)

T 2021/10/18 08:04:53.496957 194.209.160.181:41733 -> 172.31.11.79:5060 [AP] REGISTER sip:test-sip.sensefly.io:5060;transport=TCP SIP/2.0. Via: SIP/2.0/TCP 194.209.160.181:36703;branch=z9hG4bK-524287-1---0bc0be77eac080a1;rport. Max-Forwards: 70. Contact: sip:cedric@194.209.160.181:36703;rinstance=010fcab4c692c129;transport=tcp. To: sip:cedric@test-sip.sensefly.io:5060;transport=TCP. From: sip:cedric@test-sip.sensefly.io:5060;transport=TCP;tag=f8166d78. Call-ID: 9PbwQuHQ1PQQsE0-ZIrNEQ... CSeq: 1 REGISTER. Expires: 70. Allow: INVITE, ACK, CANCEL, BYE, NOTIFY, REFER, MESSAGE, OPTIONS, INFO, SUBSCRIBE. Supported: replaces, norefersub, extended-refer, timer, outbound, path, X-cisco-serviceuri. User-Agent: Z 5.5.6 v2.10.15.1-141-g4fc603c55. Allow-Events: presence, kpml, talk. Content-Length: 0. .

It seems right no? ________________________________ From: sr-users sr-users-bounces@lists.kamailio.org on behalf of Cedric Thiebault cedric.thiebault@sensefly.com Sent: Monday, October 18, 2021 9:28 AM To: Henning Westerholt hw@skalatan.de; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Kamailio on AWS Kubernetes with NLB