[SR-Users] DMQ problems
Henning Westerholt
hw at kamailio.org
Thu Aug 2 20:40:39 CEST 2018
Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin -
evosip.cloud:
> I am able to see
> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
>
> but in tcpdump no DNS query
>
> Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c
> luster.local" and starting kamailio at the same time the result is always
> the correct ip, no 42.100.109.113 coming up so this is very strange!
Hello,
without looking too much into that, I have two remarks which may worth
investigating:
- for DNS names in the configuration file Kamailio does a lookup during
startup and "compiles" this to structure which is evaluated during run time
- there is also a DNS caching functionality, which could cause something like
you observed. You can deactivate this with a core variable, I think.
Best regards,
Henning
> 2018-08-02 15:29 GMT+02:00 Charles Chance <charles.chance at sipcentric.com>:
> > Hi Paolo,
> >
> > kamailio "resolves" A host but does not query the DNS
> >
> >
> > What do you mean by this?
> >
> > I am not overly familiar with Kamailio's DNS internals, but what do you
> > see if you run the following commands?
> >
> > kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
> >
> > kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
> >
> > Cheers,
> >
> > Charles
> >
> >
> > On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
> >
> > paolo.visintin at evosip.cloud> wrote:
> >> Hello again!
> >> Another interesting thing found running in debug=3
> >>
> >> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
> >> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
> >>
> >> so , kamailio "resolves" A host but does not query the DNS
> >> this is dump of dns request inside kamailio container:
> >>
> >> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d
> >> SRV dmq-router-service.paolo.svc.cluster.local
> >> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query
> >> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10
> >> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A
> >> 172.28.0.240
> >> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904
> >> A dmq-router-service.paolo.svc.cluster.local
> >> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query
> >> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A
> >> 172.28.0.240
> >>
> >>
> >>
> >> *Paolo Visintin*
> >> *CTO*
> >> evosip.cloud
> >> [image: Risultati immagini per evosip]
> >>
> >> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
> >>
> >> paolo.visintin at evosip.cloud>:
> >>> Hello Charles,
> >>> sorry for late reply, we've had some issues rebuilding docker images
> >>> with kamailo nightly build!
> >>>
> >>> We are running on our local, private on premise kubernetes cluster so
> >>> connections from outside is simply impossible (in dev we are using a
> >>> "closed" network system)
> >>>
> >>> I've just tried to :
> >>> - open a tcpdump trace in the instance tracing only the DMQ port defined
> >>> [5062] - (tcpdump -i eth0 port 5062)
> >>> - packet coming from nowhere
> >>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
> >>> - seeing this in tcpdump :
> >>>
> >>> root at router-0:/# tcpdump -i eth0 port 5062
> >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> >>> decode
> >>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144
> >>> bytes
> >>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
> >>> 42.100.109.113.5062: UDP, length 497
> >>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
> >>> 42.100.109.113.5062: UDP, length 497
> >>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
> >>> 42.100.109.113.5062: UDP, length 497
> >>>
> >>> so seems that kamailio is trying to contact the foreign ip
> >>>
> >>> if i check inside the configuration:
> >>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
> >>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
> >>> modparam("dmq", "multi_notify", 1)
> >>> modparam("dmq", "num_workers", 4)
> >>> modparam("dmq", "ping_interval", 15)
> >>>
> >>>
> >>> ...
> >>>
> >>> if(is_method("KDMQ") && dst_port == DMQ_PORT){
> >>>
> >>> dmq_handle_message();
> >>> exit;
> >>>
> >>> }
> >>>
> >>> and in DMQ_NOTIFICATION_ADDRESS I have :
> >>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
> >>>
> >>> dmq-router-service DNS resolves:
> >>> > dmq-router-service.paolo.svc.cluster.local
> >>>
> >>> Server: 10.96.0.10
> >>> Address: 10.96.0.10#53
> >>>
> >>> Non-authoritative answer:
> >>> Name: dmq-router-service.paolo.svc.cluster.local
> >>> Address: 172.28.0.240
> >>>
> >>>
> >>> so -nowhere- i am able to find 42.100.109.113
> >>>
> >>>
> >>> and in kamailio log:
> >>> 13(821) ERROR: dmq [notification_peer.c:599]:
> >>> notification_resp_callback_f(): deleting server sip:42.100.109.113:5062
> >>> because of failed request
> >>>
> >>>
> >>>
> >>> Cheers,
> >>>
> >>> Paolo
> >>>
> >>> 2018-07-26 18:14 GMT+02:00 Charles Chance <charles.chance at sipcentric.com
> >>>
> >>>> Hi Aleksandar,
> >>>>
> >>>> The initial depopulation of the nodes (following a period of 'pending'
> >>>> state) is due to no response being received from them. Are you able to
> >>>> trace the messages to/from one of them to confirm what is happening
> >>>> there?
> >>>>
> >>>> As for the unrecognised IP, I'm afraid I can't answer that one. It must
> >>>> be coming from somewhere - it's just a case of finding out where.
> >>>> Following
> >>>> the initial population/depopulation of the node list, DMQ simply
> >>>> repeats
> >>>> the process over and over until it eventually connects to one or more
> >>>> other
> >>>> nodes. Are you running on a public cloud somewhere or is it your own,
> >>>> locally running Kubernetes cluster?
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Charles
> >>>>
> >>>>
> >>>> On 26 July 2018 at 15:30, Aleksandar Sosic <alex.sosic at evosip.cloud>
> >>>>
> >>>> wrote:
> >>>>> Hi Charles,
> >>>>>
> >>>>> We're so glad about the improvements you just committed! Thanks!
> >>>>>
> >>>>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11
> >>>>> 65+xenial
> >>>>> Kamailio starts even if the DNS record does not exist at first, that's
> >>>>> great. I'm having this nodes up and running:
> >>>>> ```
> >>>>> proxy-66f79498cc-8ws6d 3/3 Running 0
> >>>>> 4m 172.28.1.4
> >>>>> proxy-66f79498cc-b68dd 3/3 Running 0
> >>>>> 4m 172.28.1.5
> >>>>> proxy-66f79498cc-lhbxg 3/3 Running 0
> >>>>> 4m 172.28.1.6
> >>>>> proxy-66f79498cc-png6p 3/3 Running 0
> >>>>> 4m 172.28.1.7
> >>>>> ```
> >>>>>
> >>>>> In those 4-5 minutes I've experienced a population and depopulation of
> >>>>> the dmq nodes.
> >>>>>
> >>>>> For example:
> >>>>> ```
> >>>>> kamcmd> dmq.list_nodes
> >>>>> {
> >>>>> host: 172.28.1.4
> >>>>> port: 5061
> >>>>> resolved_ip: 172.28.1.4
> >>>>> status: pending
> >>>>> last_notification: 0
> >>>>> local: 0
> >>>>> }
> >>>>> {
> >>>>> host: 172.28.1.6
> >>>>> port: 5061
> >>>>> resolved_ip: 172.28.1.6
> >>>>> status: pending
> >>>>> last_notification: 0
> >>>>> local: 0
> >>>>> }
> >>>>> {
> >>>>> host: 40.100.109.113
> >>>>> port: 5061
> >>>>> resolved_ip: 40.100.109.113
> >>>>> status: pending
> >>>>> last_notification: 0
> >>>>> local: 0
> >>>>> }
> >>>>> {
> >>>>> host: 172.28.1.5
> >>>>> port: 5061
> >>>>> resolved_ip: 172.28.1.5
> >>>>> status: active
> >>>>> last_notification: 0
> >>>>> local: 1
> >>>>> }
> >>>>> ```
> >>>>>
> >>>>> And after a few seconds only the loca node itself:
> >>>>> ```
> >>>>> kamcmd> dmq.list_nodes
> >>>>> {
> >>>>> host: 172.28.1.5
> >>>>> port: 5061
> >>>>> resolved_ip: 172.28.1.5
> >>>>> status: active
> >>>>> last_notification: 0
> >>>>> local: 1
> >>>>> }
> >>>>> ```
> >>>>> And then again the 3 nodes and the 40.100.109.113 which is a Microsoft
> >>>>> IP and I don't know what's doing that IP into my dmq server list.
> >>>>>
> >>>>> The DNS record is this:
> >>>>> ```
> >>>>> root at proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service
> >>>>> Server: 10.96.0.10
> >>>>> Address: 10.96.0.10#53
> >>>>>
> >>>>> Non-authoritative answer:
> >>>>> Name: dmq-proxy-service.alex.svc.cluster.local
> >>>>> Address: 172.28.1.7
> >>>>> Name: dmq-proxy-service.alex.svc.cluster.local
> >>>>> Address: 172.28.1.6
> >>>>> Name: dmq-proxy-service.alex.svc.cluster.local
> >>>>> Address: 172.28.1.5
> >>>>> Name: dmq-proxy-service.alex.svc.cluster.local
> >>>>> Address: 172.28.1.4
> >>>>> ```
> >>>>> Any clues why that ip is in the dmq nodes list? And why .7 is missing?
> >>>>> Kind regards,
> >>>>> --
> >>>>> Aleksandar Sosic
> >>>>> mail: alex.sosic at evosip.cloud
> >>>>>
> >>>>> On Wed, Jul 25, 2018 at 3:06 PM Charles Chance
> >>>>>
> >>>>> <charles.chance at sipcentric.com> wrote:
> >>>>> > Hi,
> >>>>> >
> >>>>> > I have just pushed some changes to master - one of these allows
> >>>>>
> >>>>> startup to continue even if initial node resolution fails.
> >>>>>
> >>>>> > There are some other improvements, too, which I have been planning
> >>>>>
> >>>>> to push for some time and which should also help in your situation.
> >>>>>
> >>>>> > Can you try again with these changes applied and let me know the
> >>>>>
> >>>>> outcome?
> >>>>>
> >>>>> > Cheers,
> >>>>> >
> >>>>> > Charles
> >>>>>
> >>>>> _______________________________________________
> >>>>> Kamailio (SER) - Users Mailing List
> >>>>> sr-users at lists.kamailio.org
> >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >>>>
> >>>> --
> >>>> *Charles Chance*
> >>>> Managing Director
> >>>>
> >>>> t. 0330 120 1200 m. 07932 063 891
> >>>>
> >>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592.
> >>>> Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt
> >>>> Street,
> >>>> Birmingham Science Park, Birmingham B7 4BB.
> >>>>
> >>>> _______________________________________________
> >>>> Kamailio (SER) - Users Mailing List
> >>>> sr-users at lists.kamailio.org
> >>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >>
> >> _______________________________________________
> >> Kamailio (SER) - Users Mailing List
> >> sr-users at lists.kamailio.org
> >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> >
> > --
> > *Charles Chance*
> > Managing Director
> >
> > t. 0330 120 1200 m. 07932 063 891
> >
> > Sipcentric Ltd. Company registered in England & Wales no. 7365592.
> > Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt
> > Street,
> > Birmingham Science Park, Birmingham B7 4BB.
--
Henning Westerholt
https://skalatan.de/blog/
More information about the sr-users
mailing list