[SR-Users] DMQ problems
Paolo Visintin - evosip.cloud
paolo.visintin at evosip.cloud
Sun Aug 5 17:01:01 CEST 2018
Hello Henning,
Thanks for precious indications!
I've tried to disable dns caching but nothing changed
Also captured everything on the network interface, but nothing discovered!
I've changed also dns server (from kube DNs to core dns) without success.
I'll try to reproduce in vm environment with power dns and capturing also
on the dns side
Il Gio 2 Ago 2018, 20:40 Henning Westerholt <hw at kamailio.org> ha scritto:
> Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin -
> evosip.cloud:
> > I am able to see
> > 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
> > DMQ node A host dmq-router-service=sip:42.100.109.113:5062
> >
> > but in tcpdump no DNS query
> >
> > Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c
> > luster.local" and starting kamailio at the same time the result is always
> > the correct ip, no 42.100.109.113 coming up so this is very strange!
>
> Hello,
>
> without looking too much into that, I have two remarks which may worth
> investigating:
>
> - for DNS names in the configuration file Kamailio does a lookup during
> startup and "compiles" this to structure which is evaluated during run time
>
> - there is also a DNS caching functionality, which could cause something
> like
> you observed. You can deactivate this with a core variable, I think.
>
> Best regards,
>
> Henning
>
>
> > 2018-08-02 15:29 GMT+02:00 Charles Chance <charles.chance at sipcentric.com
> >:
> > > Hi Paolo,
> > >
> > > kamailio "resolves" A host but does not query the DNS
> > >
> > >
> > > What do you mean by this?
> > >
> > > I am not overly familiar with Kamailio's DNS internals, but what do you
> > > see if you run the following commands?
> > >
> > > kamcmd dns.lookup SRV
> _sip._udp.dmq-router-service.paolo.svc.cluster.local
> > >
> > > kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
> > >
> > > Cheers,
> > >
> > > Charles
> > >
> > >
> > > On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
> > >
> > > paolo.visintin at evosip.cloud> wrote:
> > >> Hello again!
> > >> Another interesting thing found running in debug=3
> > >>
> > >> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list():
> adding
> > >> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
> > >>
> > >> so , kamailio "resolves" A host but does not query the DNS
> > >> this is dump of dns request inside kamailio container:
> > >>
> > >> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query
> 0xf31d
> > >> SRV dmq-router-service.paolo.svc.cluster.local
> > >> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query
> > >> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10
> > >> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A
> > >> 172.28.0.240
> > >> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query
> 0x1904
> > >> A dmq-router-service.paolo.svc.cluster.local
> > >> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query
> > >> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A
> > >> 172.28.0.240
> > >>
> > >>
> > >>
> > >> *Paolo Visintin*
> > >> *CTO*
> > >> evosip.cloud
> > >> [image: Risultati immagini per evosip]
> > >>
> > >> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
> > >>
> > >> paolo.visintin at evosip.cloud>:
> > >>> Hello Charles,
> > >>> sorry for late reply, we've had some issues rebuilding docker images
> > >>> with kamailo nightly build!
> > >>>
> > >>> We are running on our local, private on premise kubernetes cluster so
> > >>> connections from outside is simply impossible (in dev we are using a
> > >>> "closed" network system)
> > >>>
> > >>> I've just tried to :
> > >>> - open a tcpdump trace in the instance tracing only the DMQ port
> defined
> > >>> [5062] - (tcpdump -i eth0 port 5062)
> > >>> - packet coming from nowhere
> > >>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
> > >>> - seeing this in tcpdump :
> > >>>
> > >>> root at router-0:/# tcpdump -i eth0 port 5062
> > >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> > >>> decode
> > >>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144
> > >>> bytes
> > >>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
> > >>> 42.100.109.113.5062: UDP, length 497
> > >>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
> > >>> 42.100.109.113.5062: UDP, length 497
> > >>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
> > >>> 42.100.109.113.5062: UDP, length 497
> > >>>
> > >>> so seems that kamailio is trying to contact the foreign ip
> > >>>
> > >>> if i check inside the configuration:
> > >>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
> > >>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
> > >>> modparam("dmq", "multi_notify", 1)
> > >>> modparam("dmq", "num_workers", 4)
> > >>> modparam("dmq", "ping_interval", 15)
> > >>>
> > >>>
> > >>> ...
> > >>>
> > >>> if(is_method("KDMQ") && dst_port == DMQ_PORT){
> > >>>
> > >>> dmq_handle_message();
> > >>> exit;
> > >>>
> > >>> }
> > >>>
> > >>> and in DMQ_NOTIFICATION_ADDRESS I have :
> > >>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
> > >>>
> > >>> dmq-router-service DNS resolves:
> > >>> > dmq-router-service.paolo.svc.cluster.local
> > >>>
> > >>> Server: 10.96.0.10
> > >>> Address: 10.96.0.10#53
> > >>>
> > >>> Non-authoritative answer:
> > >>> Name: dmq-router-service.paolo.svc.cluster.local
> > >>> Address: 172.28.0.240
> > >>>
> > >>>
> > >>> so -nowhere- i am able to find 42.100.109.113
> > >>>
> > >>>
> > >>> and in kamailio log:
> > >>> 13(821) ERROR: dmq [notification_peer.c:599]:
> > >>> notification_resp_callback_f(): deleting server sip:
> 42.100.109.113:5062
> > >>> because of failed request
> > >>>
> > >>>
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Paolo
> > >>>
> > >>> 2018-07-26 18:14 GMT+02:00 Charles Chance <
> charles.chance at sipcentric.com
> > >>>
> > >>>> Hi Aleksandar,
> > >>>>
> > >>>> The initial depopulation of the nodes (following a period of
> 'pending'
> > >>>> state) is due to no response being received from them. Are you able
> to
> > >>>> trace the messages to/from one of them to confirm what is happening
> > >>>> there?
> > >>>>
> > >>>> As for the unrecognised IP, I'm afraid I can't answer that one. It
> must
> > >>>> be coming from somewhere - it's just a case of finding out where.
> > >>>> Following
> > >>>> the initial population/depopulation of the node list, DMQ simply
> > >>>> repeats
> > >>>> the process over and over until it eventually connects to one or
> more
> > >>>> other
> > >>>> nodes. Are you running on a public cloud somewhere or is it your
> own,
> > >>>> locally running Kubernetes cluster?
> > >>>>
> > >>>> Cheers,
> > >>>>
> > >>>> Charles
> > >>>>
> > >>>>
> > >>>> On 26 July 2018 at 15:30, Aleksandar Sosic <alex.sosic at evosip.cloud
> >
> > >>>>
> > >>>> wrote:
> > >>>>> Hi Charles,
> > >>>>>
> > >>>>> We're so glad about the improvements you just committed! Thanks!
> > >>>>>
> > >>>>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11
> > >>>>> 65+xenial
> > >>>>> Kamailio starts even if the DNS record does not exist at first,
> that's
> > >>>>> great. I'm having this nodes up and running:
> > >>>>> ```
> > >>>>> proxy-66f79498cc-8ws6d 3/3 Running 0
> > >>>>> 4m 172.28.1.4
> > >>>>> proxy-66f79498cc-b68dd 3/3 Running 0
> > >>>>> 4m 172.28.1.5
> > >>>>> proxy-66f79498cc-lhbxg 3/3 Running 0
> > >>>>> 4m 172.28.1.6
> > >>>>> proxy-66f79498cc-png6p 3/3 Running 0
> > >>>>> 4m 172.28.1.7
> > >>>>> ```
> > >>>>>
> > >>>>> In those 4-5 minutes I've experienced a population and
> depopulation of
> > >>>>> the dmq nodes.
> > >>>>>
> > >>>>> For example:
> > >>>>> ```
> > >>>>> kamcmd> dmq.list_nodes
> > >>>>> {
> > >>>>> host: 172.28.1.4
> > >>>>> port: 5061
> > >>>>> resolved_ip: 172.28.1.4
> > >>>>> status: pending
> > >>>>> last_notification: 0
> > >>>>> local: 0
> > >>>>> }
> > >>>>> {
> > >>>>> host: 172.28.1.6
> > >>>>> port: 5061
> > >>>>> resolved_ip: 172.28.1.6
> > >>>>> status: pending
> > >>>>> last_notification: 0
> > >>>>> local: 0
> > >>>>> }
> > >>>>> {
> > >>>>> host: 40.100.109.113
> > >>>>> port: 5061
> > >>>>> resolved_ip: 40.100.109.113
> > >>>>> status: pending
> > >>>>> last_notification: 0
> > >>>>> local: 0
> > >>>>> }
> > >>>>> {
> > >>>>> host: 172.28.1.5
> > >>>>> port: 5061
> > >>>>> resolved_ip: 172.28.1.5
> > >>>>> status: active
> > >>>>> last_notification: 0
> > >>>>> local: 1
> > >>>>> }
> > >>>>> ```
> > >>>>>
> > >>>>> And after a few seconds only the loca node itself:
> > >>>>> ```
> > >>>>> kamcmd> dmq.list_nodes
> > >>>>> {
> > >>>>> host: 172.28.1.5
> > >>>>> port: 5061
> > >>>>> resolved_ip: 172.28.1.5
> > >>>>> status: active
> > >>>>> last_notification: 0
> > >>>>> local: 1
> > >>>>> }
> > >>>>> ```
> > >>>>> And then again the 3 nodes and the 40.100.109.113 which is a
> Microsoft
> > >>>>> IP and I don't know what's doing that IP into my dmq server list.
> > >>>>>
> > >>>>> The DNS record is this:
> > >>>>> ```
> > >>>>> root at proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service
> > >>>>> Server: 10.96.0.10
> > >>>>> Address: 10.96.0.10#53
> > >>>>>
> > >>>>> Non-authoritative answer:
> > >>>>> Name: dmq-proxy-service.alex.svc.cluster.local
> > >>>>> Address: 172.28.1.7
> > >>>>> Name: dmq-proxy-service.alex.svc.cluster.local
> > >>>>> Address: 172.28.1.6
> > >>>>> Name: dmq-proxy-service.alex.svc.cluster.local
> > >>>>> Address: 172.28.1.5
> > >>>>> Name: dmq-proxy-service.alex.svc.cluster.local
> > >>>>> Address: 172.28.1.4
> > >>>>> ```
> > >>>>> Any clues why that ip is in the dmq nodes list? And why .7 is
> missing?
> > >>>>> Kind regards,
> > >>>>> --
> > >>>>> Aleksandar Sosic
> > >>>>> mail: alex.sosic at evosip.cloud
> > >>>>>
> > >>>>> On Wed, Jul 25, 2018 at 3:06 PM Charles Chance
> > >>>>>
> > >>>>> <charles.chance at sipcentric.com> wrote:
> > >>>>> > Hi,
> > >>>>> >
> > >>>>> > I have just pushed some changes to master - one of these allows
> > >>>>>
> > >>>>> startup to continue even if initial node resolution fails.
> > >>>>>
> > >>>>> > There are some other improvements, too, which I have been
> planning
> > >>>>>
> > >>>>> to push for some time and which should also help in your situation.
> > >>>>>
> > >>>>> > Can you try again with these changes applied and let me know the
> > >>>>>
> > >>>>> outcome?
> > >>>>>
> > >>>>> > Cheers,
> > >>>>> >
> > >>>>> > Charles
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> Kamailio (SER) - Users Mailing List
> > >>>>> sr-users at lists.kamailio.org
> > >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> > >>>>
> > >>>> --
> > >>>> *Charles Chance*
> > >>>> Managing Director
> > >>>>
> > >>>> t. 0330 120 1200 m. 07932 063 891
> > >>>>
> > >>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592.
> > >>>> Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt
> > >>>> Street,
> > >>>> Birmingham Science Park, Birmingham B7 4BB.
> > >>>>
> > >>>> _______________________________________________
> > >>>> Kamailio (SER) - Users Mailing List
> > >>>> sr-users at lists.kamailio.org
> > >>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> > >>
> > >> _______________________________________________
> > >> Kamailio (SER) - Users Mailing List
> > >> sr-users at lists.kamailio.org
> > >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
> > >
> > > --
> > > *Charles Chance*
> > > Managing Director
> > >
> > > t. 0330 120 1200 m. 07932 063 891
> > >
> > > Sipcentric Ltd. Company registered in England & Wales no. 7365592.
> > > Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt
> > > Street,
> > > Birmingham Science Park, Birmingham B7 4BB.
>
>
> --
> Henning Westerholt
> https://skalatan.de/blog/
>
--
paolo visintin
direttore tecnico
------------------------------
timenet srl | www.timenet.it | tel 05711738000
assistenza tecnica: assistenza at timenet.it
ufficio commerciale: sales at timenet.it
ufficio amministrativo: contabilita at timenet.it
twitter.timenet.it | linkedin.timenet.it
<http://timenet.it/email/redirect.php>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20180805/7c88221b/attachment.html>
More information about the sr-users
mailing list