Hello Henning, <div>Thanks for precious indications! </div><div>I've tried to disable dns caching but nothing changed</div><div>Also captured everything on the network interface, but nothing discovered!</div><div>I've changed also dns server (from kube DNs to core dns) without success. </div><div><br></div><div>I'll try to reproduce in vm environment with power dns and capturing also on the dns side </div><div><br></div><div><br></div><div><br><div class="gmail_quote"><div dir="ltr">Il Gio 2 Ago 2018, 20:40 Henning Westerholt <<a href="mailto:hw@kamailio.org">hw@kamailio.org</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin - <br>
evosip.cloud:<br>
> I am able to see<br>
> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding<br>
> DMQ node A host dmq-router-service=sip:<a href="http://42.100.109.113:5062" rel="noreferrer" target="_blank">42.100.109.113:5062</a><br>
> <br>
> but in tcpdump no DNS query<br>
> <br>
> Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c<br>
> luster.local" and starting kamailio at the same time the result is always<br>
> the correct ip, no 42.100.109.113 coming up so this is very strange!<br>
<br>
Hello,<br>
<br>
without looking too much into that, I have two remarks which may worth <br>
investigating:<br>
<br>
- for DNS names in the configuration file Kamailio does a lookup during <br>
startup and "compiles" this to structure which is evaluated during run time<br>
<br>
- there is also a DNS caching functionality, which could cause something like <br>
you observed. You can deactivate this with a core variable, I think.<br>
<br>
Best regards,<br>
<br>
Henning<br>
<br>
<br>
> 2018-08-02 15:29 GMT+02:00 Charles Chance <<a href="mailto:charles.chance@sipcentric.com" target="_blank">charles.chance@sipcentric.com</a>>:<br>
> > Hi Paolo,<br>
> > <br>
> > kamailio "resolves" A host but does not query the DNS<br>
> > <br>
> > <br>
> > What do you mean by this?<br>
> > <br>
> > I am not overly familiar with Kamailio's DNS internals, but what do you<br>
> > see if you run the following commands?<br>
> > <br>
> > kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local<br>
> > <br>
> > kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local<br>
> > <br>
> > Cheers,<br>
> > <br>
> > Charles<br>
> > <br>
> > <br>
> > On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <<br>
> > <br>
> > paolo.visintin@evosip.cloud> wrote:<br>
> >> Hello again!<br>
> >> Another interesting thing found running in debug=3<br>
> >> <br>
> >> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding<br>
> >> DMQ node A host dmq-router-service=sip:<a href="http://42.100.109.113:5062" rel="noreferrer" target="_blank">42.100.109.113:5062</a><br>
> >> <br>
> >> so , kamailio "resolves" A host but does not query the DNS<br>
> >> this is dump of dns request inside kamailio container:<br>
> >> <br>
> >> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d<br>
> >> SRV dmq-router-service.paolo.svc.cluster.local<br>
> >> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query<br>
> >> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10<br>
> >> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A<br>
> >> 172.28.0.240<br>
> >> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904<br>
> >> A dmq-router-service.paolo.svc.cluster.local<br>
> >> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query<br>
> >> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A<br>
> >> 172.28.0.240<br>
> >> <br>
> >> <br>
> >> <br>
> >> *Paolo Visintin*<br>
> >> *CTO*<br>
> >> evosip.cloud<br>
> >> [image: Risultati immagini per evosip]<br>
> >> <br>
> >> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <<br>
> >> <br>
> >> paolo.visintin@evosip.cloud>:<br>
> >>> Hello Charles,<br>
> >>> sorry for late reply, we've had some issues rebuilding docker images<br>
> >>> with kamailo nightly build!<br>
> >>> <br>
> >>> We are running on our local, private on premise kubernetes cluster so<br>
> >>> connections from outside is simply impossible (in dev we are using a<br>
> >>> "closed" network system)<br>
> >>> <br>
> >>> I've just tried to :<br>
> >>> - open a tcpdump trace in the instance tracing only the DMQ port defined<br>
> >>> [5062] - (tcpdump -i eth0 port 5062)<br>
> >>> - packet coming from nowhere<br>
> >>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)<br>
> >>> - seeing this in tcpdump :<br>
> >>> <br>
> >>> root@router-0:/# tcpdump -i eth0 port 5062<br>
> >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol<br>
> >>> decode<br>
> >>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144<br>
> >>> bytes<br>
> >>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 ><br>
> >>> 42.100.109.113.5062: UDP, length 497<br>
> >>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 ><br>
> >>> 42.100.109.113.5062: UDP, length 497<br>
> >>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 ><br>
> >>> 42.100.109.113.5062: UDP, length 497<br>
> >>> <br>
> >>> so seems that kamailio is trying to contact the foreign ip<br>
> >>> <br>
> >>> if i check inside the configuration:<br>
> >>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)<br>
> >>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)<br>
> >>> modparam("dmq", "multi_notify", 1)<br>
> >>> modparam("dmq", "num_workers", 4)<br>
> >>> modparam("dmq", "ping_interval", 15)<br>
> >>> <br>
> >>> <br>
> >>> ...<br>
> >>> <br>
> >>> if(is_method("KDMQ") && dst_port == DMQ_PORT){<br>
> >>> <br>
> >>> dmq_handle_message();<br>
> >>> exit;<br>
> >>> <br>
> >>> }<br>
> >>> <br>
> >>> and in DMQ_NOTIFICATION_ADDRESS I have :<br>
> >>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"<br>
> >>> <br>
> >>> dmq-router-service DNS resolves:<br>
> >>> > dmq-router-service.paolo.svc.cluster.local<br>
> >>> <br>
> >>> Server: 10.96.0.10<br>
> >>> Address: 10.96.0.10#53<br>
> >>> <br>
> >>> Non-authoritative answer:<br>
> >>> Name: dmq-router-service.paolo.svc.cluster.local<br>
> >>> Address: 172.28.0.240<br>
> >>> <br>
> >>> <br>
> >>> so -nowhere- i am able to find 42.100.109.113<br>
> >>> <br>
> >>> <br>
> >>> and in kamailio log:<br>
> >>> 13(821) ERROR: dmq [notification_peer.c:599]:<br>
> >>> notification_resp_callback_f(): deleting server sip:<a href="http://42.100.109.113:5062" rel="noreferrer" target="_blank">42.100.109.113:5062</a><br>
> >>> because of failed request<br>
> >>> <br>
> >>> <br>
> >>> <br>
> >>> Cheers,<br>
> >>> <br>
> >>> Paolo<br>
> >>> <br>
> >>> 2018-07-26 18:14 GMT+02:00 Charles Chance <<a href="mailto:charles.chance@sipcentric.com" target="_blank">charles.chance@sipcentric.com</a><br>
> >>> <br>
> >>>> Hi Aleksandar,<br>
> >>>> <br>
> >>>> The initial depopulation of the nodes (following a period of 'pending'<br>
> >>>> state) is due to no response being received from them. Are you able to<br>
> >>>> trace the messages to/from one of them to confirm what is happening<br>
> >>>> there?<br>
> >>>> <br>
> >>>> As for the unrecognised IP, I'm afraid I can't answer that one. It must<br>
> >>>> be coming from somewhere - it's just a case of finding out where.<br>
> >>>> Following<br>
> >>>> the initial population/depopulation of the node list, DMQ simply<br>
> >>>> repeats<br>
> >>>> the process over and over until it eventually connects to one or more<br>
> >>>> other<br>
> >>>> nodes. Are you running on a public cloud somewhere or is it your own,<br>
> >>>> locally running Kubernetes cluster?<br>
> >>>> <br>
> >>>> Cheers,<br>
> >>>> <br>
> >>>> Charles<br>
> >>>> <br>
> >>>> <br>
> >>>> On 26 July 2018 at 15:30, Aleksandar Sosic <alex.sosic@evosip.cloud><br>
> >>>> <br>
> >>>> wrote:<br>
> >>>>> Hi Charles,<br>
> >>>>> <br>
> >>>>> We're so glad about the improvements you just committed! Thanks!<br>
> >>>>> <br>
> >>>>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11<br>
> >>>>> 65+xenial<br>
> >>>>> Kamailio starts even if the DNS record does not exist at first, that's<br>
> >>>>> great. I'm having this nodes up and running:<br>
> >>>>> ```<br>
> >>>>> proxy-66f79498cc-8ws6d 3/3 Running 0<br>
> >>>>> 4m 172.28.1.4<br>
> >>>>> proxy-66f79498cc-b68dd 3/3 Running 0<br>
> >>>>> 4m 172.28.1.5<br>
> >>>>> proxy-66f79498cc-lhbxg 3/3 Running 0<br>
> >>>>> 4m 172.28.1.6<br>
> >>>>> proxy-66f79498cc-png6p 3/3 Running 0<br>
> >>>>> 4m 172.28.1.7<br>
> >>>>> ```<br>
> >>>>> <br>
> >>>>> In those 4-5 minutes I've experienced a population and depopulation of<br>
> >>>>> the dmq nodes.<br>
> >>>>> <br>
> >>>>> For example:<br>
> >>>>> ```<br>
> >>>>> kamcmd> dmq.list_nodes<br>
> >>>>> {<br>
> >>>>> host: 172.28.1.4<br>
> >>>>> port: 5061<br>
> >>>>> resolved_ip: 172.28.1.4<br>
> >>>>> status: pending<br>
> >>>>> last_notification: 0<br>
> >>>>> local: 0<br>
> >>>>> }<br>
> >>>>> {<br>
> >>>>> host: 172.28.1.6<br>
> >>>>> port: 5061<br>
> >>>>> resolved_ip: 172.28.1.6<br>
> >>>>> status: pending<br>
> >>>>> last_notification: 0<br>
> >>>>> local: 0<br>
> >>>>> }<br>
> >>>>> {<br>
> >>>>> host: 40.100.109.113<br>
> >>>>> port: 5061<br>
> >>>>> resolved_ip: 40.100.109.113<br>
> >>>>> status: pending<br>
> >>>>> last_notification: 0<br>
> >>>>> local: 0<br>
> >>>>> }<br>
> >>>>> {<br>
> >>>>> host: 172.28.1.5<br>
> >>>>> port: 5061<br>
> >>>>> resolved_ip: 172.28.1.5<br>
> >>>>> status: active<br>
> >>>>> last_notification: 0<br>
> >>>>> local: 1<br>
> >>>>> }<br>
> >>>>> ```<br>
> >>>>> <br>
> >>>>> And after a few seconds only the loca node itself:<br>
> >>>>> ```<br>
> >>>>> kamcmd> dmq.list_nodes<br>
> >>>>> {<br>
> >>>>> host: 172.28.1.5<br>
> >>>>> port: 5061<br>
> >>>>> resolved_ip: 172.28.1.5<br>
> >>>>> status: active<br>
> >>>>> last_notification: 0<br>
> >>>>> local: 1<br>
> >>>>> }<br>
> >>>>> ```<br>
> >>>>> And then again the 3 nodes and the 40.100.109.113 which is a Microsoft<br>
> >>>>> IP and I don't know what's doing that IP into my dmq server list.<br>
> >>>>> <br>
> >>>>> The DNS record is this:<br>
> >>>>> ```<br>
> >>>>> root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service<br>
> >>>>> Server: 10.96.0.10<br>
> >>>>> Address: 10.96.0.10#53<br>
> >>>>> <br>
> >>>>> Non-authoritative answer:<br>
> >>>>> Name: dmq-proxy-service.alex.svc.cluster.local<br>
> >>>>> Address: 172.28.1.7<br>
> >>>>> Name: dmq-proxy-service.alex.svc.cluster.local<br>
> >>>>> Address: 172.28.1.6<br>
> >>>>> Name: dmq-proxy-service.alex.svc.cluster.local<br>
> >>>>> Address: 172.28.1.5<br>
> >>>>> Name: dmq-proxy-service.alex.svc.cluster.local<br>
> >>>>> Address: 172.28.1.4<br>
> >>>>> ```<br>
> >>>>> Any clues why that ip is in the dmq nodes list? And why .7 is missing?<br>
> >>>>> Kind regards,<br>
> >>>>> --<br>
> >>>>> Aleksandar Sosic<br>
> >>>>> mail: alex.sosic@evosip.cloud<br>
> >>>>> <br>
> >>>>> On Wed, Jul 25, 2018 at 3:06 PM Charles Chance<br>
> >>>>> <br>
> >>>>> <<a href="mailto:charles.chance@sipcentric.com" target="_blank">charles.chance@sipcentric.com</a>> wrote:<br>
> >>>>> > Hi,<br>
> >>>>> > <br>
> >>>>> > I have just pushed some changes to master - one of these allows<br>
> >>>>> <br>
> >>>>> startup to continue even if initial node resolution fails.<br>
> >>>>> <br>
> >>>>> > There are some other improvements, too, which I have been planning<br>
> >>>>> <br>
> >>>>> to push for some time and which should also help in your situation.<br>
> >>>>> <br>
> >>>>> > Can you try again with these changes applied and let me know the<br>
> >>>>> <br>
> >>>>> outcome?<br>
> >>>>> <br>
> >>>>> > Cheers,<br>
> >>>>> > <br>
> >>>>> > Charles<br>
> >>>>> <br>
> >>>>> _______________________________________________<br>
> >>>>> Kamailio (SER) - Users Mailing List<br>
> >>>>> <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
> >>>>> <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
> >>>> <br>
> >>>> --<br>
> >>>> *Charles Chance*<br>
> >>>> Managing Director<br>
> >>>> <br>
> >>>> t. 0330 120 1200 m. 07932 063 891<br>
> >>>> <br>
> >>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592.<br>
> >>>> Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt<br>
> >>>> Street,<br>
> >>>> Birmingham Science Park, Birmingham B7 4BB.<br>
> >>>> <br>
> >>>> _______________________________________________<br>
> >>>> Kamailio (SER) - Users Mailing List<br>
> >>>> <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
> >>>> <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
> >> <br>
> >> _______________________________________________<br>
> >> Kamailio (SER) - Users Mailing List<br>
> >> <a href="mailto:sr-users@lists.kamailio.org" target="_blank">sr-users@lists.kamailio.org</a><br>
> >> <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" rel="noreferrer" target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><br>
> > <br>
> > --<br>
> > *Charles Chance*<br>
> > Managing Director<br>
> > <br>
> > t. 0330 120 1200 m. 07932 063 891<br>
> > <br>
> > Sipcentric Ltd. Company registered in England & Wales no. 7365592.<br>
> > Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt<br>
> > Street,<br>
> > Birmingham Science Park, Birmingham B7 4BB.<br>
<br>
<br>
-- <br>
Henning Westerholt<br>
<a href="https://skalatan.de/blog/" rel="noreferrer" target="_blank">https://skalatan.de/blog/</a><br>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><br><img hspace="0" border="0" vspace="0" src="http://timenet.it/email/timenet.jpg"></div><div dir="ltr" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><font face="monospace, monospace">paolo visintin</font></div><div dir="ltr" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><font face="monospace, monospace">direttore tecnico</font><hr style="border-top-width:2px;min-height:1px;border-bottom:0px solid rgb(105,171,211);color:rgb(105,171,211);width:405px;margin:2px 2px 2px 0px;padding:0px;background:rgb(105,171,211)"><font face="monospace, monospace">timenet srl | <a href="http://www.timenet.it/" target="_blank" style="color:rgb(0,0,0);text-decoration:none">www.timenet.it</a> | tel <a href="tel:05711738000" value="+3905711738000" target="_blank" style="color:rgb(17,85,204)">05711738000</a></font><font face="monospace, monospace"> </font></div><div dir="ltr" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><font face="monospace, monospace">assistenza tecnica: <a href="mailto:assistenza@timenet.it" target="_blank" style="color:rgb(0,0,0);text-decoration:none">assistenza@timenet.it</a></font></div><div dir="ltr" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><font face="monospace, monospace">ufficio commerciale: <a href="mailto:sales@timenet.it" target="_blank" style="color:rgb(0,0,0);text-decoration:none">sales@timenet.it</a></font></div><div dir="ltr" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><font face="monospace, monospace">ufficio amministrativo: <a href="mailto:contabilita@timenet.it" target="_blank" style="color:rgb(0,0,0);text-decoration:none">contabilita@timenet.it</a></font><font face="monospace, monospace"> </font></div><div dir="ltr" style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;line-height:normal"><font face="monospace, monospace"><a href="http://twitter.timenet.it/" target="_blank" style="color:rgb(0,0,0);text-decoration:none">twitter.timenet.it</a> | <a href="http://linkedin.timenet.it/" target="_blank" style="color:rgb(0,0,0);text-decoration:none">linkedin.timenet.it</a><br></font><a href="http://timenet.it/email/redirect.php" target="_blank" style="color:rgb(17,85,204)"><img alt="" hspace="0" border="0" vspace="6" src="http://timenet.it/email/spot.jpg"></a></div></div></div>