Re: [SR-Users] DMQ problems

I am able to see 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062 but in tcpdump no DNS query Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c luster.local" and starting kamailio at the same time the result is always the correct ip, no 42.100.109.113 coming up so this is very strange!

2018-08-02 15:29 GMT+02:00 Charles Chance &lt;charles.chance(a)sipcentric.com&gt;om>: > Hi Paolo, > > kamailio "resolves" A host but does not query the DNS > > > What do you mean by this? > > I am not overly familiar with Kamailio's DNS internals, but what do you > see if you run the following commands? > > kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local > > kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local > > Cheers, > > Charles > > > On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud < > > paolo.visintin(a)evosip.cloud&gt; wrote: >> Hello again! >> Another interesting thing found running in debug=3 >> >> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding >> DMQ node A host dmq-router-service=sip:42.100.109.113:5062 >> >> so , kamailio "resolves" A host but does not query the DNS >> this is dump of dns request inside kamailio container: >> >> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d >> SRV dmq-router-service.paolo.svc.cluster.local >> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query >> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 >> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A >> 172.28.0.240 >> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 >> A dmq-router-service.paolo.svc.cluster.local >> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query >> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A >> 172.28.0.240 >> >> >> >> *Paolo Visintin* >> *CTO* >> evosip.cloud >> [image: Risultati immagini per evosip] >> >> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud < >> >> paolo.visintin(a)evosip.cloud&gt;gt;: >>> Hello Charles, >>> sorry for late reply, we've had some issues rebuilding docker images >>> with kamailo nightly build! >>> >>> We are running on our local, private on premise kubernetes cluster so >>> connections from outside is simply impossible (in dev we are using a >>> "closed" network system) >>> >>> I've just tried to : >>> - open a tcpdump trace in the instance tracing only the DMQ port defined >>> [5062] - (tcpdump -i eth0 port 5062) >>> - packet coming from nowhere >>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e) >>> - seeing this in tcpdump : >>> >>> root@router-0:/# tcpdump -i eth0 port 5062 >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol >>> decode >>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 >>> bytes >>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > >>> 42.100.109.113.5062: UDP, length 497 >>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > >>> 42.100.109.113.5062: UDP, length 497 >>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > >>> 42.100.109.113.5062: UDP, length 497 >>> >>> so seems that kamailio is trying to contact the foreign ip >>> >>> if i check inside the configuration: >>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) >>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) >>> modparam("dmq", "multi_notify", 1) >>> modparam("dmq", "num_workers", 4) >>> modparam("dmq", "ping_interval", 15) >>> >>> >>> ... >>> >>> if(is_method("KDMQ") && dst_port == DMQ_PORT){ >>> >>> dmq_handle_message(); >>> exit; >>> >>> } >>> >>> and in DMQ_NOTIFICATION_ADDRESS I have : >>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062" >>> >>> dmq-router-service DNS resolves: >>> > dmq-router-service.paolo.svc.cluster.local >>> >>> Server: 10.96.0.10 >>> Address: 10.96.0.10#53 >>> >>> Non-authoritative answer: >>> Name: dmq-router-service.paolo.svc.cluster.local >>> Address: 172.28.0.240 >>> >>> >>> so -nowhere- i am able to find 42.100.109.113 >>> >>> >>> and in kamailio log: >>> 13(821) ERROR: dmq [notification_peer.c:599]: >>> notification_resp_callback_f(): deleting server sip:42.100.109.113:5062 >>> because of failed request >>> >>> >>> >>> Cheers, >>> >>> Paolo >>> >>> 2018-07-26 18:14 GMT+02:00 Charles Chance &lt;charles.chance(a)sipcentric.com >>> >>>> Hi Aleksandar, >>>> >>>> The initial depopulation of the nodes (following a period of 'pending' >>>> state) is due to no response being received from them. Are you able to >>>> trace the messages to/from one of them to confirm what is happening >>>> there? >>>> >>>> As for the unrecognised IP, I'm afraid I can't answer that one. It must >>>> be coming from somewhere - it's just a case of finding out where. >>>> Following >>>> the initial population/depopulation of the node list, DMQ simply >>>> repeats >>>> the process over and over until it eventually connects to one or more >>>> other >>>> nodes. Are you running on a public cloud somewhere or is it your own, >>>> locally running Kubernetes cluster? >>>> >>>> Cheers, >>>> >>>> Charles >>>> >>>> >>>> On 26 July 2018 at 15:30, Aleksandar Sosic &lt;alex.sosic(a)evosip.cloud&gt; >>>> >>>> wrote: >>>>> Hi Charles, >>>>> >>>>> We're so glad about the improvements you just committed! Thanks! >>>>> >>>>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11 >>>>> 65+xenial >>>>> Kamailio starts even if the DNS record does not exist at first, that's >>>>> great. I'm having this nodes up and running: >>>>> ``` >>>>> proxy-66f79498cc-8ws6d 3/3 Running 0 >>>>> 4m 172.28.1.4 >>>>> proxy-66f79498cc-b68dd 3/3 Running 0 >>>>> 4m 172.28.1.5 >>>>> proxy-66f79498cc-lhbxg 3/3 Running 0 >>>>> 4m 172.28.1.6 >>>>> proxy-66f79498cc-png6p 3/3 Running 0 >>>>> 4m 172.28.1.7 >>>>> ``` >>>>> >>>>> In those 4-5 minutes I've experienced a population and depopulation of >>>>> the dmq nodes. >>>>> >>>>> For example: >>>>> ``` >>>>> kamcmd> dmq.list_nodes >>>>> { >>>>> host: 172.28.1.4 >>>>> port: 5061 >>>>> resolved_ip: 172.28.1.4 >>>>> status: pending >>>>> last_notification: 0 >>>>> local: 0 >>>>> } >>>>> { >>>>> host: 172.28.1.6 >>>>> port: 5061 >>>>> resolved_ip: 172.28.1.6 >>>>> status: pending >>>>> last_notification: 0 >>>>> local: 0 >>>>> } >>>>> { >>>>> host: 40.100.109.113 >>>>> port: 5061 >>>>> resolved_ip: 40.100.109.113 >>>>> status: pending >>>>> last_notification: 0 >>>>> local: 0 >>>>> } >>>>> { >>>>> host: 172.28.1.5 >>>>> port: 5061 >>>>> resolved_ip: 172.28.1.5 >>>>> status: active >>>>> last_notification: 0 >>>>> local: 1 >>>>> } >>>>> ``` >>>>> >>>>> And after a few seconds only the loca node itself: >>>>> ``` >>>>> kamcmd> dmq.list_nodes >>>>> { >>>>> host: 172.28.1.5 >>>>> port: 5061 >>>>> resolved_ip: 172.28.1.5 >>>>> status: active >>>>> last_notification: 0 >>>>> local: 1 >>>>> } >>>>> ``` >>>>> And then again the 3 nodes and the 40.100.109.113 which is a Microsoft >>>>> IP and I don't know what's doing that IP into my dmq server list. >>>>> >>>>> The DNS record is this: >>>>> ``` >>>>> root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service >>>>> Server: 10.96.0.10 >>>>> Address: 10.96.0.10#53 >>>>> >>>>> Non-authoritative answer: >>>>> Name: dmq-proxy-service.alex.svc.cluster.local >>>>> Address: 172.28.1.7 >>>>> Name: dmq-proxy-service.alex.svc.cluster.local >>>>> Address: 172.28.1.6 >>>>> Name: dmq-proxy-service.alex.svc.cluster.local >>>>> Address: 172.28.1.5 >>>>> Name: dmq-proxy-service.alex.svc.cluster.local >>>>> Address: 172.28.1.4 >>>>> ``` >>>>> Any clues why that ip is in the dmq nodes list? And why .7 is missing? >>>>> Kind regards, >>>>> -- >>>>> Aleksandar Sosic >>>>> mail: alex.sosic(a)evosip.cloud >>>>> >>>>> On Wed, Jul 25, 2018 at 3:06 PM Charles Chance >>>>> >>>>> &lt;charles.chance(a)sipcentric.com&gt; wrote: >>>>> > Hi, >>>>> > >>>>> > I have just pushed some changes to master - one of these allows >>>>> >>>>> startup to continue even if initial node resolution fails. >>>>> >>>>> > There are some other improvements, too, which I have been planning >>>>> >>>>> to push for some time and which should also help in your situation. >>>>> >>>>> > Can you try again with these changes applied and let me know the >>>>> >>>>> outcome? >>>>> >>>>> > Cheers, >>>>> > >>>>> > Charles >>>>> >>>>> _______________________________________________ >>>>> Kamailio (SER) - Users Mailing List >>>>> sr-users(a)lists.kamailio.org >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> -- >>>> *Charles Chance* >>>> Managing Director >>>> >>>> t. 0330 120 1200 m. 07932 063 891 >>>> >>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. >>>> Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt >>>> Street, >>>> Birmingham Science Park, Birmingham B7 4BB. >>>> >>>> _______________________________________________ >>>> Kamailio (SER) - Users Mailing List >>>> sr-users(a)lists.kamailio.org >>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> >> _______________________________________________ >> Kamailio (SER) - Users Mailing List >> sr-users(a)lists.kamailio.org >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > -- > *Charles Chance* > Managing Director > > t. 0330 120 1200 m. 07932 063 891 > > Sipcentric Ltd. Company registered in England & Wales no. 7365592. > Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt > Street, > Birmingham Science Park, Birmingham B7 4BB.