[SR-Users] DMQ problems
Paolo Visintin - evosip.cloud
paolo.visintin at evosip.cloud
Thu Aug 2 20:33:28 CEST 2018
Hi Charles,
when I say kamailio "resolves" but does not query DNS I mean that in DEBUG
I am able to see
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
DMQ node A host dmq-router-service=sip:42.100.109.113:5062
but in tcpdump no DNS query
Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c
luster.local" and starting kamailio at the same time the result is always
the correct ip, no 42.100.109.113 coming up so this is very strange!
Cheers
*Paolo Visintin*
*CTO*
evosip.cloud
[image: Risultati immagini per evosip]
2018-08-02 15:29 GMT+02:00 Charles Chance <charles.chance at sipcentric.com>:
> Hi Paolo,
>
> kamailio "resolves" A host but does not query the DNS
>
>
> What do you mean by this?
>
> I am not overly familiar with Kamailio's DNS internals, but what do you
> see if you run the following commands?
>
> kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
>
> kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
>
> Cheers,
>
> Charles
>
>
> On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
> paolo.visintin at evosip.cloud> wrote:
>
>> Hello again!
>> Another interesting thing found running in debug=3
>>
>> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
>> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
>>
>> so , kamailio "resolves" A host but does not query the DNS
>> this is dump of dns request inside kamailio container:
>>
>> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d
>> SRV dmq-router-service.paolo.svc.cluster.local
>> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query
>> response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10
>> 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A
>> 172.28.0.240
>> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904
>> A dmq-router-service.paolo.svc.cluster.local
>> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query
>> response 0x1904 A dmq-router-service.paolo.svc.cluster.local A
>> 172.28.0.240
>>
>>
>>
>> *Paolo Visintin*
>> *CTO*
>> evosip.cloud
>> [image: Risultati immagini per evosip]
>>
>> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
>> paolo.visintin at evosip.cloud>:
>>
>>>
>>> Hello Charles,
>>> sorry for late reply, we've had some issues rebuilding docker images
>>> with kamailo nightly build!
>>>
>>> We are running on our local, private on premise kubernetes cluster so
>>> connections from outside is simply impossible (in dev we are using a
>>> "closed" network system)
>>>
>>> I've just tried to :
>>> - open a tcpdump trace in the instance tracing only the DMQ port defined
>>> [5062] - (tcpdump -i eth0 port 5062)
>>> - packet coming from nowhere
>>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
>>> - seeing this in tcpdump :
>>>
>>> root at router-0:/# tcpdump -i eth0 port 5062
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
>>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
>>> 42.100.109.113.5062: UDP, length 497
>>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
>>> 42.100.109.113.5062: UDP, length 497
>>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
>>> 42.100.109.113.5062: UDP, length 497
>>>
>>> so seems that kamailio is trying to contact the foreign ip
>>>
>>> if i check inside the configuration:
>>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
>>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
>>> modparam("dmq", "multi_notify", 1)
>>> modparam("dmq", "num_workers", 4)
>>> modparam("dmq", "ping_interval", 15)
>>>
>>>
>>> ...
>>>
>>> if(is_method("KDMQ") && dst_port == DMQ_PORT){
>>> dmq_handle_message();
>>> exit;
>>> }
>>>
>>>
>>> and in DMQ_NOTIFICATION_ADDRESS I have :
>>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
>>>
>>> dmq-router-service DNS resolves:
>>> > dmq-router-service.paolo.svc.cluster.local
>>> Server: 10.96.0.10
>>> Address: 10.96.0.10#53
>>>
>>> Non-authoritative answer:
>>> Name: dmq-router-service.paolo.svc.cluster.local
>>> Address: 172.28.0.240
>>>
>>>
>>> so -nowhere- i am able to find 42.100.109.113
>>>
>>>
>>> and in kamailio log:
>>> 13(821) ERROR: dmq [notification_peer.c:599]:
>>> notification_resp_callback_f(): deleting server sip:42.100.109.113:5062
>>> because of failed request
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Paolo
>>>
>>> 2018-07-26 18:14 GMT+02:00 Charles Chance <charles.chance at sipcentric.com
>>> >:
>>>
>>>> Hi Aleksandar,
>>>>
>>>> The initial depopulation of the nodes (following a period of 'pending'
>>>> state) is due to no response being received from them. Are you able to
>>>> trace the messages to/from one of them to confirm what is happening there?
>>>>
>>>> As for the unrecognised IP, I'm afraid I can't answer that one. It must
>>>> be coming from somewhere - it's just a case of finding out where. Following
>>>> the initial population/depopulation of the node list, DMQ simply repeats
>>>> the process over and over until it eventually connects to one or more other
>>>> nodes. Are you running on a public cloud somewhere or is it your own,
>>>> locally running Kubernetes cluster?
>>>>
>>>> Cheers,
>>>>
>>>> Charles
>>>>
>>>>
>>>> On 26 July 2018 at 15:30, Aleksandar Sosic <alex.sosic at evosip.cloud>
>>>> wrote:
>>>>
>>>>> Hi Charles,
>>>>>
>>>>> We're so glad about the improvements you just committed! Thanks!
>>>>>
>>>>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11
>>>>> 65+xenial
>>>>> Kamailio starts even if the DNS record does not exist at first, that's
>>>>> great. I'm having this nodes up and running:
>>>>> ```
>>>>> proxy-66f79498cc-8ws6d 3/3 Running 0
>>>>> 4m 172.28.1.4
>>>>> proxy-66f79498cc-b68dd 3/3 Running 0
>>>>> 4m 172.28.1.5
>>>>> proxy-66f79498cc-lhbxg 3/3 Running 0
>>>>> 4m 172.28.1.6
>>>>> proxy-66f79498cc-png6p 3/3 Running 0
>>>>> 4m 172.28.1.7
>>>>> ```
>>>>>
>>>>> In those 4-5 minutes I've experienced a population and depopulation of
>>>>> the dmq nodes.
>>>>>
>>>>> For example:
>>>>> ```
>>>>> kamcmd> dmq.list_nodes
>>>>> {
>>>>> host: 172.28.1.4
>>>>> port: 5061
>>>>> resolved_ip: 172.28.1.4
>>>>> status: pending
>>>>> last_notification: 0
>>>>> local: 0
>>>>> }
>>>>> {
>>>>> host: 172.28.1.6
>>>>> port: 5061
>>>>> resolved_ip: 172.28.1.6
>>>>> status: pending
>>>>> last_notification: 0
>>>>> local: 0
>>>>> }
>>>>> {
>>>>> host: 40.100.109.113
>>>>> port: 5061
>>>>> resolved_ip: 40.100.109.113
>>>>> status: pending
>>>>> last_notification: 0
>>>>> local: 0
>>>>> }
>>>>> {
>>>>> host: 172.28.1.5
>>>>> port: 5061
>>>>> resolved_ip: 172.28.1.5
>>>>> status: active
>>>>> last_notification: 0
>>>>> local: 1
>>>>> }
>>>>> ```
>>>>>
>>>>> And after a few seconds only the loca node itself:
>>>>> ```
>>>>> kamcmd> dmq.list_nodes
>>>>> {
>>>>> host: 172.28.1.5
>>>>> port: 5061
>>>>> resolved_ip: 172.28.1.5
>>>>> status: active
>>>>> last_notification: 0
>>>>> local: 1
>>>>> }
>>>>> ```
>>>>> And then again the 3 nodes and the 40.100.109.113 which is a Microsoft
>>>>> IP and I don't know what's doing that IP into my dmq server list.
>>>>>
>>>>> The DNS record is this:
>>>>> ```
>>>>> root at proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service
>>>>> Server: 10.96.0.10
>>>>> Address: 10.96.0.10#53
>>>>>
>>>>> Non-authoritative answer:
>>>>> Name: dmq-proxy-service.alex.svc.cluster.local
>>>>> Address: 172.28.1.7
>>>>> Name: dmq-proxy-service.alex.svc.cluster.local
>>>>> Address: 172.28.1.6
>>>>> Name: dmq-proxy-service.alex.svc.cluster.local
>>>>> Address: 172.28.1.5
>>>>> Name: dmq-proxy-service.alex.svc.cluster.local
>>>>> Address: 172.28.1.4
>>>>> ```
>>>>> Any clues why that ip is in the dmq nodes list? And why .7 is missing?
>>>>> Kind regards,
>>>>> --
>>>>> Aleksandar Sosic
>>>>> mail: alex.sosic at evosip.cloud
>>>>>
>>>>> On Wed, Jul 25, 2018 at 3:06 PM Charles Chance
>>>>> <charles.chance at sipcentric.com> wrote:
>>>>> >
>>>>> > Hi,
>>>>> >
>>>>> > I have just pushed some changes to master - one of these allows
>>>>> startup to continue even if initial node resolution fails.
>>>>> >
>>>>> > There are some other improvements, too, which I have been planning
>>>>> to push for some time and which should also help in your situation.
>>>>> >
>>>>> > Can you try again with these changes applied and let me know the
>>>>> outcome?
>>>>> >
>>>>> > Cheers,
>>>>> >
>>>>> > Charles
>>>>>
>>>>> _______________________________________________
>>>>> Kamailio (SER) - Users Mailing List
>>>>> sr-users at lists.kamailio.org
>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Charles Chance*
>>>> Managing Director
>>>>
>>>> t. 0330 120 1200 m. 07932 063 891
>>>>
>>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
>>>> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
>>>> Birmingham Science Park, Birmingham B7 4BB.
>>>>
>>>> _______________________________________________
>>>> Kamailio (SER) - Users Mailing List
>>>> sr-users at lists.kamailio.org
>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Kamailio (SER) - Users Mailing List
>> sr-users at lists.kamailio.org
>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>
>>
>
>
> --
> *Charles Chance*
> Managing Director
>
> t. 0330 120 1200 m. 07932 063 891
>
> Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
> Birmingham Science Park, Birmingham B7 4BB.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20180802/7ea95324/attachment.html>
More information about the sr-users
mailing list