[SR-Users] DMQ problems
Charles Chance
charles.chance at sipcentric.com
Thu Aug 2 15:29:47 CEST 2018
Hi Paolo,
kamailio "resolves" A host but does not query the DNS
What do you mean by this?
I am not overly familiar with Kamailio's DNS internals, but what do you see
if you run the following commands?
kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
Cheers,
Charles
On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
paolo.visintin at evosip.cloud> wrote:
> Hello again!
> Another interesting thing found running in debug=3
>
> 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding
> DMQ node A host dmq-router-service=sip:42.100.109.113:5062
>
> so , kamailio "resolves" A host but does not query the DNS
> this is dump of dns request inside kamailio container:
>
> 147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d
> SRV dmq-router-service.paolo.svc.cluster.local
> 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response
> 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0
> 3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
> 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 A
> dmq-router-service.paolo.svc.cluster.local
> 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response
> 0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
>
>
>
> *Paolo Visintin*
> *CTO*
> evosip.cloud
> [image: Risultati immagini per evosip]
>
> 2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
> paolo.visintin at evosip.cloud>:
>
>>
>> Hello Charles,
>> sorry for late reply, we've had some issues rebuilding docker images with
>> kamailo nightly build!
>>
>> We are running on our local, private on premise kubernetes cluster so
>> connections from outside is simply impossible (in dev we are using a
>> "closed" network system)
>>
>> I've just tried to :
>> - open a tcpdump trace in the instance tracing only the DMQ port defined
>> [5062] - (tcpdump -i eth0 port 5062)
>> - packet coming from nowhere
>> - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
>> - seeing this in tcpdump :
>>
>> root at router-0:/# tcpdump -i eth0 port 5062
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
>> 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 >
>> 42.100.109.113.5062: UDP, length 497
>> 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 >
>> 42.100.109.113.5062: UDP, length 497
>> 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 >
>> 42.100.109.113.5062: UDP, length 497
>>
>> so seems that kamailio is trying to contact the foreign ip
>>
>> if i check inside the configuration:
>> modparam("dmq", "server_address", DMQ_SERVER_ADDRESS)
>> modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS)
>> modparam("dmq", "multi_notify", 1)
>> modparam("dmq", "num_workers", 4)
>> modparam("dmq", "ping_interval", 15)
>>
>>
>> ...
>>
>> if(is_method("KDMQ") && dst_port == DMQ_PORT){
>> dmq_handle_message();
>> exit;
>> }
>>
>>
>> and in DMQ_NOTIFICATION_ADDRESS I have :
>> #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
>>
>> dmq-router-service DNS resolves:
>> > dmq-router-service.paolo.svc.cluster.local
>> Server: 10.96.0.10
>> Address: 10.96.0.10#53
>>
>> Non-authoritative answer:
>> Name: dmq-router-service.paolo.svc.cluster.local
>> Address: 172.28.0.240
>>
>>
>> so -nowhere- i am able to find 42.100.109.113
>>
>>
>> and in kamailio log:
>> 13(821) ERROR: dmq [notification_peer.c:599]:
>> notification_resp_callback_f(): deleting server sip:42.100.109.113:5062
>> because of failed request
>>
>>
>>
>> Cheers,
>>
>> Paolo
>>
>> 2018-07-26 18:14 GMT+02:00 Charles Chance <charles.chance at sipcentric.com>
>> :
>>
>>> Hi Aleksandar,
>>>
>>> The initial depopulation of the nodes (following a period of 'pending'
>>> state) is due to no response being received from them. Are you able to
>>> trace the messages to/from one of them to confirm what is happening there?
>>>
>>> As for the unrecognised IP, I'm afraid I can't answer that one. It must
>>> be coming from somewhere - it's just a case of finding out where. Following
>>> the initial population/depopulation of the node list, DMQ simply repeats
>>> the process over and over until it eventually connects to one or more other
>>> nodes. Are you running on a public cloud somewhere or is it your own,
>>> locally running Kubernetes cluster?
>>>
>>> Cheers,
>>>
>>> Charles
>>>
>>>
>>> On 26 July 2018 at 15:30, Aleksandar Sosic <alex.sosic at evosip.cloud>
>>> wrote:
>>>
>>>> Hi Charles,
>>>>
>>>> We're so glad about the improvements you just committed! Thanks!
>>>>
>>>> Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11
>>>> 65+xenial
>>>> Kamailio starts even if the DNS record does not exist at first, that's
>>>> great. I'm having this nodes up and running:
>>>> ```
>>>> proxy-66f79498cc-8ws6d 3/3 Running 0
>>>> 4m 172.28.1.4
>>>> proxy-66f79498cc-b68dd 3/3 Running 0
>>>> 4m 172.28.1.5
>>>> proxy-66f79498cc-lhbxg 3/3 Running 0
>>>> 4m 172.28.1.6
>>>> proxy-66f79498cc-png6p 3/3 Running 0
>>>> 4m 172.28.1.7
>>>> ```
>>>>
>>>> In those 4-5 minutes I've experienced a population and depopulation of
>>>> the dmq nodes.
>>>>
>>>> For example:
>>>> ```
>>>> kamcmd> dmq.list_nodes
>>>> {
>>>> host: 172.28.1.4
>>>> port: 5061
>>>> resolved_ip: 172.28.1.4
>>>> status: pending
>>>> last_notification: 0
>>>> local: 0
>>>> }
>>>> {
>>>> host: 172.28.1.6
>>>> port: 5061
>>>> resolved_ip: 172.28.1.6
>>>> status: pending
>>>> last_notification: 0
>>>> local: 0
>>>> }
>>>> {
>>>> host: 40.100.109.113
>>>> port: 5061
>>>> resolved_ip: 40.100.109.113
>>>> status: pending
>>>> last_notification: 0
>>>> local: 0
>>>> }
>>>> {
>>>> host: 172.28.1.5
>>>> port: 5061
>>>> resolved_ip: 172.28.1.5
>>>> status: active
>>>> last_notification: 0
>>>> local: 1
>>>> }
>>>> ```
>>>>
>>>> And after a few seconds only the loca node itself:
>>>> ```
>>>> kamcmd> dmq.list_nodes
>>>> {
>>>> host: 172.28.1.5
>>>> port: 5061
>>>> resolved_ip: 172.28.1.5
>>>> status: active
>>>> last_notification: 0
>>>> local: 1
>>>> }
>>>> ```
>>>> And then again the 3 nodes and the 40.100.109.113 which is a Microsoft
>>>> IP and I don't know what's doing that IP into my dmq server list.
>>>>
>>>> The DNS record is this:
>>>> ```
>>>> root at proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service
>>>> Server: 10.96.0.10
>>>> Address: 10.96.0.10#53
>>>>
>>>> Non-authoritative answer:
>>>> Name: dmq-proxy-service.alex.svc.cluster.local
>>>> Address: 172.28.1.7
>>>> Name: dmq-proxy-service.alex.svc.cluster.local
>>>> Address: 172.28.1.6
>>>> Name: dmq-proxy-service.alex.svc.cluster.local
>>>> Address: 172.28.1.5
>>>> Name: dmq-proxy-service.alex.svc.cluster.local
>>>> Address: 172.28.1.4
>>>> ```
>>>> Any clues why that ip is in the dmq nodes list? And why .7 is missing?
>>>> Kind regards,
>>>> --
>>>> Aleksandar Sosic
>>>> mail: alex.sosic at evosip.cloud
>>>>
>>>> On Wed, Jul 25, 2018 at 3:06 PM Charles Chance
>>>> <charles.chance at sipcentric.com> wrote:
>>>> >
>>>> > Hi,
>>>> >
>>>> > I have just pushed some changes to master - one of these allows
>>>> startup to continue even if initial node resolution fails.
>>>> >
>>>> > There are some other improvements, too, which I have been planning to
>>>> push for some time and which should also help in your situation.
>>>> >
>>>> > Can you try again with these changes applied and let me know the
>>>> outcome?
>>>> >
>>>> > Cheers,
>>>> >
>>>> > Charles
>>>>
>>>> _______________________________________________
>>>> Kamailio (SER) - Users Mailing List
>>>> sr-users at lists.kamailio.org
>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>
>>>
>>>
>>> --
>>> *Charles Chance*
>>> Managing Director
>>>
>>> t. 0330 120 1200 m. 07932 063 891
>>>
>>> Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered
>>> office: Faraday Wharf, Innovation Birmingham Campus, Holt Street,
>>> Birmingham Science Park, Birmingham B7 4BB.
>>>
>>> _______________________________________________
>>> Kamailio (SER) - Users Mailing List
>>> sr-users at lists.kamailio.org
>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>>
>>
>
> _______________________________________________
> Kamailio (SER) - Users Mailing List
> sr-users at lists.kamailio.org
> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
>
>
--
*Charles Chance*
Managing Director
t. 0330 120 1200 m. 07932 063 891
--
Sipcentric Ltd.
Company registered in England & Wales no.
7365592. Registered
office: Faraday Wharf, Innovation
Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.kamailio.org/pipermail/sr-users/attachments/20180802/1868ae59/attachment.html>
More information about the sr-users
mailing list