[Feature] Node local DNS

[palma21]

Re-architect azure CNI for more resilient DNS

[timja]

Any update on this?

We deployed nodelocaldns and it made a massive difference for us, (#1326 (comment))

[palma21]

Good to know, we are designing it as we speak

[jstewart612]

@palma21 one thing you may want to take back to your folks is the currently proposed nodelocal daemonset is not tolerating itself sufficiently to prevent from inadvertent inability to assign to any nodes with custom taints on them.

I used these tolerations and they resolved that issue for us:

• operator: Exists
  effect: NoExecute
• operator: Exists
  effect: NoSchedule

[JennyLJY]

Any updates on this feature? When is the ETA?
@palma21

[palma21]

It's currently on the committed items for this semester and under design review. Will have more ideas on the concrete eta by the end of the month.

[curtdept]

Hopefully this is higher in the priority list, then would be amazing to have. I get DNS latency bursts constantly.

[bergerx]

Any update on the ETA? If there is now a rough timeline for ETA, we may prefer wait a bit more rather than investing engineering time into something which would be obsolete soon after.

[curtdept]

@bergerx this will make you happy :)

https://github.com/curtdept/aks_nodelocaldns/blob/main/nodelocaldns.yaml

[MiyKh]

My AKS cluster is actually running with kubenet, the DNS service IP is something like 172.17.80.10, but I cant' figure out what is <node-local-address> parameter, is it an IP from the service CIDR? pod CIDR?

[PSanetra]

@MiyKh the <node-local-address> is just some random and not-in-use IP in the 169.254.20.0/16 subnet. It will be used on each node and will not be reachable from outside the node.
See https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/#configuration

The local listen IP address for NodeLocal DNSCache can be any IP in the 169.254.20.0/16 space or any other IP address that can be guaranteed to not collide with any existing IP. This document uses 169.254.20.10 as an example.

[MiyKh]

@PSanetra Thanks for the clarification.The node local dns is deployed into the kube-system namespace but it is deleted by Azure sync system, how can we force AKS to skip deletion on this component?

EDIT: This can be done by removing the labels:
kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile

[PSanetra]

@MiyKh interesting. We still have those labels set on the daemonset, but it is not getting deleted. We are running AKS 1.17.11.

[curtdept]

@MiyKh interesting. We still have those labels set on the daemonset, but it is not getting deleted. We are running AKS 1.17.11.

Same here on 1.19

[MiyKh]

I have the same behavior than this issue: #1435

[curtdept]

I have the same behavior than this issue: #1435

What AKS version?

[MiyKh]

I have the same behavior than this issue: #1435

What AKS version?

This is a 1.18.8 cluster with kubenet networking, removing the labels was the solution but I guess, I would still need to redeploy it after cluter upgrade.

[4c74356b41]

@curtdept @palma21 hey folks, do you have any ETA for us? Will this be backported to 1.17.x or not? thanks!

[djsly]

Any news on an ETA ?

[joaguas]

@4c74356b41 @djsly
Azure CNI now defaults to transparent mode which uses layer 3 routing which means pod-to-pod no longer relies on conntrack and will not be affected by its race condition. This should mitigate the 5s delay on DNS lookups

Upgrading a cluster still using bridge mode will make it use transparent mode instead

[djsly]

@joaguas thanks! I asked a question about querying the status. Wondering how to check if our clusters consumed the update or not yet, since we did performed a few updates.

[4c74356b41]

you can just do az aks nodepool upgrade --node-image-only, this would rollout latest os base image with this fix for sure (no kubernetes version upgrade), if you already have it - it will just return success in 30ish seconds

[djsly]

ok so the CNI transparent mode is baked in the image. good to know. I will try to know which base image contains the fix then.

thanks @4c74356b41

[joaguas]

Hi @djsly , an easy way will be to get a shell in one of the nodes and check either the interfaces or route tables.
If there's an interface called azure0 that means it's still using a bridge.

You can also check the route table (ip route show). If you see multiple routes for the veth interfaces (azvXXXXXXXX) that means the node is already running the L3 routing (transparent mode)

Thanks for the tip @4c74356b41

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ghost]

This issue will now be closed because it hasn't had any activity for 15 days after stale. palma21 feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.