DNS lookup timeouts

[juan-lee]

Symptoms
Outbound requests from pods can see a 5 second delay during DNS lookups. This is known to impact containers based off of the alpine image.

Root Cause
https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts

Workaround
Add the following to your impacted pod's manifest.

postStart:
  exec:
    command:
    - /bin/sh
    - -c
    - "/bin/echo 'options single-request-reopen' >> /etc/resolv.conf"

What is single-request-reopen?

single-request-reopen (since glibc 2.9)
  Sets RES_SNGLKUPREOP in _res.options.  The resolver
  uses the same socket for the A and AAAA requests.  Some
  hardware mistakenly sends back only one reply.  When
  that happens the client system will sit and wait for
  the second reply.  Turning this option on changes this
  behavior so that if two requests from the same port are
  not handled correctly it will close the socket and open
  a new one before sending the second request.

[devteng]

FYI, I have not testing this personally -- this supposedly will not work for Alpine based containers since MUSL lib that Alpine uses does not support this option.

kubernetes/kubernetes#62628 (comment)

[epoyraz]

We had the same issue using alpine image.
It's not a AKS problem, but a general kubernetes issue: See
kubernetes/kubernetes#56903

I would recommend to read this: https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/

We changed our base image to jessie-slim and added this to the pod manifest.

dnsConfig:  
  options:  
    - name: single-request-reopen

more details : https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

I think it is a cleaner solution than adding a postStart hook.

[timwebster9]

I'm not sure what to think about this. Is the advice going to be: 'you can't run Alpine and need to customise DNS' if you expect proper DNS performance?

Could someone from Microsoft weigh in here?

[AXington]

This also works, and it works on alpine pods:

https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
service/networking/custom-dns.yaml
apiVersion: v1
kind: Pod
metadata:
namespace: default
name: dns-example
spec:
containers:
- name: test
image: nginx
dnsPolicy: "None"
dnsConfig:
nameservers:
- 1.2.3.4
searches:
- ns1.svc.cluster.local
- my.dns.search.suffix
options:
- name: ndots
value: "2"
- name: edns0

[wutingbupt]

We have the same problem, will AKS rollout a solution for us? I don't see an elegant workaround so far.

[wutingbupt]

@AXington Your solution reduces the delay from 5 seconds to 2.5 :), but still not fully work.

[wutingbupt]

@juan-lee your workaround didn't work perfectly for the alpine based images, any better idea?

[juan-lee]

@juan-lee your workaround didn't work perfectly for the alpine based images, any better idea?

My understanding is that we've rolled the fix out during this week's release. You will have to upgrade your cluster to a newer version of k8s to get the changes.

[wutingbupt]

Thanks very much, @juan-lee, I will let you know how does it work.
Br,
Tim

[wutingbupt]

@juan-lee The 1.11.4 solves this problem, thanks for your help.

Br,
Tim

[motarski]

I checked official change log can't find specific change that addresses this bug in the change log, but amazingly the upgrade seems to fix this issue for our AKS cluster. I hope it will not come back on next version bump.

[juan-lee]

I checked official change log can't find specific change that addresses this bug in the change log, but amazingly the upgrade seems to fix this issue for our AKS cluster. I hope it will not come back on next version bump.

The fix isn't in k8s, but rather a kernel patch that is now default for new agent nodes. The reason you're seeing the problem go away with an upgrade is because your old agent nodes are replaced with ones that have the new kernel.

[motarski]

Ok I see, thanks for explanation@juan-lee

[timwebster9]

@juan-lee is the fix only present in the new 'MobyImage' that you have to register for, or is it present in all 1.11.4 images?

[weinong]

Hi people!
As it stands today, all new VMs (through new cluster, scale, or upgrade) have the kernel fix '"netfilter: nf_conntrack: resolve clash for matching conntracks" fixes the 1st race (accepted).` described in https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts

Thanks

[nodeselector]

@weinong any info on when the second race will be addressed?

[kim0]

I just hit this, and single-request-reopen does seem to make it go away! This is on a recent fresh 1.13.5 aks cluster. Not sure why the DNS server gives the wrong answer to begin with. Is this a coredns issue ?
Thanks @juan-lee for the work-around 👍

[kim0]

Hi @weinong .. I deployed a cluster a few days back, and still hitting this. So I think it's still not fixed. Is there any update when a fix is expected to land.
Also taking a chance and pinging @jnoller

[tapanhalani]

Hi people. Is this issue fixed with kubernetes 1.13.5? I wanted to upgrade my AKS cluster to the said version, but I am using multiple deployments with alpine-based images.

[jemag]

Still seeing the issue with kernel-version 4.15.0-1063-azure

[Vandersteen]

I also am seeing this issue in kernel version 4.15.0-1060-azure

[juan-lee]

@jemag @Vandersteen Recently I've seen DNS timeouts being caused by high resource usage on the nodes where the coredns pods reside. Could either of you check your monitoring and see how saturated your CPU, memory, and disk are?

[jemag]

@juan-lee I have replicated this problem under various loads. Currently with all nodes under 18% cpu usage and memory at around 70% for each node the problem still happens intermittently. Disk usage is extremely low.

[Vandersteen]

@juan-lee I have replicated this on 2 different clusters. 1 has 40% cpu usage & 60% memory usage. The other one has 20% cpu usage and 42% memory usage.

[juan-lee]

@jemag @Vandersteen we are looking into it. By chance are your clusters using azure cni? I'm only able to reproduce the issue with azure cni clusters.

[jemag]

@juan-lee mine are indeed using azure cni

[Vandersteen]

@juan-lee Yes we are using azure cni

[juan-lee]

We are still working to get to the bottom of this issue. In the meantime, adding the following to your specs in most cases can help.

dnsConfig:  
  options:  
    - name: single-request-reopen

[guitmz]

This started to happen for me when we upgraded the cluster to v.1.14 (from 1.13). We had the same problem a year ago and it was fixed after the kernel was patched with the mitigations from https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts

is there a way we can run coredns in AKS as daemonset to try to fix this (at least try)? We need azure cni and we cant use single-request-reopen for all pods because some are alpine based

[jnoller]

Please also see this issue for intermittent nodenotready, DNS latency and other crashed related to system load: #1373

[kwaazaar]

We are still working to get to the bottom of this issue. In the meantime, adding the following to your specs in most cases can help.

dnsConfig:  
  options:  
    - name: single-request-reopen

I tried this, but keep getting this issue. I have a container doing a curl statement in a loop with a 2 second pause in between attempt and get it about once every 20 attempts. I seems to happen on 3 clusters (OT, ACC and PROD). On those last two this seems to help pretty well. On the OT-cluster (1.16.9) it does not. All clusters are using Azure CNI and use private networking and a VPN connection to on-prem DNS-servers.

[neil-south]

This is now effecting us. the workaround

dnsConfig:  
  options:  
    - name: single-request-reopen

seems to work, but obviously this is effecting lots of people (most of which wont even realise), and needs sorting!

Ive even tried creating a new v1.19 AKS cluster, and using the official mcr.microsoft.com/dotnet/core/aspnet:3.1 docker base image (as of 29/9/2020) and the issue is still present.

[davesmits]

We used the dnsCOnfig work around and initially it didnt work for us. It turned out that it only works with the non-alpine images.

[sikri-eic]

Is there a solution/workaround for this issue for Alpine based images? We tried dnsPolicy: Default to no avail. We are also using Azure CNI (we have a 2500 ms delay on nslookup).

Kubernetes cluster version: 1.18.8
OS Image: Ubuntu 16.04.7 LTS

A newer version of OS image (18.04) has been deployed very recently, would it help to upgrade?

[timja]

@sikri-eic deploying nodelocaldns fixed it for us.
18.04 didn't solve it in our case

[sikri-eic]

@timja Thank you for the prompt response. I will take a look at nodelocaldns. The documentation sounds like it may actually address this problem.

[Vandersteen]

@timja Did you have to do anything special to install nodelocaldns on aks ? Seems like you need to mess around with IPtables or something

[timja]

It depends if you're using kubenet or azure cni,

kubenet works with just the standard config,

azure cni:

add the -setupebtables flag

ref:
#1642

example config for azure cni (note it took months for a release after our changes so we're using a forked image but looks like upstream have released now):
https://github.com/hmcts/cnp-flux-config/blob/master/k8s/namespaces/kube-system/nodelocaldns/nodelocaldns.yaml

[curtdept]

Here is a version for AKS, runs on everything except virtual node.

https://github.com/curtdept/aks_nodelocaldns/blob/main/nodelocaldns.yaml

This worked amazing btw, cleaned up tons of issues I had with heartbeats and clustered services.

[palma21]

Marking these as duplicate/known-issue and adding the in-progress feature for it as well as well as @curtdept current workaround (thanks!)

https://github.com/curtdept/aks_nodelocaldns/blob/main/nodelocaldns.yaml

#1492

[ghost]

This issue has been marked as duplicate and has not had any activity for 1 day. It will be closed for housekeeping purposes.