DNS issues when using AzureIdentity

[nader-ziada]

/kind bug

What steps did you take and what happened:

• Created a management cluster running on Azure
• Tried to create a private cluster in the same vnet as the management cluster
• If the private cluster is using an AzureClusterIdentity, the nmi pod gives the following error

I0519 14:15:27.829897       1 managed.go:52] clientID in request: 4b8d##### REDACTED #####9086, capz-system/capz-controller-manager-874c54fb4-nslgz has been matched with azure identity capz-system/capz-e2e-c951p2-capz-e2e-c14w7g-cluster-identity
I0519 14:15:27.830129       1 managed.go:89] matched identityType:1 adendpoint: tenantid:b39138ca-**** auxiliaryTenantIDs:[] clientid:4b8d##### REDACTED #####9086 resource:https://management.azure.com/
E0519 14:15:57.833861       1 server.go:378] failed to get service principal token for pod: capz-system/capz-controller-manager-874c54fb4-nslgz, error: failed to refresh token, error: adal: Failed to execute the refresh request. Error = 'Post "https://login.microsoftonline.com/b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0/oauth2/token?api-version=1.0": dial tcp: i/o timeout'

• Creation of the private cluster fails

What did you expect to happen:

• Expected the private cluster to use the referenced identity and get created

Anything else you would like to add:

• After some investigations found the following
  • The dns resolution of the nmi pod is the root cause of the issue
  • the nmi pod is using hostNetwork=true and dnsPolicy=ClusterFirstWithHostNet and there seems to be a related issue in calico that I think is the root cause: TCP offloading on vxlan.calico adaptor causing 63 second delays in VXLAN communications node->nodeport or node->clusterip:port. projectcalico/calico#3145
  • More context from kubernetes dnsPolicy in hostNetwork not working as expected kubernetes/kubernetes#87852
  • The kernel version used in the capz published images doesn't have a fix for this back ported to it https://bugs.launchpad.net/ubuntu/+source/iptables/+bug/1805543
  • I think this calico fix should help, but not released yet Disable checksum offload on the VXLAN tunnel for kernels <v5.7. projectcalico/felix#2811
  • found this issue while working on this PR: Use AzureClusterIdentity when running ci e2e tests #1360

Environment:

• cluster-api-provider-azure version: main branch as of fe01418401196933c87b3835d17927adca225519
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[nader-ziada]

The dns issue in the nmi pod when using Azure identity is expected to be resolved when calico v3.20.x is out in the next few weeks. Tested with a nightly build from main brach of calico and issues seems resolved.

[CecileRobertMichon]

/assign @nader-ziada

[CecileRobertMichon]

Calico 3.19.2 and 3.20.0 were just released: https://github.com/projectcalico/felix/releases

[dkoshkin]

For anyone else who has found this issue and is still seeing an error when installing Calico with an Operator, the autodetection is still not perfect projectcalico/calico#4727 and may need to be override manually.

You can do so by setting featureDetectOverride: ChecksumOffloadBroken=true and creating it as follows. (The rest of the values are created by quay.io/tigera/operator:v1.27.7 but may change for different versions)

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  name: default
spec:
  bpfLogLevel: ""
  featureDetectOverride: ChecksumOffloadBroken=true
  floatingIPs: Disabled
  healthPort: 9099
  logSeverityScreen: Info
  reportingInterval: 0s