kubeadm init errors out kubeadm/app/phases/addons/dns/dns.go

[HankTheCrank]

Is this a request for help?

Not sure.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:08:27Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:02:01Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

• Kernel (e.g. uname -a):

Linux SERVERNAME 4.18.0-240.15.1.el8_3.x86_64 #1 SMP Mon Mar 1 17:16:16 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

What happened?

I ran kubeadm init on a fresh bare metal install and I got the error:

[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0327 17:28:24.778708 17181 kubeletfinalize.go:132] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
start version '1.8.3' not supported
unable to get list of changes to the configuration.
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.isCoreDNSConfigMapMigrationRequired
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:387
k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns.createCoreDNSAddon
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/phases/addons/dns/dns.go:274

To troubleshoot I also tried an update to the failed init and got:

[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks.
[preflight] Some fatal errors occurred:
[ERROR CoreDNSUnsupportedPlugins]: start version '1.8.3' not supported
[ERROR CoreDNSMigration]: CoreDNS will not be upgraded: start version '1.8.3' not supported
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

What you expected to happen?

I was expecting the init to finish and provide join commands for the other masters/workers

How to reproduce it (as minimally and precisely as possible)?

As far as I can tell, just run the kubeadm init for kubernetes v1.20.5.
I am setting up a bare metal multi-master

Anything else we need to know?

From what I can piece together, it looks like kubeadm downloads coreDNS container version 1.7.0 as expected, but the latest version of coreDNS is 1.8.3, which is not supported. During the finalization of the kubeadm init, it's either trying to get the latest coreDNS and failing, or it's looking at the latest version and failing. coreDNS 1.8.3 does not get downloaded into the local docker, only 1.7.0 does.

I tried manually downloading coreDNS 1.8.3:
docker pull coredns/coredns:1.8.3
and reran the init (after a reset), and it worked!

Something strange going on with the init

[neolit123]

hi,

I ran kubeadm init on a fresh bare metal install and I got the error:

this looks like the output of the kubeadm upgrade apply command, not kubeadm init.
can you show the full output of the init command?

From what I can piece together, it looks like kubeadm downloads coreDNS container version 1.7.0 as expected, but the latest version of coreDNS is 1.8.3, which is not supported. During the finalization of the kubeadm init, it's either trying to get the latest coreDNS and failing, or it's looking at the latest version and failing. coreDNS 1.8.3 does not get downloaded into the local docker, only 1.7.0 does.

kubeadm init for a kubeadm binary that is 1.20 would install coredns 1.7.0 as can be seen here:
https://github.com/kubernetes/kubernetes/blob/release-1.20/cmd/kubeadm/app/constants/constants.go#L353

I tried manually downloading coreDNS 1.8.3:
docker pull coredns/coredns:1.8.3
and reran the init (after a reset), and it worked!

downloading the coredns image would not have solved the preflight error that you are getting.
the error happens if coredns version X was installed and then you run a kubeadm binary that cannot migrate (during upgrade).
e.g. coredns was at 1.8.3 and you run the kubeadm upgrade apply of the kubeadm 1.20 binary the would not recognize it.

/kind support

[neolit123]

From what I can piece together, it looks like kubeadm downloads coreDNS container version 1.7.0 as expected, but the latest version of coreDNS is 1.8.3, which is not supported. During the finalization of the kubeadm init, it's either trying to get the latest coreDNS and failing, or it's looking at the latest version and failing. coreDNS 1.8.3 does not get downloaded into the local docker, only 1.7.0 does.

i think something else is going on in your setup.
there is no code in kubeadm 1.20 to download coredns 1.8.3 (by default)

what configuration are you passing to kubeadm --config?

[pacoxu]

https://github.com/kubernetes/kubernetes/blob/v1.20.5/cmd/kubeadm/app/phases/addons/dns/dns.go#L385-L388

If user upgrades dns to v1.8.0 before kubeadm upgrade, they may encounter this problem.
It is fixed in #97919, should we back port it to 1.20 ?

[neolit123]

If user upgrades dns to v1.8.0 before kubeadm upgrade, they may encounter this problem.

from my understanding, the user manually setting coredns to v1.8.0 in an old cluster, then running kubeadm upgrade that cannot use v1.8.0 as the base coredns version will fail.

the reason for that is that the coredns migration library for 1.20 would not recognize 1.8.x as the base version - therefore cannot migrate it.

It is fixed in #97919, should we back port it to 1.20 ?

not sure if this is the same as the v problem.
waiting for response from the user on my questions above.

i don't think there is a need to backport it, since the v change only happened in 1.21 and old clusters should not use v1.8.0

long term we want coredns to be deployed and upgrade by this operator:
https://github.com/kubernetes-sigs/cluster-addons/tree/master/coredns

[HankTheCrank]

Sorry for the delay, got busy.

This is an init for an HA install with and external ETCD on a freshly installed bare metal Centos 7. Not an upgrade.

When the error occurs there is an API call. One interesting point is:

"containers":[{
"args":["-conf","/etc/coredns/Corefile"],"image":"coredns/coredns:1.8.3","imagePullPolicy":"IfNotPresent"

The full text:

I0410 22:44:26.253319 21432 round_trippers.go:454] GET https://192.168.4.15:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s 200 OK in 4 milliseconds
I0410 22:44:26.253371 21432 round_trippers.go:460] Response Headers:
I0410 22:44:26.253387 21432 round_trippers.go:463] Content-Length: 1688
I0410 22:44:26.253394 21432 round_trippers.go:463] Date: Sun, 11 Apr 2021 02:44:26 GMT
I0410 22:44:26.253401 21432 round_trippers.go:463] Cache-Control: no-cache, private
I0410 22:44:26.253412 21432 round_trippers.go:463] Content-Type: application/json
I0410 22:44:26.253420 21432 round_trippers.go:463] X-Kubernetes-Pf-Flowschema-Uid: a79c531e-ad23-4112-b352-609d5664ae3a
I0410 22:44:26.253427 21432 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: e42fce59-bde9-4274-a9f7-27523cb30379
I0410 22:44:26.253471 21432 request.go:1123] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"coredns","namespace":"kube-system","uid":"026a0518-2e81-449c-ac57-f17a1f0d33a5","resourceVersion":"8508776","creationTimestamp":"2020-09-05T19:39:06Z","annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"v1","data":{"Corefile":".:53 {\n errors\n health {\n lameduck 5s\n }\n ready\n kubernetes cluster.local in-addr.arpa ip6.arpa {\n fallthrough in-addr.arpa ip6.arpa\n }\n prometheus :9153\n forward . /etc/resolv.conf {\n max_concurrent 1000\n }\n cache 30\n loop\n reload\n loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"name":"coredns","namespace":"kube-system"}}\n"},"managedFields":[{"manager":"kubeadm","operation":"Update","apiVersion":"v1","time":"2020-09-05T19:39:06Z","fieldsType":"FieldsV1","fieldsV1":{"f:data":{}}},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"v1","time":"2021-03-27T21:09:47Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}}}},{"manager":"kubectl-edit","operation":"Update","apiVersion":"v1","time":"2021-04-01T01:59:52Z","fieldsType":"FieldsV1","fieldsV1":{"f:data":{"f:Corefile":{}}}}]},"data":{"Corefile":".:53 {\n errors\n health {\n lameduck 5s\n }\n ready :8181\n kubernetes cluster.local in-addr.arpa ip6.arpa {\n fallthrough in-addr.arpa ip6.arpa\n }\n prometheus :9153\n forward . /etc/resolv.conf {\n max_concurrent 1000\n }\n cache 30\n loop\n reload\n loadbalance\n}\n"}}
I0410 22:44:26.253670 21432 round_trippers.go:435] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubeadm/v1.21.0 (linux/amd64) kubernetes/cb303e6" 'https://192.168.4.15:6443/apis/apps/v1/namespaces/kube-system/deployments?labelSelector=k8s-app%3Dkube-dns'
I0410 22:44:26.259484 21432 round_trippers.go:454] GET https://192.168.4.15:6443/apis/apps/v1/namespaces/kube-system/deployments?labelSelector=k8s-app%3Dkube-dns 200 OK in 5 milliseconds
I0410 22:44:26.259504 21432 round_trippers.go:460] Response Headers:
I0410 22:44:26.259513 21432 round_trippers.go:463] X-Kubernetes-Pf-Prioritylevel-Uid: e42fce59-bde9-4274-a9f7-27523cb30379
I0410 22:44:26.259521 21432 round_trippers.go:463] Date: Sun, 11 Apr 2021 02:44:26 GMT
I0410 22:44:26.259531 21432 round_trippers.go:463] Cache-Control: no-cache, private
I0410 22:44:26.259539 21432 round_trippers.go:463] Content-Type: application/json
I0410 22:44:26.259546 21432 round_trippers.go:463] X-Kubernetes-Pf-Flowschema-Uid: a79c531e-ad23-4112-b352-609d5664ae3a
I0410 22:44:26.259711 21432 request.go:1123] Response Body: {"kind":"DeploymentList","apiVersion":"apps/v1","metadata":{"resourceVersion":"8633805"},"items":[{"metadata":{"name":"coredns","namespace":"kube-system","uid":"d66323a4-40c9-4a85-9024-3bb0a34a4506","resourceVersion":"8521114","generation":10,"creationTimestamp":"2021-03-20T08:50:48Z","labels":{"k8s-app":"kube-dns","kubernetes.io/name":"CoreDNS"},"annotations":{"deployment.kubernetes.io/revision":"8","kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{"deployment.kubernetes.io/revision":"5"},"creationTimestamp":"2021-03-20T08:50:48Z","generation":6,"labels":{"k8s-app":"kube-dns","kubernetes.io/name":"CoreDNS"},"managedFields":[{"apiVersion":"apps/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:progressDeadlineSeconds":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:selector":{},"f:strategy":{"f:rollingUpdate":{".":{},"f:maxSurge":{},"f:maxUnavailable":{}},"f:type":{}},"f:template":{"f:metadata":{"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"coredns\"}":{".":{},"f:args":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":53,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":53,\"protocol\":\"UDP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9153,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:limits":{".":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:capabilities":{".":{},"f:add":{},"f:drop":{}},"f:readOnlyRootFilesystem":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{\"mountPath\":\"/etc/coredns\"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}}}}},"f:dnsPolicy":{},"f:nodeSelector":{".":{},"f:kubernetes.io/os":{}},"f:priorityClassName":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{\"name\":\"config-volume\"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:items":{},"f:name":{}},"f:name":{}}}}}}},"manager":"kubectl-create","operation":"Update","time":"2021-03-20T08:50:48Z"},{"apiVersion":"apps/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{"f:kubernetes.io/name":{}}},"f:spec":{"f:template":{"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"coredns\"}":{"f:image":{}}},"f:tolerations":{}}}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2021-03-28T00:18:58Z"},{"apiVersion":"apps/v1","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{"k:{\"name\":\"coredns\"}":{"f:securityContext":{"f:allowPrivilegeEscalation":{}}}}}}}},"manager":"kubectl-edit","operation":"Update","time":"2021-03-28T09:22:45Z"},{"apiVersion":"apps/v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:deployment.kubernetes.io/revision":{}}},"f:status":{"f:conditions":{"k:{\"type\":\"Progressing\"}":{"f:lastTransitionTime":{},"f:lastUpdateTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:observedGeneration":{},"f:replicas":{},"f:unavailableReplicas":{},"f:updatedReplicas":{}}},"manager":"kube-controller-manager","operation":"Update","time":"2021-04-01T01:29:37Z"}],"name":"coredns","namespace":"kube-system","resourceVersion":"8506123","uid":"d66323a4-40c9-4a85-9024-3bb0a34a4506"},"spec":{"progressDeadlineSeconds":600,"replicas":1,"revisionHistoryLimit":10,"selector":{"matchLabels":{"k8s-app":"kube-dns"}},"strategy":{"rollingUpdate":{"maxSurge":"25%","maxUnavailable":1},"type":"RollingUpdate"},"template":{"metadata":{"creationTimestamp":null,"labels":{"k8s-app":"kube-dns"}},"spec":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"k8s-app","operator":"In","values":["kube-dns"]}]},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"containers":[{"args":["-conf","/etc/coredns/Corefile"],"image":"coredns/coredns:1.8.3","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":5,"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":60,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":5},"name":"coredns","ports":[{"containerPort":53,"name":"dns","protocol":"UDP"},{"containerPort":53,"name":"dns-tcp","protocol":"TCP"},{"containerPort":9153,"name":"metrics","protocol":"TCP"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/ready","port":8181,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":10},"resources":{"limits":{"memory":"170Mi"},"requests":{"cpu":"100m","memory":"70Mi"}},"securityContext":{"allowPrivilegeEscalation":true,"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["all"]},"readOnlyRootFilesystem":true},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","volumeMounts":[{"mountPath":"/etc/coredns","name":"config-volume","readOnly":true}]}],"dnsPolicy":"Default","nodeSelector":{"kubernetes.io/os":"linux"},"priorityClassName":"system-cluster-critical","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"coredns","serviceAccountName":"coredns","terminationGracePeriodSeconds":30,"tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"}],"volumes":[{"configMap":{"defaultMode":420,"items":[{"key":"Corefile","path":"Corefile"}],"name":"coredns"},"name":"config-volume"}]}}},"status":{"conditions":[{"lastTransitionTime":"2021-03-20T08:50:48Z","lastUpdateTime":"2021-03-20T08:50:48Z","message":"Deployment has minimum availability.","reason":"MinimumReplicasAvailable","status":"True","type":"Available"},{"lastTransitionTime":"2021-04-01T01:39:38Z","lastUpdateTime":"2021-04-01T01:39:38Z","message":"ReplicaSet \"coredns-68d9d686d9\" has timed out progressing.","reason":"ProgressDeadlineExceeded","status":"False","type":"Progressing"}],"observedGeneration":6,"replicas":1,"unavailableReplicas":1,"updatedReplicas":1}}\n"},"managedFields":[{"manager":"kubectl-create","operation":"Update","apiVersion":"apps/v1","time":"2021-03-20T08:50:48Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:progressDeadlineSeconds":{},"f:replicas":{},"f:revisionHistoryLimit":{},"f:selector":{},"f:strategy":{"f:rollingUpdate":{".":{},"f:maxSurge":{},"f:maxUnavailable":{}},"f:type":{}},"f:template":{"f:metadata":{"f:labels":{".":{},"f:k8s-app":{}}},"f:spec":{"f:containers":{"k:{"name":"coredns"}":{".":{},"f:args":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{"containerPort":53,"protocol":"TCP"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{"containerPort":53,"protocol":"UDP"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{"containerPort":9153,"protocol":"TCP"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:resources":{".":{},"f:limits":{".":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:capabilities":{".":{},"f:add":{},"f:drop":{}},"f:readOnlyRootFilesystem":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{},"f:volumeMounts":{".":{},"k:{"mountPath":"/etc/coredns"}":{".":{},"f:mountPath":{},"f:name":{},"f:readOnly":{}}}}},"f:dnsPolicy":{},"f:nodeSelector":{".":{},"f:kubernetes.io/os":{}},"f:priorityClassName":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{},"f:volumes":{".":{},"k:{"name":"config-volume"}":{".":{},"f:configMap":{".":{},"f:defaultMode":{},"f:items":{},"f:name":{}},"f:name":{}}}}}}}},{"manager":"kubectl-client-side-apply","operation":"Update","apiVersion":"apps/v1","time":"2021-04-01T01:52:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{"f:kubernetes.io/name":{}}},"f:spec":{"f:template":{"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{"name":"coredns"}":{"f:image":{}}},"f:tolerations":{}}}}}},{"manager":"kubectl-edit","operation":"Update","apiVersion":"apps/v1","time":"2021-04-01T03:01:21Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:template":{"f:spec":{"f:containers":{"k:{"name":"coredns"}":{"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:securityContext":{"f:allowPrivilegeEscalation":{}}}}}}}}},{"manager":"kube-controller-manager","operation":"Update","apiVersion":"apps/v1","time":"2021-04-01T03:03:05Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:deployment.kubernetes.io/revision":{}}},"f:status":{"f:conditions":{"k:{"type":"Available"}":{"f:lastTransitionTime":{},"f:lastUpdateTime":{},"f:message":{},"f:reason":{},"f:status":{}},"k:{"type":"Progressing"}":{"f:lastTransitionTime":{},"f:lastUpdateTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:observedGeneration":{},"f:replicas":{},"f:unavailableReplicas":{},"f:updatedReplicas":{}}}}]},"spec":{"replicas":3,"selector":{"matchLabels":{"k8s-app":"kube-dns"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"k8s-app":"kube-dns"}},"spec":{"volumes":[{"name":"config-volume","configMap":{"name":"coredns","items":[{"key":"Corefile","path":"Corefile"}],"defaultMode":420}}],"containers":[{"name":"coredns","image":"coredns/coredns:1.8.3","args":["-conf","/etc/coredns/Corefile"],"ports":[{"name":"dns","containerPort":53,"protocol":"UDP"},{"name":"dns-tcp","containerPort":53,"protocol":"TCP"},{"name":"metrics","containerPort":9153,"protocol":"TCP"}],"resources":{"limits":{"memory":"170Mi"},"requests":{"cpu":"100m","memory":"70Mi"}},"volumeMounts":[{"name":"config-volume","readOnly":true,"mountPath":"/etc/coredns"}],"livenessProbe":{"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"initialDelaySeconds":60,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":5},"readinessProbe":{"httpGet":{"path":"/ready","port":8181,"scheme":"HTTP"},"timeoutSeconds":10,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["all"]},"readOnlyRootFilesystem":true,"allowPrivilegeEscalation":true}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"Default","nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"coredns","serviceAccount":"coredns","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"k8s-app","operator":"In","values":["kube-dns"]}]},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"CriticalAddonsOnly","operator":"Exists"}],"priorityClassName":"system-cluster-critical"}},"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxUnavailable":1,"maxSurge":"25%"}},"revisionHistoryLimit":10,"progressDeadlineSeconds":600},"status":{"observedGeneration":10,"replicas":4,"updatedReplicas":2,"unavailableReplicas":4,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2021-04-01T02:53:12Z","lastTransitionTime":"2021-04-01T02:53:12Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"False","lastUpdateTime":"2021-04-01T03:24:41Z","lastTransitionTime":"2021-04-01T03:24:41Z","reason":"ProgressDeadlineExceeded","message":"ReplicaSet "coredns-6c79c9cb8f" has timed out progressing."}]}}]}
start version '1.8.3' not supported
unable to get list of changes to the configuration.

[neolit123]

kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5",

+

start version '1.8.3' not supported

kubeadm 1.20.5 does not support coredns 1.8.3.

what is your configuration file that you pass to kubeadm init?

[neolit123]

looking at your logs seems to be the case..
you should not be setting coredns 1.8.x with this older version of kubeadm.

if you strictly want coredns 1.8.3, use kubeadm 1.21.

[HankTheCrank]

kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:30:03Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}

kubeadm init --config /kubernetes/01.kubeadm-config.yaml --upload-certs --v=10

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "192.168.4.15:6443"
apiServer:
certSANs:
- "k8.MYDOMAIN.com"
- "192.168.4.15"
advertise-address: 0.0.0.0
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
etcd:
external:
endpoints:
- https://192.168.4.177:2379
- https://192.168.4.178:2379
- https://192.168.4.179:2379
caFile: /etc/kubernetes/pki/etcd/ca.crt
certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
controllerManager:
extraArgs:
"allocate-node-cidrs": "true"
"service-cluster-ip-range": "10.96.0.0/12"
"cluster-cidr": "10.244.0.0/16"

[neolit123]

we are not seeing problems with coredns 1.8.3 and kubeadm 1.21 in our CI.

are you seeing this same replicaset problem with kubeadm 1.21?
if yes, try:

sudo kubectl describe po --kubeconfig /etc/kubernetes/admin.conf -n kube-system <some-coredns-pod>
sudo kubectl describe deployment --kubeconfig /etc/kubernetes/admin.conf -n kube-system coredns

and try to understand more about your problem.

[HankTheCrank]

okay, that's interesting. When the init had the error I thought there was no master node install done. After trying your suggested command, I looked at the running config and I see pods that were not installed as part of this install. What I think is happening is the installation is getting config information from the etcd. The external etcd is one that I am reusing from a previous cluster installation. This could be why it's looking like an upgrade when the command I am issuing is an init. I need to figure out how to flush the old data in the etcd.

Thanks for your help. I'll come back and comment after I've dealt with the etcd and let you know how it turns out.

[HankTheCrank]

Flushing the data out of the etcd solved it. I manually deleted everything using kubectl to clear out etcd.

Thanks again!