Renew cert on start if current cert has has expired #10122

anravi · 2021-01-10T17:01:01Z

Steps to reproduce the issue:

Oracle vm virtualbox version 6.1
kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-19T07:38:38Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}

minikube version
minikube version: v1.16.0
commit: 9f1e482

$ docker --version
Docker version 20.10.2, build 2291f61

minikube start
😄 minikube v1.16.0 on Darwin 10.15.7
🆕 Kubernetes 1.20.0 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.20.0
✨ Using the virtualbox driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🏃 Updating the running virtualbox "minikube" VM ...
🐳 Preparing Kubernetes v1.17.0 on Docker 19.03.5 ...| E0110 11:59:25.743916 9686 kubeadm.go:647] sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml failed - will try once more: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml": Process exited with status 1
stdout:
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk

stderr:
W0110 16:59:25.557503 17370 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0110 16:59:25.557541 17370 validation.go:28] Cannot validate kubelet config - no validator is available
error execution phase certs/apiserver-kubelet-client: failed to write or validate certificate "apiserver-kubelet-client": failure loading apiserver-kubelet-client certificate: failed to load certificate: the certificate has expired
To see the stack trace of this error execute with --v=5 or higher
/ 🤦 Unable to restart cluster, will reset it: run: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml": Process exited with status 1
stdout:
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk

stderr:
W0110 16:59:25.744187 17376 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0110 16:59:25.744235 17376 validation.go:28] Cannot validate kubelet config - no validator is available
error execution phase certs/apiserver-kubelet-client: failed to write or validate certificate "apiserver-kubelet-client": failure loading apiserver-kubelet-client certificate: failed to load certificate: the certificate has expired
To see the stack trace of this error execute with --v=5 or higher

▪ Generating certificates and keys .../ 💢  initialization failed, will try again: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap": Process exited with status 1

stdout:
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk

stderr:

▪ Generating certificates and keys ...-

💣 Error starting cluster: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk

stderr:

😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
👉 https://github.com/kubernetes/minikube/issues/new/choose

❌ Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.17.0:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.17.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk

stderr:

😿 If the above advice does not help, please let us know:
👉 https://github.com/kubernetes/minikube/issues/new/choose

Optional: Full output of minikube logs command:

The text was updated successfully, but these errors were encountered:

dabcoder · 2021-01-11T10:24:15Z

Also ran into this issue earlier today. However, it persists with the same error even if I downgrade minikube with this version for instance:

minikube v1.14.2 on Darwin 10.15.7

Same stdout and stderr output referencing to the certificate issue.
EDIT: solved it by deleting the cluster with minikube delete and start a new one but that does not reveal the underlying cause of this issue.

anravi · 2021-01-11T21:12:36Z

@dabcoder - Thanksminikube delete fixed my issue as well but don't know root cause of the issue.

dabcoder · 2021-01-25T08:00:27Z

Still unsure what caused this issue:

W0110 16:59:25.557503 17370 validation.go:28] Cannot validate kube-proxy config - no validator is available
W0110 16:59:25.557541 17370 validation.go:28] Cannot validate kubelet config - no validator is available
error execution phase certs/apiserver-kubelet-client: failed to write or validate certificate "apiserver-kubelet-client": failure loading apiserver-kubelet-client certificate: failed to load certificate: the certificate has expired

Looks certificate related, in any case I'd suggest renaming this issue to mention the error(s) from the stderr output in the title.

medyagh · 2021-01-25T23:38:16Z

this seems to be related to an expired cert, I wonder if you had this cluster for a long time ? in that case minikube should still have detected expired certs (or maybe we could provide more relaxed expiration for our certs)

I would accept any PR that would increase our certs expiration to 5 years

dabcoder · 2021-01-27T21:57:58Z

this seems to be related to an expired cert, I wonder if you had this cluster for a long time ? in that case minikube should still have detected expired certs (or maybe we could provide more relaxed expiration for our certs)

In my case, I've had it for a while yes, >= 18 months I would say.

I would accept any PR that would increase our certs expiration to 5 years.

@medyagh would the modification need to be done in https://github.com/kubernetes/minikube/blob/master/pkg/minikube/bootstrapper/certs.go?

dabcoder · 2021-01-29T07:42:39Z

Although I am seeing this expiration date set to 10 years in the future:

minikube/pkg/util/crypto.go

Line 51 in f0c59ce

NotAfter: time.Now().Add(time.Hour * 24 * 365 * 10),

.

dabcoder · 2021-02-03T11:40:09Z

Quick ping @medyagh, to see if you've had a chance to look at the above.

tstromberg · 2021-02-22T19:52:40Z

@dabcoder - seems good to me!

medyagh · 2021-03-05T17:23:55Z

@sathyaprakashmani
@dabcoder is it possible that the Date on your machine was wrong and it was set for a time in the past and minikube created certs according to the wrong time ? and the later ur machine fixed its time ?

I suggest minikube at least print the Current Date and Time and The Expiry Time of cert if this error happens

medyagh · 2021-03-05T21:32:01Z

interestingly
on my machine I see the End date in 2024 (not in 10 years) as our code suggests:

	template := x509.Certificate{
		SerialNumber: big.NewInt(1),
		Subject: pkix.Name{
			CommonName: name,
		},
		NotBefore: time.Now().Add(time.Hour * -24),
		NotAfter:  time.Now().Add(time.Hour * 24 * 365 * 10),
		KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
		BasicConstraintsValid: true,
		IsCA:                  true,
	}

$ cat ~/.minikube/certs/ca.pem | openssl x509 -noout -enddate
notAfter=Feb  9 22:45:00 2024 GMT

and removing NotBefore and NotAfter code will not change anything

	template := x509.Certificate{
		SerialNumber: big.NewInt(1),
		Subject: pkix.Name{
			CommonName: name,
		},
		NotBefore:             time.Now().Add(time.Hour * -1000),
		NotAfter:              time.Now().Add(time.Hour * 24 * 365 * 100),
		KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth, x509.ExtKeyUsageServerAuth},
		BasicConstraintsValid: true,
		IsCA:                  true,
	}


$ cat ~/.minikube/ca.pem | openssl x509 -noout -startdate
notBefore=Mar  5 21:31:00 2021 GMT
13:38:58 medya/workspace/ex1
$ cat ~/.minikube/ca.pem | openssl x509 -noout -enddate
notAfter=Feb 18 21:31:00 2024 GMT
13:39:03 medya/workspace/ex1
$ cat ~/.minikube/certs/ca.pem | openssl x509 -noout -enddate
notAfter=Feb 18 21:31:00 2024 GMT
13:39:07 medya/workspace/ex1
$ cat ~/.minikube/certs/ca.pem | openssl x509 -noout -startdate
notBefore=Mar  5 21:31:00 2021 GMT

dabcoder · 2021-03-16T08:37:32Z

@dabcoder is it possible that the Date on your machine was wrong and it was set for a time in the past and minikube created certs according to the wrong time ? and the later ur machine fixed its time ?

@medyagh I've not noticed any date/time issues on my machine, very unlikely to be the case there.

I suggest minikube at least print the Current Date and Time and The Expiry Time of cert if this error happens

That would be good yes.

jeffmaury · 2021-06-14T11:51:02Z

There'a a workaround for those who want to keep the cluster data.
Once the minikube vm refuse to start, ssh into it and go to /var/lib/minikube/certs. I deleted the certificates that were reported as expired (don't have the list as I processed them incrementally, maybe deleting the certs folder may work) as they will be re created

Osprey2021 · 2021-07-09T09:09:58Z

I ran into this issue today, deleting /var/lib/minikube/certs did not help. Is there any other workaround, please?

jeffmaury · 2021-07-09T09:12:30Z

What is your error message then ?

Osprey2021 · 2021-07-09T09:21:58Z

What is your error message then ?

Preparing Kubernetes v1.14.8 on Docker 19.03.2 ...
E0709 10:41:14.065218 12920 kubeadm.go:586] sudo env PATH=/var/lib/minikube/binaries/v1.14.8:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml failed - will try once more: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.14.8:$PATH kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml": Process exited with status 1
stdout:
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing etcd/ca certificate authority
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing ca certificate authority

stderr:
error execution phase certs/apiserver: failed to write or validate certificate "apiserver": failure loading apiserver certificate: failed to load certificate: the certificate has expired

jeffmaury · 2021-07-09T13:19:31Z

Can you

minikube ssh
cd var/lib/minikube/certs
ls -l

All files listed should have yesterday or today's date

Osprey2021 · 2021-07-13T08:00:42Z

Can you

minikube ssh

cd var/lib/minikube/certs

ls -l

All files listed should have yesterday or today's date

Some files in that folder have new date, and some have older date. I moved certs folder to certs_BC so that certs folder was recreated, so am not sure how its possible that some older files reappear here.

Osprey2021 · 2021-07-13T17:24:23Z

docker@minikube:/var/lib/minikube/certs$ ll
total 64
drwxr-xr-x. 3 root root 4096 Jul 13 17:21 ./
drwxr-xr-x. 6 root root 79 Jul 13 17:21 ../
-rw-r--r--. 1 root root 1090 Jul 13 17:21 apiserver-etcd-client.crt
-rw-------. 1 root root 1675 Jul 13 17:21 apiserver-etcd-client.key
-rw-r--r--. 1 root root 1350 May 17 2020 apiserver.crt
-rw-------. 1 root root 1675 May 17 2020 apiserver.key
-rw-r--r--. 1 root root 1066 May 17 2020 ca.crt
-rw-------. 1 root root 1679 May 17 2020 ca.key
drwxr-xr-x. 2 root root 4096 Jul 13 17:21 etcd/
-rw-r--r--. 1 root root 1038 Jul 13 17:21 front-proxy-ca.crt
-rw-------. 1 root root 1675 Jul 13 17:21 front-proxy-ca.key
-rw-r--r--. 1 root root 1058 Jul 13 17:21 front-proxy-client.crt
-rw-------. 1 root root 1675 Jul 13 17:21 front-proxy-client.key
-rw-r--r--. 1 root root 1074 May 17 2020 proxy-client-ca.crt
-rw-------. 1 root root 1675 May 17 2020 proxy-client-ca.key
-rw-r--r--. 1 root root 1103 May 17 2020 proxy-client.crt
-rw-------. 1 root root 1675 May 17 2020 proxy-client.key

As can be seen here, some certificates are still old after recreating certs folder.

sharifelgamal · 2021-07-21T18:51:04Z

Yeah, long running certs are a known issue in minikube. This is something we would like to fix, either with longer expiration, or proper certification rotation.

edemen · 2021-07-28T13:36:56Z

There'a a workaround for those who want to keep the cluster data.
Once the minikube vm refuse to start, ssh into it and go to /var/lib/minikube/certs. I deleted the certificates that were reported as expired (don't have the list as I processed them incrementally, maybe deleting the certs folder may work) as they will be re created

Deleting the whole certs folder did not help. I restored it and then deleted the problematic certs iteratively, and in the end got it working, but all deployments were lost, so might have as well deleted minikube.
In case it might be useful to somebody, here is what I did:
minikube start failed similar to the OP
Then I did

minikube ssh
su - root
cp -r /var/lib/minikube/certs /var/lib/minikube/certs.back
rm /var/lib/minikube/certs/etcd/server*
rm /var/lib/minikube/certs/etcd/peer*
rm /var/lib/minikube/certs/etcd/healthcheck*
rm /var/lib/minikube/certs/apiserver-etcd*
exit
exit
minikube stop
minikube start

This results in:

* minikube v1.9.2 on Microsoft Windows 10 Enterprise 10.0.17763 Build 17763
* Using the hyperv driver based on existing profile
* Starting control plane node m01 in cluster minikube
* Restarting existing hyperv VM for "minikube" ...
* Preparing Kubernetes v1.18.0 on Docker 19.03.8 ...
! Unable to restart cluster, will reset it: apiserver health: controlPlane never updated to v1.18.0
* Enabling addons: default-storageclass, storage-provisioner
! Enabling 'default-storageclass' returned an error: running callbacks: [Error making standard the default storage class: Error listing StorageClasses: Unauthorized]
* Done! kubectl is now configured to use "minikube"

Note that kubectl get nodes would return error: You must be logged in to the server (Unauthorized)
In my case, kubectl was configured with:

- name: minikube
  user:
    client-certificate: C:\Users\User\.minikube\profiles\minikube\client.crt
    client-key: C:\Users\User\.minikube\profiles\minikube\client.key

These certs correspond to /var/lib/minikube/certs/apiserver.[crt|key] inside the minikube VM.
So I got them with:

minikube ssh
su - root
cat /var/lib/minikube/certs/apiserver.crt
cat /var/lib/minikube/certs/apiserver.key
exit
exit
minikube stop
minikube start

And replaced the contents of client.crt and client.key in C:\Users\User\.minikube\profiles\minikube\ accordingly.
Then kubectl get nodes started working.

But, as I noted above, the cluster got reset at some point, so I had to redeploy everything, which is a big problem for using minikube for anything other than a quick-dev/test and then throw it away. We need to be able to set it up, configure and leave it working for years without having to worry about it resetting once certificates expire.

infa-ddeore · 2021-07-29T11:40:40Z

in my case deleting rm ~/.minikube/client.{crt,key} files and then minikube delete worked fine, i was okay to delete minikube vm

cppragada · 2022-02-14T02:09:13Z

Donno how, minikube delete command resolved the issue

ebdxflr · 2022-11-21T14:33:46Z

i have a few questions on this:

Has it been resolved somehow? Is there any way to keep existing cluster and renew certificates?
I have tried solving this by following some of the instructions here:

backed up /var/lib/minikube/certs/
removed /var/lib/minikube/certs/
minikube stop/start
Yet, I was prompted with a new cluster so I quickly reverted the old certificates that I have backed up, in place. Is there any chance I can retrieve my old cluster without losing data?

NOTE: the ~.minikube folder has not been removed/altered.
NOTE2: Also tried removing only expired certs in /var/lib/minikube/certs/ but at the minikube start I'm prompted with certificate not found error:

error execution phase certs/apiserver-kubelet-client: failed to write or validate certificate "apiserver-kubelet-client": failure loading apiserver-kubelet-client certificate: couldn't load the certificate file /var/lib/minikube/certs/apiserver-kubelet-client.crt: open /var/lib/minikube/certs/apiserver-kubelet-client.crt: no such file or directory

LeBakii · 2022-12-08T08:33:16Z

I delete files and folders in the "/var/lib/minikube/certs/*" and minikube stop then start fix my issue

Sergiodcm00 · 2022-12-12T10:15:29Z

kubeadm certs renew all -cert-dir /var/lib/minikube/certs

Kimi450 · 2023-03-25T14:33:35Z

@sharifelgamal can we reopen this issue? I dont think this is fixed. Or at least not as I would expect it to be fixed. Ive linked the issue from my repo here. You can check my latest comment on that issue which describes why I think its broken (and how to replicate it). Unless im doing something wrong, I think this issue still persists.

Basically, if you use --cert-expiration tag, that only affects

/var/lib/minikube/certs/proxy-client.crt
/var/lib/minikube/certs/apiserver.crt

# these are on my mounted ones, minikube start --mount --mount-string "/home/kimi450:/minikube-host" --cert-expiration="120s"
/minikube-host/.minikube/profiles/minikube/proxy-client.crt
/minikube-host/.minikube/profiles/minikube/client.crt
/minikube-host/.minikube/profiles/minikube/apiserver.crt

and the problematic (not renewed) ones are

/var/lib/minikube/certs/etcd/healthcheck-client.crt
/var/lib/minikube/certs/etcd/server.crt
/var/lib/minikube/certs/etcd/peer.crt
/var/lib/minikube/certs/apiserver-etcd-client.crt
/var/lib/minikube/certs/apiserver-kubelet-client.crt
/var/lib/minikube/certs/front-proxy-client.crt

minikube version

minikube version: v1.29.0
commit: ddac20b4b34a9c8c857fc602203b6ba2679794d3

Or is a new issue preferred?

vasilev-alex · 2023-12-05T17:05:42Z

I delete files and folders in the "/var/lib/minikube/certs/*" and minikube stop then start fix my issue

worked for me as well

mbaynton · 2023-12-21T22:27:31Z

Slightly more surgical version of "delete all files and folders in /var/lib/minikube/certs" that worked for me in the case where I could not even start the cluster:

minikube ssh
cd /var/lib/minikube/certs
sudo find . -type f ! -mtime 2 -name '*.crt' -delete
sudo find . -type f ! -mtime 2 -name '*.key' -delete

This deletes all the certs and keys that were last modified 2 or more days ago.

SteveSims2 · 2024-02-27T21:54:49Z

I experienced this issue, and using a combination of the above seems to have recreated ALL of the certs with renewed expiration datetimes.

On my windows machine under C:\Users<name>.minikube there are two certs folders, one at C:\Users<name>.minikube\certs and the other at C:\Users<name>.minikube\profiles\minikube

To recreate the first set: (Powershell syntax)

Stop current minikube cluster

minikube stop

Move main certs out of that path

mv C:\Users<name>.minikube\certs C:\Users<name>\Desktop

Start a new cluster under a different profile

minikube start -p xxx

Allow above to create a new VM on a different profile. Main certs were recreated.

Stop and delete the unneeded profile

minikube stop -p xxx
minikube delete -p xxx

Move the original minikube profile certs out of that path

mv C:\Users<name>.minikube\profile\minikube C:\Users<name>\Desktop

Start the original cluster

minikube start -p minikube

Now do the steps above posters mentioned get into the VM and move the folder at /var/lib/minikube/certs out of the way

minikube ssh
cd /var/lib/minikube
sudo su
mv certs certs.old
exit
exit

Now stop and restart the cluster

minikube stop
minikube start

end

This procedure seemed to renew all of the certs. However, a lot of these steps were guess work. Please comment if you feel something is wrong or don't work for you.

mfisch04 · 2024-04-17T19:51:48Z

I was able to restart my cluster with the following:

minikube ssh
PATH="/var/lib/minikube/binaries/v1.25.2:$PATH"
kubeadm certs renew all --cert-dir /var/lib/minikube/certs
exit
minikube start

valsv · 2024-04-26T05:23:25Z

After updating the certificates, all of my deployments were lost:

root@minikube:/home/docker# PATH="/var/lib/minikube/binaries/v1.26.3:$PATH"

root@minikube:/home/docker# kubeadm certs renew all --cert-dir /var/lib/minikube/certs
MISSING! certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
MISSING! certificate embedded in the kubeconfig file for the controller manager to use
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
MISSING! certificate embedded in the kubeconfig file for the scheduler manager to use
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Is it possible to recover my deployments somehow?

victory4software · 2024-06-24T16:20:46Z

After updating the certificates, all of my deployments were lost:

root@minikube:/home/docker# PATH="/var/lib/minikube/binaries/v1.26.3:$PATH"

root@minikube:/home/docker# kubeadm certs renew all --cert-dir /var/lib/minikube/certs MISSING! certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself certificate for serving the Kubernetes API renewed certificate the apiserver uses to access etcd renewed certificate for the API server to connect to kubelet renewed MISSING! certificate embedded in the kubeconfig file for the controller manager to use certificate for liveness probes to healthcheck etcd renewed certificate for etcd nodes to communicate with each other renewed certificate for serving etcd renewed certificate for the front proxy client renewed MISSING! certificate embedded in the kubeconfig file for the scheduler manager to use Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Is it possible to recover my deployments somehow?

Yes I faced the same issue all my deployment gone is there is any solution for that?

I delete all all the certificates in the below path
/var/lib/minikube/certs

Then stop minikube and start again on the same profile

priyawadhwa added the kind/support Categorizes issue or PR as a support question. label Jan 25, 2021

medyagh changed the title ~~minikube is not working~~ failed to load certificate: the certificate has expired Jan 25, 2021

medyagh added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Jan 25, 2021

medyagh mentioned this issue Mar 5, 2021

"failed to load certificate: the certificate has expire" #10726

Closed

spowelljr added the triage/long-term-support label Apr 21, 2021

spowelljr added long-term-support Long-term support issues that can't be fixed in code and removed triage/long-term-support labels May 19, 2021

spowelljr changed the title ~~failed to load certificate: the certificate has expired~~ Renew cert on start if current cert has has expired Jun 30, 2021

spowelljr mentioned this issue Jun 30, 2021

[support] How to regenerate client certificate #11831

Closed

sharifelgamal added the ev/certificate-errors failed due to certificate issues label Jul 21, 2021

This was referenced Aug 2, 2021

minikube1.2.0 can't start apiserver #11966

Closed

Minikube fails to deploy working cluster (expired certificate) #12093

Closed

sharifelgamal added this to the 1.24.0 milestone Sep 20, 2021

sharifelgamal self-assigned this Sep 20, 2021

sharifelgamal mentioned this issue Sep 21, 2021

renew minikube certs if expired #12534

Merged

sharifelgamal closed this as completed in #12534 Sep 24, 2021

Kimi450 mentioned this issue Mar 25, 2023

Minikube cert expiration Kimi450/ubuntu_server#4

Closed

Kimi450 mentioned this issue Mar 30, 2023

Expired certs not being renewed #16201

Closed

victory4software mentioned this issue Jun 24, 2024

After updating the certificates, all of my deployments were lost: #19134

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Renew cert on start if current cert has has expired #10122

Renew cert on start if current cert has has expired #10122

anravi commented Jan 10, 2021

dabcoder commented Jan 11, 2021 •

edited

Loading

anravi commented Jan 11, 2021

dabcoder commented Jan 25, 2021

medyagh commented Jan 25, 2021

dabcoder commented Jan 27, 2021

dabcoder commented Jan 29, 2021

dabcoder commented Feb 3, 2021

tstromberg commented Feb 22, 2021

medyagh commented Mar 5, 2021 •

edited

Loading

medyagh commented Mar 5, 2021 •

edited

Loading

dabcoder commented Mar 16, 2021

jeffmaury commented Jun 14, 2021

Osprey2021 commented Jul 9, 2021

jeffmaury commented Jul 9, 2021

Osprey2021 commented Jul 9, 2021

jeffmaury commented Jul 9, 2021

Osprey2021 commented Jul 13, 2021

Osprey2021 commented Jul 13, 2021

sharifelgamal commented Jul 21, 2021

edemen commented Jul 28, 2021

infa-ddeore commented Jul 29, 2021

cppragada commented Feb 14, 2022

ebdxflr commented Nov 21, 2022 •

edited

Loading

LeBakii commented Dec 8, 2022

Sergiodcm00 commented Dec 12, 2022

Kimi450 commented Mar 25, 2023 •

edited

Loading

vasilev-alex commented Dec 5, 2023

mbaynton commented Dec 21, 2023

SteveSims2 commented Feb 27, 2024

mfisch04 commented Apr 17, 2024

valsv commented Apr 26, 2024 •

edited

Loading

victory4software commented Jun 24, 2024

Renew cert on start if current cert has has expired #10122

Renew cert on start if current cert has has expired #10122

Comments

anravi commented Jan 10, 2021

dabcoder commented Jan 11, 2021 • edited Loading

anravi commented Jan 11, 2021

dabcoder commented Jan 25, 2021

medyagh commented Jan 25, 2021

dabcoder commented Jan 27, 2021

dabcoder commented Jan 29, 2021

dabcoder commented Feb 3, 2021

tstromberg commented Feb 22, 2021

medyagh commented Mar 5, 2021 • edited Loading

medyagh commented Mar 5, 2021 • edited Loading

dabcoder commented Mar 16, 2021

jeffmaury commented Jun 14, 2021

Osprey2021 commented Jul 9, 2021

jeffmaury commented Jul 9, 2021

Osprey2021 commented Jul 9, 2021

jeffmaury commented Jul 9, 2021

Osprey2021 commented Jul 13, 2021

Osprey2021 commented Jul 13, 2021

sharifelgamal commented Jul 21, 2021

edemen commented Jul 28, 2021

infa-ddeore commented Jul 29, 2021

cppragada commented Feb 14, 2022

ebdxflr commented Nov 21, 2022 • edited Loading

LeBakii commented Dec 8, 2022

Sergiodcm00 commented Dec 12, 2022

Kimi450 commented Mar 25, 2023 • edited Loading

vasilev-alex commented Dec 5, 2023

mbaynton commented Dec 21, 2023

SteveSims2 commented Feb 27, 2024

Stop current minikube cluster

Move main certs out of that path

Start a new cluster under a different profile

Allow above to create a new VM on a different profile. Main certs were recreated.

Stop and delete the unneeded profile

Move the original minikube profile certs out of that path

Start the original cluster

Now do the steps above posters mentioned get into the VM and move the folder at /var/lib/minikube/certs out of the way

Now stop and restart the cluster

end

mfisch04 commented Apr 17, 2024

valsv commented Apr 26, 2024 • edited Loading

victory4software commented Jun 24, 2024

dabcoder commented Jan 11, 2021 •

edited

Loading

medyagh commented Mar 5, 2021 •

edited

Loading

medyagh commented Mar 5, 2021 •

edited

Loading

ebdxflr commented Nov 21, 2022 •

edited

Loading

Kimi450 commented Mar 25, 2023 •

edited

Loading

valsv commented Apr 26, 2024 •

edited

Loading