Expired certs not being renewed

[Kimi450]

What Happened?

This was supposed fixed in issue 10122 but I dont think this is fixed. Or at least not as I would expect it to be fixed. Ive linked the issue from my repo here. You can check my Kimi450/ubuntu_server#4 (comment) that issue which describes why I think its broken (and how to replicate it). Unless im doing something wrong, I think this issue still persists.

Basically, if you use --cert-expiration tag, that only affects

/var/lib/minikube/certs/proxy-client.crt
/var/lib/minikube/certs/apiserver.crt

# these are on my mounted ones, minikube start --mount --mount-string "/home/kimi450:/minikube-host" --cert-expiration="120s"
/minikube-host/.minikube/profiles/minikube/proxy-client.crt
/minikube-host/.minikube/profiles/minikube/client.crt
/minikube-host/.minikube/profiles/minikube/apiserver.crt

and the problematic (not renewed) ones are

/var/lib/minikube/certs/etcd/healthcheck-client.crt
/var/lib/minikube/certs/etcd/server.crt
/var/lib/minikube/certs/etcd/peer.crt
/var/lib/minikube/certs/apiserver-etcd-client.crt
/var/lib/minikube/certs/apiserver-kubelet-client.crt
/var/lib/minikube/certs/front-proxy-client.crt

Without these being renewed, the minikube instance cannot be accessed (unless you change the system time to a date in the past)

minikube version

minikube version: v1.29.0
commit: ddac20b4b34a9c8c857fc602203b6ba2679794d3

Reproduce by doing the following

• install minikube (no special setting required)

• change system time

  timedatectl
  timedatectl set-ntp no
  
  # some time in the future, more than a year 
  timedatectl set-time 2025-08-10
  

• restart minikube (optional)

  minikube stop
  minikube start
  

Attach the log file

I do not have this at the moment, but I can provide later if absolutely necessary

Operating System

Ubuntu

Driver

Docker

[kundan2707]

/kind support

[spowelljr]

Thanks for the information @Kimi450, we occasionally got issues related to certs after we thought it was fixed, but this confirms why it's failing.

[Kimi450]

Thanks for the information @Kimi450, we occasionally got issues related to certs after we thought it was fixed, but this confirms why it's failing.

No worries! I am not sure which ones from the list above need to be renewed, but at least after renewing all of them I think this issue should be fixed. Im hoping that it will be straight forward (the same logic from the other certs being renewed can be used here too) to do this.

[spowelljr]

Im hoping that it will be straight forward (the same logic from the other certs being renewed can be used here too) to do this.

The certs that are not being renewed are actually created by kubeadm on kubeadm init so we can just loop through all those certs, check if they're expired, and if they are run: kubeadm certs renew <cert_path> --config <kubeadm_config_path>.

[Kimi450]

I was surprised to not find an initial config to set an expiration for these in the first place, but it seems like its designed to be this way to encourage upgrades.

When would the looping be done? Would have to be some sort of a cron job that runs daily to see if theyre close to expiration and renew with some buffer I guess? Theres a caveat for static pods though. Im not sure which pods are static in the control plane but this will have to be catered too Im guessing. So it might not be as straight forward as we might have been hoping.

[spowelljr]

minikube currently checks for outdated certs on minikube start so I'd recommend we add the new check there as well. In regard to static pods, If minikube is stopped all pods should be stopped, as the container/VM is shut down so I don't think that will be an issue. I just tried a POC and it worked great, I'll polish it up and create a PR for it.

[Kimi450]

minikube currently checks for outdated certs on minikube start so I'd recommend we add the new check there as well. In regard to static pods, If minikube is stopped all pods should be stopped, as the container/VM is shut down so I don't think that will be an issue. I just tried a POC and it worked great, I'll polish it up and create a PR for it.

Great, thanks for putting in the work!

[medyagh]

this bug has been bugging me for years ! good job @Kimi450 and @spowelljr on helping to investigate it

[Kimi450]

this bug has been bugging me for years ! good job @Kimi450 and @spowelljr on helping to investigate it

Same lol, after it bricked my server once I decided to finally look into it before it happens again.

EDIT: Does this mean Ive a job at Google secured? /jk

[spowelljr]

I've finalized the PR with reproduction steps in the description if you want to rest it for yourself @Kimi450

[Kimi450]

I've finalized the PR with reproduction steps in the description if you want to rest it for yourself @Kimi450

Nice, thanks. I'll try to take it out for a test run over the weekend but I'm sure it's fine anyway. Appreciate the fix, thanks again!

[Kimi450]

I've finalized the PR with reproduction steps in the description if you want to rest it for yourself @Kimi450

Ngl, Im not sure how to test it out myself just from the src code changes. However, the changes youve made look good to me (I added a small comment that I think is worthwhile changing). And your description shows that it works too. Id be happy to see this released so I can upgrade my server :)

[Kimi450]

@spowelljr thanks for the fix, do you know what release this change will be included in?

[spowelljr]

The next release will include the fix (v1.31.0), we don't yet have an ETA for the release though, I can provide you a link to the binary if you'd like though

[Kimi450]

If you wouldnt mind, Id appreciate it! Just in case :) Thanks!

[spowelljr]

https://storage.googleapis.com/minikube-builds/16249/minikube-linux-amd64

If you run into any cert related issues let me know

[Kimi450]

Thanks! I might try a test build if I get some free time