Container runtime containerd with insecure registry breaks new pods

[ahamilton55]

What Happened?

When creating an new minikube cluster locally with Docker and specifying the --insecure-registry option, pods created after the registry is added to the config fail to start. This breaks with the 1.26.0 minikube release but works with release versions 1.25.2 and older.

For example, the coredns pod has been stuck in the ContainerCreating stage but the other pods have started. Any other pods that are created now stay in the ContainerCreating stage.

kubectl get pods -A
NAMESPACE     NAME                               READY   STATUS              RESTARTS   AGE
kube-system   coredns-558bd4d5db-fh95p           0/1     ContainerCreating   0          14m
kube-system   etcd-minikube                      1/1     Running             0          14m
kube-system   kindnet-xgb9c                      1/1     Running             0          14m
kube-system   kube-apiserver-minikube            1/1     Running             0          14m
kube-system   kube-controller-manager-minikube   1/1     Running             0          14m
kube-system   kube-proxy-s66zf                   1/1     Running             0          14m
kube-system   kube-scheduler-minikube            1/1     Running             0          14m
kube-system   storage-provisioner                1/1     Running             1          14m

Another error that is seen in my tilt environment is the following:

[event: pod atm/atmr-db-migrate-q75zf] (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b7f3f841cd2efa4a0fd20a2a79e4a44745db4e10d0eccce2804c0a87e5d63932": plugin type="bridge" name="crio" failed (add): failed to set bridge addr: could not add IP address to "cni0": permission denied

The error seems to occur because the deprecated configuration setup for adding registry mirrors is used and breaks the containerd service after being applied and restarted.

I built a local version of minikube and updated to the newer containerd mirror structure works as expected and fixes the issue. (I'm currently trying to figure out if I can open the PR for it or not).

Containerd Configure Image Registry
Containerd Registry Configuration

Attach the log file

minikube.log

Operating System

macOS (Default)

Driver

Docker

[klaases]

I built a local version of minikube and updated to the newer containerd mirror structure works as expected and fixes the issue. (I'm currently trying to figure out if I can open the PR for it or not).

Hi @ahamilton55, can you confirm if minikube is working as expected? By the way, we've updated to v.1.26.1, so please give it a try.

[ahamilton55]

Hi @klaases . I just downloaded 1.26.1 and built a new local cluster but I'm still seeing the same issue.

My PR is still available if someone is able to review it.