Flag "image-repository" prevents kube-proxy pod from running

[clamprecht01]

Note: Eventual goal of this environment is for an entirely air-gapped minikube environment with Ingress addon enabled. Without "image-repository" flag, minikube starts and runs, but we don't have the "ingress" addon enabled (which we need). Solution to install "ingress" addon from offline source was taken from PR-9551.

Pre-requisites:

• Copy "cache" directory from online minikube environment to offline
• Upload "Kicbase" image to local_docker_registry

Minikube start command:
minikube start --vm-driver=docker --base-image="<local_docker_registry>:5000/gcr.io/k8s-minikube/kicbase:latest" --insecure-registry="<local_docker_registry>:5000"

Note: Minikube will start correctly with above command

Steps to reproduce the issue:

1. kube-proxy pod gets into "CrashLoopBackoff" state when started with "--image-repository" flag
2. minikube start --vm-driver=docker --base-image="<local_docker_registry>:5000/gcr.io/k8s-minikube/kicbase:latest" --insecure-registry="<local_docker_registry>:5000" --image-repository="<local_docker_registry>:5000"

PS C:> kubectl get pods -A (without image-repository):

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-74ff55c5b-4rhr6 1/1 Running 0 2m1s
kube-system etcd-minikube 1/1 Running 0 2m14s
kube-system kube-apiserver-minikube 1/1 Running 0 2m14s
kube-system kube-controller-manager-minikube 1/1 Running 0 2m14s
kube-system kube-proxy-gk8km 1/1 Running 0 2m1s
kube-system kube-scheduler-minikube 1/1 Running 0 2m14s
kube-system storage-provisioner 1/1 Running 1 2m12s

PS C:> kubectl get pods -A (with image-repository flag):

PS C:\minikube_reqd_images> kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-5b77899479-x68gr 1/1 Running 0 69s
kube-system etcd-minikube 1/1 Running 2 88s
kube-system kube-apiserver-minikube 1/1 Running 0 2m8s
kube-system kube-controller-manager-minikube 1/1 Running 1 88s
kube-system kube-proxy-98rmh 0/1 CrashLoopBackOff 3 58s
kube-system kube-scheduler-minikube 1/1 Running 0 2m8s
kube-system storage-provisioner 1/1 Running 4 9m2s

Full output of minikube start command used, if not already included:
Attached log files:

minikube start --vm-driver=docker --base-image="<local_docker_registry>:5000/gcr.io/k8s-minikube/kicbase:latest" --insecure-registry="<local_docker_registry>:5000" --image-repository="<local_docker_registry>:5000" >>
minikube_start_not_working.log

minikube start --vm-driver=docker --base-image="<local_docker_registry>:5000/gcr.io/k8s-minikube/kicbase:latest" --insecure-registry="<local_docker_registry>:5000" >> minikube_start_working.log

Optional: Full output of minikube logs command:

[clamprecht01]

Additional info
kubectl describe kube-proxy -n kube-system kube-proxy_kubelet.log

kubectl describe storage-provision -n kube-system storage-provision_kubelet.log

[sharifelgamal]

 Message:      OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "/usr/local/bin/kube-proxy": stat /usr/local/bin/kube-proxy: no such file or directory: unknown

That error is clearly what's causing the kube-proxy pod to crash. I wonder if we're not mounting in that binary properly in certain cases?

[clamprecht01]

@sharifelgamal I had noticed that as well at one point, but I guess I forgot about it.

Would I expect to find that binary on the Minikube container (currently I don't see it)?

[medyagh]

@clamprecht01 is there a reason you are providing your own --base-image ?

[medyagh]

/triage needs-information
/kind support

[clamprecht01]

I found that the core issue was actually related to the kube-proxy pod was using the kube-scheduler image.

Bad kube proxy output (offline):

kubectl describe pod kube-proxy-zxv2w -n kube-system | select-string "Image"

Image:        <internal_IP>:5000/kube-proxy:v1.20.0
Image ID:      docker-pullable://k8s.gcr.io/kube-scheduler@sha256:beaa710325047fa9c867eff4ab9af38d9c2acec05ac5b416c708c304f76bdbef

Good kube proxy output (online):

kubectl describe pod kube-proxy-lmx55 -n kube-system | Select-String "Image"

Image:         k8s.gcr.io/kube-proxy:v1.20.2
Image ID:      docker-pullable://k8s.gcr.io/kube-proxy@sha256:326fe8a4508a5db91cf234c4867eff5ba458bc4107c2a7e15c827a74faa19be9

**Note:
The "correct" kube-proxy image is available on the internal image repository.

[clamprecht01]

@medyagh

I have to use the --base-image flag or else the creation step fails. Even though this step isn't detailed documentation, I found another ticket that informed me of this flag

"
PS C:\Docker_images\Wolfpack_TM\Integ> minikube start --vm-driver=docker --insecure-registry="<local_docker_IP>:5000" --embed-certs

• minikube v1.18.1 on Microsoft Windows 10 Enterprise 10.0.19042 Build 19042
• Using the docker driver based on user configuration
• Starting control plane node minikube in cluster minikube
• Pulling base image ...
  E0316 09:31:04.734404 9412 cache.go:180] Error downloading kic artifacts: failed to download kic base image or any fallback image
  "

[clamprecht01]

Found that issue was either with:

• Minikube Version 16 (no longer issue when upgraded to version 18.1)
• Testing environment had bad images (possibly when transferring to offline system, I named/tagged images incorrectly)

Closing issue.