Qemu2 driver, Can not Start after a stop

[medyagh]

What Happened?

I just tried Minikube Qemu driver on M1, and it starts nicely, but when I stop and start it again it wont start again

19:23:17 medya
$ minikube start --driver=qemu2
😄  minikube v1.26.0-beta.0 on Darwin 12.3.1 (arm64)
✨  Using the qemu2 (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.23.6 on Docker 20.10.14 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
19:24:30 medya
$ minikube stop
✋  Stopping node "minikube"  ...

🛑  1 node stopped.
19:24:47 medya
$ 
19:24:47 medya
$ minikube start 
😄  minikube v1.26.0-beta.0 on Darwin 12.3.1 (arm64)
✨  Using the qemu2 (experimental) driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing qemu2 VM for "minikube" ...
🐳  Preparing Kubernetes v1.23.6 on Docker 20.10.14 ...\ ^C

@afbjorklund I suspect this is because the Random Port, here is the log of minikube start (after a stop)

Attach the log file

14 19:26:26.976485 22066 api_server.go:256] stopped: https://localhost:64024/healthz: Get "https://localhost:64024/healthz": dial tcp [::1]:64024: connect: connection refused
I0514 19:26:27.475184 22066 api_server.go:240] Checking apiserver healthz at https://localhost:64024/healthz ...
I0514 19:26:27.476992 22066 api_server.go:256] stopped: https://localhost:64024/healthz: Get "https://localhost:64024/healthz": dial tcp [::1]:64024: connect: connection refused
I0514 19:26:27.975299 22066 api_server.go:240] Checking apiserver healthz at https://localhost:64024/healthz ...
I0514 19:26:27.976082 22066 api_server.go:256] stopped: https://localhost:64024/healthz: Get "https://localhost:64024/healthz": dial tcp [::1]:64024: connect: connection refused
I0514 19:26:28.475373 22066 api_server.go:240] Checking apiserver healthz at https://localhost:64024/healthz ...
I0514 19:26:28.476598 22066 api_server.go:256] stopped: https://localhost:64024/healthz: Get "https://localhost:64024/healthz": dial tcp [::1]:64024: connect: connection refused
I0514 19:26:28.975479 22066 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0514 19:26:29.010638 22066 logs.go:274] 2 containers: [2dc1caca367a 3b05de79db6e]
I0514 19:26:29.010718 22066 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}

Operating System

macOS (Default)

Driver

N/A

[afbjorklund]

Yeah, there is some duplicated check here that doesn't start the ssh tunneling of the 8443 port properly.

start, code path 1

🔥 Creating qemu2 VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
post-start starting for "minikube" (driver="qemu2")

APIServerPort:39899
Checking apiserver healthz at https://localhost:39899/healthz ...

stop

The ssh tunnel is stopped with the node:

✋  Stopping node "minikube"  ...
Connection to localhost closed by remote host.
🛑  1 node stopped.

start, code path 2

🔄 Restarting existing qemu2 VM for "minikube" ...
post-start starting for "minikube" (driver="qemu2")

APIServerPort:34455
Checking apiserver healthz at https://localhost:34455/healthz ...

This will get better when it is moved to the same place as the 22 port, so that it gets started with the vm.

[afbjorklund]

But it did start ok (second time) on Linux, eventually

[sharifelgamal]

Yeah, it takes a bit longer, but it does eventually start up. I tested on M1 as well.

[sharifelgamal]

Yeah, so the source of the problem is that when there's an existing cluster we're restarting, we check of the api server is healthy or not. Since the config has the updated api server port, but hasn't done the tunneling yet (since that normally happens after everything in the cluster is up on first start), the check just times out over and over and fails. It's not a hard failure, since that causes a hard restart eventually which succeeds.

[afbjorklund]

Ah, so it was the self-destruct I was seeing