CI jobs often fail with minikube error X Exiting due to GUEST_PROVISION

[nixpanic]

Describe the bug

When runninc CI jobs, minikube regularly fails with the following error:

* Failed to start kvm2 VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute

X Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute

This prevents the job from continuing, and a /retest ... is needed.

Logs

From mini-e2e_k8s-1.20/1523:

* Creating kvm2 VM (CPUs=8, Memory=14336MB, Disk=20000MB) ...
* Deleting "minikube" in kvm2 ...
! StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute
* Creating kvm2 VM (CPUs=8, Memory=14336MB, Disk=20000MB) ...
* Failed to start kvm2 VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute

X Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute

[nixpanic]

#2343 (comment) failed again (logs)

[nixpanic]

#2308 (comment) failed here too (logs)

[nixpanic]

#2354 (comment) as well (logs)

[nixpanic]

failed in #2350 (comment) too (logs

[nixpanic]

#2341 (comment) hit this too (logs)

[Madhu-1]

@nixpanic as we are hitting frequently is it a good idea to add a wrapper to retry?

[nixpanic]

#2322 (comment) hit this (logs)

[nixpanic]

@nixpanic as we are hitting frequently is it a good idea to add a wrapper to retry?

I don't know. There is already an automated retry by minikube. We should try to identify the cause and work on preventing it. Gathering the hostnames of the CentOS CI bare-metal machines might give a clue. Different groups of hosts have different hardware, and are in different subnets (https://wiki.centos.org/QaWiki/PubHardware).

[nixpanic]

#2341 (comment) as well (logs)

[nixpanic]

#2339 (comment) hit this too (logs)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[nixpanic]

This is definitely not fixed yet 😞

[nixpanic]

Seems like kubernetes/minikube#11459 is reported too.

[nixpanic]

This has not happened since a long time.