Minikube start failure in CentOS

[furkatgofurov7]

Hi, we are running CentOS jobs with Minikube in our CI, and builds flake out when minikube starts 20-30% of the time. Below are the steps to reproduce the issue:

1. Centos 8.3 VM running in the Cloud

$cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

2. Installed minikube version: v1.20.0 configured with
   minikube config set vm-driver kvm2
minikube config set memory 4096
3. Below script is run to reproduce the issue.

 #!/bin/bash

MINIKUBE="minikube"

$MINIKUBE config set vm-driver kvm2
$MINIKUBE config set memory 4096

while /bin/true; do  
  $MINIKUBE delete  
  if ! $MINIKUBE start -v=9 --alsologtostderr > /tmp/minikubefail 2>&1; then    
    exit 0  
  fi
done

Full output of minikube logs command:
Logs from Minikube v1.18

`+ sudo su -l -c 'minikube start --insecure-registry 192.168.111.1:5000' ****
* minikube v1.18.1 on Centos 8.3.2011 (kvm/amd64)
* minikube 1.19.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.19.0
* To disable this notice, run: 'minikube config set WantUpdateNotification false'

* Using the kvm2 driver based on user configuration

X Exiting due to PROVIDER_KVM2_ERROR: /usr/bin/virsh domcapabilities --virttype kvm failed:

* Suggestion: Follow your Linux distribution instructions for configuring KVM
* Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/kvm2/ `

Logs from Minikube v1.19

`+ sudo su -l -c 'minikube start --insecure-registry 192.168.111.1:5000' ****
* minikube v1.19.0 on Centos 8.3.2011 (kvm/amd64)
* Using the kvm2 driver based on user configuration
* Downloading VM boot image ...
    > minikube-v1.19.0.iso.sha256: 65 B / 65 B [-------------] 100.00% ? p/s 0s    > minikube-v1.19.0.iso: 16.00 MiB / 244.49 MiB [>___________] 6.54% ? p/s ?    > minikube-v1.19.0.iso: 43.95 MiB / 244.49 MiB [->_________] 17.98% ? p/s ?    > minikube-v1.19.0.iso: 48.00 MiB / 244.49 MiB [-->________] 19.63% ? p/s ?    > minikube-v1.19.0.iso: 71.01 MiB / 244.49 MiB  29.05% 91.82 MiB p/s ETA 1s    > minikube-v1.19.0.iso: 92.25 MiB / 244.49 MiB  37.73% 91.82 MiB p/s ETA 1s    > minikube-v1.19.0.iso: 129.89 MiB / 244.49 MiB  53.13% 91.82 MiB p/s ETA 1    > minikube-v1.19.0.iso: 182.14 MiB / 244.49 MiB  74.50% 97.85 MiB p/s ETA 0    > minikube-v1.19.0.iso: 229.50 MiB / 244.49 MiB  93.87% 97.85 MiB p/s ETA 0    > minikube-v1.19.0.iso: 244.49 MiB / 244.49 MiB  100.00% 168.40 MiB p/s 1.6* Starting control plane node minikube in cluster minikube
* Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v10-v1...: 16.00 MiB / 491.71 MiB [>_] 3.25% ? p/s ?    > preloaded-images-k8s-v10-v1...: 48.00 MiB / 491.71 MiB [>_] 9.76% ? p/s ?    > preloaded-images-k8s-v10-v1...: 77.75 MiB / 491.71 MiB [>] 15.81% ? p/s ?    > preloaded-images-k8s-v10-v1...: 120.00 MiB / 491.71 MiB  24.40% 173.19 Mi    > preloaded-images-k8s-v10-v1...: 159.17 MiB / 491.71 MiB  32.37% 173.19 Mi    > preloaded-images-k8s-v10-v1...: 190.86 MiB / 491.71 MiB  38.82% 173.19 Mi    > preloaded-images-k8s-v10-v1...: 238.47 MiB / 491.71 MiB  48.50% 174.77 Mi    > preloaded-images-k8s-v10-v1...: 257.19 MiB / 491.71 MiB  52.30% 174.77 Mi    > preloaded-images-k8s-v10-v1...: 304.64 MiB / 491.71 MiB  61.95% 174.77 Mi    > preloaded-images-k8s-v10-v1...: 344.00 MiB / 491.71 MiB  69.96% 174.82 Mi    > preloaded-images-k8s-v10-v1...: 349.69 MiB / 491.71 MiB  71.12% 174.82 Mi    > preloaded-images-k8s-v10-v1...: 379.53 MiB / 491.71 MiB  77.19% 174.82 Mi    > preloaded-images-k8s-v10-v1...: 430.19 MiB / 491.71 MiB  87.49% 172.82 Mi    > preloaded-images-k8s-v10-v1...: 438.08 MiB / 491.71 MiB  89.09% 172.82 Mi    > preloaded-images-k8s-v10-v1...: 480.00 MiB / 491.71 MiB  97.62% 172.82 Mi    > preloaded-images-k8s-v10-v1...: 491.71 MiB / 491.71 MiB  100.00% 164.22 M* Creating kvm2 VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
* Deleting "minikube" in kvm2 ...
! StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute
* Creating kvm2 VM (CPUs=2, Memory=4096MB, Disk=20000MB) ...
* Failed to start kvm2 VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute

X Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute
* 
* If the above advice does not help, please let us know: 
  https://github.com/kubernetes/minikube/issues/new/choose  `

Full output of failed command:

Output of minikubefail file: paste.openstack

[furkatgofurov7]

Thanks for response beforehand! I can fetch more logs from the environment where we are encountering the problem if needed.

[furkatgofurov7]

hi! Can someone please take a look and if possible give some pointers regarding this issue, it seems to be not flaky but every other build is failing with same error in our CI. Thank you.

/cc @tstromberg @medyagh @afbjorklund

[ilya-zuyev]

@furkatgofurov7 Thanks for creating this issue!
Is your CI minikube the only process on the host that uses kvm/libvirt? Is it only one minikube, or multiple instances running in parallel? We're still investigating the problem, in the meantime you can try to use minikube with --driver=docker, not sure if it's a valid option for you.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[anoopcs9]

While trying to create a cluster using minikube on CentOS we are also encountering the exact same issue:

🔥  Creating kvm2 VM (CPUs=2, Memory=4096MB, Disk=20480MB) ...
🔥  Deleting "minikube" in kvm2 ...
🤦  StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute
🔥  Creating kvm2 VM (CPUs=2, Memory=4096MB, Disk=20480MB) ...
😿  Failed to start kvm2 VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute

❌  Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute

Platform and binary details:

# cat /etc/os-release 
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

# minikube version
minikube version: v1.24.0
commit: 76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b

minikube options used are --driver=kvm2, --nodes=3, --memory=4096, --cpus=2, --extra-disks=2, --disk-size=20g

In fact minikube VM itself is running and IP is assigned to it.

# virsh list
 Id   Name       State
--------------------------
 2    minikube   running

# virsh domifaddr minikube
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet3      52:54:00:6b:be:75    ipv4         192.168.124.208/24

What are we missing here?

[spowelljr]

Hi @anoopcs9, could you run minikube logs --file=logs.txt and upload the generated log file, thanks!

[anoopcs9]

Hi @anoopcs9, could you run minikube logs --file=logs.txt and upload the generated log file, thanks!

logs.txt

[anoopcs9]

Hi @spowelljr, let me know if more logs are required.

[anoopcs9]

I have a new finding to share.

To me it looks like ip address assigned for mk-minikube network is reflected only after a specified amount of time. At the same time it is show when --source is changed to arp as shown below:

# virsh domifaddr minikube
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet3      52:54:00:68:87:66    ipv4         192.168.122.152/24

# cat /var/lib/libvirt/dnsmasq/virbr1.status

# virsh domifaddr minikube --source arp
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet2      52:54:00:fc:2c:83    ipv4         192.168.39.246/0
 vnet3      52:54:00:68:87:66    ipv4         192.168.122.152/0

Bridge interface status file is not updated immediately. If I manually do virsh net-update mk-minikube ip is shown afterwards and status gets updated.

# virsh domifaddr minikube
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 vnet2      52:54:00:fc:2c:83    ipv4         192.168.39.246/24
 vnet3      52:54:00:68:87:66    ipv4         192.168.122.152/24

# cat /var/lib/libvirt/dnsmasq/virbr1.status 
[
  {
    "ip-address": "192.168.39.246",
    "mac-address": "52:54:00:fc:2c:83",
    "hostname": "minikube",
    "client-id": "01:52:54:00:fc:2c:83",
    "expiry-time": 1643050981
  }
]

In Fedora bridge status file gets updated along with domain creation it seems.

What kind of time period are we looking at here? Even if we don't execute virsh net-update status file gets updates afterwards.

And I was thinking whether it is OK to change source to arp inside minikube(DOMAIN_INTERFACE_ADDRESSES_SRC_LEASE -> DOMAIN_INTERFACE_ADDRESSES_SRC_ARP) which should work independent of libvirt versions on different platforms.

@spowelljr thoughts?

[spowelljr]

Hi @anoopcs9, sorry for the delayed response.

I'm not very familiar with this part of minikube but I'll do my best to help.

What kind of time period are we looking at here?

Assuming you're asking how long it waits, it retries for one minutes with an exponential backoff.

And I was thinking whether it is OK to change source to arp inside minikube

I don't know the answer to that, but we could create a PR and run tests against it and see if it past tests.