kvm2 driver vs. docker-machine #9453

darkn3rd · 2020-10-12T23:30:15Z

The machine driver kvm2 with the default options to docker-machine no longer work. There's no documentation around how minikube interacts with docker-machine to get the correction options. Currently, there is no working driver at this moment that will support KVM with docker-machine, so I hope there can be consideration to support this, as there is no solution path.

Steps to reproduce the issue:

Install Docker Machine

curl -sL https://github.com/docker/machine/releases/download/v0.16.2/docker-machine-Linux-x86_64 \
    > /tmp/docker-machine
sudo mv /tmp/docker-machine /usr/local/bin/docker-machine &&
  chmod +x /usr/local/bin/docker-machine

Install KVM2 driver

sudo apt install ./docker-machine-driver-kvm2_1.14.0-0_amd64.deb 
export MACHINE_DRIVER="kvm2"

Delete all other systems running (did this with Virtual Machine Manager)

Create new docker machine instance

docker-machine --debug create --driver kvm2 default

Full output of failed command:

Docker Machine Version:  0.16.2, build bd45ab13
Found binary path at /usr/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:41235
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(flag-lookup) Calling .GetMachineName
(flag-lookup) Calling .DriverName
(flag-lookup) Calling .GetCreateFlags
Found binary path at /usr/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:46701
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(default) Calling .GetMachineName
(default) Calling .DriverName
(default) Calling .GetCreateFlags
(default) Calling .SetConfigFromFlags
Reading certificate data from /home/joaquin/.docker/machine/certs/ca.pem
Decoding PEM data...
Parsing certificate...
Reading certificate data from /home/joaquin/.docker/machine/certs/cert.pem
Decoding PEM data...
Parsing certificate...
Running pre-create checks...
(default) Calling .PreCreateCheck
(default) Calling .GetConfigRaw
Creating machine...
(default) Calling .Create
(default) Creating KVM machine...
(default) Setting up store path in /home/joaquin/.docker/machine/machines/default ...
(default) Building disk image from 
(default) DBG | ERROR: logging before flag.Parse: I1012 16:26:50.026862   98864 common.go:100] Making disk image using store path: /home/joaquin/.docker/machine
(default) DBG | local Boot2Docker ISO version:  v19.03.12
(default) Copying /home/joaquin/.docker/machine/cache/boot2docker.iso to /home/joaquin/.docker/machine/machines/default/boot2docker.iso...
(default) DBG | ERROR: logging before flag.Parse: I1012 16:26:50.347365   98864 common.go:107] Creating ssh key: /home/joaquin/.docker/machine/machines/default/id_rsa...
(default) DBG | ERROR: logging before flag.Parse: I1012 16:26:50.555036   98864 common.go:113] Creating raw disk image: /home/joaquin/.docker/machine/machines/default/default.rawdisk...
(default) DBG | Writing magic tar header
(default) DBG | Writing SSH key tar header
(default) DBG | ERROR: logging before flag.Parse: I1012 16:26:50.555157   98864 common.go:127] Fixing permissions on /home/joaquin/.docker/machine/machines/default ...
(default) DBG | Checking permissions on dir: /home/joaquin/.docker/machine/machines/default
(default) Setting executable bit set on /home/joaquin/.docker/machine/machines/default (perms=drwx------)
(default) DBG | Checking permissions on dir: /home/joaquin/.docker/machine/machines
(default) Setting executable bit set on /home/joaquin (perms=drwxr-xr-x)
(default) DBG | Checking permissions on dir: /home/joaquin/.docker/machine
(default) DBG | Checking permissions on dir: /home/joaquin/.docker
(default) DBG | Checking permissions on dir: /home/joaquin
(default) Creating domain...
(default) DBG | Checking permissions on dir: /home
(default) DBG | Skipping /home - not owner
(default) KVM machine creation complete!
Error creating machine: Error in driver during machine creation: creating domain: error defining domain xml: 
<domain type='kvm'>
  <name>default</name>
  <memory unit='MiB'>0</memory>
  <vcpu>0</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    
  </features>
  <cpu mode='host-passthrough'/>
  <os>
    <type>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
  </os>
  <devices>
    <disk type='file' device='cdrom'>
      <source file=''/>
      <target dev='hdc' bus='scsi'/>
      <readonly/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='default' io='threads' />
      <source file=''/>
      <target dev='hda' bus='virtio'/>
    </disk>
    <interface type='network'>
      <source network='default'/>
      <mac address='38:dd:69:8f:f2:3d'/>
      <model type='virtio'/>
    </interface>
    <interface type='network'>
      <source network='minikube-net'/>
      <mac address='80:02:5c:bd:b1:b7'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
    </rng>
    
  </devices>
</domain>
: virError(Code=1, Domain=20, Message='internal error: CPU IDs in <numa> exceed the <vcpu> count')
notifying bugsnag: [Error creating machine: Error in driver during machine creation: creating domain: error defining domain xml: 
<domain type='kvm'>
  <name>default</name>
  <memory unit='MiB'>0</memory>
  <vcpu>0</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    
  </features>
  <cpu mode='host-passthrough'/>
  <os>
    <type>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
  </os>
  <devices>
    <disk type='file' device='cdrom'>
      <source file=''/>
      <target dev='hdc' bus='scsi'/>
      <readonly/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='default' io='threads' />
      <source file=''/>
      <target dev='hda' bus='virtio'/>
    </disk>
    <interface type='network'>
      <source network='default'/>
      <mac address='38:dd:69:8f:f2:3d'/>
      <model type='virtio'/>
    </interface>
    <interface type='network'>
      <source network='minikube-net'/>
      <mac address='80:02:5c:bd:b1:b7'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
    </rng>
    
  </devices>
</domain>
: virError(Code=1, Domain=20, Message='internal error: CPU IDs in <numa> exceed the <vcpu> count')]

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

The text was updated successfully, but these errors were encountered:

medyagh · 2020-10-13T00:46:35Z

thank you for creaitng an issue @darkn3rd I am not clear what you are trying to do,
are you trying to install KVM2 ? or are you trying to install docker using KVM2 as a backend?

if you are trying to install kvm2 so you can use minikube's kvm2 driver here, does the installation docs here help ? https://minikube.sigs.k8s.io/docs/drivers/kvm2/#installing-prerequisites

you dont need to do docker-machine create !

btw currently docker driver is our newest and most supported driver, I recommend docker driver instead of KVM2 on linux.

medyagh · 2020-10-13T00:46:44Z

/triage needs-information
/kind support

darkn3rd · 2020-10-13T07:45:25Z

I just wanted to use docker essentially, but not on the host, in a virtual machine.

would like to run docker-compose to automate docker that doesn't use host's docker daemon.
run tests, docs, tutorials for those using docker-compose across win/lin/mac w/ consistent version of docker-compose used across all platforms.
would like to use docker-compose on system that has libvirt/kvm for faster performance, as virtualbox is slow.

I had installed the following and I am able to create/destroy virtual machines.

sudo apt-get install -y \
 bridge-utils \
 cpu-checker \
 libguestfs-tools \
 libvirt-dev \
 qemu-kvm \
 virt-manager

I ran virt-host-validate and had all pass with these two warnings (on clean Ubuntu 20.04.1):

  QEMU: Checking if IOMMU is enabled by kernel                               : WARN (IOMMU appears to be disabled in kernel. Add intel_iommu=on to kernel cmdline arguments)
  QEMU: Checking for secure guest support                                    : WARN (Unknown if this platform has Secure Guest support)

If I don't need to docker-machine create, how do I run docker commands that use docker from the VM(s) created with minikube? For example, the docker-machine env equivalent? Is there anyway to tell what minikube sets to docker-machine to create the k8s nodes?

afbjorklund · 2020-10-13T09:30:15Z

We had some of the discussion of this in dhiltgen/docker-machine-kvm#77 already, but I will take it here too.

Basically minikube has forked the "kvm" driver, so the kvm2 is now only used for minikube and doesn't work with docker-machine.
The libmachine library is also forked, but part of it is still using the same code - i.e. the changes to the project code are smaller

The docker-machine project itself, including the boot2docker iso and particularly the kvm (libvirt) driver, is now deprecated (dead).
We have applied some bug fixes in the so-called "machine-drivers" project, that is not affiliated with Docker Inc. (like the others):

https://github.com/machine-drivers/machine

https://github.com/machine-drivers/docker-machine-kvm

As suggested, it would be possible to issue a fixed/backported version of the old "docker-machine-driver-kvm" driver binary there.
Neither project has released any binaries, so there is no infrastructure for that like it was in the old/upstream projects from docker.

~~https://github.com/docker/machine~~ v0.16.2 (Sep, 2019)

~~https://github.com/dhiltgen/docker-machine-kvm~~ v0.10.0 (Apr, 2017)

Currently I would recommend Vagrant as the environment going forward, it has a libvirt provisioner if you dislike the virtualbox.
It has a much larger footprint (than Tiny Core Linux), but you would get a supported distribution (currently Ubuntu) and tools...

https://boot2podman.github.io/2020/07/22/machine-replacement.html

https://github.com/vagrant-libvirt/vagrant-libvirt

Going forward, we will integrate the old libmachine and drivers codebase further into minikube itself and drop the dependency.
There will be no further releases from either project, so Docker 19.03 and Podman 1.9.3 are the last releases available (ever).

afbjorklund · 2020-10-13T09:47:01Z

If I don't need to docker-machine create, how do I run docker commands that use docker from the VM(s) created with minikube? For example, the docker-machine env equivalent?

minikube docker-env

Going forward, we will just use SSH to connect to the VM... Like export DOCKER_HOST="ssh://docker@$(minikube ip):22"

It's a little more complicated than that, since you would also have to add the ssh host keys and the ssh user keys. But only once.

See #9229

afbjorklund · 2020-11-04T17:06:23Z

This actually sounds more of a question for docker-machine and docker-compose, than for minikube ?

The primary goal is to provide a kubernetes environment, not an environment to run other legacy tools.

ilya-zuyev · 2020-11-25T19:42:39Z

@darkn3rd staring version v1.15.2 minikube supports docker driver with docker running in docker-machine. Would you mind to check if this version helps with your issue?

medyagh · 2020-12-09T19:31:24Z

@darkn3rd I close this issue, please feel free to reopen if the answers @afbjorklund didnt sovle the issue

afbjorklund · 2020-12-09T20:19:43Z

As detailed above, the "kvm2" driver is only for use with minikube. It will not work with the regular docker-machine...

If you need the other one, for docker-machine, you can find it here: https://github.com/afbjorklund/docker-machine-kvm

k8s-ci-robot added triage/needs-information Indicates an issue needs more information in order to work on it. kind/support Categorizes issue or PR as a support question. labels Oct 13, 2020

medyagh closed this as completed Dec 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvm2 driver vs. docker-machine #9453

kvm2 driver vs. docker-machine #9453

darkn3rd commented Oct 12, 2020 •

edited

Loading

medyagh commented Oct 13, 2020

medyagh commented Oct 13, 2020

darkn3rd commented Oct 13, 2020

afbjorklund commented Oct 13, 2020 •

edited

Loading

afbjorklund commented Oct 13, 2020 •

edited

Loading

afbjorklund commented Nov 4, 2020

ilya-zuyev commented Nov 25, 2020

medyagh commented Dec 9, 2020

afbjorklund commented Dec 9, 2020

kvm2 driver vs. docker-machine #9453

kvm2 driver vs. docker-machine #9453

Comments

darkn3rd commented Oct 12, 2020 • edited Loading

medyagh commented Oct 13, 2020

medyagh commented Oct 13, 2020

darkn3rd commented Oct 13, 2020

afbjorklund commented Oct 13, 2020 • edited Loading

afbjorklund commented Oct 13, 2020 • edited Loading

afbjorklund commented Nov 4, 2020

ilya-zuyev commented Nov 25, 2020

medyagh commented Dec 9, 2020

afbjorklund commented Dec 9, 2020

darkn3rd commented Oct 12, 2020 •

edited

Loading

afbjorklund commented Oct 13, 2020 •

edited

Loading

afbjorklund commented Oct 13, 2020 •

edited

Loading