Skip to content
This repository has been archived by the owner on May 6, 2020. It is now read-only.

Start qemu failed in CC 3.0.0 (works fine in CC 2.2.4) #637

Open
jovizhangwei opened this issue Sep 27, 2017 · 11 comments
Open

Start qemu failed in CC 3.0.0 (works fine in CC 2.2.4) #637

jovizhangwei opened this issue Sep 27, 2017 · 11 comments

Comments

@jovizhangwei
Copy link

Below error logs indicated there have some issue in qemu command line. Interesting my machine can ran CC 2.2.4 successfully.

Is there anyone can give me some tips? Thanks.

$sudo cat /var/lib/clear-containers/runtime/runtime.log
2017-09-27 15:34:39.388233786 +0800 CST:22918:cc-runtime:info:cc-runtime (version 3.0.0, commit 7fc11d5) called as: [create --bundle /var/run/docker/libcontainerd/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e --console-socket /tmp/pty454214293/pty.sock --pid-file /run/docker/libcontainerd/containerd/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/init/pid 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e]
2017-09-27 15:34:39.414433824 +0800 CST:22918:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-27 15:34:40.933751823 +0800 CST:22918:cc-runtime:info:No sockets from configuration
2017-09-27 15:34:40.956700885 +0800 CST:22918:cc-runtime:info:Device details for container 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e: Major:253, Minor:1, MountPoint:/var/lib/docker/devicemapper/mnt/7d7c67d334d72e9bf34b2792c68e692cb966afca78c88a0ec99dea2a4794d6b5
2017-09-27 15:34:40.982031478 +0800 CST:22918:cc-runtime:info:Block Device path /dev/mapper/docker-8:2-2402074-7d7c67d334d72e9bf34b2792c68e692cb966afca78c88a0ec99dea2a4794d6b5 detected for container with fstype : 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e
%!(EXTRA string=xfs)
2017-09-27 15:34:41.310825787 +0800 CST:22918:cc-runtime:info:launching qemu with: [-name pod-7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e -uuid 37643065-6562-3030-3234-636330323236 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=97517M -smp 24,cores=24,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/devicemapper/mnt/7d7c67d334d72e9bf34b2792c68e692cb966afca78c88a0ec99dea2a4794d6b5/rootfs,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e,security_model=none -device virtio-blk,drive=drive-7d0eeb0024cc02261c1592260,scsi=off,config-wce=off -drive id=drive-7d0eeb0024cc02261c1592260,file=/dev/mapper/docker-8:2-2402074-7d7c67d334d72e9bf34b2792c68e692cb966afca78c88a0ec99dea2a4794d6b5,aio=threads,format=raw,if=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e::off::]
2017-09-27 15:34:41.350440609 +0800 CST:22918:cc-runtime:error:Unable to launch qemu: exit status 1
2017-09-27 15:34:41.374005133 +0800 CST:22918:cc-runtime:error:qemu-system-x86_64: -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on: tap: open vhost char device failed: No such file or directory

2017-09-27 15:34:42.310571277 +0800 CST:22918:cc-runtime:error:Did not receive the pod started notification
2017-09-27 15:34:42.357430479 +0800 CST:23032:cc-runtime:info:cc-runtime (version 3.0.0, commit 7fc11d5) called as: [delete 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e]
2017-09-27 15:34:42.374332063 +0800 CST:23032:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-27 15:34:43.911450546 +0800 CST:23032:cc-runtime:info:Shim(s) stopped
2017-09-27 15:34:43.943192904 +0800 CST:23032:cc-runtime:error:dial unix /var/run/clear-containers/proxy.sock: connect: connection refused
2017-09-27 15:34:44.010752112 +0800 CST:23487:cc-runtime:info:cc-runtime (version 3.0.0, commit 7fc11d5) called as: [delete 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e]
2017-09-27 15:34:44.044182951 +0800 CST:23487:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-27 15:34:45.612318276 +0800 CST:23487:cc-runtime:error:Can not move from stopped to stopped
@jovizhangwei jovizhangwei mentioned this issue Sep 27, 2017
Open
@jodh-intel
Copy link
Contributor

Is this the same system that cc-check fails on (your #634 issue)?

Note that that log shows you are using /etc/clear-containers/configuration.toml. Have you seen:

Also, it would be extremely useful if you could fill out the template you should have seen. Specifically, if you can run the cc-collect-data.sh script, that would be helpful. I notice that you are using the latest OBS release. That script isn't available in those package versions, but you can either copy the built file and run on your system (or just paste the output of sudo cc-runtime cc-env to this issue).

@jovizhangwei
Copy link
Author

@jodh-intel it is the same machine with the cc-check failed.

I've changed the configuration file to /usr/share/defaults/clear-containers/configuration.toml

$sudo cc-runtime cc-env
[Meta]
  Version = "1.0.3"

[Runtime]
  [Runtime.Version]
    Semver = "3.0.0"
    Commit = "7fc11d5"
    OCI = "1.0.0-rc5"
  [Runtime.Config]
    GlobalLogPath = "/var/lib/clear-containers/runtime/runtime.log"
    [Runtime.Config.Location]
      Path = "/usr/share/defaults/clear-containers/configuration.toml"
      Resolved = "/usr/share/defaults/clear-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
  [Hypervisor.Location]
    Path = "/usr/bin/qemu-system-x86_64"
    Resolved = "/usr/bin/qemu-system-x86_64"

[Image]
  Path = "/usr/share/clear-containers/clear-containers.img"
  Resolved = "/usr/share/clear-containers/clear-17270-containers.img"

[Kernel]
  Parameters = ""
  [Kernel.Location]
    Path = "/usr/share/clear-containers/vmlinuz.container"
    Resolved = "/usr/share/clear-containers/vmlinuz-4.9.47-77.2.container"

[Proxy]
  Type = "ccProxy"
  Version = "Version: 3.0.0+git.c66f21e"
  URL = "unix:///var/run/clear-containers/proxy.sock"

[Shim]
  Type = "ccShim"
  Version = "shim version: 3.0.0-alpha.4 (commit: 739e5e2)"
  [Shim.Location]
    Path = "/usr/libexec/clear-containers/cc-shim"
    Resolved = "/usr/libexec/clear-containers/cc-shim"

[Agent]
  Type = "hyperstart"
  Version = "<<unknown>>"
  [Agent.PauseBin]
    Path = "/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"
    Resolved = "/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"

[Host]
  Kernel = "3.10.0-327.ali2011.alios7.x86_64"
  CCCapable = false
  [Host.Distro]
    Name = "Alibaba Group Enterprise Linux Server"
    Version = "7.2"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz"

@Ace-Tang
Copy link
Contributor

@jovizhangwei i got the same error
here is my configuration.toml file

#cat /etc/clear-containers/configuration.toml
# XXX: Warning: this file is auto-generated from file "".

[hypervisor.qemu]
path = "/usr/local/bin/qemu-lite-system-x86_64"
kernel = "/usr/local/share/clear-containers/vmlinuz.container"
image = "/usr/local/share/clear-containers/clear-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc
kernel_params = ""

# Default number of vCPUs per POD/VM:
# unspecified or 0 --> will be set to 1
# < 0              --> will be set to the actual number of physical cores
# > 0 <= 255       --> will be set to the specified number
# > 255            --> will be set to 255
default_vcpus = -1

# Default memory size in MiB for POD/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
disable_block_device_use = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# Debug changes the default hypervisor and kernel parameters to
# enable debug output where available.
# Default false
# these logs can be obtained in the cc-proxy logs  when the 
# proxy is set to run in debug mode
# /usr/libexec/clear-containers/cc-proxy -log debug
# or by stopping the cc-proxy service and running the cc-proxy 
# explicitly using the same command line
# 
enable_debug = true

[proxy.cc]
url = "unix:///var/run/clear-containers/proxy.sock"

[shim.cc]
path = "/usr/local/libexec/clear-containers/cc-shim"

[agent.hyperstart]
pause_root_path = "/usr/local/var/lib/clear-containers/runtime/bundles/pause_bundle"

## Uncomment to enable the global logging to the default path.
#[runtime]
global_log_path = "/usr/local/var/lib/clear-containers/runtime/runtime.log"

cc-runtime cc-check output

#cc-runtime cc-check
INFO[0000] Found CPU attribute "Intel Architecture CPU" (GenuineIntel) 
INFO[0000] Found CPU flag "Virtualization support" (vmx) 
INFO[0000] Found CPU flag "64Bit CPU" (lm)              
INFO[0000] Found CPU flag "SSE4.1" (sse4_1)             
INFO[0000] Found kernel module "Kernel-based Virtual Machine" (kvm) 
INFO[0000] Found kernel module "Intel KVM" (kvm_intel)  
ERRO[0000] ERROR: kernel module "Intel KVM" parameter "nested" has value "N" (expected "Y") 
ERROR: kernel module "Intel KVM" parameter "nested" has value "N" (expected "Y")

here is my cc-runtime cc-env output

#cc-runtime --cc-config=/etc/clear-containers/configuration.toml cc-env
[Meta]
  Version = "1.0.4"

[Runtime]
  [Runtime.Version]
    Semver = "3.0.0-beta.2"
    Commit = "d9897e1fa8579d456799c11d3a82e0acc6e7f1ec"
    OCI = "1.0.0-rc5"
  [Runtime.Config]
    Path = "/etc/clear-containers/configuration.toml"
    GlobalLogPath = ""

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
  Path = "/opt/vlinux/2.2.4/qemu-lite/2.7.0-e84eba1324/bin/qemu-system-x86_64"

[Image]
  Path = "/usr/local/share/clear-containers/clear-containers.img"

[Kernel]
  Path = "/usr/local/share/clear-containers/vmlinuz.container"
  Parameters = ""

[Proxy]
  Type = "ccProxy"
  Version = "<<unknown>>"
  URL = "unix:///var/run/clear-containers/proxy.sock"

[Shim]
  Type = "ccShim"
  Version = "shim version: 3.0.0-alpha.4 (commit: 4f2b7daf521934bdefc2748be235cfb7bc265736)"
  Path = "/usr/local/libexec/clear-containers/cc-shim"

[Agent]
  Type = "hyperstart"
  Version = "<<unknown>>"
  PauseBinPath = "/usr/local/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"

[Host]
  Kernel = "3.10.0-327.ali2012_cgroup_pids_test_713.alios7.x86_64"
  CCCapable = false
  [Host.Distro]
    Name = "Alibaba Group Enterprise Linux Server"
    Version = "7.2"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz"

@jodh-intel
Copy link
Contributor

Hi @jovizhangwei - the problem is likely to be that your kernel doesn't have vhost capabilities. However, due to the way cc-check currently works (and because you have nesting disabled), you don't get to see that error from cc-check. I've raised #638 so we can improve cc-check.

@jodh-intel
Copy link
Contributor

Hi @jovizhangwei - can you try running the following and then seeing if you can start a container?

$ for m in vhost vhost_net; do sudo modprobe $m; done

@jodh-intel
Copy link
Contributor

@Ace-Tang - could you also try the above please?

@jovizhangwei
Copy link
Author

@jodh-intel

$for m in vhost vhost_net; do sudo modprobe $m; done
modprobe: FATAL: Module vhost not found.
modprobe: FATAL: Module vhost_net not found.

Why I can ran CC 2.2.4? without vhost.

@sameo
Copy link

sameo commented Sep 27, 2017

@jovizhangwei With 2.2.4 we were not enabling vhost by default. With 3.0 we assume vhost is available.

I just opened a virtcontainers issue for that: containers/virtcontainers#385 and will get a fix for that in the next few days.

@sameo
Copy link

sameo commented Sep 27, 2017

@jovizhangwei FWIW, vhost-net provides significant networking performance improvements with KVM. Is there any reason why it is not enabled with your kernel ?

@jovizhangwei
Copy link
Author

@sameo After upgraded my kernel, vhost and vhost_net modules was loaded, then I found another error, please take a look at #642

Thanks

@jodh-intel
Copy link
Contributor

Hi @jovizhangwei - Is this problem now resolved for you?

mcastelino pushed a commit to mcastelino/runtime that referenced this issue Dec 6, 2018
@flyflypeng
vendor: update govmm to be compatible with qemu 2.8
f841e89 
govmm has ExecuteBlockdevAdd() function and ExecuteBlockdevDel() function
doesn't compatible with qemu 2.8,because blockdev-add and x-blockdev-del usages
are different between qemu 2.7 and qemu 2.8

shortlog:

ce070d1 govmm: modify govmm to be compatible with qemu 2.8
0286ff9 qemu/qmp: support hotplug a nic whose qdisc is mq
8515ae4 qmp: Remind users that you must first call ExecuteQMPCapabilities()
21504d3 qemu/qmp: Add netdev_add with chardev support
ed34f61 Add some negative test cases for qmp.go
17cacc7 Add negative test cases for qemu.go

fixes: clearcontainers#637

Signed-off-by: flyflypeng <jiangpengfei9@huawei.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sameo @Ace-Tang @jodh-intel @jovizhangwei