Hypocaust-2 is an experimental type-1 hypervisor with H extension run on RISC-V machine. It depends on the RISC -V H extension, which currently runs on QEMU 7.1.0 or above. It is the successor of the hypocaust project.
My plan is to build a high-performance riscv64 hypervisor that physically maps the cpu cores, so there is no need to schedule guests in the hypervisor. In addition, the passthrough method for IO devices has achieved good performance.
The purpose of this project is to run on bare metal or embedded devices, but it is not ruled out that kvm technology will be used and run on linux in the future.
- QEMU 7.1.0
- RustSBI-QEMU Prereleased 2023-02-01
- Rust 1.66.0
./srcipts/rCore-Tutorial-v3.sh && make qemu PLATFORM=rCore-Tutorial-v3
./srcipts/rt-thread.sh && make qemu PLATFORM=rt-thread
Toolchains:
$ sudo apt install autoconf automake autotools-dev curl libmpc-dev libmpfr-dev libgmp-dev \
gawk build-essential bison flex texinfo gperf libtool patchutils bc \
zlib1g-dev libexpat-dev git \
libglib2.0-dev libfdt-dev libpixman-1-dev \
libncurses5-dev libncursesw5-dev
# install riscv linux toolchain
$ git clone https://gitee.com/mirrors/riscv-gnu-toolchain --depth=1
$ cd riscv-gnu-toolchain
$ git rm qemu
$ git submodule update --init --recursive
$ ./configure --prefix=/opt/riscv64 --with-arch=rv64imac --with-abi=lp64
$ sudo make linux -j8
$ export PATH=$PATH:/opt/riscv64/bin
Build Linux:
$ git clone https://github.com/torvalds/linux -b v6.2
$ cd linux
$ git am ${HYPOCAUST_2_DIR}/guest/linux/patches/*.patch
$ make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- defconfig
$ make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- menuconfig
$ make ARCH=riscv CROSS_COMPILE=riscv64-unknown-linux-gnu- all -j8
Run bare Linux on qemu:
$ qemu-system-riscv64 -M virt -m 256M -nographic -bios $(BOOTLOADER)/rustsbi-qemu.bin -kernel $(linux)/arch/riscv/boot/Image
Docker Command（MacOS/Windows）:
# run docker container, mount workspace in docker
docker run -itd --name riscv-env --privileged -v {WORKSPACE}:/workspace riscv-gnu-toolchain /bin/bash
# run docker
docker exec -it riscv-env /bin/bash
Make rootfs:
git clone https://gitee.com/mirrors/busyboxsource.git
cd busyboxsource
# Select: Settings -> Build Options -> Build static binary
CROSS_COMPILE=riscv64-unknown-linux-gnu- make menuconfig
## Build && Install
CROSS_COMPILE=riscv64-unknown-linux-gnu- make -j10
CROSS_COMPILE=riscv64-unknown-linux-gnu- make install
# Make minimal root file system
cd ../
qemu-img create rootfs.img 1g
mkfs.ext4 rootfs.img
# mount file system && copy busybox
mkdir rootfs
mount -o loop rootfs.img rootfs
cd rootfs
cp -r ../busyboxsource/_install/* .
mkdir proc dev tec etc/init.d
cd etc/init.d/
touch rcS
vim rcS
#####
#!/bin/sh
mount -t proc none /proc
mount -t sysfs none /sys
/sbin/mdev -s
#####
chmod +x rcS
umount rootfs
qemu-system-riscv64 -M virt -m 256M -nographic -bios {BOOTLOADR} -kernel {KERNEL_ELF} -drive file=rootfs.img,format=raw,id=hd0 -device virtio-blk-device,drive=hd0 -append "root=/dev/vda rw console=ttyS0"
- Load guest elf image.
- Jump guest loaded to a VM while enabling guest physical address translation by
hgatp. - Run a tiny kernel that does not require any external hardware like disk devices.
- Handle read/write requests for CSRs from a guest
- Handle SBI calls(currently only
console_putchar,console_getcharandset_timerandbaserelated) - Guest enable paging & setup 2-stage page table translation.
- Jump VU mode and run user applications
- Timers
- Passthrough virtio block device
- Configure hypervisor and guest memory addresses and peripheral space mapping by device tree.
- Emulate PLIC && Forward interrupts
- Expose and/or emulate peripherals
- run rCore-Tutorial-v3
- run RT-Thread
- run Linux
- IOMMU enabled
- AIA support
- multicore supported
- multiguest supported
- VCPU and Host Interrupt Affinity
- Spatial and Temporal Memory Isolation
- Pass-through device support(enable IOMMU)
- Block device virtualization
- Network device virtualization
- Input device virtualization
- Display device virtualization
Two types of device tree(DT):
- Host DT:
- Device tree which describes underlying host HW to hypocaust-2
- Used by hypocaust-2 at boot-time
- Guest DT:
- Device tree which dscribes Guest virtual HW to hypocaust-2
- Used by hypocaust-2 to create Guest
- When the hypervisor is initialized, it is necessary to write the
hcounterenregister to all 1, because it is possible to read thetimeregister in VU mode or VS mode.(refs: The counter-enable registerhcounterenis a 32-bit register that controls the availability of the hardware performance monitoring counters to the guest virtual machine.
When the CY, TM, IR, or HPMn bit in the hcounteren register is clear, attempts to read the cycle, time, instret, or hpmcountern register while V=1 will cause a virtual instruction exception if the same bit in mcounteren is 1. When one of these bits is set, access to the corresponding register is permitted when V=1, unless prevented for some other reason. In VU-mode, a counter is not readable unless the applicable bits are set in bothhcounterenandscounteren.
hcounterenmust be implemented. However, any of the bits may be read-only zero, indicating reads to the corresponding counter will cause an exception when V=1. Hence, they are effectively WARL fields.) - When the hypervisor initializes the memory for the guest, it needs to set all the mapping flags of the guest memory to RWX, although it needs to be modified in the end. Otherwise, when the guest allocates memory for the application, it will not be executable, causing
InstructionGuestPageFault. - The hypervisor currently does not support IOMMU, so it is necessary to have all its memory configured with identify mapping when guest wishes to use a DMA device.