OpenBSD takes much longer time to boot than FreeBSD. Why?

[Xottab-DUTY]

That's it.
I noticed that OpenBSD boots longer. Is that something related to OpenBSD itself and there's nothing to do about it.. Or is it something we can improve?

[manxorist]

What boot times are you seeing? I am seeing ~30s for FreeBSD 14.0 and ~45s for OpenBSD 7.5, which looks ok to me. Both are on amd64/Ubuntu-22.04/QEMU.

[Xottab-DUTY]

Even in your timings I can see a bit of a slowdown, but 15s is ok. My results are significantly worse.
I'm using amd64/Ubuntu-22.04/QEMU too.

My whole setup step (boot + install packages):

• FreeBSD: 53s
  • 34s to boot
  • 19s to install packages
• OpenBSD: 2m 33s
  • 1m 14s to boot
  • 1m 19s to install packages

The versions are FreeBSD 14.0 and OpenBSD 7.4. I'll try the newer version of OpenBSD.

Actions logs (don't forget to enable timestamps)
FreeBSD log: https://github.com/OpenXRay/xray-16/actions/runs/8614276648/job/23607363938
OpenBSD log: https://github.com/OpenXRay/xray-16/actions/runs/8614276648/job/23607365457

[Xottab-DUTY]

Nope, OpenBSD 7.5 didn't improved the situation. The setup still takes from 2 to 3 minutes.

I compile C++ code using CPA and notice that compilation in OpenBSD also takes much longer time than FreeBSD (only for Debug builds, Release builds are even).
Maybe it's something to do with OpenBSD filesystem, etc.? (I saw in OpenBSD chat in Telegram some messages complaining about FS)

[manxorist]

I guess that's still inside normal variance, depending on how busy the host runners are. My OpenBSD 7.4 timings are about the same as OpenBSD 7.5.

As for installing packages, that is expected I would say. OpenBSD filesystem access is way slower than any other modern OS in my experience. Enabling softdep helped a bit in the past, but that is no longer available since OpenBSD 7.4 (see https://www.openbsd.org/74.html Make the softdep mount(8) option a no-op. Softdep was a significant impediment to improving the vfs layer.).

[Xottab-DUTY]

That's a pity then! Well, I guess we can't do much in this case...

[jacob-carlborg]

In my experience (by running VMs locally) OpenBSD is a bit slower to boot. But as @manxorist mentioned, it should be within normal variance. For the latest release of this action the fastest FreeBSD job took 29 seconds (full job, doesn't include much more than booting). The fastest OpenBSD job took 34 seconds. There are definitely cases where FreeBSD takes longer than that. IIRC, OpenBSD is doing some extra work, like relinking the kernel, or something similar, for every boot.

That's a pity then! Well, I guess we can't do much in this case...

When it comes to solutions, I think there are a lot to do or at least try and experiment with. It's just a matter how much work and time one wants to spend on the issue. Firecracker can boot a Linux VM in 125 ms, so there's no technical reason why booting OpenBSD, or FreeBSD for that matter, can't be made faster. Here are a few things:

• Remove things from the host operating system so it boots quicker, like removing unneeded drivers. *BSD are general purpose operating systems that are intended to run on a variety of hardware. But if you run in a very controlled environment like in a VM with this action, it opens up for removing support for a lot of unnecessary hardware.

• If booting is slow, skip booting 😄. This is what Firecracker is doing, it just loads the kernel without a BIOS or booting. QEMU supports this as well. FreeBSD does not support this, I don't know about OpenBSD.

• Try another hypervisor, like Firecracker. This is now possible since the Linux runners support hardware accelerated nested virtualization, which is required by Firecracker. Unfortunately *BSD won't run inside Firecracker out of the box and it seems the Firecracker team is not interested in supporting anything outside of their use case (Linux VMs for AWS Lambda). That means modifying the host operating system until it works inside Firecracker. I read a blog post about someone trying to get FreeBSD running inside Firecracker. In the process, they found other things to improve during the booting of FreeBSD that was not directly related to Firecracker.

• QEMU has a microVM machine type inspired by Firecracker. There's a blog post for NetBSD

• It's not only booting the VM that is an overhead. There are other things, like downloading the VM image. These are currently up to 500 MB in size. If those could be slimed down to be more like the same sized of a container that would be an improvement

Unfortunately I'm not an expert on these things. I had not used FreeBSD or OpenBSD before staring this project. It seems like a fun project to get *BSD running inside Firecracker, but I'm doing all this work on my very limited spare time. Adding another project that will take up a lot of time might not be the best idea at this point. If someone wants to help an experiment with these things that would be great.

[manxorist]

Maybe using async mount option for the OpenBSD's filesystem in /etc/fstab might help. Although I think any sync I/O overhead should already be mitigated by QEMU unsafe. It still may be worth a try. I would not expect it to matter much for boot time (if at all), but installing packages might benefit.

See https://man.openbsd.org/mount

async
Metadata I/O to the file system should be done asynchronously. By default, only regular data is read/written asynchronously.
This is a dangerous flag to set since it does not guarantee to keep a consistent file system structure on the disk. You should not use this flag unless you are prepared to recreate the file system should your system crash. The most common use of this flag is to speed up restore(8) where it can give a factor of two speed increase.

async has very similar implications as unsafe has (see #67 (comment))