Add support for Alpine guest OS

[jandubois]

Requires Alpine image with cloud-init, but configures lima-guestagent service with openrc and pretends /bin/ash can stand in for /bin/bash.

Does not work with containerd; not sure where to put the OpenRC scripts for it (maybe nerdctl?).

Persistent disk when booting from ISO image

When booting from a LiveCD image there is no point in generating a diffDisk because the OS is running from RAM, not from the disk.

In that case just add diffDisk as an independent data volume and move directories that should be persisted on there. Currently: /home, /usr/local, /etc/containerd, and /var/lib/containerd.

This data-volume logic is currently only enabled for Alpine because it hasn't been tested anywhere else yet.

Additional, if the configured disk size is 0, no diffDisk will be allocated and changes will be written back to the baseDisk (or discarded, if the baseDisk is an ISO image).

This is still WIP

Things seem to be working fine, except limactl stop so far doesn't manage to shut down Alpine gracefully.

[AkihiroSuda]

Thanks! But why not use openSUSE? 😛

[AkihiroSuda]

apk update is probably not needed

[jandubois]

It is needed, at least in my image, which only includes a static apkovl in the repositories by default.

It is still 🚧, but the script I use to build (various different) ISOs is at https://github.com/rancher-sandbox/alpine-lima

[jandubois]

When I tested it again right now, it seemed like apk update is indeed not needed. Not sure which image I was testing when I did need it. I can remove it, if you prefer, or leave it in. It doesn't really hurt, as apk add sshfs will still update the repos anyways, so I would prefer to leave it as-is.

[AkihiroSuda]

On systemd host, we have lima-guestagent install-systemd, so maybe we should have lima-guestagent install-openrc?

[jandubois]

I actually started that way, and may even still have the branch here locally, but then realized that including the file in user-data is much easier.

Added complexity is that the OpenRC script must be installed by root, and lima-guestagent doesn't allow running as root, so that logic also became more complicated.

Is there a reason not to generate the systemd config via template in user-data (or as a separate file) as well?

[AkihiroSuda]

I think we can modify the guest agent to allow running as the root.

Is there a reason not to generate the systemd config via template in user-data (or as a separate file) as well?

No specific reason

[jandubois]

Is there a reason not to generate the systemd config via template in user-data (or as a separate file) as well?

No specific reason

In that case I would want to simplify the systemd config in a separate commit, once the user-data refactoring is done.

[AkihiroSuda]

.Containerd.System and .Containerd.User should be ignored with a warning message on non-systemd hosts, until we can have openrc scripts

[jandubois]

Yes, but the "warning message" will only be in /var/log/cloud-init-output.log on the guest. We don't know if the OS is using systemd or openrc at the time the template is rendered, and I intentionally didn't want to add another config setting for it.

I guess I could add a "requirement" for it though, to check once the VM is up and running.

[jandubois]

I've tried to implement this in d981c4f but am not really happy with it:

1. The fatal field seems like a kludge, but there is no point in retrying the check for systemd; it isn't going to magically appear eventually.
2. The failure output is really ugly/hard to read. I suspect all requirements failures look like this now though, so maybe not something to address in this PR?

INFO[0035] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available"
INFO[0035] [hostagent] Forwarding "/run/user/501/lima-guestagent.sock" (guest) to "/Users/jan/.lima/lima/ga.sock" (host)
INFO[0035] [hostagent] No further optional requirements will be checked
ERRO[0035] [1 error occurred:
	* failed to satisfy the optional requirement 1 of 2 "systemd must be available": systemd is required to run containerd, but does not seem to be available.
Make sure that you use an image that supports systemd. If you do not want to run
containerd, please make sure that both 'container.system' and 'containerd.user'
are set to 'false' in the config file.
; skipping further checks: stdout="", stderr="+ command -v systemctl\n+ echo 'systemd is not available on this OS'\nsystemd is not available on this OS\n+ exit 1\n": failed to execute script "systemd must be available": stdout="", stderr="+ command -v systemctl\n+ echo 'systemd is not available on this OS'\nsystemd is not available on this OS\n+ exit 1\n": exit status 1

]
WARN[0035] DEGRADED. The VM seems running, but file sharing and port forwarding may not work. (hint: see "/Users/jan/.lima/lima/ha.stderr.log")
FATA[0035] degraded, status={Running:true Degraded:true Exiting:false Errors:[1 error occurred:
	* failed to satisfy the optional requirement 1 of 2 "systemd must be available": systemd is required to run containerd, but does not seem to be available.
Make sure that you use an image that supports systemd. If you do not want to run
containerd, please make sure that both 'container.system' and 'containerd.user'
are set to 'false' in the config file.
; skipping further checks: stdout="", stderr="+ command -v systemctl\n+ echo 'systemd is not available on this OS'\nsystemd is not available on this OS\n+ exit 1\n": failed to execute script "systemd must be available": stdout="", stderr="+ command -v systemctl\n+ echo 'systemd is not available on this OS'\nsystemd is not available on this OS\n+ exit 1\n": exit status 1

] SSHLocalPort:60022}

Suggestions welcome!

[AkihiroSuda]

Looks good for now.

I agree that we should improve UI in future.

[jandubois]

Thanks! But why not use openSUSE? 😛

I'm looking at replacing the macOS VM management in Rancher Desktop with lima (the first alpha used a hacked up minikube build with a k3s bootstrapper, the current version uses docker-machine-driver-hyperkit with a custom control program).

For that I want the smallest possible image that can run k3s, which means little more than a kernel with support for /dev, /proc, and /sys, an init process, and /bin/sh. On Windows we rely on WSL, so our distro is just

$ ls -lh distro.tar
-rw-r--r--  1 jan  staff   3.3M 10 Mar 17:29 distro.tar
$ tar tvfz distro.tar | grep -v ^[ld]
-rwxr-xr-x  0 root   root   303088 17 Feb 17:23 bin/find
-rwxr-xr-x  0 root   root   812632 17 Feb 17:23 bin/busybox
-rwxr-xr-x  0 root   root   610884 10 Mar 17:29 bin/screen
-rwxr-xr-x  0 root   root  1376352 17 Feb 17:23 bin/coreutils
-rw-r--r--  0 root   root       30 10 Mar 17:29 etc/passwd
-rw-r--r--  0 root   root   214057 22 Feb 11:51 etc/ssl/certs/ca-certificates.crt
-rw-r--r--  0 root   root     3810 10 Mar 17:29 usr/share/terminfo/x/xterm-256color
-rwxr-xr-x  0 root   root      428 10 Mar 17:29 usr/local/bin/kubeconfig
-rwxr-xr-x  0 root   root     1437 10 Mar 17:29 usr/local/bin/run-k3s

From that bin/find, bin/screen, and the terminfo entry aren't even used anymore.

The reason to slim down the distro is so that we are not affected by CVEs in software that we don't even run. Having a 500MB Linux distro means we have to update it frequently, the updates will be huge, and the users not happy...

Obviously we'll need more than 3MB on macOS, but I think 40-50MB instead of 550MB is a reasonable target. The "virt" image for Alpine is 41MB, and grows to 61MB when you add cloud-init. This obviously is also ridiculous, and should be replaced by a small bash script instead. 😄

We don't want to bother with having the OS update itself; we just want to replace the OS and keep the kube objects and container images on a separate volume. This means using an ISO image is ideal, as it is also much smaller than the corresponding QCOW2 image.

There is another project/experiment building OS images from containers: rancher-sandbox/cOS-toolkit. I will challenge them to produce an openSUSE based ISO that is at most 50% larger than my Alpine image. :) rancher/elemental-toolkit#26

Anyways, this is probably much more than you wanted to know. 😄

[jandubois]

limactl stop so far doesn't manage to shut down Alpine gracefully.

Turns out acpid was not running because the image was based on the virt flavour.

[jandubois]

@AkihiroSuda I've switched the PR to non-draft now because I think it works as-is.

I will need to spend some more effort on the alpine-lima repo to document how to create images compatible with lima etc. Not sure if you want to wait for that, or review/merge now and then update later with e.g. an example/alpine.yaml?

[AkihiroSuda]

Thanks. The doc can be added later, but could you add the example YAML so that I can test it?

Would be great if we can test it on CI, too. https://github.com/AkihiroSuda/lima/blob/36b2ab60cd8ebf18bd232833c78e3603aae953c4/.github/workflows/test.yml#L55

[AkihiroSuda]

Thanks!