[RFE] Utilize systemd's systext for image composability / extensability

[blixtra]

Issue #443 is about deprecating Torcx in favor of systemd's sysext. But it focuses on the limited use case that Torcx was for, installing multiple versions of an application, usually a container runtime. But I think systemd sysext could be leveraged in a much broader and useful way now that systemd 249 is rolling though the channels.

Flatcar has always been a bare minimum Linux distro for containers. In essence all we need is kernel + systemd + update agent (of course, a small set of support tools). But we we see many usecases where it is viable to create an image rather than use containers for everything. I’d like to create a general mechanism that could be used to create customized image rather than just adding single application versions.

An issues with the torcx method is that it's not something that gets auto-updated. So that if I maintain my own Nebraska, I can have a pipeline where I consider an update to be the core Flatcar image + whatever sysext overlay(s) I have. Alternatively, it would be cool to have a tool where I could take my overlay(s) and create a merged image from that. That would simply retain our current update mechanism.

Ideally, as a project, we could maintain only a core image + a container overlay, and maybe a kubernetes overlay. Others could create their own overlays which the Flatcar team does not have to support or worry about but greatly expands the applicability of the Flatcar project and brings its features of auto-updates and immutability to many more areas.

But the key is that it would retain all those features: immutable, auto-updatable, minimal. In this way we're not just making another package manager. The focus should be on create images and having those be able to update.

Roadmap

• Deprecate Torcx in favour of e.g. systemd-sysext #443 (Preshipped Docker as Docker+containerd sysext, docs for providing own sysext images)
• Extend usage of sysext for alternative container runtimes (preshipped on /usr): Support arbitrary container runtimes via provisioning-time images #447
• Use sysext for OEM tools and update them through update-engine (offered as additional payloads in the Nebraska Flatcar channel): Enable updating of OEM specific tools #60
• Support optional sysext images, e.g., for Kubernetes binaries, either auto-updated similar to the OEM tools (offered in the Nebraska Flatcar channel), or updated independently through systemd-sysupdate (possibly with Nebraska if Omaha support is added)

[pothos]

This is also interesting for the OEM tools because they coupled with the Flatcar release version and tested together (currently they are not updated which causes problems, see #60). Having the OEM tools packaged as systemd-sysext image helps to be able to update them in a reliable way.
I like the proposal use extend Nebraska to be able to update a sysext image independently from or maybe even coupled to the Flatcar version. However, this does not enable rollbacks which I see as essential. Therefore, to cover automatic rollback, I would suggest to explore if the compressed OEM tools sysext images could even be stored in the /usr partition and loaded from there. This gives the rollback support for free and does not need any changes in Nebraska nor additional awareness when maintaining an own Nebraska server. The Challenge: The size limitation of the /usr partition has to be kept in mind but when we enable filesystem compression on the /usr partition we get more free space than now, and with sysext we can also remove some library duplications which currently increase the OEM tools size.

In addition, using sysext also works for the initramfs when passed by the boot loader. This would allow to inject custom OEM behavior into the early boot stage to load non-default kernel modules or similar.

[vbatts]

one thing to work out about systemd-sysext, is:

• how to make a workflow for having extensions available that are not enabled to be loaded
• figure out how racy the merge/unmerge/refresh are. Say we land a new/updated extension, and run refresh, then it will momentarily unmount all extensions, and then re-merge and mount them. Possibly order dependent? seems racy.

[joaquimrocha]

one thing to work out about systemd-sysext, is:

• how to make a workflow for having extensions available that are not enabled to be loaded

There's no mechanism to mark extensions as enabled or not. Either they are present and compatible with the system's version or not. So the recommended way to handle this seems to be (as I understood from our chat with Lennart) by directly adding/removing symlinks (in the expected extension locations) to the actual images.

• figure out how racy the merge/unmerge/refresh are. Say we land a new/updated extension, and run refresh, then it will momentarily unmount all extensions, and then re-merge and mount them. Possibly order dependent? seems racy.

We could make it so the recommended way to apply images will require a reboot. If an admin calls refresh/merge directly, that would be considered an approach for development.

[t-lo]

We could make it so the recommended way to apply images will require a reboot.
If an admin calls refresh/merge directly, that would be considered an approach for development.

I strongly second this. Even more, we could consider making sysexts a provision time feature.
Otherwise we risk just re-inventing package management.

[pothos]

Regarding the raciness of refresh, the new mount beneath feature in the kernel is planned to be used for replacing the sysexts in one go instead of having a window where the overlay is missing.

[till]

From a user perspective, updating components via sysext and not having to reboot nodes is a great thing.