Support systemd-sysext OEM update payloads in update-engine

[pothos]

Current situation

The OEM partition doesn't get updated and contains a base Ignition config and config files that get copied over to the rootfs or binaries/tarballs that get used directly from the OEM partition.

Impact

The OEM tools get outdated and may even be break.

Ideal future situation

Change update-engine to recognize the additional files (systemd-sysext images) in the Omaha response and pick the one for the OEM ID. It gets verified through an attached signature of the update key (gpg allows piping the file but we need the key in the right format). For an A/B update with a way to roll back to the old partition, both the current active and the new image should be kept while the inactive image should be deleted. Since the OEM partition may be too small to hold both sysext images, the root filesystem can act as secondary storage. However, it should not hold the current active image because it may be reformatted, or files accidentally deleted. The update reboot is likely not reformatting the rootfs and the previously downloaded file is expected to be still there, and on the reboot into the new /usr partition the swap of old and new sysext image can happen so that the now active sysext gets placed on the OEM partition before being enabled through a symlink. If the OEM partition has enough free space, we may skip using the root filesystem which makes the A/B updates/rollbacks safer against lost files on the root filesystem. If the OEM partition is too small, the rootfs will be used as fallback. The files are named by the OEM ID and Flatcar version.

Since old instances without the update-engine support for the OEM files will not download them, we use this opportunity to define a migration path where the old OEM partition content is kept and the sysext gets optionally downloaded in the postinstall handler but stays inactive when the old Flatcar OEM systemd units on /etc are detected until the current running /usr partition has sysext OEM support and the newly installed /usr partition, too (we can check the Flatcar version of both partition and check the presence of sysext images for both). After the flag file for migration is created, the bootengine logic will delete the known old Flatcar OEM partition contents and the old Flatcar OEM systemd units on /etc to make space for the sysext image and allow its services to take over, i.e., we need to load the inactive sysext at this point. This delayed migration is needed to make sure we only delete the old OEM contents and the systemd units on /etc when they won't be used again in a rollback to the other /usr partition.

NOTE The used file name above is not final. We will probably have something like oem-OEMID.gz for the OEM images and flatcar-EXTENSION.gz for the optional Flatcar extensions. Update-engine should fetch max 1 OEM image as specified in the ID= field of /oem/oem-release and as many extensions as specified in /etc/flatcar/enabled-sysext.conf. The update payloads are signed protobufs and will be extracted to disk with a .raw suffix as /oem/sysext/oem-${OEMID}-${VERSION}.raw (or as fallback /etc/flatcar/oem-sysext/oem-${OEMID}-${VERSION}.raw if the OEM partition is too small) for the OEM or /etc/flatcar/sysext/flatcar-${NAME}-${VERSION}.raw for the optional Flatcar extensions. If both old and new partition support sysexts for this OEM, /oem/sysext/active-oem-OEMID is created to use sysext and migrate the instance.

Instead of a oem-sysext.service we have bootlogic in bootengine under 99setup-root/initrd-setup-root-after-ignitio.

Implementation options

• The postinstall action can be done with a special binary: Integration of ue-rs with update-engine post-install script #1028

Additional information

Nebraska already supports the extra files but flatcar-update not. It makes sense to add support to flatcar-update for easier development.
Until this is done one needs to add the extra files to the Nebraska package and edit the version to be higher than the Flatcar version. I'm using these commands to run a local instance:

docker run --rm -d --name nebraska-postgres-dev --tmpfs /var/lib/postgresql/data:rw,noexec,nosuid,size=5G --network host -e POSTGRES_PASSWORD=nebraska docker.io/postgres && (sleep 10 && psql postgres://postgres:nebraska@localhost:5432/postgres -c 'create database nebraska;' && psql postgres://postgres:nebraska@localhost:5432/nebraska -c 'set timezone = "utc";' )
ulimit -n 65000
LOGXI=* bin/nebraska -auth-mode noop -http-static-dir "$PWD"/frontend/build -http-log "$@"

[pothos]

The migration logic to remove the old OEM files is now done in the bootengine repo under dracut/99setup-root/initrd-setup-root-after-ignition. The fallback download logic should also be done there - we could still have it in the post-inst hook as well already in case the initrd has no networking.

[pothos]

I've created an own issue for the signing: #1032

[pothos]

The post-inst action should also write the list of old OEM files to delete to /oem/sysext/migrate-oem-${OEMID when both the current and new partition have a matching OEM sysext file. On the next boot this will be cleaned up from the initrd when the OEM sysext gets activated.

[pothos]

PR in flatcar/update_engine#24