Deprecate Torcx in favour of e.g. systemd-sysext

[t-lo]

NOTE: This item is about deprecating torcx the item, not torcx the feature. We will continue to support the feature of installing custom core binaries at provisioning time. We will also provide a seamless migration path to the new implementation of this feature.

Issue

• The torcx project was discontinued upstream and is unmaintained
• Customisation features provided by torcx are desired in Flatcar user community

Impact

• Torcx bugs do not get fixed, security issues remain open
• Torcx regularly interferes with other Flatcar key components

Ideal future state

• A new implementation of the feature provided by torcx (e.g. systemd-sysext) is integrated with Flatcar
  • build process
  • Testing
  • Run-time, i.e. ignition & friends
• Missing items (e.g. download of extensions from custom URLs) are added to the new implementation
• Seamless migration path for existing automation based on torcx is defined and implemented

Tasks

• Define a SYSEXT_LEVEL= for Flatcar, e.g. 1.0 (and work with upstream on making the matching more flexible, plus match for architecture), so that users can provide their own sysext images (with static binaries, not coupled to /usr), and provide user docs (rework https://www.flatcar.org/docs/latest/container-runtimes/use-a-custom-docker-or-containerd-version/ and advise to create the /dev/null symlink to disable the upcoming shipped Docker and conatinerd sysext in advance): [RFE] Define sysext level for Flatcar #643
• Investigate automatic conversion of custom Torcx images to sysext for backwards compat - maybe limited to simple cases, maybe even out of scope
• Announce deprecation of Torcx, with instructions on how to convert Torcx images to sysext. Users should also ensure that they don't use the Torcx paths if they customized their containerd systemd unit. [RFE] Prepare for Torcx deprecation #644
• Remove Torcx but convert the Torcx image building to a sysext image building step (splitting it into Docker and containerd, making services and default configs show up under /usr and making it easy for users to drop-in additional configs under /etc/containerd/runtime_*.toml) and ship them on the image as before (placed under /usr/lib/extensions/ to automatically enable them, users can disable them through a /dev/null symlink on /etc/extensions/ with the same name maybe we can still use a symlink on /usr to have a stable name under /usr/lib/extensions but encode the Docker version in the image file name). Make sure the sysext images are enabled by default and the user can disable them when bringing an own sysext image or binaries on /opt

Additional info:
The file names for the split Docker and containerd sysext have already been agreed on here (docker-flatcar.raw and containerd-flatcar.raw):
https://www.flatcar.org/docs/latest/provisioning/sysext/#supplying-your-sysext-image-from-ignition

Related

• Enable updating of OEM specific tools #60
• [RFE] Utilize systemd's systext for image composability / extensability #548
• Support arbitrary container runtimes via provisioning-time images #447

[pothos]

/usr/lib/extensions/ is not supported anymore as location to prevent confusion. For us this means we need a tmpfile rule that sets up a symlink in /run/extensions/ to point to the docker/containerd sysext image on /usr.

[pothos]

Ah, no, tmpfiles also runs after systemd-sysext.service and we anyway want to set up things from the initrd eventually. Having it in /etc/extensions/ is the best then but it means we need to also change the documentation on how to get rid of it. In the past it was a folder under /etc/extensions but now the user would have to create an own symlink to /dev/null or an empty file in /etc/extensions/ to overwrite what the lowerdir of the /etc overlay provides.

Edit: Docs PR flatcar-archive/flatcar-docs#298

[pothos]

The convert_torcx_image.sh script is covered in the docs.
While we talked a lot about deprecating/removing Torcx, we didn't announce removal yet with an aimed date but I think that makes most sense when we work on removing it from Alpha. The deprecation notice is somehow present in the docs already because since a long time we state the preference for either static binaries on /opt/bin or sysext.

[t-lo]

There's some traction coming into this; we fleshed out the immediate path forward in

• [RFE] Include Docker packages and dependencies in portage-stable #1091
• [RFE] Build and ship Docker sysext #1092
• [RFE] Remove torcx from the image build and remove the torcx package from the base OS #1093

(the last 2 are co-dependent)