Move and rename repos, upgrade to Catalyst 4, support SDK on arm64 #2093

Sorry for doing all this in one giant commit, but it was hard to separate it out. We had no arm64 SDK, so some cross-compiling or emulation was most likely going to be needed to produce one. Catalyst 4 adds support for building with QEMU, so I looked into upgrading. This turned out to be very much slower than emulating the amd64 SDK on arm64, where an arm64 build could then be mostly run without emulation. We can't stay on Catalyst 3 forever though, so I continued with the upgrade. Despite being slow, I have kept support for building with QEMU using Catalyst since it requires little code and may be useful to somebody. Catalyst 4 has totally changed the way repositories are handled. It only works when the name of the directory containing the repository matches the configured name of that repository. This was not the case for us, with the coreos repository residing in the coreos-overlay directory. We wanted to move and rename our repositories anyway, so they are now known as gentoo-subset and flatcar-overlay, and they live under scripts/repos. Using the same name as upstream Gentoo would have been problematic, and just "flatcar" would have looked awkward in documentation. Catalyst 4 also ingests the main repository snapshot as a squashfs rather than a tarball. It features a utility to generate such a snapshot, but it doesn't fit Flatcar well, particularly because it expects each ebuild repository to reside at the top level of its own git repository. It was very easy to call tar2sqfs manually though. There were several places where we assumed that amd64 was native and arm64 required emulation via QEMU. The scripts are now more architecture-agnostic, paving the way for riscv support later. We no longer set QEMU_LD_PREFIX because it prevents the SDK itself from being emulated. It also assumes there is only one non-native target, which won't be the case soon. bubblewrap does a better job of running binaries under QEMU. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

This hasn't been needed for a while, and it now breaks util-linux, installing modules under /usr/lib64 when they should be under /usr/lib. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

@world

This is what upstream Gentoo does. They would previously update the entire seed, but this took a long time. Our seeds are much bigger, so we kept repo snapshots to build stage1 against these instead. The new method of only rebuilding packages with changed sub-slots is a good compromise and removes the need to write stage1 hooks that selectively catch the repository up. This also avoids some conflicts by adding the `--ignore-world` option. Gentoo seeds have nothing in @world. We have much more, but none of that is needed for stage1. This continues to exclude cross-*-cros-linux-gnu/* as that is not needed for stage1. It now also excludes dev-lang/rust, because it is never a DEPEND, so it would not break other packages in this way. It may fail to run due to a sub-slot change in one of its own dependencies, but it is also unlikely to be needed in stage1 and it is not configured to use the system LLVM. If needs be, we could improve the behaviour of Portage's @changed-subslot to respect `--with-bdeps`. In my testing, it was unable to handle an SDK from 17 months ago, but one from 7 months ago did work. In practise, we will always use a much more recent one, which is far more likely to work. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

From https://wiki.gentoo.org/wiki/Catalyst/Stage_Creation#Build_Stage3: > It is not necessary to build stage2 in order to build stage3. Gentoo > release engineering does not build stage2, and you should not need to > unless you're intentionally building a stage2 as your goal. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

We stopped using profiles with a lib->lib64 symlink a while ago, so there is no point in checking for this any more. We weren't checking against the target SDK architecture anyway. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

We currently put an os-release symlink in lib64, but we shouldn't assume that the architecture will even have a lib64 directory. I doubt this compatibility symlink was needed anyway. Gentoo doesn't have one, and applications are supposed to check /etc/os-release. I can find almost no reference to /usr/lib64/os-release anywhere, let alone in Flatcar. Signed-off-by: James Le Cuirot <jlecuirot@microsoft.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move and rename repos, upgrade to Catalyst 4, support SDK on arm64 #2093

Move and rename repos, upgrade to Catalyst 4, support SDK on arm64 #2093

Commits on Jul 5, 2024