stdenv: aarch64-linux: gcc9 -> gcc12

[wegank]

Description of changes

A roadblock for bumping the aarch64-linux GCC version is #167726 (comment), or simply #36947. A proper fix would then be to add an extra stage5 to build glibc against the newer GCC.

But, what if the bootstrap version of libgcc_s.so is recent enough? This PR therefore does the following:

1. Bump the GCC version to 12.
   Observe that icu does not build at this time.
2. nix build -f ./pkgs/stdenv/linux/make-bootstrap-tools.nix bootstrapFiles
   The bootstrap files build successfully.
3. Upload and use the new bootstrap files.
   Observe that icu, dejagnu and protobuf now build successfully.

$ nix hash file /nix/store/5vj209irqzvmzajsnj9wam1mcnrsk7ml-bootstrap-tools.tar.xz
sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
$ nix hash file /nix/store/w39cbankzyc8fs5mdlikwndi3a3kgzaz-busybox
sha256-jtAPxLmSw6CoU3/CPZzZg4Q8h61EDfbz5yZXJV1cOm4=

Closes #108305.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[tpwrules]

I think it is no secret that updating the bootstraps to a more recent GCC will (probably, I haven't independently confirmed yet) work, the question is if we want to do it? I am not sure what the official process is for updating the bootstraps, or what constraints there might be on GCC versions in them.

And even right now there is a draft PR to upgrade to GCC 12 by default which might cause the bootstrap to need to be re-updated if there is a big change which needs a new bootstrap libgcc_s.so again. Unfortunately this is not really a correct solution which is why I think there has been hesitance to do it.

[tpwrules]

Looks like this does in fact fix icu and avoid the compilation failure due to the mismatched libgcc_s.so. I will try to build my simple GUI system completely and see if anything else shows problems. Thanks for actually doing the legwork to test whether this is a viable approach.

I propose we wait and see how the GCC 12 upgrade moves. If that goes smoothly we can simply bump the bootstrap and mitigate this problem for longer than if we did it now. Though at some point pragmatism has to outweigh purity and I have heard there are problems with modern packages (particularly KDE) and GCC 9, so this really needs to be solved soon.

While that is happening, we can explore the possibility of revamping the bootstrap sequence to put this issue to bed. I don't personally feel too confident about having time/skill to do that, and there are some concerns about excessive iteration time for people who need to touch the stdenv. But it's the right thing to do in the end.

[tpwrules]

Also worth noting, this is the last time the aarch64-linux bootstrap tarballs were updated and there is some info on people and process if we decide to go that route: #151399

[wegank]

I added a third second commit on my machine, bumping the GCC version to 12, and icu, dejagnu and protobuf all build well. So I guess the bootstrap files won't need updating very soon.

Also, the bootstrap version of GCC is already at 9.3.0 after #151399.

[K900]

I think we should do this if it get us GCC 12, but I also think we should really fix the bootstrap properly before things explode again.

[tpwrules]

I can confirm that a basic desktop system with Xfce and Firefox builds, boots, and seems to work okay.

One comment is that I would prefer this be one commit instead of two, with the message clarifying that the bootstrap tools in the commit are generated after the commit's other changes. This would avoid a unique and broken environment during e.g. bisection.

[wegank]

@ofborg build stdenv stdenv.passthru.tests

[gbpdt]

I've tested this change and it appears to work well for me (e.g. it fixes the icu build problem). I'd support merging it ASAP.

[tpwrules]

Just reminding that this cannot be merged until the files are uploaded to nixos.org and the appropriate URLs inserted.

I am also working on independently verifying the hashes here.

[tpwrules]

I cannot reproduce your hashes. It looks like you built glibc against kernel 6.0 and I built it against kernel 6.1. I made the change which removes the aarch64-linux exception to the default GCC on top of nixpkgs commit dac3b0d87329edf225ea7d2ed68b9ab79ca8c53b then built the bootstrap files as described.

It's great that we've shown this is a viable path forward to fix the issue but it's definitely not mergeable as-is. We need reliable provenance for the bootstrap components.

I'm not sure of the protocol for that, looking through the history most times they have been built by Hydra as the "official" versions. But it would be awkward to break half of aarch64-linux for a week trying to get the bootstrap tools rebuilt, even on staging. Someone might be able to create a special branch and job though.

[wegank]

@tpwrules I'll update the commit once I finished the build on dac3b0d87329edf225ea7d2ed68b9ab79ca8c53b. If the hashes still don't match, I guess it might be better to build the bootstrap files on a Hydra branch.

[vcunat]

@lovesegfault: so, let me ask explicitly for the upload.

Target paths are a bit trickier than usual. Customarily we use name of the commit from which the new tools were generated and which was already merged at that point. Here we need to switch the gcc version first to build the bootstrap tools, but that commit isn't well usable until the new tools get used. (That's why aarch64-linux has been stuck on old gcc in the first place.) Still, I think we can include such a commit in history; in particular I'll suggest this commit and upload path: vcunat@7c026ba

[tpwrules]

Built my VM test environment which is a stock Xfce + Firefox + a few miscellaneous daemons and packages. Everything seems to build and work except for two simple issues:

• orc has a known test failure on aarch64 and GCC 12 quite possibly due to a GCC bug. Just disabled the tests and it seems to work fine.
• vte won't link under clangStdenv anymore due to some C++ string ABI mismatch, reverting that change and just using the regular now-GCC-12 stdenv and it works.

[vcunat]

Hmm, let me try pinging again: @lovesegfault (I've tried other people with permissions in the meantime.)

[lovesegfault]

I don't have access to my AWS keys to upload right now, since I'm traveling, and it's not on my work laptop.

I'm home next week, so that's the earliest I'd be able to help.

I'll try to see if I have them in a backup or something, if so, I can do it today.

Furthermore, I do need a bit more detail to do the upload, just because it's very hard not to mess up. Here's an example of a good request: #183487

[wegank]

Furthermore, I do need a bit more detail to do the upload, just because it's very hard not to mess up. Here's an example of a good request: #183487

I've put the hashes in the description of changes. This should roughly translate to

$ sha256sum ./bootstrap-tools.tar.xz ./busybox
689bedb167ae4076dbd7804667612238ace3427e3a877c7776eb8f11ac06d1e1  ./bootstrap-tools.tar.xz
8ed00fc4b992c3a0a8537fc23d9cd983843c87ad440df6f3e72657255d5c3a6e  ./busybox

The targeted commit is 7244b4babe85d158883c3bf949abf7947a050f38.

Let me know if you need more information.

[vcunat]

This upcoming week would be nice, given the other (non-)responses I've got so far.

Perhaps you missed some description in the commit which I had linked and proposed to merge later (vcunat@7c026ba). I prefer to have these important bits directly in there, signed and not (just) in some discussion thread which might be harder to find or even disappear.

You don't need so complicated downloads from Hydra and adding. I think it's much better to directly get /nix/store paths from cache by nix-store --realize $path or nix-build $path. (I had thought that obvious and didn't add it into the commit message.) Anyway, ask for whatever you're missing.

[lovesegfault]

Alright, I've uploaded them:

❯ aws s3 ls --recursive s3://nixpkgs-tarballs/stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38
2023-02-26 13:23:59   20461272 stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/bootstrap-tools.tar.xz
2023-02-26 13:23:59     257384 stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/busybox

❯ aws s3 cp --recursive s3://nixpkgs-tarballs/stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/ ./
download: s3://nixpkgs-tarballs/stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/busybox to ./busybox
download: s3://nixpkgs-tarballs/stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/bootstrap-tools.tar.xz to ./bootstrap-tools.tar.xz

❯ sha256sum bootstrap-tools.tar.xz busybox
689bedb167ae4076dbd7804667612238ace3427e3a877c7776eb8f11ac06d1e1  bootstrap-tools.tar.xz
8ed00fc4b992c3a0a8537fc23d9cd983843c87ad440df6f3e72657255d5c3a6e  busybox

[wegank]

Alright, I've uploaded them:

Thank you! I guess these files should be available soon from the links below?

http://tarballs.nixos.org/stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/busybox
http://tarballs.nixos.org/stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/bootstrap-tools.tar.xz

[lovesegfault]

Alright, I've uploaded them:

Thank you! I guess these files should be available soon from the links below?

http://tarballs.nixos.org/stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/busybox
http://tarballs.nixos.org/stdenv-linux/aarch64/7244b4babe85d158883c3bf949abf7947a050f38/bootstrap-tools.tar.xz

Yup! It can take a couple of hours for them to pop-up on the http endpoint, for whatever reason.

[lovesegfault]

Okay, the issue was: I had forgotten to set the ACL to public-read.

Now fixed, and the links are working. Once you update this PR with the right bootstrap links, I think it can be landed?

[K900]

Anyone want to go through the GCC9 workarounds after this?

[vcunat]

Well, different commit/URL than I posted and explained, but it's just a convention and the files are the same, so I don't really care.

[lovesegfault]

Well, different commit/URL than I posted and explained, but it's just a convention and the files are the same, so I don't really care.

Well, that's what I get for trying to do anything with the flu 🤧

If you tell me where you want them, I can re-upload to the correct place.

Sorry for the mistake, wanted to get this done ASAP and ended up messing up.

[wegank]

If you tell me where you want them, I can re-upload to the correct place.

The commit would be 21ec906463ea8f11abf3f9091ddd4c3276516e58, with the same files. Sorry about that!

[lovesegfault]

If you tell me where you want them, I can re-upload to the correct place.

The commit would be 21ec906463ea8f11abf3f9091ddd4c3276516e58, with the same files. Sorry about that!

❯ aws s3 ls --recursive s3://nixpkgs-tarballs/stdenv-linux/aarch64/21ec906463ea8f11abf3f9091ddd4c3276516e58/
2023-02-26 15:57:34   20461272 stdenv-linux/aarch64/21ec906463ea8f11abf3f9091ddd4c3276516e58/bootstrap-tools.tar.xz
2023-02-26 15:57:34     257384 stdenv-linux/aarch64/21ec906463ea8f11abf3f9091ddd4c3276516e58/busybox

Hopefully, I got it right now, heading to sleep.

[wegank]

@ofborg build icu

[wegank]

@vcunat Is the PR good to go?

[vcunat]

Yes, I think so, after another look.