Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stdenv, gcc: rename postPostInstall -> preFixup #223634

Closed
wants to merge 13 commits into from
Closed

stdenv, gcc: rename postPostInstall -> preFixup #223634

wants to merge 13 commits into from

Commits on Mar 17, 2023

  1. Revert "nixos/lib/testing: set default timeout for VM tests"

    This reverts commit 254426e.
    Adam Joseph committed Mar 17, 2023
    Configuration menu
    Copy the full SHA
    8b5d412 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2023

  1. trivial-builders/test/references.nix: fix eval

    The command
    
    ```
    nix-build -A tests.trivial-builders.references --show-trace
    ```
    
    fails eval with
    
    ```
    in job ‘nixpkgs.tests.trivial-builders.references’:
    error: The option `meta.description' does not exist. Definition values:
           - In `makeTest parameters': "Run the Nixpkgs trivial builders tests"
    ```
    
    because `meta.description` and `meta.license` are not valid for
    `nixosTest`s (they are valid for `mkDerivation` of course).
    
    This has been causing Hydra eval failures:
    
      https://hydra.nixos.org/jobset/nixos/pr-209870-gcc-external-bootstrap#tabs-errors
    
    Let's fix eval by removing these attributes.
    Adam Joseph committed Mar 19, 2023
    Configuration menu
    Copy the full SHA
    e33f2d1 View commit details
    Browse the repository at this point in the history
  2. test-defaultPkgConfigPackages.nix: filter out recurseForDerivations

    The command at the top of this file fails to evaluate:
    
    ```
    $ nix-build -A tests.pkg-config.defaultPkgConfigPackages
    in job ‘nixpkgs.tests.pkg-config.defaultPkgConfigPackages.tests-combined.x86_64-linux’:
    error: pkg-config module `recurseForDerivations` is not defined to be a derivation. Please check the attribute value for `recurseForDerivations` in `pkgs/top-level/pkg-config-packages.nix` in Nixpkgs.
    ```
    
    This is also causing eval errors on Hydra:
    
      https://hydra.nixos.org/jobset/nixos/pr-209870-gcc-external-bootstrap#tabs-errors
    
    Let's filter out `recurseForDerivations=true` from the attrset,
    since it exists mainly as a flag to signal special handling when
    recursing.
    Adam Joseph committed Mar 19, 2023
    Configuration menu
    Copy the full SHA
    44137a7 View commit details
    Browse the repository at this point in the history
  3. cc-wrapper: if isClang, add -L${gccForLibs.libgcc}/lib

    When wrapping `clang` and using a `gccForLibs` whose `libgcc` is in
    its own output (rather than the `lib` output), this commit will adds
    `-L${gccForLibs.libgcc}/lib` to `cc-ldflags`.
    
    If that flag is not added, `firefox` will fail to compile because it
    invokes `clang-wrapper` with `-fuse-ld=lld` and passes `-lgcc_s` to
    `lld`, but does not tell `lld` where to find `libgcc_s.so`.  In that
    situation, firefox will fail to link.
    Adam Joseph committed Mar 19, 2023
    Configuration menu
    Copy the full SHA
    dc2a208 View commit details
    Browse the repository at this point in the history
  4. ccache-links: inherit version to keep cc-wrapper happy

    The Nix-driven bootstrap of gcc expects that `stdenv.cc.cc` has a
    `version` attribute, except if `stdenv.cc` is the `bootstrapFiles`.
    
    This commit causes `ccache-links`'s `cc-wrapper` to `inherit
    version` in order to satisfy that requirement.  Without this commit,
    ofborg CI will fail.
    
    Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
    Adam Joseph and SuperSandro2000 committed Mar 19, 2023
    Configuration menu
    Copy the full SHA
    3cddb6a View commit details
    Browse the repository at this point in the history
  5. stdenv/linux: factor out commonGccOverrides

    This commit has no effect on eval.  It simply factors out a common
    subexpression.
    Adam Joseph committed Mar 19, 2023
    Configuration menu
    Copy the full SHA
    0f8923c View commit details
    Browse the repository at this point in the history
  6. gcc/{11,12}: use lib.pipe

    This commit has no effect on eval.  It simply reorganizes the
    `gcc11` and `gcc12` expressions so they apply a list of
    `overrideAttrs`.  The list is currently empty.
    Adam Joseph committed Mar 19, 2023
    Configuration menu
    Copy the full SHA
    f0be5a9 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2023

  1. stdenv: Nix-driven bootstrap of gcc

     #### Summary
    
    By default, when you type `make`, GCC will compile itself three
    times.  This PR inhibits that behavior by configuring GCC with
    `--disable-bootstrap`, and reimplements the triple-rebuild using
    Nix rather than `make`/`sh`.
    
     #### Immediate Benefits
    
    - Allow `gcc11` and `gcc12` on `aarch64` (without needing new
      `bootstrapFiles`)
    - Faster stdenv rebuilds: the third compilation of gcc
      (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
      This allows Nix to build stageCompare in parallel with the rest of
      nixpkgs instead of in series.
    - No more copying `libgcc_s` out of the bootstrap-files or other
      derivations
    - No more Frankenstein compiler: the final gcc and the libraries it
      links against (mpfr, mpc, isl, glibc) are all built by the same
      compiler (xgcc) instead of a mixture of the bootstrapFiles'
      compiler and xgcc.
    - No more [static lib{mpfr,mpc,gmp,isl}.a hack]
    - Many other small `stdenv` hacks eliminated
    - `gcc` and `clang` share the same codepath for more of `cc-wrapper`.
    
     #### Future Benefits
    
    - This should allow using a [foreign] `bootstrap-files` so long as
      `hostPlatform.canExecute bootstrapFiles`.
    - This should allow each of the libraries that ship with `gcc`
      (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
      offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
      built in separate (one-liner) derivations which `inherit src;`
      from `gcc`, much like #132343
    
     #### Incorporates
    
    - #210004
    - #36948 (unreverted)
    - #210325
    - #210118
    - #210132
    - #210109
    - #213909
    - #216136
    - #216237
    - #210019
    - #216232
    - #216016
    - #217977
    - #217995
    
     #### Closes
    
    - Closes #108305
    - Closes #108111
    - Closes #201254
    - Closes #208412
    
     #### Credits
    
    This project was made possible by three important insights, none of
    which were mine:
    
    1. @Ericson2314 was the first to advocate for this change, and
       probably the first to appreciate its advantages.  Nix-driven
       (external) bootstrap is "cross by default".
    
    2. @trofi has figured out a lot about how to get gcc to not mix up
       the copy of `libstdc++` that it depends on with the copy that it
       builds, by moving the `bootstrapFiles`' `libstdc++` into a
       [versioned directory].  This allows a Nix-driven bootstrap of gcc
       without the final gcc would still having references to the
       `bootstrapFiles`.
    
    3. Using the undocumented variable [`user-defined-trusted-dirs`]
       when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
       uses a completely different and totally special set of rules for
       finding `libgcc_s.so`.  This trick is the only way we can put
       `libgcc_s.so` in its own separate outpath without creating
       circular dependencies or dependencies on the bootstrapFiles.  I
       would never have guessed to use this (or that it existed!) if it
       were not for a [comment in guix] which @Mic92 [mentioned].
    
    My own role in this PR was basically: being available to go on a
    coding binge at an opportune moment, so we wouldn't waste a
    [crisis].
    
    [aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
    [amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
    [nonexistent sysroot]: #210004
    [versioned directory]: #209054
    [`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
    [comment in guix]: https://github.com/guix-mirror/guix/blob/5e4ec8218142eee8e6e148e787381a5ef891c5b1/gnu/packages/gcc.scm#L253
    [mentioned]: #210112 (comment)
    [crisis]: #108305
    [foreign]: #170857 (comment)
    [static lib{mpfr,mpc,gmp,isl}.a hack]: https://github.com/NixOS/nixpkgs/blob/2f1948af9c984ebb82dfd618e67dc949755823e2/pkgs/stdenv/linux/default.nix#L380
    Adam Joseph committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    131aa21 View commit details
    Browse the repository at this point in the history
  2. gcc: add common/checksum.nix

    This commit adds `gcc/common/checksum.nix`, which contains code
    common to both gcc11 and gcc12, implementing the `enableChecksum`
    feature.
    
    When gcc's built-in bootstrap (`--enable-bootstrap`) is used, gcc
    compiles itself three times and compares a hash of the unlinked `.o`
    files from the second and third compilation.  The
    `enableChecksum=true` parameter performs the same comparison as part
    of the `postInstall` phase.
    
    Notably, `enableChecksum=true` can be used with `enableBootstrap=false`.
    
    Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
    Adam Joseph and SuperSandro2000 committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    d193b8a View commit details
    Browse the repository at this point in the history
  3. pkgs/test/stdenv/default.nix: add gcc-stageCompare

    This commit adds a derivation `gcc-stageCompare` to
    `pkgs/test/stdenv/default.nix`.
    
    It is important to always build this derivation whenever building
    `stdenv`!  Because we are using a Nix-driven bootstrap instead of
    gcc's built-in `--enable-bootstrap`, the `gcc` derivation no longer
    performs the post-self-compilation sanity check.  You must build
    this derivation in order to perform that sanity check.
    
    The major benefit of this new approach is that the sanity check
    (which involves a third compilation of gcc) can be performed
    *concurrently* with all packages that depend on `stdenv`, rather
    than serially.  Since `stdenv` has very little derivation-level
    parallelism it cannot take advantage of more than one or perhaps two
    builders.  If you have three or more builders this commit will
    reduce the time-to-rebuild-stdenv by around 20% (one of three gcc
    rebuilds is removed from the critical path, and stdenv's build time
    is dominated by roughly 3*gcc + 1*binutils + 1*bison-test-suite).
    
    Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
    Adam Joseph and SuperSandro2000 committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    09d06cc View commit details
    Browse the repository at this point in the history
  4. emacs: path fixes resulting from libgccjit changes

    The Nix-driven bootstrap of gcc resulted in some changes to the
    structure of the `libgccjit` outpaths, and also added an additional
    output (`libgcc`) to `gcc`.
    
    This commit makes the corresponding changes in the `emacs`
    derivation in order to not break emacs.
    
    Emacs is the only user of `libgccjit` in nixpkgs at the moment.
    Adam Joseph committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    bf99025 View commit details
    Browse the repository at this point in the history
  5. make-bootstrap-tools.nix: follow libgcc_s symlink

    Adam Joseph committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    4b2dc7f View commit details
    Browse the repository at this point in the history
  6. stdenv, gcc: rename postPostInstall -> preFixup

    This commit implements @SuperSandro2000's suggestion here:
    
      #209870 (review)
    
    I am submitting it as a separate PR in order to avoid a full rebuild
    on Hydra.
    Adam Joseph committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    132e7d1 View commit details
    Browse the repository at this point in the history