cc-wrapper: deunify clang/gcc treatment of -isystem #225220

ghost · 2023-04-07T20:59:03Z

Problem

Failures reported among packages which use gfortran:

Methodology

Add -v to NIX_CFLAGS_COMPILE
Run builds of python3Packages.scipy with and without #209870 reverted
Run nuke-refs on the resulting logs
Run wdiff -n | colordiff on the nuked logs

Diagnosis

The fix

commit 77239eb5b09e5f0c507887524557b987a5508e50
Author: Adam Joseph <adam@westernsemico.com>
Date:   Fri Apr 7 21:44:28 2023 -0700

    In https://github.com/NixOS/nixpkgs/pull/209870 I tried to unify the
    treatment of clang and gcc in cc-wrapper as much as possible.
    However it appears that I went too far.

    Clang requires -isystem flags in order to be able to find gcc's
    libstdc++.  Gcc does not need these flags.  If they are added,
    gfortran will get confused:

      https://github.com/NixOS/nixpkgs/pull/209870#issuecomment-1500550903

    This commit deunifies the chunk of code that adds the -isystem
    flags, and explains why this chunk applies only to clang.

Validation

I hacked in this PR's fix to scipy (only) using wrapCCWith and scipy built correctly. Currently verifying a global rebuild with this PR, but it won't finish until after I go to sleep.

@vcunat, if the situation is urgent we can send the wrapCCWith-hacked version to staging-next. Otherwise the fix has to wait for this PR's global rebuild ☹️

ghost · 2023-04-07T22:28:46Z

Hrm, it got farther but failed (differently). Investigating.

python3.10-scipy>   FAILED: scipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p/wright.cc.o
python3.10-scipy>   g++ -Iscipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p -Iscipy/special -I../../scipy/special -I/nix/store/l8ylq3skndl7xs7zwsmdlhxwi4ik5vzy-python3.10-numpy-1.24.2/lib/python3.10/site-packages/numpy/core/include -I../../scipy/_lib/boost -Iscipy/_lib -I../../scipy/_lib -I../../scipy/_build_utils/src -I/nix/store/45v06dfypiw75jn6dwffyv4y10d3pdhy-python3-3.10.10/include/python3.10 -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++14 -O2 -fPIC -Wno-cpp -MD -MQ scipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p/wright.cc.o -MF scipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p/wright.cc.o.d -o scipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p/wright.cc.o -c ../../scipy/special/wright.cc
python3.10-scipy>   In file included from ../../scipy/special/_round.h:53,
python3.10-scipy>                    from ../../scipy/special/wright.cc:79:
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:58:11: error: ‘fenv_t’ has not been declared in ‘::’
python3.10-scipy>      58 |   using ::fenv_t;
python3.10-scipy>         |           ^~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:59:11: error: ‘fexcept_t’ has not been declared in ‘::’
python3.10-scipy>      59 |   using ::fexcept_t;
python3.10-scipy>         |           ^~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:62:11: error: ‘feclearexcept’ has not been declared in ‘::’
python3.10-scipy>      62 |   using ::feclearexcept;
python3.10-scipy>         |           ^~~~~~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:63:11: error: ‘fegetexceptflag’ has not been declared in ‘::’
python3.10-scipy>      63 |   using ::fegetexceptflag;
python3.10-scipy>         |           ^~~~~~~~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:64:11: error: ‘feraiseexcept’ has not been declared in ‘::’
python3.10-scipy>      64 |   using ::feraiseexcept;
python3.10-scipy>         |           ^~~~~~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:65:11: error: ‘fesetexceptflag’ has not been declared in ‘::’
python3.10-scipy>      65 |   using ::fesetexceptflag;
python3.10-scipy>         |           ^~~~~~~~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:66:11: error: ‘fetestexcept’ has not been declared in ‘::’
python3.10-scipy>      66 |   using ::fetestexcept;
python3.10-scipy>         |           ^~~~~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:68:11: error: ‘fegetround’ has not been declared in ‘::’
python3.10-scipy>      68 |   using ::fegetround;
python3.10-scipy>         |           ^~~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:69:11: error: ‘fesetround’ has not been declared in ‘::’
python3.10-scipy>      69 |   using ::fesetround;
python3.10-scipy>         |           ^~~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:71:11: error: ‘fegetenv’ has not been declared in ‘::’
python3.10-scipy>      71 |   using ::fegetenv;
python3.10-scipy>         |           ^~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:72:11: error: ‘feholdexcept’ has not been declared in ‘::’
python3.10-scipy>      72 |   using ::feholdexcept;
python3.10-scipy>         |           ^~~~~~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:73:11: error: ‘fesetenv’ has not been declared in ‘::’
python3.10-scipy>      73 |   using ::fesetenv;
python3.10-scipy>         |           ^~~~~~~~
python3.10-scipy>   /nix/store/8sphbc0jiba0ky3ghq7zrd4gg2ybabab-gcc-12.2.0/include/c++/12.2.0/fenv.h:74:11: error: ‘feupdateenv’ has not been declared in ‘::’
python3.10-scipy>      74 |   using ::feupdateenv;
python3.10-scipy>         |           ^~~~~~~~~~~

lovesegfault · 2023-04-08T07:04:03Z

@amjoseph-nixpkgs i think you want to target staging-next?

vcunat · 2023-04-08T07:05:48Z

I think I'll go for a workaround using this just for the specific breakage(s) in the current staging-next, as rebuilding linux stdenvs is quite costly.

ghost · 2023-04-08T08:00:23Z

@amjoseph-nixpkgs i think you want to target staging-next?

This causes a global rebuild; is that okay? I thought staging-next was sort of like master -- no mass-rebuilds allowed. Did I misunderstand?

Anyways it doesn't matter to me.

Also, my rebuild went way faster than I thought; I should have a rebuilt-from-scratch scipy in the next few minutes.

vcunat · 2023-04-08T08:09:23Z

I'm a bit confused that you mention scipy all the time, as that package ~~worked~~ built successfully: https://hydra.nixos.org/job/nixpkgs/staging-next/python310Packages.scipy.x86_64-linux

vcunat · 2023-04-08T08:12:13Z

(it was python3.pkgs.scikit-learn that I knew to be broken)

vcunat · 2023-04-08T11:39:08Z

No problem :-) Fortunately it still seems like simple hacks will suffice for now, so a proper solution isn't urgent and could be a mass rebuild.

ghost · 2023-04-08T17:56:52Z

✔️ This PR is ready. @vcunat @trofi please consider merging this.

It contains two commits, and both are needed. If you omit either one of them, scikit-learn will fail to build (although the failure might be different, or one of its dependencies might fail before it even starts building).

Got it! To fix scikit-learn you need both trofi's suggestion and also the change to -isystem.

Proof:

nix@moore:/git/work/fix/scikit/with-trofi-idea$ git rev-parse HEAD
de7cbf714b8562d4dddbe284a205e5d92f7b7bf7
nix@moore:/git/work/fix/scikit/with-trofi-idea$ nix-instantiate . -A python3Packages.scikit-learn
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/j9jd3mad0624pfhn9vlmc17rvk9017a2-python3.10-scikit-learn-1.2.1.drv
nix@moore:/git/work/fix/scikit/with-trofi-idea$ nix-build . -A python3Packages.scikit-learn
/nix/store/z3fxmiam657acwl10jwkhygvvgda44ll-python3.10-scikit-learn-1.2.1

scipy fails to build if you omit cc-wrapper: deunify clang/gcc treatment of -isystem

nix@moore:/git/work/fix/scikit/with-trofi-idea$ git log | head -n 8 commit 9d5d1f7cb3aa329b2029b0cdf97334b26b4118e9 Author: Adam Joseph Date: Sat Apr 8 10:54:19 2023 -0700 Revert "cc-wrapper: deunify clang/gcc treatment of -isystem" This reverts commit de7cbf714b8562d4dddbe284a205e5d92f7b7bf7. nix@moore:/git/work/fix/scikit/with-trofi-idea$ nix build -f . -L python3Packages.scikit-learn ... python3.10-scipy> g++ -Iscipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p -Iscipy/special -I../../scipy/special -I/nix/store/15ln23q9jqphzp2py537819y5p4hs4rl-python3.10-numpy-1.24.2/lib/python3.10/site-packages/numpy/core/include -I../../scipy/_lib/boost -Iscipy/_lib -I../../scipy/_lib -I../../scipy/_build_utils/src -I/nix/store/aaqc8qslczqpjbyl9l3djrcaylm0kxlk-python3-3.10.11/include/python3.10 -fvisibility=hidden -fvisibility-inlines-hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=c++14 -O2 -fPIC -Wno-cpp -MD -MQ scipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p/wright.cc.o -MF scipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p/wright.cc.o.d -o scipy/special/_ufuncs_cxx.cpython-310-x86_64-linux-gnu.so.p/wright.cc.o -c ../../scipy/special/wright.cc python3.10-scipy> In file included from ../../scipy/special/_round.h:53, python3.10-scipy> from ../../scipy/special/wright.cc:79: python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:58:11: error: ‘fenv_t’ has not been declared in ‘::’ python3.10-scipy> 58 | using ::fenv_t; python3.10-scipy> | ^~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:59:11: error: ‘fexcept_t’ has not been declared in ‘::’ python3.10-scipy> 59 | using ::fexcept_t; python3.10-scipy> | ^~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:62:11: error: ‘feclearexcept’ has not been declared in ‘::’ python3.10-scipy> 62 | using ::feclearexcept; python3.10-scipy> | ^~~~~~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:63:11: error: ‘fegetexceptflag’ has not been declared in ‘::’ python3.10-scipy> 63 | using ::fegetexceptflag; python3.10-scipy> | ^~~~~~~~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:64:11: error: ‘feraiseexcept’ has not been declared in ‘::’ python3.10-scipy> 64 | using ::feraiseexcept; python3.10-scipy> | ^~~~~~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:65:11: error: ‘fesetexceptflag’ has not been declared in ‘::’ python3.10-scipy> 65 | using ::fesetexceptflag; python3.10-scipy> | ^~~~~~~~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:66:11: error: ‘fetestexcept’ has not been declared in ‘::’ python3.10-scipy> 66 | using ::fetestexcept; python3.10-scipy> | ^~~~~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:68:11: error: ‘fegetround’ has not been declared in ‘::’ python3.10-scipy> 68 | using ::fegetround; python3.10-scipy> | ^~~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:69:11: error: ‘fesetround’ has not been declared in ‘::’ python3.10-scipy> 69 | using ::fesetround; python3.10-scipy> | ^~~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:71:11: error: ‘fegetenv’ has not been declared in ‘::’ python3.10-scipy> 71 | using ::fegetenv; python3.10-scipy> | ^~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:72:11: error: ‘feholdexcept’ has not been declared in ‘::’ python3.10-scipy> 72 | using ::feholdexcept; python3.10-scipy> | ^~~~~~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:73:11: error: ‘fesetenv’ has not been declared in ‘::’ python3.10-scipy> 73 | using ::fesetenv; python3.10-scipy> | ^~~~~~~~ python3.10-scipy> /nix/store/g69llwf0yh385r5gr545zckkfz7g0fq0-gcc-12.2.0/include/c++/12.2.0/fenv.h:74:11: error: ‘feupdateenv’ has not been declared in ‘::’ python3.10-scipy> 74 | using ::feupdateenv; python3.10-scipy> | ^~~~~~~~~~~ ... python3.10-scipy> ninja: build stopped: subcommand failed. python3.10-scipy> INFO: autodetecting backend as ninja python3.10-scipy> INFO: calculating backend command to run: /nix/store/z13jdpnmh5qb7w2k3hb86d7nnrvz322c-ninja-1.11.1/bin/ninja python3.10-scipy> + meson setup --prefix=/nix/store/aaqc8qslczqpjbyl9l3djrcaylm0kxlk-python3-3.10.11 /build/scipy-1.10.1 /build/scipy-1.10.1/.mesonpy-1dniauw0/build --native-file=/build/scipy-1.10.1/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 python3.10-scipy> + meson compile python3.10-scipy> error: subprocess-exited-with-error python3.10-scipy> × Preparing metadata (pyproject.toml) did not run successfully. python3.10-scipy> │ exit code: 1 python3.10-scipy> ╰─> See above for output. python3.10-scipy> note: This error originates from a subprocess, and is likely not a problem with pip. python3.10-scipy> full command: /nix/store/aaqc8qslczqpjbyl9l3djrcaylm0kxlk-python3-3.10.11/bin/python3.10 /nix/store/9mc17h3wsdrb2gfflr6a0475dx7vy6mz-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmp_lylr9n5 python3.10-scipy> cwd: /build/scipy-1.10.1 python3.10-scipy> Preparing metadata (pyproject.toml) ... error python3.10-scipy> error: metadata-generation-failed python3.10-scipy> × Encountered error while generating package metadata. python3.10-scipy> ╰─> See above for output. python3.10-scipy> note: This is an issue with the package mentioned above, not pip. python3.10-scipy> hint: See above for details. error: build of '/nix/store/djx1wxzwdy7hn2r50i5rrrdwlw0maqyp-python3.10-scipy-1.10.1.drv' on 'ssh://root@192.168.22.102' failed: builder for '/nix/store/djx1wxzwdy7hn2r50i5rrrdwlw0maqyp-python3.10-scipy-1.10.1.drv' failed with exit code 1 error: builder for '/nix/store/djx1wxzwdy7hn2r50i5rrrdwlw0maqyp-python3.10-scipy-1.10.1.drv' failed with exit code 1; last 10 log lines: > full command: /nix/store/aaqc8qslczqpjbyl9l3djrcaylm0kxlk-python3-3.10.11/bin/python3.10 /nix/store/9mc17h3wsdrb2gfflr6a0475dx7vy6mz-python3.10-pip-23.0.1/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py prepare_metadata_for_build_wheel /build/tmp_lylr9n5 > cwd: /build/scipy-1.10.1 > Preparing metadata (pyproject.toml) ... error > error: metadata-generation-failed > > × Encountered error while generating package metadata. > ╰─> See above for output. > > note: This is an issue with the package mentioned above, not pip. > hint: See above for details. For full logs, run 'nix log /nix/store/djx1wxzwdy7hn2r50i5rrrdwlw0maqyp-python3.10-scipy-1.10.1.drv'. error: 1 dependencies of derivation '/nix/store/zwc76q4cjq1jd1pzsbyvz63la57dlf14-python3.10-scikit-learn-1.2.1.drv' failed to build

scikit-learn fails to build if you omit gcc: never disableBootstrap for gfortran

nix@moore:/git/work/fix/scikit/with-trofi-idea$ git reset --hard de7cbf714b8562d4dddbe284a205e5d92f7b7bf7
HEAD is now at de7cbf714b85 cc-wrapper: deunify clang/gcc treatment of -isystem
nix@moore:/git/work/fix/scikit/with-trofi-idea$ git revert HEAD^
[with-trofi-idea 797d9b6aaad1] Revert "gcc: never disableBootstrap for gfortran"
 1 file changed, 1 insertion(+), 4 deletions(-)
nix@moore:/git/work/fix/scikit/with-trofi-idea$ nix-build . -A python3Packages.scikit-learn
...
       > gcc -shared -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.13/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib -L/nix/store/2scdlrgd8sqmm1sgsg9b7sjnwlmr9s1p-libxcrypt-4.4.33/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.41.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.8/lib -L/nix/store/1whrncscgc90j7dh08wcyvygjgzcjkgm-tzdata-2023c/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.13/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.8/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-expat-2.5.0/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.4.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libffi-3.4.4/lib -L/nix/store/2scdlrgd8sqmm1sgsg9b7sjnwlmr9s1p-libxcrypt-4.4.33/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-gdbm-1.23/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.41.2/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-readline-8.2p1/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ncurses-6.4/lib -L/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-3.0.8/lib -L/nix/store/1whrncscgc90j7dh08wcyvygjgzcjkgm-tzdata-2023c/lib build/temp.linux-x86_64-cpython-310/sklearn/_loss/_loss.o -L/nix/store/xasxsk7n57j3mx50vz848gd04i95kj4v-python3-3.10.11/lib -Lbuild/temp.linux-x86_64-cpython-310 -lm -llibsvm-skl -lliblinear-skl -o build/lib.linux-x86_64-cpython-310/sklearn/_loss/_loss.cpython-310-x86_64-linux-gnu.so -fopenmp
       > error: command '/nix/store/02nzdnalvzi8mf6p2l8svl6swy9bp5by-gfortran-wrapper-12.2.0/bin/gcc' failed with exit code 1
       > /nix/store/fqvpf04307g05fibxjg287z5cga3pcd9-stdenv-linux/setup: line 1594: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/0khrvashssrr1rr7z0cyx8b77xjiyy7a-python3.10-scikit-learn-1.2.1.drv'.

trofi

LGTM, to state the obvious it's a full linux rebuild:

$ ./maintainers/scripts/rebuild-amount.sh HEAD^
Estimating rebuild amount by counting changed Hydra jobs (parallel=unset).
      1 pkgs-lib-tests
      2 x86_64-darwin
  57773 x86_64-linux

ghost · 2023-04-08T19:19:03Z

LGTM, to state the obvious it's a full linux rebuild:

Yes indeed.

The low-rebuilds version in #225273 has been updated; I'm just waiting to make sure it works before undraftifying it. If it works, I think the best plan is to merge that one to staging-next, and then in the next full-rebuild cycle revert #225273 and merge this. Edit: @vcunat intends to put this into the next full-rebuild cycle, so the low-rebuilds version is not needed.

ghost · 2023-04-08T19:39:11Z

I have cherry-picked 4b9280f8fd008b7bd22aeff50ede5195dcf355fb and then reverted it so we won't forget to do so.

Once staging-next merges to staging github will drop the cherry-picked commit, and then when this merges it will revert 4b9280f8fd008b7bd22aeff50ede5195dcf355fb.

This is a low-rebuild version of PR #225273 /cc the proper and hopefully complete fix in PR #225220

Artturin · 2023-04-10T18:21:42Z

Needs rebase

vcunat · 2023-04-10T18:25:30Z

Well yes, it needs a revert of e269122.

Artturin · 2023-04-10T18:40:31Z

Well yes, it needs a revert of e269122.

i did

git rebase --onto staging HEAD~4 --interactive

then added exec git revert e269122 at the top, exited, then got the following

$ git rebase --onto staging HEAD~4 --interactive
Executing: git revert e269122
Auto-merging pkgs/top-level/all-packages.nix
[detached HEAD a8c95cf146b] Revert "julia{18,19,}: fix build by a temporary hack"
 3 files changed, 4 insertions(+), 22 deletions(-)
The previous cherry-pick is now empty, possibly due to conflict resolution.
If you wish to commit it anyway, use:

    git commit --allow-empty

Otherwise, please use 'git rebase --skip'
interactive rebase in progress; onto da9d5848454
Last commands done (2 commands done):
   exec git revert e269122
   pick e64a751c475 python3Packages.scikit-learn: hack-fix missing libstdc++
Next commands to do (3 remaining commands):
   pick 3d904285202 Revert "python3Packages.scikit-learn: hack-fix missing libstdc++"
   pick 68ed28c77d6 gcc: never disableBootstrap for gfortran
  (use "git rebase --edit-todo" to view and edit)
You are currently rebasing branch 'pr/gfortran/do-not-disable-bootstrap' on 'da9d5848454'.
  (all conflicts fixed: run "git rebase --continue")

then ran git rebase --continue

and i think it worked

$ git log staging...HEAD
commit 613e199c0d05457114dab4a2925032dae27a1578 (HEAD -> pr/gfortran/do-not-disable-bootstrap)
Author: Adam Joseph <adam@westernsemico.com>
Date:   Fri Apr 7 21:44:28 2023 -0700

    cc-wrapper: deunify clang/gcc treatment of -isystem

    In https://github.com/NixOS/nixpkgs/pull/209870 I tried to unify the
    treatment of clang and gcc in cc-wrapper as much as possible.
    However it appears that I went too far.

    Clang requires -isystem flags in order to be able to find gcc's
    libstdc++.  Gcc does not need these flags.  If they are added,
    gfortran will get confused:

      https://github.com/NixOS/nixpkgs/pull/209870#issuecomment-1500550903

    This commit deunifies the chunk of code that adds the -isystem
    flags, and explains why this chunk applies only to clang.

commit ba3792975a42f75a54c27a1c8bb9c2f57decf270
Author: Adam Joseph <adam@westernsemico.com>
Date:   Fri Apr 7 13:50:35 2023 -0700

    gcc: never disableBootstrap for gfortran

    As suggested by @trofi here:

      https://github.com/NixOS/nixpkgs/pull/209870#issuecomment-1500635687

    This should fix failures among packages which use gfortran:

      https://github.com/NixOS/nixpkgs/pull/209870#issuecomment-1500550903
      https://hydra.nixos.org/build/215195834

commit a83e1a0d194fb88931ba037c74e97e75c1014c93
Author: Adam Joseph <adam@westernsemico.com>
Date:   Sat Apr 8 12:30:33 2023 -0700

    Revert "python3Packages.scikit-learn: hack-fix missing libstdc++"

    This reverts commit 46f29d4bd6305a9f7c743d3f9022a8d5f6d3cb12.

commit a8c95cf146bc927b73904778d88863ee9a420e50
Author: Artturin <Artturin@artturin.com>
Date:   Mon Apr 10 21:37:31 2023 +0300

    Revert "julia{18,19,}: fix build by a temporary hack"

    This reverts commit e2691227cdc424c643511d40fd6234acdf77372e.

Again, temporary until PR #225220

ghost · 2023-04-11T17:07:37Z

git rebase --onto staging HEAD~4 --interactive

BTW thanks for telling me about this trick earlier. I figured out how to do it without manually counting commits (I often rebase 100+-long branches):

git-rebase-from-to() {
  echo "rebasing from $1 to $2"
  git rebase --onto $2 $(git merge-base $1 HEAD) HEAD
}

and then you can do things like

$ git-rebase-from-to origin/master origin/staging
rebasing from origin/master to origin/staging
...

This reverts commit 398b54b.

This reverts commit e269122.

This reverts commit 46f29d4.

@trofi

As suggested by @trofi here: #209870 (comment) This should fix failures among packages which use gfortran: #209870 (comment) https://hydra.nixos.org/build/215195834

In #209870 I tried to unify the treatment of clang and gcc in cc-wrapper as much as possible. However it appears that I went too far. Clang requires -isystem flags in order to be able to find gcc's libstdc++. Gcc does not need these flags. If they are added, gfortran will get confused: #209870 (comment) This commit deunifies the chunk of code that adds the -isystem flags, and explains why this chunk applies only to clang.

Artturin · 2023-04-11T17:22:14Z

added another revert

This comment was marked as outdated.

Sign in to view

ghost marked this pull request as ready for review April 7, 2023 21:24

ghost requested a review from matthewbauer as a code owner April 7, 2023 21:24

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels Apr 7, 2023

This comment was marked as outdated.

Sign in to view

ghost marked this pull request as draft April 8, 2023 00:22

This comment was marked as outdated.

Sign in to view

ghost changed the base branch from staging-next to staging April 8, 2023 04:53

This comment was marked as outdated.

Sign in to view

ghost changed the title ~~gcc: never disableBootstrap for gfortran~~ cc-wrapper: deunify clang/gcc treatment of -isystem Apr 8, 2023

ghost marked this pull request as ready for review April 8, 2023 05:29

ghost requested a review from Ericson2314 as a code owner April 8, 2023 05:29

ghost mentioned this pull request Apr 8, 2023

stdenv: Nix-driven bootstrap of gcc #209870

Merged

4 tasks

ghost requested review from trofi and vcunat April 8, 2023 05:30

ofborg bot added 10.rebuild-linux-stdenv and removed 10.rebuild-darwin: 1 labels Apr 8, 2023

trofi approved these changes Apr 8, 2023

View reviewed changes

This comment was marked as outdated.

Sign in to view

ghost mentioned this pull request Apr 8, 2023

cc-wrapper: gross hack for staging-next to fix scipy **revert this when merging staging-next to staging** #225273

Closed

This comment was marked as outdated.

Sign in to view

ghost marked this pull request as draft April 8, 2023 10:19

ghost marked this pull request as ready for review April 8, 2023 17:56

This comment was marked as outdated.

Sign in to view

trofi approved these changes Apr 8, 2023

View reviewed changes

vcunat added a commit that referenced this pull request Apr 10, 2023

julia{18,19,}: fix build by a temporary hack

e269122

This is a low-rebuild version of PR #225273 /cc the proper and hopefully complete fix in PR #225220

github-actions bot added the 6.topic: python label Apr 10, 2023

ofborg bot added the 8.has: clean-up label Apr 10, 2023

vcunat added a commit that referenced this pull request Apr 11, 2023

python310Packages.kaldi-active-grammar: fixup build

398b54b

Again, temporary until PR #225220

Artturin and others added 5 commits April 11, 2023 20:19

Revert "python310Packages.kaldi-active-grammar: fixup build"

fd004f1

This reverts commit 398b54b.

Revert "julia{18,19,}: fix build by a temporary hack"

b1d4dfd

This reverts commit e269122.

Revert "python3Packages.scikit-learn: hack-fix missing libstdc++"

88fc226

This reverts commit 46f29d4.

gcc: never disableBootstrap for gfortran

f06ab17

As suggested by @trofi here: #209870 (comment) This should fix failures among packages which use gfortran: #209870 (comment) https://hydra.nixos.org/build/215195834

This was referenced Apr 11, 2023

#209870 broke llvmPackages_rocm.clang #225780

Closed

#209870 broke llvmPackages_13.compiler-rt-libc #225779

Closed

Artturin merged commit a2da3d6 into NixOS:staging Apr 11, 2023

ghost mentioned this pull request Apr 12, 2023

cc-wrapper: deunify clang/gcc handling of -B flag #225846

Merged

This was referenced Apr 17, 2023

WIP staging->master: Update flake.lock Nix-QChem/NixOS-QChem#278

Merged

Build failure: python310Packages.openmm #226895

Closed

ghost deleted the pr/gfortran/do-not-disable-bootstrap branch January 23, 2024 06:48

Artturin mentioned this pull request Jan 29, 2024

List of @amjoseph-nixpkgs Pull requests before account removal #284796

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cc-wrapper: deunify clang/gcc treatment of -isystem #225220

cc-wrapper: deunify clang/gcc treatment of -isystem #225220

ghost commented Apr 7, 2023 •

edited by ghost

Loading

This comment was marked as outdated.

ghost commented Apr 7, 2023 •

edited by ghost

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

lovesegfault commented Apr 8, 2023

vcunat commented Apr 8, 2023

ghost commented Apr 8, 2023

This comment was marked as outdated.

This comment was marked as outdated.

vcunat commented Apr 8, 2023 •

edited

Loading

vcunat commented Apr 8, 2023

vcunat commented Apr 8, 2023

ghost commented Apr 8, 2023 •

edited by ghost

Loading

This comment was marked as outdated.

trofi left a comment

ghost commented Apr 8, 2023 •

edited by ghost

Loading

ghost commented Apr 8, 2023 •

edited by ghost

Loading

Artturin commented Apr 10, 2023

vcunat commented Apr 10, 2023

Artturin commented Apr 10, 2023 •

edited

Loading

ghost commented Apr 11, 2023

Artturin commented Apr 11, 2023

cc-wrapper: deunify clang/gcc treatment of -isystem #225220

cc-wrapper: deunify clang/gcc treatment of -isystem #225220

Conversation

ghost commented Apr 7, 2023 • edited by ghost Loading

Problem

Methodology

Diagnosis

The fix

Validation

This comment was marked as outdated.

ghost commented Apr 7, 2023 • edited by ghost Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

lovesegfault commented Apr 8, 2023

vcunat commented Apr 8, 2023

ghost commented Apr 8, 2023

This comment was marked as outdated.

This comment was marked as outdated.

vcunat commented Apr 8, 2023 • edited Loading

vcunat commented Apr 8, 2023

vcunat commented Apr 8, 2023

ghost commented Apr 8, 2023 • edited by ghost Loading

This comment was marked as outdated.

trofi left a comment

Choose a reason for hiding this comment

ghost commented Apr 8, 2023 • edited by ghost Loading

ghost commented Apr 8, 2023 • edited by ghost Loading

Artturin commented Apr 10, 2023

vcunat commented Apr 10, 2023

Artturin commented Apr 10, 2023 • edited Loading

ghost commented Apr 11, 2023

Artturin commented Apr 11, 2023

ghost commented Apr 7, 2023 •

edited by ghost

Loading

ghost commented Apr 7, 2023 •

edited by ghost

Loading

vcunat commented Apr 8, 2023 •

edited

Loading

ghost commented Apr 8, 2023 •

edited by ghost

Loading

ghost commented Apr 8, 2023 •

edited by ghost

Loading

ghost commented Apr 8, 2023 •

edited by ghost

Loading

Artturin commented Apr 10, 2023 •

edited

Loading