libgcc_s is from bootstrap tools??

[dtzWill]

Issue description

Time for a libgcc package? 😁

We probably want to have libgcc_s matching our compiler (and not frozen to whatever is in our bootstrap tools).

diffoscope between libgcc_s.so.1 in stdenv.cc.cc.lib and glibc: https://gist.github.com/dtzWill/596b43d5c16178f570b5357e941ea587

Note that this libgcc_s (the old one from gcc 5.3 apparently) is what everything links to (that needs it), easy example is lzip.

nixpkgs/pkgs/development/libraries/glibc/default.nix

Line 36 in 25edc47

# When building glibc from bootstrap-tools, we need libgcc_s at RPATH for

(Comment suggests the glibc used in stdenv won't be copied like this, but that's not true)

A separate libgcc package (or compiler-rt? :D) would be best, but if nothing else we probably should use a libgcc that matches our compiler.

Steps to reproduce

$ nix build -f channel:nixos-unstable glibc
$ readelf -a ./result/lib/libgcc_s.so.1
...
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libgcc_s.so.1]
 0x000000000000001d (RUNPATH)            Library runpath: [/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.23/lib]
...

[dtzWill]

Update: This is actually the reason gcc7 is broken with i686 everywhere! ("undefined refeences to __divmoddi4 and such).

Sure enough the libgcc_s we should be using has that symbol, but it is missing in the one from bootstrap tools (that we manage to use instead).

[Ekleog]

Relevant to #36453 ; for nixos:release-18.03:nixos.closures.xfce.i686-linux (https://hydra.nixos.org/build/71018796)

[dezgeg]

I have a solution for this based on #36948, but the hairy problem is that a number of packages that used to find libgcc_s.so from Glibc and would need to be modified to find it from the new location. I don't quite want to debug this in time to be in 18.09, so I guess we can just upgrade the bootstrap tools in the meantime.

[vcunat]

👍 sounds updating them is the easiest workaround ATM. It would be really nice to have i686 in a better shape for 18.09.

[Ericson2314]

We actually do have a libgcc package, haha. It's sort of lying in waiting until I package enough of the other ones to redo the bootstrapping.

[dezgeg]

There is no way the abomination that grovels deeply in the internals of the GCC build system can or should be used. Especially considering the fact that the current problems are caused by a version mismatch between libgcc_s.so and the GCC version used, that is the exact opposite approach that should be taken.

libgcc (and other compiler support libraries like libatomic) are GCC's own private property. Programs should be finding it from ${gcc.lib}/lib/libgcc.so so that it always matches the version of GCC that was used to compile it. The real annoying problem that makes fixing this not so easy is various binary packages that used to find libgcc.so from the glibc output break and need to be fixed by having ${gcc.lib}/lib added to their RPATH.

[dtzWill]

Huge 👍 (warranting a comment ;)).

[dtzWill]

Also fixing up binary packages doesn't seem so bad-- can we fix this for 18.09? This is the foundation of everything else in nixpkgs (more or less), and it's worth ensuring it's solid.

[Ericson2314]

Yes that should be down for 18.09. And if I make the libs output a depsTargetTargetPropagated of the wrapped GCC or similar, it will begin to alleviate some of the issues we have when mixing compilers

[vcunat]

There was the plan to just update bootstrap tools to work around the worst for 18.09 – what phase is that in? (Have I missed some thread?) EDIT: the longterm plan is to fork 18.09 during the next weekend, you know...

[dezgeg]

@vcunat I have the branch at https://github.com/dezgeg/nixpkgs/tree/update-bootstrap. IT seemed to work for me, as in fixing the i686 packages that used to be broken. Just need someone to upload them to an official location, for which the commit message has a script for whoever is responsible for that.

(edit: https://github.com/dezgeg/nixpkgs/tree/update-bootstrap-rebased rebased to today's staging to fix a conflict, not tested)

[vcunat]

OK, I tested stdenv and some other builds on the rebased branch: x86_64, i686, and aarch64 (via the community machine). I confirm that i686 problem doesn't appear anymore.

@edolstra: can you upload the tarballs? I don't know who else has credentials for this. Paths to Hydra builds are linked in that commit. I suppose it might be enough to update i686 to work around the worst part, but it's probably better to keep them all in sync.

[woffs]

I've built my system based on this and I'm very happy to have recent qt and stuff now again on my old i686 laptop!

[bobvanderlinden]

This is needed for #45960. Is uploading the tarballs blocking progress? Are there any other people, apart from @edolstra, able to upload them?

[woffs]

When things get automatically mirrored to tarballs.nixos.org once hydra built them, what is to be done to have the changes made in dezgeg@2dff64e available in official places? I found some docs by @domenkozar, maybe he wants to help.
And, maybe this has to be merged first?

[vcunat]

First should be tarball upload, then pushing the equivalent of dezgeg@2dff64e (with different URLs) to staging branch(es) to rebuild the world, and finally merging to the main branch(es).

[woffs]

Maybe I'm missing something, but why not push those two commits as-is to staging and change the URLs later? And, will this get into 18.09 at all?

[vcunat]

That would be possible, but not nice, and we might just forget to upload it for a very long time. 18.09 certainly won't contain this at release time (days away), but I think it might be possible to pick it later, as update of bootstrapping tools should be very non-intrusive (it's mainly a mass rebuild).

[dtzWill]

Rebased the glibc fix, since it had conflicts. Why do bootstrap tool bundles need updating for this, again?
I'm all in favor of doing so but not sure why the two are connected.

If/when we merge this we may wish to include immediately also pick the commit from @dezgeg that removes the treewide kludgery that works around the issues this resolves.

[dtzWill]

Found the commit, touched it up and added it.

[sg2002]

Any updates on this?

@dtzWill can you please rebase your branch, I've been trying to rebuild from it and got hit with #45902.

[worldofpeace]

webkitgtk is still built with clangStdenv even though now it does not fail to build with gcc stdenv.

Ahh great. I can open a PR to change this. I believe it was in GNOME 3.34 we had to do this (woefully).

[orivej]

To summarize, glibc still ships bootstrap libgcc_s:

$ patchelf --print-rpath $(nix-build --no-out-link '<nixpkgs>' -A glibc)/lib/libgcc_s.so.1
/nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-glibc-2.27/lib

but any final stage package should use libgcc_s from gccForLibs instead:

$ patchelf --print-rpath $(nix-build --no-out-link '<nixpkgs>' -A gccForLibs.lib)/lib/libgcc_s.so.1
/nix/store/2pi6zgkwnr3zdslvlv16nixpzvbyjx1n-glibc-2.31/lib
$ ldd $(nix-build --no-out-link '<nixpkgs>' -A lzip)/bin/lzip | grep gcc_s
libgcc_s.so.1 => /nix/store/2pi6zgkwnr3zdslvlv16nixpzvbyjx1n-glibc-2.31/lib/libgcc_s.so.1 (0x00007ff8ede7c000)

[worldofpeace]

Aargh, I seem to have used the wrong keyword. Sorry for closing 😁

[attila-lendvai]

FYI, i have arrived here while trying to use clang to build an i686 executable on an x86_64 platform. it's probably not directly related, but still, this might be useful input for you.

it's a project that can also be built without linking to libc (directly talking to the linux kernel), hence the crt1.o mess in the shell hook. the nix part can most probably be done better, but i'm new to nix.

lang -O3 -Qunused-arguments -m32 --target=i686-linux-gnu -o build/llvm-libc/i686-linux-gnu/eval0  build/llvm-libc/32bit-le/eval0.ll
warning: overriding the module target triple with i386-unknown-linux-gnu [-Woverride-module]
1 warning generated.
/nix/store/b10shv9yqbgps47y0n8x7l7bq8fmp1i6-binutils-2.31.1/bin/ld: skipping incompatible /nix/store/hq3ps9kak12kw08hgacpjr4y4i9ias2l-gcc-9.3.0/lib/gcc/x86_64-unknown-linux-gnu/9.3.0/libgcc.a when searching for -lgcc
/nix/store/b10shv9yqbgps47y0n8x7l7bq8fmp1i6-binutils-2.31.1/bin/ld: cannot find -lgcc
/nix/store/b10shv9yqbgps47y0n8x7l7bq8fmp1i6-binutils-2.31.1/bin/ld: skipping incompatible /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/libgcc_s.so.1 when searching for libgcc_s.so.1
/nix/store/b10shv9yqbgps47y0n8x7l7bq8fmp1i6-binutils-2.31.1/bin/ld: cannot find libgcc_s.so.1
/nix/store/b10shv9yqbgps47y0n8x7l7bq8fmp1i6-binutils-2.31.1/bin/ld: skipping incompatible /nix/store/hq3ps9kak12kw08hgacpjr4y4i9ias2l-gcc-9.3.0/lib/gcc/x86_64-unknown-linux-gnu/9.3.0/libgcc.a when searching for -lgcc
/nix/store/b10shv9yqbgps47y0n8x7l7bq8fmp1i6-binutils-2.31.1/bin/ld: cannot find -lgcc
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

my shell.nix:

{ pkgs            ? import <nixpkgs> {},
  pkgsx86_64Linux ? import <nixpkgs> { system = "x86_64-linux"; },
  pkgsi686Linux   ? import <nixpkgs> { system = "i686-linux"; }
}:

pkgs.multiStdenv.mkDerivation {
  name = "maru";

  buildInputs = with pkgs;
    [
      coreutils time ncurses posix_man_pages bash-completion less
      gitFull diffutils
      gnumake
      gdb binutils
      llvm_11 clang_11

      # keep this line if you use bash
      bashInteractive
    ];

  # TODO this doesn't work: make TARGET_CPU=i686 TARGET_VENDOR=linux TARGET_OS=gnu test-bootstrap-llvm eval-llvm || beep
  # https://github.com/NixOS/nixpkgs/issues/36947
  # Programs should be finding it from ${gcc.lib}/lib/libgcc.so
  # 
  # NIX_LD_LIBRARY_PATH = pkgs.lib.makeLibraryPath (with pkgsi686Linux; [
  #   libgcc.out
  # ]);
  # NIX_LDFLAGS_FOR_TARGET = "-L ${pkgsi686Linux.libgcc.out}/lib/ -L ${pkgsi686Linux.glibc.out}/lib/";

  shellHook = ''
    mkdir -p build
    # KLUDGE? or not? is there a better way?
    mkdir -p build/x86-linux/i386-linux-gnu/ \
             build/llvm-linux/x86_64-linux-gnu/ \
             build/llvm-linux/i686-linux-gnu/
    cp ${pkgsi686Linux.glibc}/lib/crt1.o build/x86-linux/i386-linux-gnu/
    cp ${pkgsi686Linux.glibc}/lib/crt1.o build/llvm-linux/i686-linux-gnu/
    cp ${pkgsx86_64Linux.glibc}/lib/crt1.o build/llvm-linux/x86_64-linux-gnu/
    chmod +w build/x86-linux/i386-linux-gnu/* build/llvm-linux/x86_64-linux-gnu/* build/llvm-linux/i686-linux-gnu/*
  '';
}

[stale]

I marked this as stale due to inactivity. → More info

[Ericson2314]

#132343 should help.

[queezle42]

I stumbled on this while debugging #136574. The problem is, that recent GCCs add more symbols to libgcc_s which are not available in the version copied to glibc. In particular this breaks usage of atomics on aarch64, which breaks llvmPackages_13 and chromium. The linker fails with missing symbols, so patchelf is not an option.

A previous fix has removed libgcc from glibc which would solve the problem but was reverted because it caused other issues.

We (@BBBSnowball and I) have explored another option: Instead of removing libgcc_s, we have replaced it with the correct version whenever the compiler is replaced in the stdenv. It needs more polish (e.g. it does not fix it for the normal stdenv right now), but it works for llvm: queezle42@2cdd8e4

[georgefst]

Is there any sort of workaround available here?

I've become blocked on needing Chromium on my RPi, which is marked broken due to #136629. If I've read everything correctly, this appears to be the underlying issue.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/chromium-on-rpi-workaround/18557/1

[tobim]

Can this be closed?

[vcunat]

Yes, I believe so.