Cross compilation from x86_64 to x86_64 with custom gcc arch/tune fails

[jys1670]

Steps to reproduce

I have small VPS which could benefit greatly from AES-NI and other fancy instructions. It can't be used for building due to load limitations, so I've attempted to do compilation on local machine and push that on remote target:

nixos-rebuild --flake .#myServer \
  --target-host servername \
  --build-host localhost \
  boot

Wiki suggest tweaking configuration for such scenario in this way:

nixpkgs = {
    # building ON
    localSystem = {
        system = "x86_64-linux";
    };
    # building FOR
    crossSystem = {
        gcc.arch = "znver2";
        gcc.tune = "znver2";
        system = "x86_64-linux";
    };
};

However it fails while trying to compile bash:

checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu

Beginning configuration for bash-5.2-release for x86_64-unknown-linux-gnu

checking for x86_64-unknown-linux-gnu-gcc... x86_64-unknown-linux-gnu-gcc
checking whether the C compiler works... no
configure: error: in `/build/bash-5.2':
configure: error: C compiler cannot create executables

After that I've tried alternative approach that doesn't touch localSystem / crossSystem and modifies compiler wrapper instead:

crossOverlays = [
    (final: prev: {
        wrapCC = cc: prev.wrapCCWith {
            inherit cc;
            extraBuildCommands = ''
                echo "-O3 -march=znver2 -mtune=znver2" >> "$out/nix-support/cc-cflags"
            '';
        };
    })
];

Sadly that failed on python (bash worked though):

Found duplicated packages in closure for dependency 'setuptools':
        setuptools 68.2.2.post0 (/nix/store/rk7kwr8c0v8dwk64442q8c0lp7car7qn-python3.11...
        setuptools 68.2.2.post0 (/nix/store/y8yq448xka665bqyd5gvdhajqdbsnyys-python3.11...

Notify maintainers

I've seen @amjoseph-nixpkgs working on cross compilation a lot, I assume it's his area of expertise

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.62, NixOS, 23.11 (Tapir), 23.11.20231117.c757e9b`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `""`
 - nixpkgs: `/nix/store/rj6pqzx66pliiwdfk2wjnnlk40rm3mpd-source`

[Artturin]

Dupe of #243164

localSystem should work

https://github.com/search?q=%2Fgcc%5C.tune%2F+lang%3Anix+NOT+is%3Afork&type=code

There's a simpler way to add flags than what you're doing

nixpkgs/pkgs/stdenv/adapters.nix

Line 287 in 4ccb3e0

withCFlags = compilerFlags: stdenv:

https://github.com/search?q=withCFlags+lang%3Anix+NOT+is%3Afork&type=code

[jys1670]

@Artturin stdenv adapters stuff seems to be broken. Doing it this way in latest nixos-unstable (global overlay):

self: super: {
    stdenv = super.withCFlags [ "-O3" "-march=znver2" "-mtune=znver2" "-flto" ] super.stdenv;
}

always results in error

error: The ‘env’ attribute set cannot contain any attributes passed to derivation. The following attributes are overlapping: NIX_CFLAGS_COMPILE`

I've tried other adapters mentioned here, it's all the same

As for localSystem, I think last time I tried it just built optimized packages locally, but normal packages were copied to target. Same goes for overlays (that's why crossOverlays). It could be that I simply forgot to reboot though, I am going to retest.

But anyways, man nixos-rebuild has such line:

--target-host host
...
Note that nixos-rebuild honors the nixpkgs.crossSystem setting of the given configuration but disregards the  true  architecture  of  the  target host. Hence the nixpkgs.crossSystem setting has to match the target platform or else activation will fail.

It implies that crossSystem should be used, not localSystem.

[Artturin]

@Artturin stdenv adapters stuff seems to be broken. Doing it this way in latest nixos-unstable (global overlay):

self: super: {
    stdenv = super.withCFlags [ "-O3" "-march=znver2" "-mtune=znver2" "-flto" ] super.stdenv;
}

always results in error

error: The ‘env’ attribute set cannot contain any attributes passed to derivation. The following attributes are overlapping: NIX_CFLAGS_COMPILE`

This is because something is setting NIX_CFLAGS_COMPILE instead of env.NIX_CFLAGS_COMPILE, it would probably be possible to choose which one to use with some conditionals, but I'm not inclined to do it because people should switch to using the env attrset for environmental variables within builds to support structuredAttrs.

Found a issue #269539

Answer before realizing the error was about overlapping attrs

Works fine here

nix build --expr '(import ./. { overlays = [(self: super: {stdenv = super.withCFlags [ "-O3" "-march=znver2" "-mtune=znver2" "-flto" ] super.stdenv;  })]; }).zsh' --impure --keep-failed

because toString is run on it

nixpkgs/pkgs/stdenv/adapters.nix

Line 290 in 3ccf65f

env = (args.env or {}) // { NIX_CFLAGS_COMPILE = toString (args.env.NIX_CFLAGS_COMPILE or "") + " ${toString compilerFlags}"; };

however it fails with stuff like

expand-response-params> cc1: error: bad value ('znver2') for '-march=' switch
expand-response-params> cc1: note: valid arguments to '-march=' switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client icelake-server bonnell atom silvermont slm knl knm x86-64 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 btver1 btver2 native; did you mean 'znver1'?

(likely related #253713)

To avoid the bootstrapping stuff do import ./. { config = { replaceStdenv = ({ pkgs }: pkgs.withCFlags [ "-O3" "-march=znver2" "-mtune=znver2" "-flto" ] pkgs.stdenv); }; };

This still fails with errors like

> g++ -g -O2  -o gperf version.o positions.o options.o keyword.o keyword-list.o input.o bool-array.o hash-table.o search.o output.o main.o ../lib/libgp.a -lm
> /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: /build/ccPAfibH.ltrans0.ltrans.o: in function `Input::read_input()':
> /build/gperf-3.1/src/./input.cc:254: undefined reference to `get_delim(char**, unsigned long*, int, _IO_FILE*)'
> /nix/store/1fn92b0783crypjcxvdv6ycmvi27by0j-binutils-2.40/bin/ld: /build/ccPAfibH.ltrans0.ltrans.o: in function `Hash_Table::insert(KeywordExt*)':
> /build/gperf-3.1/src/./hash-table.cc:143: undefined reference to `hashpjw(unsigned char const*, unsigned int)'