Build script not rerun when target rustflags change

[Nemo157]

Problem

Some crates such as anyhow utilize the CARGO_ENCODED_RUSTFLAGS to customize their build. When a build is performed with only the target rustflags changed the build script does not get rerun, so the build customization from it is inconsistent with the build environment for the library itself.

Steps

> cargo dl -e anyhow@=1.0.75 && cd anyhow-1.0.75
                          anyhow@=1.0.75   extracted anyhow 1.0.75 to anyhow-1.0.75

> cargo build --target x86_64-unknown-linux-gnu -v
   Compiling anyhow v1.0.75 (/tmp/tmp.7WdQVxA9Y6/anyhow-1.0.75)
     Running `rustc --crate-name build_script_build --edition=2018 build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=201 --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=ec4d899bd2351612 -C extra-filename=-ec4d899bd2351612 --out-dir /run/user/1000/cargo-home/target/shared/debug/build/anyhow-ec4d899bd2351612 -C incremental=/run/user/1000/cargo-home/target/shared/debug/incremental -L dependency=/run/user/1000/cargo-home/target/shared/debug/deps`
     Running `/run/user/1000/cargo-home/target/shared/debug/build/anyhow-ec4d899bd2351612/build-script-build`
     Running `rustc --crate-name anyhow --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=201 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=7ea480d8b9426c5d -C extra-filename=-7ea480d8b9426c5d --out-dir /run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -C incremental=/run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/incremental -L dependency=/run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/deps -L dependency=/run/user/1000/cargo-home/target/shared/debug/deps --cfg backtrace`
    Finished dev [unoptimized + debuginfo] target(s) in 0.46s

> RUSTFLAGS=-Zallow-features= cargo build --target x86_64-unknown-linux-gnu -v
       Dirty anyhow v1.0.75 (/tmp/tmp.7WdQVxA9Y6/anyhow-1.0.75): the rustflags changed
   Compiling anyhow v1.0.75 (/tmp/tmp.7WdQVxA9Y6/anyhow-1.0.75)
     Running `rustc --crate-name anyhow --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=201 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=7ea480d8b9426c5d -C extra-filename=-7ea480d8b9426c5d --out-dir /run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -C incremental=/run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/incremental -L dependency=/run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/deps -L dependency=/run/user/1000/cargo-home/target/shared/debug/deps -Zallow-features= --cfg backtrace`
error[E0725]: the feature `error_generic_member_access` is not in the list of allowed features
   --> src/lib.rs:214:32
    |
214 | #![cfg_attr(backtrace, feature(error_generic_member_access))]
    |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0658]: use of unstable library feature 'error_generic_member_access'
 --> src/context.rs:7:5
  |
7 | use std::error::Request;
  |     ^^^^^^^^^^^^^^^^^^^
  |
  = note: see issue #99301 <https://github.com/rust-lang/rust/issues/99301> for more information
  = help: add `#![feature(error_generic_member_access)]` to the crate attributes to enable

...

error: could not compile `anyhow` (lib) due to 24 previous errors

Caused by:
  process didn't exit successfully: `rustc --crate-name anyhow --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=201 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=7ea480d8b9426c5d -C extra-filename=-7ea480d8b9426c5d --out-dir /run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -C incremental=/run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/incremental -L dependency=/run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/deps -L dependency=/run/user/1000/cargo-home/target/shared/debug/deps -Zallow-features= --cfg backtrace` (exit status: 1)

> cargo clean
     Removed 138 files, 20.5MiB total

> RUSTFLAGS=-Zallow-features= cargo build --target x86_64-unknown-linux-gnu -v
   Compiling anyhow v1.0.75 (/tmp/tmp.7WdQVxA9Y6/anyhow-1.0.75)
     Running `rustc --crate-name build_script_build --edition=2018 build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=201 --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=ec4d899bd2351612 -C extra-filename=-ec4d899bd2351612 --out-dir /run/user/1000/cargo-home/target/shared/debug/build/anyhow-ec4d899bd2351612 -C incremental=/run/user/1000/cargo-home/target/shared/debug/incremental -L dependency=/run/user/1000/cargo-home/target/shared/debug/deps`
     Running `/run/user/1000/cargo-home/target/shared/debug/build/anyhow-ec4d899bd2351612/build-script-build`
     Running `rustc --crate-name anyhow --edition=2018 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=201 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=7ea480d8b9426c5d -C extra-filename=-7ea480d8b9426c5d --out-dir /run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -C incremental=/run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/incremental -L dependency=/run/user/1000/cargo-home/target/shared/x86_64-unknown-linux-gnu/debug/deps -L dependency=/run/user/1000/cargo-home/target/shared/debug/deps -Zallow-features=`
    Finished dev [unoptimized + debuginfo] target(s) in 0.40s

Possible Solution(s)

It seems like Cargo's input environment variables to the build-script-execution should be part of its fingerprint, since by my understanding it's not possible for the build script to use rerun-if-env-changed with these variables.

Notes

Without --target the build-script-build is influenced by the RUSTFLAGS and so implicitly the build-script-execution is rerun after rebuilding the script itself.

Version

cargo 1.76.0-nightly (6790a5127 2023-11-10)
release: 1.76.0-nightly
commit-hash: 6790a5127895debec95c24aefaeb18e059270df3
commit-date: 2023-11-10
host: x86_64-unknown-linux-gnu
libgit2: 1.7.1 (sys:0.18.1 vendored)
libcurl: 8.4.0-DEV (sys:0.4.68+curl-8.4.0 vendored ssl:OpenSSL/1.1.1u)
ssl: OpenSSL 1.1.1u  30 May 2023
os: NixOS 23.5.0 [64-bit]

[heisen-li]

@rustbot claim

[epage]

From #13502

If I'm reading this correctly, we are blindly adding the target rustflags to the fingerprint for build script units. This will cause build scripts to rebuild/re-run more often than is needed.

We should be limiting this to when the build script cares about it.

#13560

If you want to limit the environment variables to those required by the build script, then you need to determine which environment variables are required by the build script.

I haven't come up with a better solution other than progressive scanning (and I don't think this is the right kind of solution), and it's harder than I thought it would be if I wanted to do it.

Also, more information about whether unnecessary reconstruction occurred might have been better.

Honorable @weihanglo @epage

@heisen-li bringing this back here because this is more a question of problem and design, rather than implementation. For me, the first step would be to dig into this part of the Issue:

since by my understanding it's not possible for the build script to use rerun-if-env-changed with these variables.

I admit, I'm not aware of what this is talking about but we should make sure we understand that so we can consider all options for addressing this issue.

[Nemo157]

The docs for rerun-if-env-changed says:

Note that the environment variables here are intended for global environment variables like CC and such, it is not possible to use this for environment variables like TARGET that Cargo sets for build scripts. The environment variables in use are those received by cargo invocations, not those received by the executable of the build script.

CARGO_ENCODED_RUSTFLAGS is of the latter type, env-vars set by Cargo for the build script.

[epage]

So our options are

• Always re-run host build scripts when the target-platform env changes, to support this
• Reject this, saying that the build script shouldn't care
• Allow people to tell us what to do with re-run-if-env-changed.

@ehuss when you marked this as S-accepted, did you have a particular approach in mind?

[ehuss]

Sorry I wasn't clear about what was expected.

I was expecting that it would re-run the build script whenever the CARGO_ENCODED_RUSTFLAGS value changes.

Requiring the user to set rerun-if-env-changed is an option, but it might be a bit of a footgun, since we don't require that for any other environment variable (AFAIK.. not counting CARGO_MAKEFLAGS which shouldn't matter).

[epage]

In most of those cases, the variable is assumed to be a dependency. Here, we are depending on something that is technically unrelated to the actual build unit which feels wrong. However, the over-builds from this are not likely to be a problem so not willing to block this over it.