Provide a way for build scripts to invoke rustc the way cargo would

[RalfJung]

Problem

It is a common pattern in build scripts to run a rustc 'build probe': compile some code with rustc, see if that works, and then enable some cfg flags depending on the result. Crates use this to automatically detect if some API they optionally use is available on the current compiler. Examples of this include anhow, thiserror, eyre, and the autocfg crate which abstracts that pattern into a library.

However, invoking rustc in the right way is non-trivial. build scripts have to remember to check RUSTC and RUSTC_WRAPPER, to pass on the RUSTFLAGS (using CARGO_ENCODED_RUSTFLAGS where possible so spaces are preserved properly), and to set --target based on the TARGET env var. And even then the final rustc invocation is not always the same as what cargo itself uses, causing issues like the one described here: it is impossible to tell, from the information a build script has, whether cargo will pass --target to rustc or not, and there are some situations (rustc bootstrap and Miri, for instance) where this really matters. As a consequence, the aforementioned crates sometimes break in rustc bootstrap and Miri when they are used as dependency of a proc macro.

Proposed Solution

Cargo should just tell the build script how it will invoke rustc. I'm imagining an env var like CARGO_ENCODED_RUSTC (name to be bikeshed), which build scripts could use like this:

use std::path::Path;
use std::env;
use std::fs;
use std::process::Command;

fn probe(probe: &str) -> bool {
    let out_dir = env::var_os("OUT_DIR").unwrap();
    let probefile = Path::new(&out_dir).join("probe.rs");
    fs::write(&probefile, probe).expect("failed to write in OUT_DIR");

    let rustc = env::var("CARGO_ENCODED_RUSTC").unwrap();
    let mut rustc = rustc.split('\x1f');
    let mut cmd = Command::new(rustc.next().unwrap());
    cmd
        .args(rustc)
        .arg("--crate-name=build_probe")
        .arg("--crate-type=lib")
        .arg("--emit=metadata")
        .arg("--out-dir")
        .arg(out_dir)
        .arg(probefile);
    cmd.status().expect("failed to invoke rustc").success()
}

Notes

Does this need an RFC?

[RalfJung]

With issues like rust-lang/rust#114839, I am not even sure any more if merely providing all the right arguments to rustc is sufficient. If the build script invokes rustc, that also means cargo is unable to track the dependencies of that build in the usual way -- so even if some dependency used during the probe build changes, cargo won't know to re-run the build script.

Maybe there is a way this can be resolved with more flags though, i.e. if the build script instructed rustc to put build-dep information into a particular location and then cargo would pick it up there -- then the CARGO_ENCODED_RUSTC interface could still work.

[weihanglo]

A solution might be #12432, if there is an API can give you the ability of "build probe". And people might be happy and avoid re-implement argparse for rustc.

[epage]

Sounds like options include

• Cargo providing some solution
• The build script API capturing the generic "call rustc" pattern
• The build script API more encapsulating the build probe pattern

I'd like to throw out another one: we focus on #[cfg(accessible)]. That seems like the best long-term strategy, for what it can encompass. For things not covered by it, I suspect we should explore what those should be.

[RalfJung]

cfg(accessible) does not help when a nightly API changes its type. Then the item might be "accessible" but the crate will still fail to build. Build probes test more than just whether the item exists.

• The build script API capturing the generic "call rustc" pattern
• The build script API more encapsulating the build probe pattern

These will require cooperation with cargo, since currently build scripts just do not have the information required to generate the right rustc invocation.

[epage]

How big of a use case is supporting multiple flavors of a nightly feature? To me, that sounds like a relatively small use case. Could you expand your uses for doing that?

[RalfJung]

How big of a use case is supporting multiple flavors of a nightly feature? To me, that sounds like a relatively small use case. Could you expand your uses for doing that?

I did not talk about supporting multiple flavors of a nightly feature. I talked about what happens when a nightly feature changes the type of an item. Code written cfg(accessible) will now fail to build, because the item still exists but has the wrong type. The entire point of build probes is to avoid that situation, where when the nightly feature changes in a way that client code needs to be adjusted, it just gets disabled.

[epage]

For myself, I've grown increasingly frustrated with build probes being used in this way (implicit opt-in to nightly features) to the point that I feel its an anti-pattern. Also been discussing that in dtolnay/anyhow#323.

[RalfJung]

I share some of that frustration. But what's even more frustrating is having to deal with build probes that don't work properly.^^ The build probes exist and so these issues show up, e.g. as Miri bugreports or as hard-to-debug failures in rustc bootstrapping when a nightly API changes. Wishing the build probes away won't fix those issues...

[epage]

We talked about this in today's cargo meeting. The use cases seem to be focused on build probes. We are not inclined to add features specifically for build probes at this time but would instead encourage use of

• cfg(accessible) and cfg(version) where possible (interest from t-lang to stabilize, just needs a champion to check it over and drive stabilization)
• features everywhere else (or maybe even global features), including enabling of nightly features

While cfg(accessible), cfg(version), and global features would involve MSRV bumps, I suggest considering whether the soon-to-stabilize MSRV-aware resolver might be sufficient to re-evaluate an MSRV policy in light of a feature like this. For a foundational package, one could bump minor and leave room for critical patches of old versions while allowing new versions to use these features.

[cuviper]

While cfg(accessible), cfg(version), and global features would involve MSRV bumps, I suggest considering whether the soon-to-stabilize MSRV-aware resolver might be sufficient to re-evaluate an MSRV policy in light of a feature like this.

The MSRV-aware resolver only helps once you've transitioned to an MSRV high enough to always have that resolver. I suppose if that comes before those other features, then that will indeed add some leeway and allow policy adjustment.

[epage]

The MSRV-aware resolver only helps once you've transitioned to an MSRV high enough to always have that resolver. I suppose if that comes before those other features, then that will indeed add some leeway and allow policy adjustment.

I'm assuming the standard workflow is that of running stable for development and then testing against MSRV. Even if you develop on your MSRV, all you need is to run stable to resolve your Cargo.lock. No MSRV bump is needed if you opt-in via config rather than package.resolver. I know of people who run nightly to get an MSRV aware resolver to run their deps on stable or their MSRV today.

[RalfJung]

cfg(accessible) and cfg(version) where possible (interest from t-lang to stabilize, just needs a champion to check it over and drive stabilization)

This doesn't satisfy many of the cases people use build probes for.

features everywhere else (or maybe even global features), including enabling of nightly features

This would need a concerted effort by the project to convince the community of this. Right now, the status quo is that we have a lot of bad build probes out there, and they are unlikely to go anywhere. They cause trouble for cross-compilation, Miri, rsutc bootstrap, -Zbuild-std -- just a whole lot of pain that many people have to deal with. We have even more code that doesn't even use a build probe but just blindly enables features on nightly, which is causing even more pain -- we can't change some nightly-only aspects of rustc any more without causing major disruption, which completely subverts the point of features being nightly-only. For the sake of keeping the rustc development process functional, such code should be migrated to using build probes, making the problems mentioned above even worse.

There is clearly a need for this. We can either satisfy that need, or we don't and then we suffer from the hacks people use when they satisfy their need their own way. This is already a real problem today, and it's only going to get worse.

[RalfJung]

Global features sound nice; if/when that happens, maybe we can convince crate authors to gate some of these things behind a global nightly flag or so. But I we'll probably have to deal with some maintainers that will refuse to use this flag...

[hkBst]

@epage I have my own reasons for wanting to disable autodetection of nightly sometimes, so I wrote a small utility to override version-based detection. Let me know if it is useful for you.