x.py can't build individual crates

[jessicah]

I'm trying to test building just librustc_llvm for the purposes of git bisecting, but am getting the following error:

./x.py build --stage 1 src/librustc_llvm
Warning: no rules matched /home/jessicah/rust/src/librustc_llvm

[Mark-Simulacrum]

The logic in src/bootstrap/compile.rs needs to be include the crate being passed. Similar logic can be found in check.rs, specifically within make_run and during the build itself. Let me know if you need any assistance!

[rozuur]

can i work on this?

[tamird]

I took a look at this - it strikes me as not as easy as @Mark-Simulacrum's description.

TL;DR we need a type that implements Step whose should_run returns true when given a crate path. Then that type needs to be registered in this macro invocation.

It's possible this can be done by modifying an existing type, but I didn't find a suitable one.

[Mark-Simulacrum]

The logic will want to be similar to what is in test.rs:

rust/src/bootstrap/test.rs

Lines 1333 to 1344 in b7da1aa

	fn should_run(mut run: ShouldRun) -> ShouldRun {
	let builder = run.builder;
	run = run.krate("test");
	for krate in run.builder.in_tree_crates("std") {
	if krate.is_local(&run.builder) &&
	!krate.name.contains("jemalloc") &&
	!(krate.name.starts_with("rustc_") && krate.name.ends_with("san")) &&
	krate.name != "dlmalloc" {
	run = run.path(krate.local_path(&builder).to_str().unwrap());
	}
	}
	run

. Something like that should work for the compile steps, though we'd probably want to make sure the default is (unlike for tests) to build everything instead of a step-by-step build. This might be somewhat difficult to implement with the current design; I think it'd involve having the first call on run be something like path("default-path")...

[jyn514]

I have a patch for this, but I feel a little discouraged about how much duplicate code it introduces ... anyway, I'll make a PR for it soonish if I can't think of something better. I think I could make it a little better at least by introducing a path_to_crate function that does the same thing as the in_trees_crates loop but caches it all at startup.

diff --git a/src/bootstrap/builder.rs b/src/bootstrap/builder.rs
index e7511888114..a9c2d38b644 100644
--- a/src/bootstrap/builder.rs
+++ b/src/bootstrap/builder.rs
@@ -431,7 +431,7 @@ fn parse(string: &str) -> Kind {
         }
     }
 
-    fn as_str(&self) -> &'static str {
+    pub(crate) fn as_str(&self) -> &'static str {
         match self {
             Kind::Build => "build",
             Kind::Check => "check",
@@ -456,6 +456,8 @@ fn get_step_descriptions(kind: Kind) -> Vec<StepDescription> {
         }
         match kind {
             Kind::Build => describe!(
+                compile::Crate,
+                compile::CrateLibrustc,
                 compile::Std,
                 compile::Assemble,
                 compile::CodegenBackend,
diff --git a/src/bootstrap/compile.rs b/src/bootstrap/compile.rs
index 00fc1f04342..aa6429e6b80 100644
--- a/src/bootstrap/compile.rs
+++ b/src/bootstrap/compile.rs
@@ -517,6 +517,115 @@ fn run(self, builder: &Builder<'_>) -> Vec<(PathBuf, DependencyType)> {
     }
 }
 
+#[derive(Debug, PartialOrd, Ord, Copy, Clone, PartialEq, Eq, Hash)]
+pub struct CrateLibrustc {}
+impl Step for CrateLibrustc {
+    type Output = ();
+    const DEFAULT: bool = true;
+    const ONLY_HOSTS: bool = true;
+
+    fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
+        run.krate("rustc-main")
+    }
+
+    fn make_run(run: RunConfig<'_>) {
+        let builder = run.builder;
+        let host = builder.config.build;
+        let compiler = builder.compiler_for(builder.top_stage, host, host);
+
+        for krate in builder.in_tree_crates("rustc-main", Some(run.target)) {
+            if krate.path.ends_with(&run.path) {
+                builder.ensure(Crate {
+                    compiler,
+                    target: run.target,
+                    mode: Mode::Rustc,
+                    krate: krate.name,
+                });
+            }
+        }
+    }
+
+    fn run(self, _: &Builder<'_>) {
+        unreachable!()
+    }
+}
+
+/// Build a single crate using cargo.
+///
+/// FIXME: once test flags are unified under `builder.cargo`,
+/// unify this with `test::Crate`.
+#[derive(Debug, PartialOrd, Ord, Copy, Clone, PartialEq, Eq, Hash)]
+pub struct Crate {
+    pub target: TargetSelection,
+    pub compiler: Compiler,
+    pub mode: Mode,
+    pub krate: Interned<String>,
+}
+
+impl Step for Crate {
+    type Output = ();
+    const DEFAULT: bool = true;
+    const ONLY_HOSTS: bool = true;
+
+    fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
+        run.krate("test")
+    }
+
+    fn make_run(run: RunConfig<'_>) {
+        let builder = run.builder;
+        let host = builder.config.build;
+        let compiler = builder.compiler_for(builder.top_stage, host, host);
+
+        for krate in builder.in_tree_crates("test", Some(run.target)) {
+            if krate.path.ends_with(&run.path) {
+                builder.ensure(Crate {
+                    compiler,
+                    target: run.target,
+                    // NOTE: ok to hard-code `Std`, even though `CrateRustc` uses `ensure(Crate)`,
+                    // because `make_run` is only called for dependencies of `test`.
+                    mode: Mode::Std,
+                    krate: krate.name,
+                });
+            }
+        }
+    }
+
+    fn run(self, builder: &Builder<'_>) {
+        let Crate { compiler, mode, target, krate } = self;
+
+        let mut cargo =
+            builder.cargo(compiler, mode, SourceType::InTree, target, builder.kind.as_str());
+        let stamp = match mode {
+            Mode::Std => {
+                std_cargo(builder, target, compiler.stage, &mut cargo);
+                libstd_stamp(builder, compiler, target)
+            }
+            Mode::Rustc => {
+                builder.ensure(Std { compiler, target });
+                rustc_cargo(builder, &mut cargo, target);
+                librustc_stamp(builder, compiler, target)
+            }
+            _ => panic!("crate-specific builds only supported for library and compiler"),
+        };
+
+        cargo.arg("-p").arg(krate);
+
+        builder.info(&format!(
+            "{} {} stage{} ({} -> {})",
+            builder.kind.as_str(), krate, compiler.stage, &compiler.host, target
+        ));
+
+        run_cargo(
+            builder,
+            cargo,
+            vec![],
+            &stamp,
+            vec![],
+            false,
+        );
+    }
+}
+

[krasimirgg]

We started seeing a strange error to find the proc-macro crate during stage1 bootstrapping starting at this rust commit 0a7f2c3.
The command we're using is python3 x.py build library/std compiler/rustc src/tools/cargo.
The errors look like this:

...
Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
...
error[E0463]: can't find crate for `proc_macro`
   --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro-hack-0.5.19/src/lib.rs:150:1
    |
150 | extern crate proc_macro;
    | ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

error[E0463]: can't find crate for `proc_macro`
   --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.37/src/lib.rs:119:1
    |
119 | extern crate proc_macro;
    | ^^^^^^^^^^^^^^^^^^^^^^^^ can't find crate

   Compiling unic-emoji-char v0.9.0
error[E0463]: can't find crate for `proc_macro`

error[E0635]: unknown feature `proc_macro_span`
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-macro2-1.0.37/src/lib.rs:90:59
   |
90 | #![cfg_attr(any(proc_macro_span, super_unstable), feature(proc_macro_span))]
   |                                                           ^^^^^^^^^^^^^^^

This is currently affecting our experimental rust + llvm@head build bot (https://buildkite.com/llvm-project/rust-llvm-integrate-prototype/builds/11778#0181d3d0-a73a-4852-8a31-edcc608fbb6f), but this error seems more general. I've set up a reproducer on Ubuntu in Docker over at https://github.com/krasimirgg/bootstrap-proc-macro-issue, which triggers this error via python3 x.py build library/std compiler/rustc src/tools/cargo using freshly imported rust.

It seems like python3 x.py build library/std compiler/rustc triggers this error; running python3 x.py build library/std or python3 x.py build compiler/rustc by themselves works.

[jyn514]

@krasimirgg build compiler/rustc implies build library/std; there should be no reason to use both in the same command. The error you're running into is since #95502 only std is being built, instead of all standard library crates (proc_macro doesn't depend on std).

That said, this is still a bug, compiler/rustc should be building the crates it cares about.