Consolidate staging for `rustc_private` tools #144303

Kobzol · 2025-07-22T12:05:51Z

This PR continues bootstrap refactoring, this time by consolidating staging for Mode::ToolRustc tools. This refactoring was in the critical path of refactoring test/dist/clippy/doc steps, and getting rid of the rmeta/rlib sysroot copy, because tools are pervasive and they are being used for a lot of things in bootstrap.

The main idea is to explicitly model the fact that a stage N Mode::ToolRustc tool always works with two different compilers:

Stage N-1 rustc (build_compiler) builds stage N rustc (target_compiler)
Rlib artifacts from stage N rustc are copied to the sysroot of stage N-1 rustc
Stage N-1 rustc builds the (stage N) tool itself, the tool links to the rlib artifacts of the stage N rustc

Before, the code often used compiler, which meant sometimes the build compiler, sometimes the target compiler, and sometimes neither (looking at you, download-rustc). This is especially annoying when you get to a situation where you have an install step that invokes a dist step that invokes a tool build step, where some compiler is being propagated through, without it being clear what does that compiler represent. This refactoring hopefully makes that clearer and more explicit. It also gets rid of a few builder.ensure(Rustc(...)) calls within bootstrap, which is always nice.

Rustdoc needs to be handled a bit specially, because it acts as a compiler itself, I documented that in the changes.

It wasn't practical to do these refactorings in multiple PRs, so I did it all in one PR. The meat of the change is 9ee6d1c.

I tested manually that x build rustdoc and x build miri still works even with download-rustc, although I cannot promise any extra support for download-rustc, IMO we will just have to reimplement it from scratch in a different way.

As usually, I did some drive-by refactorings to bootstrap, trying to document and clarify things, add more step metadata and tests.

Since these changes broke Cargo, which was incorrectly using Mode::ToolRustc, I also changed cargo to ToolTarget in this PR.

Best reviewed commit-by-commit (note that I renamed link_compiler to target_compiler, in accordance to the rest of bootstrap, in the last commit).

r? @jieyouxu

try-job: x86_64-gnu-aux

rustbot · 2025-07-22T12:05:57Z

jieyouxu is not on the review rotation at the moment.
They may take a while to respond.

bors · 2025-07-23T14:01:35Z

☔ The latest upstream changes (presumably #143412) made this pull request unmergeable. Please resolve the merge conflicts.

To clarify what it does.

…` tools

jieyouxu · 2025-07-25T09:03:03Z

Will start looking at this today, might take me some time to get through.

jieyouxu

Thanks, looks good overall. You can r=me after comment nit.

src/bootstrap/src/core/build_steps/tool.rs

src/bootstrap/src/core/builder/tests.rs

jieyouxu · 2025-07-28T09:59:32Z

@rustbot author

Kobzol · 2025-07-28T10:08:24Z

@bors r=jieyouxu

bors · 2025-07-28T10:08:26Z

📌 Commit 7bf4a1a has been approved by jieyouxu

It is now in the queue for this repository.

rust-bors · 2025-07-29T06:46:32Z

💔 Test failed (CI). Failed jobs:

bors build finished (web logs, extended logs)
try - x86_64-gnu-aux (web logs, extended logs)

Kobzol · 2025-07-29T08:25:37Z

Uaah, some issues with cargo. I had a branch stacked on top of this that also fixes up Cargo, but there were some additional issues with cargo tests.. well, let's land it here, these changes are really hard to do piecemeal..

Kobzol · 2025-07-29T08:31:31Z

@bors try

rust-bors · 2025-07-29T08:31:35Z

⌛ Trying commit c4fef48 with merge 2802fd0…

To cancel the try build, run the command @bors try cancel.

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux

rust-bors · 2025-07-29T09:04:54Z

💔 Test failed (CI). Failed jobs:

bors build finished (web logs, extended logs)
try - x86_64-gnu-aux (web logs, extended logs)

Kobzol · 2025-07-29T11:34:14Z

@bors try

rust-bors · 2025-07-29T11:34:18Z

⌛ Trying commit 0438a46 with merge 2f3f0ce…

To cancel the try build, run the command @bors try cancel.

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux

rust-log-analyzer · 2025-07-29T11:42:02Z

The job x86_64-gnu-aux failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

rust-bors · 2025-07-29T11:42:12Z

💔 Test failed (CI). Failed jobs:

bors build finished (web logs, extended logs)
try - x86_64-gnu-aux (web logs, extended logs)

Kobzol · 2025-07-29T11:44:35Z

@bors try

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux

rust-bors · 2025-07-29T11:44:39Z

⌛ Trying commit f731cd1 with merge 180dfb2…

To cancel the try build, run the command @bors try cancel.

rust-bors · 2025-07-29T13:38:11Z

☀️ Try build successful (CI)
Build commit: 180dfb2 (180dfb2915c406778b9b7484f8ac8e564f5a0d5d, parent: 552904134b564a74489db50aebe7070fdcce895c)

Kobzol · 2025-07-29T13:43:16Z

@rustbot ready

jieyouxu

I have some questions regarding cargo...

jieyouxu · 2025-07-30T07:09:48Z

src/bootstrap/src/core/build_steps/test.rs

+        // FIXME: support testing stage 0 cargo?
+        assert!(run.builder.top_stage > 0);


Question: should we just exit here (gracefully) instead of an assert panic? Or rather, declare this case unsupported, as otherwise the cargo test suite would have to work with both possibly-beta rustc and in-tree rustc?

src/bootstrap/src/core/builder/tests.rs

jieyouxu · 2025-07-30T07:17:24Z

src/bootstrap/src/core/build_steps/tool.rs

+            // Cargo is compilable with a stable compiler, but since we run in bootstrap,
+            // with RUSTC_BOOTSTRAP being set, some "clever" build scripts enable specialization
+            // based on this, which breaks stuff. We thus have to explicitly allow these features
+            // here.
+            allow_features: "min_specialization,specialization",


😭 Do we know which ones? ahash?

Yup, ahash and friends.

jieyouxu · 2025-07-30T07:20:14Z

src/bootstrap/src/core/builder/tests.rs

+        [build] rustc 0 <host> -> cargo 1 <host>
+        [build] rustdoc 0 <host>


Question [CARGO_TEST_STAGING 1/2]: there's some funky staging going on with the staging, I think? Usually ./x test tests/ui --stage=1 means to test xxx using the stage 1 (compiler, library). However, ./x test cargo here means... --stage=0?

jieyouxu · 2025-07-30T07:21:04Z

src/bootstrap/src/core/builder/tests.rs

+        [build] rustc 1 <host> -> cargo 2 <host>
+        [build] rustdoc 1 <host>


Question [CARGO_TEST_STAGING 2/2]: but here, ./x test cargo --stage=2 doesn't build stage 2 (compiler, library) 🤔

jieyouxu · 2025-07-30T07:22:17Z

@rustbot author

rustbot assigned jieyouxu Jul 22, 2025

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jul 22, 2025

This comment has been minimized.

Sign in to view

Kobzol added 13 commits July 23, 2025 16:12

Rename extended rustc tool macros

c89fe80

Add metadata to Cargo and RustAnalyzer tools

d1683e7

Cleanup ensure_if_default to not require Option output

ca66698

Add step metadata to RustAnalyzerProcMacroSrv

1783af7

Rename Builder::rustdoc to Builder::rustdoc_for_compiler

533a78d

To clarify what it does.

Implement RustcPrivateCompilers to unify building of `rustc_private…

7a5f9cb

…` tools

Add step metadata and a few tests for Doc steps

d5ac269

Refactor Rustdoc

6fecbb9

Fix ToolRustc build with download-rustc

006a9bd

Add step metadata and a simple test for codegen backends

acb2379

Port codegen backends to RustcPrivateCompilers

efbaa77

Rename link_compiler to target_compiler

3f53fba

Appease Clippy

7bf4a1a

Kobzol force-pushed the bootstrap-tool-cleanup branch from 2e63ce8 to 7bf4a1a Compare July 23, 2025 14:14

Kobzol mentioned this pull request Jul 26, 2025

incorrect comment in bootstrap: stage 1 rustdoc *can* be used #142745

Open

jieyouxu approved these changes Jul 28, 2025

View reviewed changes

src/bootstrap/src/core/build_steps/tool.rs Outdated Show resolved Hide resolved

src/bootstrap/src/core/builder/tests.rs Outdated Show resolved Hide resolved

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 28, 2025

bors removed the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Jul 28, 2025

This comment has been minimized.

Sign in to view

Make build compiler explicit in dist::Cargo

c4fef48

rust-bors bot added a commit that referenced this pull request Jul 29, 2025

Auto merge of #144303 - Kobzol:bootstrap-tool-cleanup, r=<try>

2802fd0

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux

This comment has been minimized.

Sign in to view

rust-bors bot added a commit that referenced this pull request Jul 29, 2025

Auto merge of #144303 - Kobzol:bootstrap-tool-cleanup, r=<try>

2f3f0ce

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux

This comment has been minimized.

Sign in to view

Fix x test cargo

f731cd1

Kobzol force-pushed the bootstrap-tool-cleanup branch from 0438a46 to f731cd1 Compare July 29, 2025 11:36

rust-bors bot added a commit that referenced this pull request Jul 29, 2025

Auto merge of #144303 - Kobzol:bootstrap-tool-cleanup, r=<try>

180dfb2

Consolidate staging for `rustc_private` tools try-job: x86_64-gnu-aux

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 29, 2025

Kobzol mentioned this pull request Jul 29, 2025

Ignore bors build finished failed jobs rust-lang/bors#398

Merged

jieyouxu reviewed Jul 30, 2025

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 30, 2025

		// FIXME: support testing stage 0 cargo?
		assert!(run.builder.top_stage > 0);

		[build] rustc 0 <host> -> cargo 1 <host>
		[build] rustdoc 0 <host>

		[build] rustc 1 <host> -> cargo 2 <host>
		[build] rustdoc 1 <host>

Consolidate staging for rustc_private tools #144303

Are you sure you want to change the base?

Consolidate staging for rustc_private tools #144303

Conversation

Kobzol commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jul 22, 2025

Uh oh!

This comment has been minimized.

bors commented Jul 23, 2025

Uh oh!

jieyouxu commented Jul 25, 2025

Uh oh!

jieyouxu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jieyouxu commented Jul 28, 2025

Uh oh!

Kobzol commented Jul 28, 2025

Uh oh!

bors commented Jul 28, 2025

Uh oh!

rust-bors bot commented Jul 29, 2025

Uh oh!

This comment has been minimized.

Kobzol commented Jul 29, 2025

Uh oh!

Kobzol commented Jul 29, 2025

Uh oh!

rust-bors bot commented Jul 29, 2025

Uh oh!

This comment has been minimized.

rust-bors bot commented Jul 29, 2025

Uh oh!

This comment has been minimized.

Kobzol commented Jul 29, 2025

Uh oh!

rust-bors bot commented Jul 29, 2025

Uh oh!

This comment has been minimized.

rust-log-analyzer commented Jul 29, 2025

Uh oh!

rust-bors bot commented Jul 29, 2025

Uh oh!

Kobzol commented Jul 29, 2025

Uh oh!

rust-bors bot commented Jul 29, 2025

Uh oh!

rust-bors bot commented Jul 29, 2025

Uh oh!

Kobzol commented Jul 29, 2025

Uh oh!

jieyouxu left a comment

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jieyouxu Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Kobzol Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

jieyouxu commented Jul 30, 2025

Uh oh!

Uh oh!

Consolidate staging for `rustc_private` tools #144303

Consolidate staging for `rustc_private` tools #144303

Kobzol commented Jul 22, 2025 •

edited

Loading

jieyouxu left a comment •

edited

Loading