From 8363e2a30b80676a820c206717f2c2cc4dfa5db8 Mon Sep 17 00:00:00 2001 From: Joshua Nelson Date: Tue, 8 Sep 2020 17:39:48 -0400 Subject: [PATCH 1/5] Explain stages in terms of the compiler currently running MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Address some confusing points + stage N+1 -> stage N artifacts + Use more likely examples of an ABI break + stage N -> stage N compiler - Mention why rustc occasionally uses `cfg(bootstrap)` - Note that stage1 is built using two different versions - Add lots of examples + `test src/test/ui` and `test compiler/rustc` run different compilers 😢 + Separate examples of what to do from examples of what not to do - 'ship stage 1 artifacts' -> 'ship stage 2 compiler' This is hopefully less confusing. --- src/building/bootstrapping.md | 228 +++++++++++++++++----------------- 1 file changed, 114 insertions(+), 114 deletions(-) diff --git a/src/building/bootstrapping.md b/src/building/bootstrapping.md index 7116d908a..a3c0a6d2a 100644 --- a/src/building/bootstrapping.md +++ b/src/building/bootstrapping.md @@ -14,7 +14,7 @@ It must have been written in a different language. In Rust's case it was only way to build a modern version of rustc is a slightly less modern version. -This is exactly how `x.py` works: it downloads the current `beta` release of +This is exactly how `x.py` works: it downloads the current beta release of rustc, then uses it to compile the new compiler. ## Stages of bootstrapping @@ -71,6 +71,8 @@ These defaults are as follows: You can always override the stage by passing `--stage N` explicitly. +For more information about stages, [see below](#understanding-stages-of-bootstrap). + ## Complications of bootstrapping Since the build system uses the current beta compiler to build the stage-1 @@ -122,43 +124,63 @@ contribution [here][bootstrap-build]. ## Understanding stages of bootstrap -This is a detailed look into the separate bootstrap stages. When running -`x.py` you will see output such as: - -```txt -Building stage0 std artifacts -Copying stage0 std from stage0 -Building stage0 compiler artifacts -Copying stage0 rustc from stage0 -Building LLVM for x86_64-apple-darwin -Building stage0 codegen artifacts -Assembling stage1 compiler -Building stage1 std artifacts -Copying stage1 std from stage1 -Building stage1 compiler artifacts -Copying stage1 rustc from stage1 -Building stage1 codegen artifacts -Assembling stage2 compiler -Uplifting stage1 std -Copying stage2 std from stage1 -Generating unstable book md files -Building stage0 tool unstable-book-gen -Building stage0 tool rustbook -Documenting standalone -Building rustdoc for stage2 -Documenting book redirect pages -Documenting stage2 std -Building rustdoc for stage1 -Documenting stage2 whitelisted compiler -Documenting stage2 compiler -Documenting stage2 rustdoc -Documenting error index -Uplifting stage1 rustc -Copying stage2 rustc from stage1 -Building stage2 tool error_index_generator -``` +### Overview + +This is a detailed look into the separate bootstrap stages. + +The convention `x.py` uses is that: +- A `--stage N` flag means to run the stage N compiler (`stageN/rustc`). +- A "stage N artifact" is a build artifact that is _produced_ by the stage N compiler. +- The "stage (N+1) compiler" is assembled from "stage N artifacts". This + process is called _uplifting_. + +#### Examples + +- `x.py build --stage 0` means to build with the beta `rustc`. +- `x.py doc --stage 0` means to document using the beta `rustdoc`. +- `x.py test --stage 0 library/std` means to run tests on the standard library + without building `rustc` from source ('build with stage 0, then test the + artifacts'). If you're working on the standard library, this is normally the + test command you want. +- `x.py test src/test/ui` means to build the stage 1 compiler and run + `compiletest` on it. If you're working on the compiler, this is normally the + test command you want. + +#### Examples of what *not* to do + +- `x.py test --stage 0 src/test/ui` is not meaningful: it runs tests on the + _beta_ compiler and doesn't build `rustc` from source. Use `test src/test/ui` + instead, which builds stage 1 from source. +- `x.py test --stage 0 compiler/rustc` builds the compiler but runs no tests: + it's running `cargo test -p rustc`, but cargo doesn't understand Rust's + tests. You shouldn't need to use this, use `test` instead (without arguments). +- `x.py build --stage 0 compiler/rustc` builds the compiler, but does not make + it usable: the build artifacts are not uplifted ([#73519]). Use `build + library/std` instead which puts the compiler in `stage1/rustc`. + +[#73519]: https://github.com/rust-lang/rust/issues/73519 + +### Building vs. Running + + +Note that `build --stage N compiler/rustc` **does not** build the stage N compiler: +instead it builds the stage _N+1_ compiler _using_ the stage N compiler. + +In short, _stage 0 uses the stage0 compiler to create stage0 artifacts which +will later be uplifted to be the stage1 compiler_. -A deeper look into `x.py`'s phases can be seen here: +In each stage, two major steps are performed: + +1. `std` is compiled by the stage N compiler. +2. That `std` is linked to programs built by the stage N compiler, including + the stage N artifacts (stage (N+1) compiler). + +This is somewhat intuitive if one thinks of the stage N artifacts as "just" +another program we are building with the stage N compiler: +`build --stage N compiler/rustc` is linking the stage N artifacts to the `std` +built by the stage N compiler. + +Here is a chart of a full build using `x.py`: A diagram of the rustc compilation phases @@ -166,6 +188,58 @@ Keep in mind this diagram is a simplification, i.e. `rustdoc` can be built at different stages, the process is a bit different when passing flags such as `--keep-stage`, or if there are non-host targets. +The stage 2 compiler is what is shipped to end-users. + +### Stages and libstd + +Note that there are two `std` libraries in play here: +1. The library _linked_ to `stageN/rustc`, which was built by stage N-1 (stage N-1 `std`) +2. The library _used to compile programs_ with `stageN/rustc`, which was + built by stage N (stage N `std`). + +Stage N `std` is pretty much necessary for any useful work with the stage N compiler. +Without it, you can only compile programs with `#![no_core]` -- not terribly useful! + +The reason these need to be different is because they aren't necessarily ABI-compatible: +there could be a new layout optimizations, changes to MIR, or other changes +to Rust metadata on nightly that aren't present in beta. + +This is also where `--keep-stage 1 library/std` comes into play. Since most +changes to the compiler don't actually change the ABI, once you've produced a +`std` in stage 1, you can probably just reuse it with a different compiler. +If the ABI hasn't changed, you're good to go, no need to spend time +recompiling that `std`. +`--keep-stage` simply assumes the previous compile is fine and copies those +artifacts into the appropriate place, skipping the cargo invocation. + +### Cross-compiling + +Building stage2 `std` is different depending on whether you are cross-compiling or not +(see in the table how stage2 only builds non-host `std` targets). +This is because `x.py` uses a trick: if `HOST` and `TARGET` are the same, +it will reuse stage1 `std` for stage2! This is sound because stage1 `std` +was compiled with the stage1 compiler, i.e. a compiler using the source code +you currently have checked out. So it should be identical (and therefore ABI-compatible) +to the `std` that `stage2/rustc` would compile. + +However, when cross-compiling, stage1 `std` will only run on the host. +So the stage2 compiler has to recompile `std` for the target. + +### Why does only libstd use `cfg(bootstrap)`? + +The `rustc` generated by the stage0 compiler is linked to the freshly-built +`std`, which means that for the most part only `std` needs to be cfg-gated, +so that `rustc` can use features added to std immediately after their addition, +without need for them to get into the downloaded beta. + +Note this is different from any other Rust program: stage1 `rustc` +is built by the _beta_ compiler, but using the _master_ version of libstd! + +The only time `rustc` uses `cfg(bootstrap)` is when it adds internal lints +that use diagnostic items. This happens very rarely. + +### Directories and artifacts generated by x.py + The following tables indicate the outputs of various stage actions: | Stage 0 Action | Output | @@ -178,7 +252,7 @@ The following tables indicate the outputs of various stage actions: | copy `stage0-rustc (except executable)` | `build/HOST/stage0-sysroot/lib/rustlib/HOST` | | build `llvm` | `build/HOST/llvm` | | `stage0` builds `codegen` with `stage0-sysroot` | `build/HOST/stage0-codegen/HOST` | -| `stage0` builds `rustdoc` with `stage0-sysroot` | `build/HOST/stage0-tools/HOST` | +| `stage0` builds `rustdoc`, `clippy`, `miri`, with `stage0-sysroot` | `build/HOST/stage0-tools/HOST` | `--stage=0` stops here. @@ -201,85 +275,11 @@ The following tables indicate the outputs of various stage actions: | copy (uplift) `stage1-sysroot` | `build/HOST/stage2/lib and build/HOST/stage2/lib/rustlib/HOST` | | `stage2` builds `test`/`std` (not HOST targets) | `build/HOST/stage2-std/TARGET` | | copy `stage2-std` (not HOST targets) | `build/HOST/stage2/lib/rustlib/TARGET` | -| `stage2` builds `rustdoc` | `build/HOST/stage2-tools/HOST` | +| `stage2` builds `rustdoc`, `clippy`, `miri` | `build/HOST/stage2-tools/HOST` | | copy `rustdoc` | `build/HOST/stage2/bin` | `--stage=2` stops here. -Note that the convention `x.py` uses is that: -- A "stage N artifact" is an artifact that is _produced_ by the stage N compiler. -- The "stage (N+1) compiler" is assembled from "stage N artifacts". -- A `--stage N` flag means build _with_ stage N. - -In short, _stage 0 uses the stage0 compiler to create stage0 artifacts which -will later be uplifted to stage1_. - -Every time any of the main artifacts (`std` and `rustc`) are compiled, two -steps are performed. -When `std` is compiled by a stage N compiler, that `std` will be linked to -programs built by the stage N compiler (including `rustc` built later -on). It will also be used by the stage (N+1) compiler to link against itself. -This is somewhat intuitive if one thinks of the stage (N+1) compiler as "just" -another program we are building with the stage N compiler. In some ways, `rustc` -(the binary, not the `rustbuild` step) could be thought of as one of the few -`no_core` binaries out there. - -So "stage0 std artifacts" are in fact the output of the downloaded stage0 -compiler, and are going to be used for anything built by the stage0 compiler: -e.g. `rustc` artifacts. When it announces that it is "building stage1 -std artifacts" it has moved on to the next bootstrapping phase. This pattern -continues in latter stages. - -Also note that building host `std` and target `std` are different based on the -stage (e.g. see in the table how stage2 only builds non-host `std` targets. -This is because during stage2, the host `std` is uplifted from the "stage 1" -`std` -- specifically, when "Building stage 1 artifacts" is announced, it is -later copied into stage2 as well (both the compiler's `libdir` and the -`sysroot`). - -This `std` is pretty much necessary for any useful work with the compiler. -Specifically, it's used as the `std` for programs compiled by the newly compiled -compiler (so when you compile `fn main() { }` it is linked to the last `std` -compiled with `x.py build library/std`). - -The `rustc` generated by the stage0 compiler is linked to the freshly-built -`std`, which means that for the most part only `std` needs to be cfg-gated, -so that `rustc` can use featured added to std immediately after their addition, -without need for them to get into the downloaded beta. The `std` built by the -`stage1/bin/rustc` compiler, also known as "stage1 std artifacts", is not -necessarily ABI-compatible with that compiler. -That is, the `rustc` binary most likely could not use this `std` itself. -It is however ABI-compatible with any programs that the `stage1/bin/rustc` -binary builds (including itself), so in that sense they're paired. - -This is also where `--keep-stage 1 library/std` comes into play. Since most -changes to the compiler don't actually change the ABI, once you've produced a -`std` in stage 1, you can probably just reuse it with a different compiler. -If the ABI hasn't changed, you're good to go, no need to spend the time -recompiling that `std`. -`--keep-stage` simply assumes the previous compile is fine and copies those -artifacts into the appropriate place, skipping the cargo invocation. - -The reason we first build `std`, then `rustc`, is largely just -because we want to minimize `cfg(stage0)` in the code for `rustc`. -Currently `rustc` is always linked against a "new" `std` so it doesn't -ever need to be concerned with differences in std; it can assume that the std is -as fresh as possible. - -The reason we need to build it twice is because of ABI compatibility. -The beta compiler has it's own ABI, and then the `stage1/bin/rustc` compiler -will produce programs/libraries with the new ABI. -We used to build three times, but because we assume that the ABI is constant -within a codebase, we presume that the libraries produced by the "stage2" -compiler (produced by the `stage1/bin/rustc` compiler) is ABI-compatible with -the `stage1/bin/rustc` compiler's produced libraries. -What this means is that we can skip that final compilation -- and simply use the -same libraries as the `stage2/bin/rustc` compiler uses itself for programs it -links against. - -This `stage2/bin/rustc` compiler is shipped to end-users, along with the -`stage 1 {std,rustc}` artifacts. - ## Passing stage-specific flags to `rustc` `x.py` allows you to pass stage-specific flags to `rustc` when bootstrapping. @@ -287,7 +287,7 @@ The `RUSTFLAGS_STAGE_0`, `RUSTFLAGS_STAGE_1` and `RUSTFLAGS_STAGE_2` environment variables pass the given flags when building stage 0, 1, and 2 artifacts respectively. -Additionally, the `RUSTFLAGS_STAGE_NOT_0` variable, as its name suggests, pass +Additionally, the `RUSTFLAGS_STAGE_NOT_0` variable, as its name suggests, passes the given arguments if the stage is not 0. ## Environment Variables From 8e7585bfd39e76ae39722ff0d447f45051960b62 Mon Sep 17 00:00:00 2001 From: Joshua Nelson Date: Thu, 1 Oct 2020 17:59:19 -0400 Subject: [PATCH 2/5] build -> x.py build --- src/building/bootstrapping.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/building/bootstrapping.md b/src/building/bootstrapping.md index a3c0a6d2a..da0eb45fa 100644 --- a/src/building/bootstrapping.md +++ b/src/building/bootstrapping.md @@ -155,7 +155,7 @@ The convention `x.py` uses is that: it's running `cargo test -p rustc`, but cargo doesn't understand Rust's tests. You shouldn't need to use this, use `test` instead (without arguments). - `x.py build --stage 0 compiler/rustc` builds the compiler, but does not make - it usable: the build artifacts are not uplifted ([#73519]). Use `build + it usable: the build artifacts are not uplifted ([#73519]). Use `x.py build library/std` instead which puts the compiler in `stage1/rustc`. [#73519]: https://github.com/rust-lang/rust/issues/73519 From 6730eba52f9fbbda2ccf8a7bff24ad93f78622a8 Mon Sep 17 00:00:00 2001 From: Joshua Nelson Date: Thu, 1 Oct 2020 18:12:24 -0400 Subject: [PATCH 3/5] Add section on build artifacts --- src/building/bootstrapping.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/building/bootstrapping.md b/src/building/bootstrapping.md index da0eb45fa..bfd6a2503 100644 --- a/src/building/bootstrapping.md +++ b/src/building/bootstrapping.md @@ -134,6 +134,18 @@ The convention `x.py` uses is that: - The "stage (N+1) compiler" is assembled from "stage N artifacts". This process is called _uplifting_. +#### Build artifacts + +Anything you can build with `x.py` is a build artifact. +Build artifacts include, but are not limited to: + +- binaries, like `stage0-rustc/rustc-main` +- shared objects, like `stage0-sysroot/rustlib/libstd-6fae108520cf72fe.so` +- [rlib] files, like `stage0-sysroot/rustlib/libstd-6fae108520cf72fe.rlib` +- HTML files generated by rustdoc, like `doc/std` + +[rlib]: ../serialization.md + #### Examples - `x.py build --stage 0` means to build with the beta `rustc`. From e563a4d516aef668041a69134c762329a143e82f Mon Sep 17 00:00:00 2001 From: Joshua Nelson Date: Thu, 1 Oct 2020 19:20:16 -0400 Subject: [PATCH 4/5] Improve wording Co-authored-by: Camelid <37223377+camelid@users.noreply.github.com> --- src/building/bootstrapping.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/building/bootstrapping.md b/src/building/bootstrapping.md index bfd6a2503..a6c439e55 100644 --- a/src/building/bootstrapping.md +++ b/src/building/bootstrapping.md @@ -136,7 +136,7 @@ The convention `x.py` uses is that: #### Build artifacts -Anything you can build with `x.py` is a build artifact. +Anything you can build with `x.py` is a _build artifact_. Build artifacts include, but are not limited to: - binaries, like `stage0-rustc/rustc-main` @@ -168,7 +168,7 @@ Build artifacts include, but are not limited to: tests. You shouldn't need to use this, use `test` instead (without arguments). - `x.py build --stage 0 compiler/rustc` builds the compiler, but does not make it usable: the build artifacts are not uplifted ([#73519]). Use `x.py build - library/std` instead which puts the compiler in `stage1/rustc`. + library/std` instead, which puts the compiler in `stage1/rustc`. [#73519]: https://github.com/rust-lang/rust/issues/73519 @@ -202,7 +202,7 @@ different stages, the process is a bit different when passing flags such as The stage 2 compiler is what is shipped to end-users. -### Stages and libstd +### Stages and `std` Note that there are two `std` libraries in play here: 1. The library _linked_ to `stageN/rustc`, which was built by stage N-1 (stage N-1 `std`) From ddf743f81f563c1c4d04b25b96b95f5719823aff Mon Sep 17 00:00:00 2001 From: Joshua Nelson Date: Thu, 1 Oct 2020 19:48:07 -0400 Subject: [PATCH 5/5] uplifted -> assembled --- src/building/bootstrapping.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/building/bootstrapping.md b/src/building/bootstrapping.md index a6c439e55..24f86d940 100644 --- a/src/building/bootstrapping.md +++ b/src/building/bootstrapping.md @@ -167,8 +167,9 @@ Build artifacts include, but are not limited to: it's running `cargo test -p rustc`, but cargo doesn't understand Rust's tests. You shouldn't need to use this, use `test` instead (without arguments). - `x.py build --stage 0 compiler/rustc` builds the compiler, but does not make - it usable: the build artifacts are not uplifted ([#73519]). Use `x.py build - library/std` instead, which puts the compiler in `stage1/rustc`. + it usable: the build artifacts are not assembled into the final compiler + ([#73519]). Use `x.py build library/std` instead, which puts the compiler in + `stage1/rustc`. [#73519]: https://github.com/rust-lang/rust/issues/73519