Tracking issue for -Z binary-dep-depinfo #63012
Comments
ehuss
added
B-unstable
Centril
added
T-compiler
…xcrichton Utilize -Zbinary-dep-depinfo in rustbuild The last commit moves us over to using binary-dep-depinfo while the first two permit us to bootstrap from what will become future beta, to be released in the next week (it's the `cfg(bootstrap)` processing). We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles. Both of these are fixable in the longer term but this existing patch gives us the following benefits: * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes rust-lang#54712. * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes rust-lang#54008, fixes rust-lang#50481. * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes rust-lang#53284, and fixes rust-lang#52719. * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes rust-lang#49979 (but no reproduction steps in that issue). Fixes rust-lang#59105. cc rust-lang#63012
…xcrichton Utilize -Zbinary-dep-depinfo in rustbuild The last commit moves us over to using binary-dep-depinfo while the first two permit us to bootstrap from what will become future beta, to be released in the next week (it's the `cfg(bootstrap)` processing). We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles. Both of these are fixable in the longer term but this existing patch gives us the following benefits: * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes rust-lang#54712. * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes rust-lang#54008, fixes rust-lang#50481. * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes rust-lang#53284, and fixes rust-lang#52719. * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes rust-lang#49979 (but no reproduction steps in that issue). Fixes rust-lang#59105. cc rust-lang#63012
…xcrichton Utilize -Zbinary-dep-depinfo in rustbuild The last commit moves us over to using binary-dep-depinfo while the first two permit us to bootstrap from what will become future beta, to be released in the next week (it's the `cfg(bootstrap)` processing). We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles. Both of these are fixable in the longer term but this existing patch gives us the following benefits: * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes rust-lang#54712. * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes rust-lang#54008, fixes rust-lang#50481. * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes rust-lang#53284, and fixes rust-lang#52719. * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes rust-lang#49979 (but no reproduction steps in that issue). Fixes rust-lang#59105. cc rust-lang#63012
…xcrichton Utilize -Zbinary-dep-depinfo in rustbuild The last commit moves us over to using binary-dep-depinfo while the first two permit us to bootstrap from what will become future beta, to be released in the next week (it's the `cfg(bootstrap)` processing). We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles. Both of these are fixable in the longer term but this existing patch gives us the following benefits: * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes rust-lang#54712. * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes rust-lang#54008, fixes rust-lang#50481. * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes rust-lang#53284, and fixes rust-lang#52719. * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes rust-lang#49979 (but no reproduction steps in that issue). Fixes rust-lang#59105. cc rust-lang#63012
…xcrichton Utilize -Zbinary-dep-depinfo in rustbuild The last commit moves us over to using binary-dep-depinfo while the first two permit us to bootstrap from what will become future beta, to be released in the next week (it's the `cfg(bootstrap)` processing). We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles. Both of these are fixable in the longer term but this existing patch gives us the following benefits: * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes rust-lang#54712. * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes rust-lang#54008, fixes rust-lang#50481. * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes rust-lang#53284, and fixes rust-lang#52719. * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes rust-lang#49979 (but no reproduction steps in that issue). Fixes rust-lang#59105. cc rust-lang#63012
…xcrichton Utilize -Zbinary-dep-depinfo in rustbuild The last commit moves us over to using binary-dep-depinfo while the first two permit us to bootstrap from what will become future beta, to be released in the next week (it's the `cfg(bootstrap)` processing). We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles. Both of these are fixable in the longer term but this existing patch gives us the following benefits: * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes rust-lang#54712. * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes rust-lang#54008, fixes rust-lang#50481. * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes rust-lang#53284, and fixes rust-lang#52719. * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes rust-lang#49979 (but no reproduction steps in that issue). Fixes rust-lang#59105. cc rust-lang#63012
Utilize -Zbinary-dep-depinfo in rustbuild We no longer utilize stamp-file mtimes at all inside rustbuild, and a future PR may be able to entirely eliminate them by eagerly copying to the appropriate sysroot. The only mtime-based dependency tracking left is for documentation because we lie to Cargo about the rustdoc binary, so Cargo does not track changes to the real binary, and codegen-backends because binary-dep-depinfo does not emit that information into the depfiles. Both of these are fixable in the longer term but this existing patch gives us the following benefits: * We no longer delete Cargo target directories manually within a stage. Cross-stage, changes to codegen backends will still clear out target directories. This means that incremental state persists across individual steps (e.g., rebuilding libstd does not clear out librustc incremental state). Fixes #54712. * Dependency tracking across steps within a given stage is now fully precise. We will not clear out all codegen backend dependencies due to changes in librustc_driver, for example, only deleting the final librustc_codegen_llvm crate. Fixes #54008, fixes #50481. * We properly track codegen backends as a dependency (equivalent to rustc) across changes. Fixes #53284, and fixes #52719. * Cross-stage dependency tracking of crates is also much more accurate and reliable. Most likely fixes #49979 (but no reproduction steps in that issue). Fixes #59105. cc #63012
|
In #68298 we fixed binary dep-depinfo to be less eager to emit dependencies on dylib/rlib files when emitting rlibs and rmeta files, as we only need rmeta input in that case. It was also noted that we currently do not correctly emit plugin dependencies (I'm not entirely sure of this, but seems not implausible). |
|
I wonder if the Cargo side of |
|
Separately, repeating the sentiment from rust-lang/cargo#10664 (comment) that tracking codegen backends should probably not be considered a blocker for landing this — the feature can very much be useful without that I think, and it could be added later on. |
|
|
Discussed in T-compiler backlog bonanza We had some confusion about who the target audience for this feature is. From reading the comments, we understand it was added for certain things in Independent of that, there were design questions, e.g. what things should be included. (It doesn't track native dependencies, for example.) So, I'm adding the design concerns label below, but I would be interested to know if I should also be adding S-tracking-perma-unstable @rustbot label: S-tracking-design-concerns |
rustbot
added
the
S-tracking-design-concerns
|
Given the confusion we had in our conversation about this, I'm also going to request a summary of this feature and its status. :) @rustbot label: S-tracking-needs-summary |
rustbot
added
the
S-tracking-needs-summary
|
I can speak to the use-case in my case: we're using a large distributed build system at $work in which packages are automatically re-built if their dependencies change. Which, for example, means that if we for example need a patch to |
|
|
|
The linux kernel uses the flag too: Rust-for-Linux/linux#2
|
Apparently it is necessary to avoid recompiling in some scenarios: Rust-for-Linux/linux#2 (comment)
|
I am confused by this statement. binary-dep-depinfo does not emit the path to the toolchain currently: https://github.com/rust-lang/rust/blob/1fa1f5932b487a2ac4105deca26493bb8013a9a6/compiler/rustc_interface/src/passes.rs#L490-L511 |
|
Ah, so, to be clear, we don't currently use |
|
We do currently use |
|
Cc #111329 (comment) which I believe should help explain why it's not currently enough for dependency version changes, I suspect a similar change for incremental may also be helpful but I haven't looked at incremental encoding/decoding. |
|
For incremental it wouldn't help. Even if the encoding is exactly identical the encoded query results may have changed between versions. We need to clear the incr cache unconditionally if rustc changes. |
|
To loop in some other tickets that have touched on "re-building if rustc changes" in the past: rust-lang/cargo#10664 and rust-lang/cargo#10367 are possibly relevant. |
|
rust-lang/cargo#10367 is a different problem. That's about if the path to the compiler changes; the problem in this issue happens when the path and version output stay the same and only the mtine is different. |
This is a tracking issue for
-Z binary-dep-depinfoadded in #61727.The cargo side is implemented in rust-lang/cargo#7137.
Blockers:
\\?\) paths, and I think we want to use only one style (whatever is compatible with make and other tools). See the PR for details.cc @Mark-Simulacrum @alexcrichton
The text was updated successfully, but these errors were encountered: