tidy: skip submodules if not present for non-CI environments #126230

onur-ozkan · 2024-06-10T14:10:10Z

Right now tidy requires rustc-perf to be fetched as it checks its license, but this doesn't make sense for most contributors since rustc-perf is a dist-specific tool and its license will not change frequently, especially during compiler development. This PR makes tidy to skip rustc-perf if it's not fetched and if it's not running in CI.

Applies #126225 (comment) and reverts #126225.

This reverts commit faac70b.

rustbot · 2024-06-10T14:10:19Z

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2024-06-10T14:10:22Z

The list of allowed third-party dependencies may have been modified! You must ensure that any new dependencies have compatible licenses before merging.

cc @davidtwco, @wesleywiser

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

rustbot · 2024-06-10T14:11:37Z

The list of allowed third-party dependencies may have been modified! You must ensure that any new dependencies have compatible licenses before merging.

cc @davidtwco, @wesleywiser

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

onur-ozkan · 2024-06-10T14:12:31Z

src/tools/tidy/src/extdeps.rs

+        // Skip if it's a submodule, not initialized, and not in a CI environment.
+        //
+        // This prevents enforcing developers to fetch submodules for tidy.
+        if is_submodule && !root.join(workspace).join(".git").exists() && !CiEnv::is_ci() {


This won't work in tarball sources, will change it.

Changed it to check if the directory is empty or not. For non-tarball sources, it's not possible to have an initialized submodule as an empty directory (because .git will be present). Even if it's a tarball source, we don't have any tool that is a submodule but an empty directory anyway. We can do this check this by using git, but it makes things overly complicated and it wouldn't work on tarball sources.

src/tools/tidy/src/deps.rs

ehuss · 2024-06-10T20:33:05Z

src/tools/tidy/src/deps.rs

    //("src/etc/test-float-parse", &[], None), // FIXME uncomment once all deps are vendored
-    ("src/tools/cargo", EXCEPTIONS_CARGO, None),
+    ("src/tools/cargo", EXCEPTIONS_CARGO, None, false),


I'm not quite clear why this is false. src/tools/cargo is a submodule.

Overall I'm confused by this PR. tidy has depended on submodules existing for some time. From my understanding, things like the cargo submodule implicitly work via this check which ensures a subset of submodules are available.

Why not just leave #126225 (or some variation of that), and always check rustc-perf? Is there some significant downside of cloning rustc-perf?

@lqd It seemed like you didn't like that we would force contributors to download rustc-perf. Do you have any specific concerns? :)

Overall I'm confused by this PR. tidy has depended on submodules existing for some time. From my understanding, things like the cargo submodule implicitly work via this check which ensures a subset of submodules are available.

Fetching a dist specific tool and continuously checking its license with tidy on development doesn't seem like a good idea overall, at least in my opinion (I am pretty sure some of the active contributors like @RalfJung don't like this kind of approach either).

I'm not quite clear why this is false. src/tools/cargo is a submodule.

Added automatic submodule detection to prevent issues like this in the future.

cc @Kobzol

I think ideally we'd require as few submodules as possible to be checked out. Submodules are annoying as updating them is always a separate step -- even if x.py does it automatically (which currently fails badly for me), the checkout is in a somewhat degraded state until the next x.py invocation.

If we can get that number down to 0, that would be perfect. :) But currently that's probably not yet possible.

That's fine if there is a significant burden. Would it be possible to update the PR description to explain what the PR changes and the motivation for the change? That way, we're at least clear as to why it is changing (such as "avoids a significant download not relevant to most contributors", or whatever the actual reason is).

Updated the PR description.

Mark-Simulacrum · 2024-06-16T20:16:24Z

src/tools/tidy/src/deps.rs

@@ -515,6 +517,19 @@ pub fn check(root: &Path, cargo: &Path, bad: &mut bool) {
    let mut checked_runtime_licenses = false;

    for &(workspace, exceptions, permitted_deps) in WORKSPACES {
+        let gitmodules = root.join(".gitmodules");


Can we move this into the function? It seems like the function could take the directory rather than the file path, which simplifies it and avoids duplicating the exact filename everywhere.

Mark-Simulacrum · 2024-06-16T20:21:23Z

src/tools/tidy/src/extdeps.rs

@@ -14,6 +15,19 @@ const ALLOWED_SOURCES: &[&str] = &[
 /// workspace `Cargo.toml`.
 pub fn check(root: &Path, bad: &mut bool) {
    for &(workspace, _, _) in crate::deps::WORKSPACES {
+        let gitmodules = root.join(".gitmodules");
+        let submodules = build_helper::util::parse_gitmodules(&gitmodules);


Can we move this out of the loop? No need to re-parse for every workspace.

Same with the other call site in deps.

Signed-off-by: onur-ozkan <work@onurozkan.dev>

onur-ozkan · 2024-06-17T05:55:05Z

Applied the review notes above.

@rustbot ready

Mark-Simulacrum · 2024-06-23T13:53:55Z

@bors r+

bors · 2024-06-23T13:53:57Z

📌 Commit e9e3c38 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

…-Simulacrum tidy: skip submodules if not present for non-CI environments Right now tidy requires rustc-perf to be fetched as it checks its license, but this doesn't make sense for most contributors since rustc-perf is a dist-specific tool and its license will not change frequently, especially during compiler development. This PR makes tidy to skip rustc-perf if it's not fetched and if it's not running in CI. Applies rust-lang#126225 (comment) and reverts rust-lang#126225.

…mpiler-errors Rollup of 9 pull requests Successful merges: - rust-lang#126230 (tidy: skip submodules if not present for non-CI environments) - rust-lang#126523 (std: refactor the TLS implementation) - rust-lang#126612 (Update outdated README in build-manifest.) - rust-lang#126616 (less bootstrap warnings) - rust-lang#126663 (remove `GIT_DIR` handling in pre-push hook) - rust-lang#126830 (make unsized_fn_params an internal feature) - rust-lang#126833 (don't ICE when encountering an extern type field during validation) - rust-lang#126837 (delegation: Do not crash on qpaths without a trait) - rust-lang#126851 (Rework pattern and expression nonterminal kinds.) r? `@ghost` `@rustbot` modify labels: rollup

…-Simulacrum tidy: skip submodules if not present for non-CI environments Right now tidy requires rustc-perf to be fetched as it checks its license, but this doesn't make sense for most contributors since rustc-perf is a dist-specific tool and its license will not change frequently, especially during compiler development. This PR makes tidy to skip rustc-perf if it's not fetched and if it's not running in CI. Applies rust-lang#126225 (comment) and reverts rust-lang#126225.

…mpiler-errors Rollup of 9 pull requests Successful merges: - rust-lang#126230 (tidy: skip submodules if not present for non-CI environments) - rust-lang#126608 (Add more constants, functions, and tests for `f16` and `f128`) - rust-lang#126612 (Update outdated README in build-manifest.) - rust-lang#126616 (less bootstrap warnings) - rust-lang#126663 (remove `GIT_DIR` handling in pre-push hook) - rust-lang#126830 (make unsized_fn_params an internal feature) - rust-lang#126833 (don't ICE when encountering an extern type field during validation) - rust-lang#126837 (delegation: Do not crash on qpaths without a trait) - rust-lang#126851 (Rework pattern and expression nonterminal kinds.) r? `@ghost` `@rustbot` modify labels: rollup

…iaskrgr Rollup of 9 pull requests Successful merges: - rust-lang#126230 (tidy: skip submodules if not present for non-CI environments) - rust-lang#126612 (Update outdated README in build-manifest.) - rust-lang#126616 (less bootstrap warnings) - rust-lang#126663 (remove `GIT_DIR` handling in pre-push hook) - rust-lang#126830 (make unsized_fn_params an internal feature) - rust-lang#126833 (don't ICE when encountering an extern type field during validation) - rust-lang#126837 (delegation: Do not crash on qpaths without a trait) - rust-lang#126851 (Rework pattern and expression nonterminal kinds.) - rust-lang#126862 (Add needs-symlink directive to compiletest) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#126230 - onur-ozkan:followup-126225, r=Mark-Simulacrum tidy: skip submodules if not present for non-CI environments Right now tidy requires rustc-perf to be fetched as it checks its license, but this doesn't make sense for most contributors since rustc-perf is a dist-specific tool and its license will not change frequently, especially during compiler development. This PR makes tidy to skip rustc-perf if it's not fetched and if it's not running in CI. Applies rust-lang#126225 (comment) and reverts rust-lang#126225.

Revert "Update rustc-perf submodule before running tidy"

dd50e18

This reverts commit faac70b.

rustbot assigned Mark-Simulacrum Jun 10, 2024

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 10, 2024

onur-ozkan force-pushed the followup-126225 branch from 8cebd4e to fb2a528 Compare June 10, 2024 14:11

onur-ozkan commented Jun 10, 2024

View reviewed changes

onur-ozkan force-pushed the followup-126225 branch from fb2a528 to 91e1ef1 Compare June 10, 2024 14:22

Kobzol reviewed Jun 10, 2024

View reviewed changes

src/tools/tidy/src/deps.rs Outdated Show resolved Hide resolved

ehuss reviewed Jun 10, 2024

View reviewed changes

onur-ozkan force-pushed the followup-126225 branch 2 times, most recently from 7e55123 to af35904 Compare June 10, 2024 21:08

onur-ozkan mentioned this pull request Jun 14, 2024

make cargo submodule optional #126470

Merged

jieyouxu mentioned this pull request Jun 16, 2024

tidy requires src/tools/rustc-perf? #126559

Closed

Mark-Simulacrum requested changes Jun 16, 2024

View reviewed changes

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 16, 2024

tidy: skip submodules if not present for non-CI environments

e9e3c38

Signed-off-by: onur-ozkan <work@onurozkan.dev>

onur-ozkan force-pushed the followup-126225 branch from af35904 to e9e3c38 Compare June 17, 2024 04:56

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 17, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 23, 2024

compiler-errors mentioned this pull request Jun 23, 2024

Rollup of 9 pull requests #126867

Closed

compiler-errors mentioned this pull request Jun 23, 2024

Rollup of 9 pull requests #126870

Closed

matthiaskrgr mentioned this pull request Jun 23, 2024

Rollup of 9 pull requests #126878

Merged

bors merged commit f016552 into rust-lang:master Jun 23, 2024
6 checks passed

rustbot added this to the 1.81.0 milestone Jun 23, 2024

onur-ozkan deleted the followup-126225 branch June 24, 2024 04:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tidy: skip submodules if not present for non-CI environments #126230

tidy: skip submodules if not present for non-CI environments #126230

onur-ozkan commented Jun 10, 2024 •

edited

Loading

rustbot commented Jun 10, 2024

rustbot commented Jun 10, 2024

rustbot commented Jun 10, 2024

onur-ozkan Jun 10, 2024

onur-ozkan Jun 10, 2024

ehuss Jun 10, 2024

Kobzol Jun 10, 2024

onur-ozkan Jun 10, 2024

onur-ozkan Jun 10, 2024

RalfJung Jun 10, 2024 •

edited

Loading

ehuss Jun 10, 2024

onur-ozkan Jun 11, 2024

Mark-Simulacrum Jun 16, 2024

Mark-Simulacrum Jun 16, 2024

onur-ozkan commented Jun 17, 2024

Mark-Simulacrum commented Jun 23, 2024

bors commented Jun 23, 2024

tidy: skip submodules if not present for non-CI environments #126230

tidy: skip submodules if not present for non-CI environments #126230

Conversation

onur-ozkan commented Jun 10, 2024 • edited Loading

rustbot commented Jun 10, 2024

rustbot commented Jun 10, 2024

rustbot commented Jun 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RalfJung Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

onur-ozkan commented Jun 17, 2024

Mark-Simulacrum commented Jun 23, 2024

bors commented Jun 23, 2024

onur-ozkan commented Jun 10, 2024 •

edited

Loading

RalfJung Jun 10, 2024 •

edited

Loading