Improve test suite for `-Zbuild-std` #7350

alexcrichton · 2019-09-10T18:10:23Z

This commit is aimed directly at rust-lang/wg-cargo-std-aware#33 and in
general making the -Zbuild-std tests more robust. The main change here
is that a new source tree is checked in, tests/testsuite/mock-std,
which mirrors rust-lang/rust's own tree for libstd. This mock tree is as
empty as it can be, ideally duplicating almost nothing but for not
requiring duplication of Cargo metadata about patches and such.

The end result here looks like:

All -Zbuild-std tests are now run in parallel
All tests run much more quickly since they're compiling tiny crates
instead of actually compiling libstd/libcore
No tests require network access
We verify that crates have access to the "custom" libraries
that we build

Coverage of tests is not currently expanded, but it's hoped that we
could add that shortly afterwards. Coverage has actually gone down
slightly since the custom target test was commented out temporarily and
the full integration test of running -Zbuild-std isn't run on CI any
more.

Closes rust-lang/wg-cargo-std-aware#33

rust-highfive · 2019-09-10T18:10:26Z

r? @ehuss

(rust_highfive has picked a reviewer for you, use r? to override)

ehuss

Nice! It's a bit elaborate, but is pleasantly fast.

Will this still work if we switch cargo to create its own sysroot? I assume we'd just remove the line that adds --sysroot.

r=me with the above addressed, and if you're comfortable moving forward.

tests/testsuite/standard_lib.rs

alexcrichton · 2019-09-12T15:12:26Z

It's a bit elaborate

I'm curious to get a bit more of your feeling on this, is this maybe too elaborate? I stared with a blank slate and basically just kept hitting "compile" and filled out all the dependencies and pub use and whatnot. It did end up needing a bit more than I was hoping for (especially having to mark something as #![panic_runtime]).

I'm not sure how we'd simplify this much more though without broader changes to rustc/cargo, which I'm pretty unwilling to do just for the sake of tests.

Will this still work if we switch cargo to create its own sysroot?

I believe so yeah, I made sure that the tests currently do e.env("RUSTFLAGS", "--sysroot=/path/to/nowhere"); which should make them resilient to a switching --sysroot.

ehuss · 2019-09-12T16:02:03Z

is this maybe too elaborate?

Eh, I think the only concern is the maintenance burden going forward, updating it to match upstream as needed. I suspect the rate of change should be small, so I don't think it is too much of a problem. If the tests end up being fussy or break often, we can rethink it later if needed. Otherwise it will just be a directory full of files nobody will ever need to look at.

alexcrichton · 2019-09-12T16:48:00Z

Ok that sounds good. If this is hard to manage then let's deal with it then.

alexcrichton · 2019-09-12T17:39:36Z

Ok I've pushed up what I'm thinking for a full integration test doing everything, want to double check me @ehuss?

ehuss · 2019-09-12T17:51:01Z

Looks like the rustup component add rust-src step needs to be moved around.

alexcrichton · 2019-09-12T20:53:50Z

Ok green now!

bors · 2019-09-16T16:42:17Z

☔ The latest upstream changes (presumably #7360) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2019-09-16T18:44:40Z

☔ The latest upstream changes (presumably #7159) made this pull request unmergeable. Please resolve the merge conflicts.

This commit is aimed directly at rust-lang/wg-cargo-std-aware#33 and in general making the `-Zbuild-std` tests more robust. The main change here is that a new source tree is checked in, `tests/testsuite/mock-std`, which mirrors rust-lang/rust's own tree for libstd. This mock tree is as empty as it can be, ideally duplicating almost nothing but for not requiring duplication of Cargo metadata about patches and such. The end result here looks like: * All `-Zbuild-std` tests are now run in parallel * All tests run much more quickly since they're compiling tiny crates instead of actually compiling libstd/libcore * No tests require network access * We verify that crates have access to the "custom" libraries that we build Coverage of tests is not currently expanded, but it's hoped that we could add that shortly afterwards. Coverage has actually gone down slightly since the custom target test was commented out temporarily and the full integration test of running `-Zbuild-std` isn't run on CI any more. Closes rust-lang/wg-cargo-std-aware#33

Extract out all our test support code to its own standalone crate so it can be shared between multiple test suites if necessary.

Only run these tests on one CI builder (not all platforms) though. This is extremely resource intensive since it rebuilds libstd. Currently this does not share a build directly like before because the number of tests are supposed to be small, but if necessary we can add that in later too.

ehuss · 2019-09-16T20:22:05Z

@bors r+

bors · 2019-09-16T20:22:07Z

📌 Commit ebd1052 has been approved by ehuss

bors · 2019-09-16T21:10:55Z

⌛ Testing commit ebd1052 with merge 13bc9a1...

Improve test suite for `-Zbuild-std` This commit is aimed directly at rust-lang/wg-cargo-std-aware#33 and in general making the `-Zbuild-std` tests more robust. The main change here is that a new source tree is checked in, `tests/testsuite/mock-std`, which mirrors rust-lang/rust's own tree for libstd. This mock tree is as empty as it can be, ideally duplicating almost nothing but for not requiring duplication of Cargo metadata about patches and such. The end result here looks like: * All `-Zbuild-std` tests are now run in parallel * All tests run much more quickly since they're compiling tiny crates instead of actually compiling libstd/libcore * No tests require network access * We verify that crates have access to the "custom" libraries that we build Coverage of tests is not currently expanded, but it's hoped that we could add that shortly afterwards. Coverage has actually gone down slightly since the custom target test was commented out temporarily and the full integration test of running `-Zbuild-std` isn't run on CI any more. Closes rust-lang/wg-cargo-std-aware#33

bors · 2019-09-16T21:43:34Z

☀️ Test successful - checks-azure
Approved by: ehuss
Pushing 13bc9a1 to master...

rust-highfive assigned ehuss Sep 10, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 10, 2019

alexcrichton force-pushed the mock-std branch 2 times, most recently from 5c21b64 to 0c8631b Compare September 10, 2019 18:34

ehuss reviewed Sep 12, 2019

View reviewed changes

tests/testsuite/standard_lib.rs Outdated Show resolved Hide resolved

tests/testsuite/standard_lib.rs Outdated Show resolved Hide resolved

alexcrichton force-pushed the mock-std branch from 0c8631b to dfa61e5 Compare September 12, 2019 17:38

alexcrichton force-pushed the mock-std branch 2 times, most recently from affe70a to a1da038 Compare September 12, 2019 19:53

alexcrichton force-pushed the mock-std branch from a4184c4 to f2e6a04 Compare September 16, 2019 18:37

alexcrichton added 3 commits September 16, 2019 11:46

Extract support directory to its own crate

9115b2c

Extract out all our test support code to its own standalone crate so it can be shared between multiple test suites if necessary.

alexcrichton force-pushed the mock-std branch from f2e6a04 to 2180c18 Compare September 16, 2019 18:47

alexcrichton added 2 commits September 16, 2019 12:00

Don't install rust-src on nightly for normal tests

40be3bb

Run rustfmt

ebd1052

alexcrichton force-pushed the mock-std branch from 2180c18 to ebd1052 Compare September 16, 2019 19:00

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Sep 16, 2019

bors merged commit ebd1052 into rust-lang:master Sep 16, 2019

This was referenced Sep 16, 2019

check subcommand should force rebuild #6664

Closed

WIP: Support target configuration in workspace modules #7022

Closed

alexcrichton deleted the mock-std branch September 16, 2019 23:46

ehuss mentioned this pull request Jun 18, 2020

Improve support for non-master main branches #8364

Merged

ehuss added this to the 1.39.0 milestone Feb 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve test suite for `-Zbuild-std` #7350

Improve test suite for `-Zbuild-std` #7350

alexcrichton commented Sep 10, 2019

rust-highfive commented Sep 10, 2019

ehuss left a comment

alexcrichton commented Sep 12, 2019

ehuss commented Sep 12, 2019

alexcrichton commented Sep 12, 2019

alexcrichton commented Sep 12, 2019

ehuss commented Sep 12, 2019

alexcrichton commented Sep 12, 2019

bors commented Sep 16, 2019

bors commented Sep 16, 2019

ehuss commented Sep 16, 2019

bors commented Sep 16, 2019

bors commented Sep 16, 2019

bors commented Sep 16, 2019

Improve test suite for -Zbuild-std #7350

Improve test suite for -Zbuild-std #7350

Conversation

alexcrichton commented Sep 10, 2019

rust-highfive commented Sep 10, 2019

ehuss left a comment

Choose a reason for hiding this comment

alexcrichton commented Sep 12, 2019

ehuss commented Sep 12, 2019

alexcrichton commented Sep 12, 2019

alexcrichton commented Sep 12, 2019

ehuss commented Sep 12, 2019

alexcrichton commented Sep 12, 2019

bors commented Sep 16, 2019

bors commented Sep 16, 2019

ehuss commented Sep 16, 2019

bors commented Sep 16, 2019

bors commented Sep 16, 2019

bors commented Sep 16, 2019

Improve test suite for `-Zbuild-std` #7350

Improve test suite for `-Zbuild-std` #7350