Cargo runs unit tests when they are disabled and --lib is passed

[RalfJung]

Problem

When a crate has no unit tests (via test = false in its manifest), I would expect cargo to never ever try to run unit tests for that crate.

And yet it turns out that cargo test --lib will run the unit tests.

Steps

1. Run cargo test --lib --bins --examples --tests --benches (in an attempt to run "everything except for doc tests"), or just cargo test --lib
2. Note how even if the library has unit tests are disabled, cargo will run them

     Running unittests src/lib.rs (build/x86_64-unknown-linux-gnu/stage0-std/x86_64-unknown-linux-gnu/release/deps/core-04ceb2ced8c36de5)

running 0 tests


test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 7.54µs

Possible Solution(s)

--lib should just be a filter, removing from the usual test set everything that's not a "library unit test". It should not add things that were not present to begin with.

Notes

This is related to but different from #10936

Version

No response

[RalfJung]

Something similar seems to happen with benchmarks. With a manifest like this

[[bench]]
name = "allocbenches"
path = "benches/lib.rs"
test = true

[[bench]]
name = "vec_deque_append_bench"
path = "benches/vec_deque_append.rs"
harness = false

Running cargo test will only run the first benchmark, but running cargo test --lib --bins --examples --tests --benches will run both -- even though it's not a test and shouldn't be run as a test.

[weihanglo]

This seems to be covered by rust-lang/testing-devex-team#1> Once we have a new JSON output format, Cargo could get more controls over tests than relying on libtest.

[RalfJung]

I don't think this issue has anything to do with handling the output from the tests. It's all about which tests cargo decides to run.

[weihanglo]

I see. Sorry I understood it wrong.

--lib is a flag for user to explicitly requests tests in library target to be run, so that is expected. So does other flags like --test, --bench, etc.

test = false controls only the default set of tests being run. See the target selection section.

Setting targets to test = false will stop them from being tested by default. Target selection options that take a target by name (such as --example foo) ignore the test flag and will always test the given target.

[weihanglo]

--lib should just be a filter, removing from the usual test set everything that's not a "library unit test". It should not add things that were not present to begin with.

The proposed solution can only be solved with rust-lang/testing-devex-team#1, from my understanding, because Cargo needs to run libtest to know if there is any unit test defined.

[RalfJung]

test = false controls only the default set of tests being run. See the target selection section.

That's very surprising... but it explains a lot, thanks.