Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit coverage to requested files #1467

Merged

Conversation

keith
Copy link
Member

@keith keith commented Jul 14, 2022

Bazel provides COVERAGE_MANIFEST which is a file we can pass through to
llvm-cov to limit the files that we produce coverage for. Using this
revealed the path-equivalence was backwards because we're using the
coverage prefix map to disable absolute paths in code coverage info,
but llvm-cov operates on absolute paths, so we have to remap the . in
the coverage data to the full path so that llvm-cov understands the
relative paths (which it absolutizes internal) from the manifest file.
This feels a bit weird and ideally it could operate entirely on these
relative paths instead.

This has the side effect of excluding files that aren't part of bazel
sources, which fixes #1466

This should also improve interaction with --instrumentation_filter.

Bazel provides COVERAGE_MANIFEST which is a file we can pass through to
llvm-cov to limit the files that we produce coverage for. Using this
revealed the path-equivalence was backwards because we're using the
coverage prefix map to disable absolute paths in code coverage info,
but llvm-cov operates on absolute paths, so we have to remap the `.` in
the coverage data to the full path so that llvm-cov understands the
relative paths (which it absolutizes internal) from the manifest file.
This feels a bit weird and ideally it could operate entirely on these
relative paths instead.

This has the side effect of excluding files that aren't part of bazel
sources, which fixes bazelbuild#1466

This should also improve interaction with `--instrumentation_filter`.
@UebelAndre
Copy link
Collaborator

This file feels a bit cryptic to me. I'm happy to hear it's working but don't know enough about what it's doing to think about how to test it. Could this potentially be something that's replaced with a rust_binary? I would imagine the use of bash here adds an unnecessary requirement to windows hosts and also makes testing just a bit harder to author.

@keith
Copy link
Member Author

keith commented Jul 15, 2022

Currently with bazel's infra that reads this file I don't think we have a choice, unless we add a custom test runner which we talked about briefly on the other PR I think. It's also bash so realistically I guess it won't work on windows either way until bazel becomes more flexible for coverage.

I wonder if we could write an integration test that could transition on coverage and depend on a rust_test and validate the coverage file contents 🤔

@keith
Copy link
Member Author

keith commented Jul 15, 2022

Note as far as this change goes the important part is limiting the files to the ones bazel tells us. The order argument was wrong in the first place it was just being ignored until we started limiting the file.

@scentini
Copy link
Collaborator

I wonder if we could write an integration test that could transition on coverage and depend on a rust_test and validate the coverage file contents

This should be easily doable.

I would very much like to
a) have tests for coverage in general before declaring it supported, as it is hard to see what is being fixed here; or
b) make it guarded by an experimental flag as long as a) is not the case.

Copy link
Collaborator

@krasimirgg krasimirgg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixes look good, +1 to coming up with coverage tests robust enough to be able to verify fixes like these (possibly as follow-ups).

@UebelAndre UebelAndre enabled auto-merge (squash) July 15, 2022 13:09
@fmorency
Copy link
Contributor

I tested my project, and everything worked as expected. Thanks a lot 👍🏻

@UebelAndre UebelAndre merged commit e83d5f3 into bazelbuild:main Jul 15, 2022
@keith keith deleted the ks/limit-coverage-to-requested-files branch July 15, 2022 18:52
@keith keith restored the ks/limit-coverage-to-requested-files branch July 15, 2022 18:52
@keith keith deleted the ks/limit-coverage-to-requested-files branch July 15, 2022 18:52
@keith keith restored the ks/limit-coverage-to-requested-files branch July 15, 2022 18:52
@keith
Copy link
Member Author

keith commented Jul 18, 2022

I tried to spike out a quick test, it doesn't seem like there is any way to access the coverage.dat file from dependencies of a test, I was envisioning something like:

rust_test(
    name = "foo",
...
)

coverage_validation_test(
     name = "foo.coverage_validation",
     target = "foo",
)

but in this test's implementation I don't see a way to get my hands on that file to do something with it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Coverage includes reference to non-project file when using thread_local!
5 participants