Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new CoverageKind::Branch to MIR #115061

Closed
wants to merge 16 commits into from

Conversation

Swatinem
Copy link
Contributor

@Swatinem Swatinem commented Aug 21, 2023

This adds a couple intermediate types, and threads branch coverage from MIR through to codegen, and emits branch coverage regions via LLVM.

Based on top of #114399 and #115058, and so far untested, as we do not (yet) generate any branches in MIR.

…LLVM

We compile each test file to LLVM IR assembly, and then pass that IR to a
dedicated program that can decode LLVM coverage maps and print them in a more
human-readable format. We can then check that output against known-good
snapshots.

This test suite has some advantages over the existing `run-coverage` tests:

- We can test coverage instrumentation without needing to run target binaries.

- We can observe subtle improvements/regressions in the underlying coverage
mappings that don't make a visible difference to coverage reports.
The output of these tests is too complicated to comfortably verify by hand, but
we can still use them to observe changes to the underlying mappings produced by
codegen/LLVM.
After coverage instrumentation and MIR transformations, we can sometimes end up
with coverage expressions that always have a value of zero. Any expression
operand that refers to an always-zero expression can be replaced with a literal
`Operand::Zero`, making the emitted coverage mapping data smaller and simpler.

This simplification step is mostly redundant with the simplifications performed
inline in `expressions_with_regions`, except that it does a slightly more
thorough job in some cases (because it checks for always-zero expressions
*after* other simplifications).

However, adding this simplification step will then let us greatly simplify that
code, without affecting the quality of the emitted coverage maps.
The LLVM API that we use to encode coverage mappings already has its own code
for removing unused coverage expressions and renumbering the rest.

This lets us get rid of our own complex renumbering code, making it easier to
refactor our coverage code in other ways.
@rustbot
Copy link
Collaborator

rustbot commented Aug 21, 2023

r? @davidtwco

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 21, 2023
@Zalathar
Copy link
Contributor

@rustbot label +A-code-coverage

@rustbot rustbot added the A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) label Aug 22, 2023
This removes quite a bit of indirection and duplicated code related to getting the `FunctionCoverage`.
This adds a couple intermediate types, and threads branch coverage from MIR through codegen.
@rustbot rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Aug 22, 2023
@Swatinem
Copy link
Contributor Author

Rebased on top of #114399 which makes the changes here a lot simpler.

@Swatinem
Copy link
Contributor Author

A first proof of concept seems to be functional and it outputs the following counters / mappings (see 7f985c4):

- Branch at (prev + 2, 8) to (start + 0, 12)
    true  = c1
    false = (c0 - c1)

As well as llvm-cov output:

  ------------------
  |  Branch (LL:13): [True: 1, False: 5]
  ------------------

There are still a couple of problems with this POC that I will take a look at this week:

  • code_regions are not yet correct for macros and derives and match statements.
  • llvm-cov is also outputting different lines for generic instantiation, but I believe that is rather the way how llvm-cov reports these things.
  • The column numbers seem to be way off sometimes, reporting thing like Branch (LL:192675): [True: 0, False: 0].
  • This still needs to be hooked up to the existing expression simplification, as it is currently outputting (Zero - Zero) expressions for unreachable branches.
  • It seems like the code is outputting duplicates for some Code(Zero) regions.

@Zalathar
Copy link
Contributor

I was surprised to see changes to coverage-dump, because it should already work with branch regions as-is. (I tested it against a C program compiled with clang.)

In the existing implementation, it prints the raw true/false counters inline, and then also expands expressions (if any) on separate lines as needed.

Comment on lines +171 to +172
/// Additionally, instrument branches and output branch overage
Branch,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To save anyone else the trouble of checking whether this is accidentally stable (it's not):

New values of InstrumentCoverage are automatically treated as nightly-only unless they are explicitly added to the check shown below on line 2674, so Branch is correctly treated as a nightly-only value. 👍

// Handle both `-Z instrument-coverage` and `-C instrument-coverage`; the latter takes
// precedence.
match (cg.instrument_coverage, unstable_opts.instrument_coverage) {
(Some(ic_c), Some(ic_z)) if ic_c != ic_z => {
handler.early_error(
"incompatible values passed for `-C instrument-coverage` \
and `-Z instrument-coverage`",
);
}
(Some(InstrumentCoverage::Off | InstrumentCoverage::All), _) => {}
(Some(_), _) if !unstable_opts.unstable_options => {
handler.early_error("`-C instrument-coverage=except-*` requires `-Z unstable-options`");
}
(None, None) => {}
(None, ic) => {
handler
.early_warn("`-Z instrument-coverage` is deprecated; use `-C instrument-coverage`");
cg.instrument_coverage = ic;
}
_ => {}
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I haven’t yet hooked up the option, so it doesn’t do anything right now, yet.

@Swatinem
Copy link
Contributor Author

In the existing implementation, it prints the raw true/false counters inline, and then also expands expressions (if any) on separate lines as needed.

I didn’t know this was intentional, as the inline Debug output looks a bit untidy.
I reverted these changes, but renamed the identifiers to r#true to have slightly tidier output.

@Zalathar
Copy link
Contributor

Zalathar commented Aug 23, 2023

In that case, I’ll switch over to r#true and r#false in the original branch as well.

How do you feel about only showing the resolved lines when the true/false counter is an expression, versus always showing them? I chose to hide them (to keep the output compact), but if you think always showing them is better overall, I can make that change in my branch too.

@Swatinem
Copy link
Contributor Author

How do you feel about only showing the resolved lines when the true/false counter is an expression, versus always showing them? I chose to hide them (to keep the output compact), but if you think always showing them is better overall, I can make that change in my branch too.

IMO for Branch it does make sense to have the true/false expressions on separate lines, but it does not make such a difference bth.

This extends the current simplification code to not only replace operands by `Zero`, but also to remove trivial `Counter + Zero` expressions and replace those with just `Counter`.
@davidtwco davidtwco added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 28, 2023
@davidtwco
Copy link
Member

Use @rustbot ready when you want this to be reviewed.

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Oct 23, 2023
…esleywiser

Introduce `-C instrument-coverage=branch` to gate branch coverage

This was extracted from rust-lang#115061 and can land independently from other coverage related work.

The flag is unused for now, but is added in advance of adding branch coverage support.
It is an unstable, nightly only flag that needs to be used in combination with `-Zunstable-options`, like so: `-Zunstable-options -C instrument-coverage=branch`.

The goal is to develop branch coverage as an unstable opt-in feature first, before it matures and can be turned on by default.
bors added a commit to rust-lang-ci/rust that referenced this pull request Oct 24, 2023
…leywiser

Introduce `-C instrument-coverage=branch` to gate branch coverage

This was extracted from rust-lang#115061 and can land independently from other coverage related work.

The flag is unused for now, but is added in advance of adding branch coverage support.
It is an unstable, nightly only flag that needs to be used in combination with `-Zunstable-options`, like so: `-Zunstable-options -C instrument-coverage=branch`.

The goal is to develop branch coverage as an unstable opt-in feature first, before it matures and can be turned on by default.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Oct 24, 2023
…esleywiser

Introduce `-C instrument-coverage=branch` to gate branch coverage

This was extracted from rust-lang#115061 and can land independently from other coverage related work.

The flag is unused for now, but is added in advance of adding branch coverage support.
It is an unstable, nightly only flag that needs to be used in combination with `-Zunstable-options`, like so: `-Zunstable-options -C instrument-coverage=branch`.

The goal is to develop branch coverage as an unstable opt-in feature first, before it matures and can be turned on by default.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Oct 24, 2023
…esleywiser

Introduce `-C instrument-coverage=branch` to gate branch coverage

This was extracted from rust-lang#115061 and can land independently from other coverage related work.

The flag is unused for now, but is added in advance of adding branch coverage support.
It is an unstable, nightly only flag that needs to be used in combination with `-Zunstable-options`, like so: `-Zunstable-options -C instrument-coverage=branch`.

The goal is to develop branch coverage as an unstable opt-in feature first, before it matures and can be turned on by default.
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Oct 24, 2023
Rollup merge of rust-lang#116094 - Swatinem:coverage-branch-gate, r=wesleywiser

Introduce `-C instrument-coverage=branch` to gate branch coverage

This was extracted from rust-lang#115061 and can land independently from other coverage related work.

The flag is unused for now, but is added in advance of adding branch coverage support.
It is an unstable, nightly only flag that needs to be used in combination with `-Zunstable-options`, like so: `-Zunstable-options -C instrument-coverage=branch`.

The goal is to develop branch coverage as an unstable opt-in feature first, before it matures and can be turned on by default.
@Dylan-DPC
Copy link
Member

@Swatinem any updates on this? thanks

@Swatinem
Copy link
Contributor Author

Swatinem commented Nov 5, 2023

I will close this PR for now.
Smaller parts of it have landed independently, and @Zalathar is still hard at work overhauling the guts of coverage instrumentation, which I am keeping track of and reviewing. Apart from that I don’t have too much time to dedicate to this right now, though I might pick that up again once I do.

@Swatinem Swatinem closed this Nov 5, 2023
@Swatinem Swatinem deleted the counter-mapping-region branch May 25, 2024 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-code-coverage Area: Source-based code coverage (-Cinstrument-coverage) A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants