Generate gcov coverage data

[steveklabnik]

Issue by brson
Thursday Jul 14, 2011 at 17:33 GMT

For earlier discussion, see rust-lang/rust#690

This issue was labelled with: A-an-interesting-project, A-debuginfo, A-testsuite in the Rust repository

---

It would be nice to know what our test coverage looks like. I believe this depends on rust-lang/rust#689

[reem]

Relevant hyper issue (linking): hyperium/hyper#44

[lifthrasiir]

Kcov had an issue with position-independent executables that rustc generates by default. This should be fixed by a PR at SimonKagstrom/kcov#77. (EDIT: This is now merged, upcoming kcov v26 or later should be fine.)

[farcaller]

Just for the reference, SimonKagstrom/kcov#77 allows to get coverage output from a casual binary produced by cargo build/test (on linux host only, obviously).

[shahn]

Recently a compiler-rt commit removed the explicit profile targets, and I'm not having any luck with just adding the required functions to FUNCTIONS.builtins in make/platform/triple.mk: compiler-rt builds and seems usable, but rustc segfaults when using -C passes=insert-gcov-profiling

[shahn]

Turns out the segfault is fixed by rust-lang/rust#31176 - so this might again be a possibility to investigate!

[frewsxcv]

@shahn Have you tried running insert-gcov-profiling after rust-lang/rust#31176 got merged?

[shahn]

Yes, but I don't have it working (yet, maybe?). Currently fighting with llvm (https://llvm.org/bugs/show_bug.cgi?id=26294)

[shahn]

So, I have it working locally. It requires a few things: A custom-built (nightly) rustc and compiler-rt with support for profiling, not using more than one codegen-units, adding the insert-gcov-profiling pass to the rustc invocation as well as linking with the custom compiler-rt.
I also had to symlink the stdlib source into my project's src directory because I couldn't figure out how to ignore these. Another issue happened around the lazy_static crate I used, because lcov really wanted a file <lazy_static macros> to exist in my project's top level directory.

The generated coverage data looks mostly ok, but has some strange things:

• it reports all derives as unused
• in all struct declarations, it reports the declaration of members as unused
• it reports "pub mod xy" lines as unused

The picture gets a bit worse when looking at branch coverage, which seems basically unusable for Rust in its current state. use statements sometimes creates tons of branches, mostly in never-executed basic blocks, but sometimes even in executed basic blocks. No idea what's up there. Generally rust code seems to be very branch heavy.

[shahn]

Oh, one more thing. One cannot use gcc's gcov tool to interpret the data depending on gcc version, but rather llvm-cov (to be built from the make generated during rustc's build process) because the data format varies wildly between gcov versions. gcov from new GCC just segfaults when given such a gcda file.

What's the process to proceed here? I would love to add support for this to be easier and actually be eventually shipped by default with rustc (I'm thinking "cargo coverage" or something). This would need build system changes, I have absolutely zero idea about windows compatibility (it works on OSX and Linux), and also an idea how to distribute the necessary compiler-rt as well as llvm-cov (perhaps relabeled to rustcov or something to avoid name clashes with a concurrently installed llvm). Is it worthwhile to start the process of implementing the necessary changes?

[nrc]

@shahn Could you link to your branch please? It would be good to see what changes are necessary to rustc.

The ideal way forward for this is to have a crate which does it outside of the main Rust repo and to develop it there until it is a bit more mature. Deep changes to the compiler should land if needed, but a lot of stuff can be done by using the compiler as a library and wrapping calls to it. You can also make Cargo plugins to experiment with cargo coverage. If there is not much to do other than compiler changes then the next step would be to put a PR together and discuss on that. If it is a lot of changes, you could make a compiler RFC, but that is probably not necessary.

This sounds really cool! Thanks for investigating it.

[shahn]

Yes, of course. Sorry for the delay, I was busy otherwise. Note that this is not even in pre-PR state, it is just enough to get the coverage build working for rustc on OSX.

• compiler-rt commit: shahn/compiler-rt@e46ba70
• rustc commit: shahn/rust@421f978

[frewsxcv]

@shahn Out of curiosity, do you remember why you removed the -std=c99 flag in the rustc commit? Also, after those changes, what did you do to actually generate coverage? Just run gcov?

[shahn]

the -std=c99 was added in a commit for msvc compat iirc (I'm not sure if it is required), and breaks the build for compiler-rt with coverage. I don't have a windows box to test this on. To actually generate the coverage, you need to:
RUSTC=$repo/$target/stage1/bin/rustc cargo rustc -- -C passes=insert-gcov-profiling -C link-args=$rustc/$target/rt/compiler-rt/triple/builtins/libcompiler_rt.a

where $rustc is your rust git checkout and $target is the target triple

[briansmith]

Consider this:

x = try!(foo());

I would like the code coverage to report which of the Ok and Err branches were considered. i.e. like coverage tools would do if macros were expanded. In general, every branch hidden by macros should be reported separately.

Similarly, in the case of the ? operator, every hidden branch of the ? should be reported separately.

[whitequark]

@shahn ping?

[shahn]

@whitequark I linked the branches above, I wasn't ready to commit more because i still am not clear how it all fits together. What exactly are you pinging about?

[whitequark]

@shahn I wondered if you did any more work on this--but nevermind, I already have an idea of how to proceed and will try to send some PRs shortly.

[whitequark]

See rust-lang/rust#38608 writeup on how to generate useful information from this is incoming.

[whitequark]

See https://users.rust-lang.org/t/howto-generating-a-branch-coverage-report/8524.

[marco-c]

rust-lang/rust#42433 (a rebase of rust-lang/rust#38608) was merged.
rust-lang/rust#42524 is tracking the stabilization of the feature.

[steveklabnik]

https://github.com/kennytm/cov

[marco-c]

https://github.com/kennytm/cov

There's also https://github.com/marco-c/grcov, which is relying on LLVM to parse the GCOV files instead of reimplementing the parsing.

[Centril]

I believe the crates referenced above suffice for now. Closing this therefore.