support code coverage when testing

[andrewrk]

Add a --coverage argument to zig test and have it generate a nice report.

Related:

• debug info audit - many virtual memory addresses have strange source locations #20989
• unreachable code paths need to be excluded from having coverage instrumentation #20992

[PavelVozenilek]

Based on my unpleasant experience with gcov (GCC coverage tool) in foolish drive for 100% coverage: there should be some easy way to label certain lines or whole functions as "doesn't need coverage". All kinds of unlikely situations, impossible to emulate.

gcov doesn't support anything like this, so I write a tool which scanned all C source files, searched for "no coverage" tags and removed these lines from gcov output.

It looked like:

if (...)  {
    exit(1); //--   (comment "//--" means no coverage for this line)
}

// The whole function is taken out of coverage via the //-->> and //<<-- brackets
void foo(void) { //-->>
  ...
} //<<--

There should be also some way to skip coverage checking for 3pp libraries, w/o touching their source code.

[tiehuis]

Just though I'd mention that kcov works perfectly as is with zig at the moment. I think coverage in general is slightly more useful than it normally would be for zig as well, since it clearly shows what segments of code were never compiled. This helps with compile-time evaluation heavy bits.

Only steps required are:

zig test bigint.zig
kcov kcov-out ./zig-cache/test
firefox kcov-out/index.html

[andrewrk]

Oh that's really interesting with the defers in yellow. The number tells you how many different ways there were to exit the block, and it only goes green if you manage to hit them all (in this case by having each try activate the error handling). Same thing with try, it'll be yellow unless you managed to test the error return.

The defer thing might change with #283

[andrewrk]

I tried this using zig test on the std lib. It's really quite useful, but I see some confusing output. In my test results I see:

Test 341/466 os.path.join...OK
Test 342/466 os.path.isAbsoluteWindows...OK
Test 343/466 os.path.isAbsolutePosix...OK
Test 344/466 os.path.windowsParsePath...OK
Test 345/466 os.path.resolve...OK
Test 346/466 os.path.resolveWindows...OK
Test 347/466 os.path.resolvePosix...OK
Test 348/466 os.path.dirnamePosix...OK
Test 349/466 os.path.dirnameWindows...OK
Test 350/466 os.path.basename...OK
Test 351/466 os.path.relative...OK
Test 352/466 os.path.real...OK

but in the coverage report I see no coverage:

I wonder what's going on here.

[piyushchauhan2011]

Looking forward to this option 🚀

[Pyrolistical]

kcov still works, the instructions have changed slightly as ./zig-cache/test doesn't exist.

rm -rf zig-cache/o
zig test lib/std/bit_set.zig
kcov kcov-out ./zig-cache/o/*/test
open kcov-out/index.html

It is very slow on my M1 Macbook Air. Took over 5 minutes.

[paulocoghi]

It would be interesting to generate a coverage report based on the kcov one* where the different logical possibilities and ramifications are informed in detail, as Jest does:
(informing the coverage of "statements", "branch", "functions", "lines" and their logical ramifications)

-----------|---------|----------|---------|---------|-------------------
File       | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s 
-----------|---------|----------|---------|---------|-------------------
All files  |     100 |      100 |     100 |     100 |                   
 index.js  |     100 |      100 |     100 |     100 |                   
 server.js |     100 |      100 |     100 |     100 |                   
-----------|---------|----------|---------|---------|-------------------
Test Suites: 2 passed, 2 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        3.735 s, estimated 5 s
Ran all test suites.

*(kcov seems to provide everything that's necessary for this)

[perillo]

I tried this using zig test on the std lib. It's really quite useful, but I see some confusing output. In my test results I see:

Test 341/466 os.path.join...OK
Test 342/466 os.path.isAbsoluteWindows...OK
Test 343/466 os.path.isAbsolutePosix...OK
Test 344/466 os.path.windowsParsePath...OK
Test 345/466 os.path.resolve...OK
Test 346/466 os.path.resolveWindows...OK
Test 347/466 os.path.resolvePosix...OK
Test 348/466 os.path.dirnamePosix...OK
Test 349/466 os.path.dirnameWindows...OK
Test 350/466 os.path.basename...OK
Test 351/466 os.path.relative...OK
Test 352/466 os.path.real...OK

but in the coverage report I see no coverage:

I wonder what's going on here.

kcov uses DWARF debugging information for compiled programs to make it possible to collect coverage information without special compiler switches.

In https://www.youtube.com/watch?v=1QMHbp5LUKg Simon Kagstrom says that he found that debugging information is often broken on Linux systems, because kcov uses debug information that are not usually used by debuggers (and thus not well tested).