Skip to content

Commit a5d633d

Browse files
bkhouribkhouri
committed
SwiftPM: HTML Coverage Report
1 parent 79b68f9 commit a5d633d

File tree

2 files changed

+186
-1
lines changed

2 files changed

+186
-1
lines changed

‎proposal-templates/0000-swiftpm-template.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77

88
*During the review process, add the following fields as needed:*
99

10-
* Implementation: [apple/swift-package-manager#NNNNN](https://github.com/apple/swift-package-manager/pull/NNNNN)
10+
* Implementation: [swiftlang/swift-package-manager#NNNNN](https://github.com/swiftlang/swift-package-manager/pull/NNNNN)
1111
* Decision Notes: [Rationale](https://forums.swift.org/), [Additional Commentary](https://forums.swift.org/)
1212
* Bugs: [SR-NNNN](https://bugs.swift.org/browse/SR-NNNN), [SR-MMMM](https://bugs.swift.org/browse/SR-MMMM)
1313
* Previous Revision: [1](https://github.com/swiftlang/swift-evolution/blob/...commit-ID.../proposals/NNNN-filename.md)

‎proposals/NNNN-swiftpm-html-coverage-report.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
# HTML Coverage Report
2+
3+
* Proposal: [SE-NNNN](NNNN-swiftpm-html-coverage-report.md)
4+
* Authors: [Sam Khouri](https://github.com/bkhouri)
5+
* Review Manager: TBD
6+
* Status: **Awaiting implementation**
7+
8+
*During the review process, add the following fields as needed:*
9+
10+
* Implementation: [swiftlang/swift-package-manager#9076][PR]
11+
* Decision Notes: [Pitch](https://forums.swift.org/t/pitch-adding-html-coverage-support/82358)
12+
13+
## Introduction
14+
15+
Currently, `swift test` supports generating a JSON coverage report, which is
16+
great for ingesting into various systems. The JSON, however, is not very "human
17+
consumable friendly" while iterating at-desk.
18+
19+
This proposes adding an additional command line argument to `swift test` that
20+
would allow the caller to select the generation of an HTML coverage report.
21+
22+
23+
## Motivation
24+
25+
JSON coverage report is great for ingesting into exgernal tools that post-process
26+
the coverage data. Allowing users to generate an HTML coverage report allows
27+
the HTML coverage report to be uploaded to CI system for visual inspection, and
28+
even allows users to generate HTML coverage report at-desk, allow faser feedback
29+
to determine if the current changes are sufficiently covered to their liking.
30+
31+
## Proposed solution
32+
33+
if users currently want an HTML report, custom scripts are written to get access
34+
to the coverage data, and then invoking the `llvm-cov` binary directly
35+
36+
e.g.:
37+
38+
```
39+
❯ swift test --enable-code-coverage
40+
41+
❯ swift test --show-codecov-path
42+
43+
❯ llvm-cov show \
44+
--project-title="HelloWorld" \
45+
--format="html" \
46+
--output-dir=".coverage" \
47+
--instr-profile=".build/arm64-apple-macosx/debug/codecov/default.profdata" \
48+
".build/.../HelloWorldPackageTests.xctest/Contents/MacOS/HelloWorldPackageTests" \
49+
"Sources"
50+
```
51+
52+
Since SwiftPM currently orchestrates the JSON coverage data, the solution adds a
53+
new command line argument, e.g.: `--coverage-format` to `swift test` which can
54+
be specified multiple times, to generate multiple coverage report type from a
55+
single test execution.
56+
57+
While processing the coverage data, SwiftPM will loop on all the unique coverage
58+
format options to generate the specified reports.
59+
60+
Unless otherwise specified, this proposal applies only to the HTML Coverage
61+
report. The generation of the JSON Coverage report is unchanged and is out of
62+
scope.
63+
64+
## Detailed design
65+
66+
Existing tools in LLVM have been around for several years (and maybe even decades),
67+
and provides robust tools for code coverage analysis. The LLVM tools are
68+
well-documented, and have been used in production for many years. SwiftPM will
69+
make use of this and contruct the proper command line arguments to the
70+
`llvm-cov show` utility, which will generate the HTML report.
71+
72+
### Format Selection
73+
74+
The `swift test` command line will have an option named `--coverage-format`,
75+
which accepts either `json` or `html`. This option can be specified multipe
76+
times on the command line, and a report will be generated for each format
77+
specified.
78+
79+
The command line option will be defined as
80+
81+
```swift
82+
@Option(
83+
name: [
84+
.customLong("codecov-format"),
85+
.customLong("code-coverage-format"),
86+
.customLong("coverage-format"),
87+
],
88+
help: ArgumentHelp(
89+
"Format of the code coverage output. Can be specified multiple times.",
90+
valueName: "format",
91+
)
92+
)
93+
var formats: [CoverageFormat] = [.json]
94+
```
95+
where `CoverageFormat` is implemented as
96+
97+
```swift
98+
package enum CoverageFormat: String, ExpressibleByArgument, CaseIterable, Comparable {
99+
case json
100+
case html
101+
102+
package var defaultValueDescription: String {
103+
switch self {
104+
case .json: "Produces a JSON coverage report."
105+
case .html: "Produces an HTML report producd by llvm-cov. TODO: Document the response file location."
106+
}
107+
}
108+
109+
package static func < (lhs: CoverageFormat, rhs: CoverageFormat) -> Bool {
110+
return lhs.rawValue < rhs.rawValue
111+
}
112+
}
113+
```
114+
115+
116+
### Coverage Report configuration
117+
118+
`llvm-cov show` has several report configurability options. In order to
119+
prevent a "command line arguments" explosion to `swift test`, a response file
120+
will be supported. If the response file is found in a location relaltive to the
121+
Swift package, namely in `<repo>/.swiftpm/configuration/coverage.html.report.args.txt`,
122+
the contents of the file will be used.
123+
124+
The only `llvm-cov show` command-line argument SwiftPM will enforce is
125+
`--format=html`, as it contradictory to have specified `--coverage-format html`
126+
on the `swift test` command line and a different format be generated.
127+
128+
SwiftPM will not perform any validation on the response file contents, except
129+
to determine the output location.
130+
131+
### Coverage report location
132+
133+
By default, the HTML report will be created in location under the Scratch Path.
134+
However, this can be overriden using the response file.
135+
136+
137+
### Show coverage path
138+
Prior to this proposal `swift test --show-coverage-path` would display an single
139+
absolute path location to the JSON coverage report.
140+
141+
Since `--coverage-format` can be specified multiple times, it's output must be
142+
changed to reflect the new functionality.
143+
144+
If the `--coverage-format` option is specified on the `swift test` command line
145+
a single time (or is not specified at all), there is no change to the output.
146+
147+
148+
if `--coverage-format` is specified multiple times, the output must reflect this.
149+
A new command line argument `--print-coverage-path-mode` option will be available
150+
to describe the visual representation of the output paths. The options accepts
151+
`json`json` or `text` and applies whenever `--show-coverage-path` is selected.
152+
153+
A value of `json` will output a JSON object with the key representing the format,
154+
and the value representing the output location of said format.
155+
156+
A value of `text` will omit the format in the output if a single `coverage-format`
157+
is requested. Otherwise, the output will be similar to
158+
159+
```
160+
❯ swift test -c release --build-system swiftbuild --show-coverage-path --coverage-format html --coverage-format json
161+
Building for debugging...
162+
[5/5] Write swift-version-4B9677C1F510A69F.txt
163+
Build complete! (0.37s)
164+
Html: /swift-package-manager/.build/arm64-apple-macosx/Products/Release/codecov/Simple-html
165+
Json: /swift-package-manager/.build/arm64-apple-macosx/Products/Release/codecov/Simple.json
166+
```
167+
168+
## Security
169+
170+
There are no security implication.
171+
172+
## Impact on existing packages
173+
174+
No impact is expected.
175+
176+
## Alternatives considered
177+
178+
- In addition to the response file, the coverage report generation can support
179+
a command line argument similar to `-Xlinker`, `-Xcc` and others, which will
180+
pass the arguments to `llvm-cov show` and override the values in the response
181+
file. This has NOT be implemented in the [PR].
182+
183+
184+
185+
[PR]: https://github.com/swiftlang/swift-package-manager/pull/9076

0 commit comments

Comments
 (0)