Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve reporting of tests coverage #1893

Closed
Tracked by #1739
skosito opened this issue Mar 13, 2024 · 3 comments · Fixed by #1899
Closed
Tracked by #1739

Improve reporting of tests coverage #1893

skosito opened this issue Mar 13, 2024 · 3 comments · Fixed by #1899
Assignees
Labels
test Tests related

Comments

@skosito
Copy link
Contributor

skosito commented Mar 13, 2024

Describe the Issue

Currently, for packages without any _test.go file, go test reports [no-test-files] and coverage is not included at all (eg. common/bitcoin package, not present in codecov report, because there are no tests, latest develop report: https://app.codecov.io/gh/zeta-chain/node/tree/develop/common). Reason for this is that packages that are without test files are not included in coverage.out which is uploaded to codecov.

Expected Outcome

These type of packages should be reported with 0% coverage, so they are detected and included in overall coverage, otherwise coverage is not correct.

Details

There are a lot of details in this issue: golang/go#24570 and it seems that this works as expected in go1.22 https://tip.golang.org/doc/go1.22#tools where 0% coverage is reported instead. But for now we can do the following (summary from various github issues):

  • prefered solution: add empty _test.go file in packages that are reported with [no test files] and we want to include in the coverage, for example adding empty _test.go file in common/bitcoin package shows it in the coverage.
    While this is not super clean, it seems like the most accurate solution (also recommended throughout this issue, some projects even implemented CI to automatically add this).

  • use -coverpkg=./... option with go test - from docs:

Apply coverage analysis in each test to packages matching the patterns.
The default is for each test to analyze only the package being tested.

This kinda works, but not quite. If package doesn't contain any test, but test of some other package calls it, it will still be covered. Eg:
common/chain.go function IsHeaderSupportedEvmChain is not covered with unit tests in common package, but because it is called in message_add_block_header.go, which has tests, it is also reported as covered when this flag is used. This is not correct in context of unit tests coverage, where each package that is reporting coverage is expected to have its own tests.

Also, it doesn't solve the problem completely - if package is not referenced in any test, it still won't be included in coverage.

Furthermore, this flag has multiple bugs after go1.19, which is solved by using GOEXPERIMENT=nocoverageredesign (different coverage results are reported with and without this flag: golang/go#58770 and it is confirmed locally).

@skosito skosito self-assigned this Mar 13, 2024
@skosito skosito added the test Tests related label Mar 13, 2024
@lumtis
Copy link
Member

lumtis commented Mar 14, 2024

prefered solution: add empty _test.go file in packages that are reported with [no test files] and we want to include in the coverage, for example adding empty _test.go file in common/bitcoin package shows it in the coverage.
While this is not super clean, it seems like the most accurate solution (also recommended throughout this issue, some projects even implemented CI to automatically add this).

Any drawback on this solution? It seems trivial and also I don't think cleanliness is a concerns, we need to add the tests so good thing to add a placeholder

@skosito
Copy link
Contributor Author

skosito commented Mar 14, 2024

prefered solution: add empty _test.go file in packages that are reported with [no test files] and we want to include in the coverage, for example adding empty _test.go file in common/bitcoin package shows it in the coverage.
While this is not super clean, it seems like the most accurate solution (also recommended throughout this issue, some projects even implemented CI to automatically add this).

Any drawback on this solution? It seems trivial and also I don't think cleanliness is a concerns, we need to add the tests so good thing to add a placeholder

I dont think there are any drawbacks, it should be straightforward. Probably can be added manually now, and for future if new package is added without any test files, codecov PR check should catch it, so we dont need any additional CI checks for this?

@lumtis
Copy link
Member

lumtis commented Mar 14, 2024

I dont think there are any drawbacks, it should be straightforward. Probably can be added manually now, and for future if new package is added without any test files, codecov PR check should catch it, so we dont need any additional CI checks for this?

We should prevent in our code reviews to let someone add a new package without any tests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test Tests related
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants