You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have been having issues with debugging integration tests, that often fail without unit tests failing alongside them. In order to fix this, we need to increase our code coverage for unit tests specifically.
In addition, the project was kickstarted by the Hacken report, that was suggesting we implement some bare minimum of code coverage.
Why should NEAR One work on this
Nearcore should see a change in the way of thinking about tests. If each new change added relevant tests, then we would be able to develop much more confidently than we are right now, and fear breakage less. We would also spend less time debugging failures in production.
All this, together, would lead to much improved developer productivity.
What needs to be accomplished
We need to:
report numbers on code coverage
change our culture, so that all our new code at least is properly tested
enforce the testing culture, by rejecting PRs that do not have enough tests
Work on 1 has already started for the Hacken report remediation, and I worked on it on-and-off depending on other issues that took priority. We currently report overall numbers, but should get more precise numbers depending on the part of the codebase affected, as we do not need full coverage for eg. the tools folder.
Main use case
Reliability is the main use case. However, security will also benefit indirectly, as fewer bugs usually means fewer vulnerabilities.
Links to external documentations and discussions
None
Estimated effort
The security team, and in particular @Ekleog-NEAR, is expected to work on this project. The estimated time is 2-3 weeks-person, that will need to be spread out to let enough time for culture changes to percolate.
Assumptions
None
Pre-requisites
None
Out of scope
Catching up on our backlog of untested code base is out of the scope of this project, but it will be enabled by proper measurement of code coverage.
Task list
The content you are editing has changed. Please copy your edits and refresh the page.
Since the last report, I configured code coverage properly. Most of the technical changes are done here, now remains what’s most important: changing the mentality so that people do not land PRs that do not meet the requirements for code coverage.
The main thing missing is this missing feature request I sent to codecov, that I noticed while writing this comment: it’s currently hard to find which lines are missing unit tests in particular.
This being said, the mentality change is going to be nontrivial: over the last week (during which code coverage was configured properly), excluding automated PRs:
4 PRs landed with all green status checks
4 PRs landed with spuriously red status checks (due to not testing some Debug impl, or having merged master into the branch itself)
0 PRs landed with red status checks but green code coverage, due to some other nonblocking check (eg. url validity)
10 PRs landed with red code coverage checks that had no obvious reason I could find
This is basically done, and I implemented the unit test coverage reporting.
I’ll leave this open for now just to not forget making a proper announcement for it, and then we can close: the only thing that’s missing will be incrementally ramping up our code coverage :)
Goals
Background
We have been having issues with debugging integration tests, that often fail without unit tests failing alongside them. In order to fix this, we need to increase our code coverage for unit tests specifically.
In addition, the project was kickstarted by the Hacken report, that was suggesting we implement some bare minimum of code coverage.
Why should NEAR One work on this
Nearcore should see a change in the way of thinking about tests. If each new change added relevant tests, then we would be able to develop much more confidently than we are right now, and fear breakage less. We would also spend less time debugging failures in production.
All this, together, would lead to much improved developer productivity.
What needs to be accomplished
We need to:
Work on 1 has already started for the Hacken report remediation, and I worked on it on-and-off depending on other issues that took priority. We currently report overall numbers, but should get more precise numbers depending on the part of the codebase affected, as we do not need full coverage for eg. the tools folder.
Main use case
Reliability is the main use case. However, security will also benefit indirectly, as fewer bugs usually means fewer vulnerabilities.
Links to external documentations and discussions
None
Estimated effort
The security team, and in particular @Ekleog-NEAR, is expected to work on this project. The estimated time is 2-3 weeks-person, that will need to be spread out to let enough time for culture changes to percolate.
Assumptions
None
Pre-requisites
None
Out of scope
Catching up on our backlog of untested code base is out of the scope of this project, but it will be enabled by proper measurement of code coverage.
Task list
Tasks
The text was updated successfully, but these errors were encountered: