Add timeout as configurable coverage criteria #1118
Conversation
mbj
force-pushed
the
timeout/configures
branch
6 times, most recently
from
b234ad1
to
f4c40aa
Compare
mbj
force-pushed
the
timeout/configures
branch
from
adf0be4
to
75a82a2
Compare
| private_constant(*constants(false)) | ||
|
|
||
| class CoverageCriteria | ||
| include Anima.new(:timeout, :test_result) |
There was a problem hiding this comment.
@dgollahon The next step is to allow to configure the coverage criteria per subject in the config file an in-line.
Same for the mutation timeout.
The ideal setting than is a global "high" timeout with coverage_criteria.timeout = false and a per subject one after manual review (From an alive, example an infinite loop) setting it to true.
There was a problem hiding this comment.
Also there are more possible criteria like warnings etc.
There was a problem hiding this comment.
And the most important criteria for @pawelpacana counting ruby aborts as killed.
Next major feature for 0.10.8. The infrastructure exists now.
Per subject config will be added in 0.10.9 possibly also in-line configuration.
| # | ||
| # @return [undefined] | ||
| def run | ||
| visit(MutationResult, object.mutation_result) |
There was a problem hiding this comment.
This is a pass through ATM, I could evision a verbose mode where we show covered mutations, and tell why they where killed.
| @@ -7,7 +7,7 @@ class Printer | |||
| # Subject result printer | |||
| class SubjectResult < self | |||
|
|
|||
| delegate :subject, :alive_mutation_results, :tests | |||
| delegate :subject, :uncovered_results, :tests | |||
There was a problem hiding this comment.
I'm slowly moving the wording away from "alive" and "killed" to "uncovered" and "covered".
| else | ||
| 0.0 | ||
| end | ||
| isolation_result.value&.runtime || 0.0 |
There was a problem hiding this comment.
Its now perfectly valid there is no value returned from the isolation. And the status of the islation result cannot be determined anymore without the coverage criteria coming from the outside.
|
|
||
| ```yml | ||
| --- | ||
| coverage_criteria: |
There was a problem hiding this comment.
Is it possible to set this via CLI flag? Once we have per-subject / per file config then customizing it that way would be good but I am thinking of doing something like having the fork abort coverage off in my local config (so I can find and report bugs) but considering it as a kill in CI so that PRs are never gated by segfaults.
There was a problem hiding this comment.
Allowing to configure each subject separately is the next step.
But: Its unlikely it makes sense in the CLI, simply because the CLI would become very long. My next initiative would be to support inheriting (prefix match) based per subject configuration in the code, and use the config file as input.
Than the overnext step to support it from in-line code comments.
Only after that I'll consider making CLI (per subject) support for these.
There was a problem hiding this comment.
Ok, yeah, I think that makes sense. Looking forward to it 👍
No description provided.