Coverage analysis based on mutants instead of code coverage

[nicojs]

Our coverage analysis is a brilliant feat of engineering (if I do say so myself). This is how it works:

• It will use instanbul to instrument files to be mutated
• It will inject a piece of code to run before each test to extract the current coverage results so we know the coverage per test
• It will communicate the coverage per test between the test runner (browser or separate node process)
  • Some lossless compression is involved in order to not overload and crash.

However, this has a lot of downsides:

• Files are instrumented by Istanbul, which only supports JavaScript. So in order to support TypeScript we had to support sourcemaps. Which is a somewhat expensive part of the code to maintain.
• It will only work for transpiled languages if it is transpiled using the typescript transpiler plugin. We could add support for other transpiler plugins, however, most users working with transpiled languages do not use the transpiler plugins. Instead, they rely on their test runner to do the transpiling (for example using ng test, karma-webpack or jest) or by require hooks (ts-node, @babel/register).

Solution

In order to add support for more use cases in a maintainable way, we should implement our own "mutant coverage instrumentation" in the source code (copy) itself.

With the unification on one mutator infrastructure in #1893, we should make sure to also align on 1 AST (preferably babel or estree. Furthermore, we should make sure that the new Mutator API will communicate mutants based on their AST nodes. Stryker would then be able to transform the AST in order to instrument for mutation coverage.

[nicojs]

Should be fixed when we implement mutation switching #1514

[nicojs]

Implemented in Stryker 4