🚀 Feature: Run last failed tests

[AuHau]

Is your feature request related to a problem or a nice-to-have?? Please describe.
During development of new features, there is usually only a small subset of tests failing. With big test-suite, it can be a lengthy process to run the whole test-suite.

Coming from Python background, one of the testing framework that I was using ( pytest) supported the ability to rerun only failed tests or even better run failed tests prior rest of the test suite. See its documentation here: Cache: working with cross-testrun state. This is a feature that I am very much missing in mocha.

Describe the solution you'd like
Same ability as pytest supports, both for "last failed" and "failed first" options.

Storing cross-run state of failed tests, that is stored on every mocha run. When mocha is run with proper parameter (for example --last-failed or --failed-first) it will use this stored state to limit the list of executed tests to those that failed prior.

Describe alternatives you've considered
The only current solution to this problem known to me is using grep or fgrep, yet this is not flexible enough to completely cover this usecase.

There is a package that does this on top of mocha called mocha-broken yet I would prefer to have natively supported this by mocha.

If this feature would be accepted, I would be happy to discuss more deeply its implementation and then provide a patch for it.

[AuHau]

Any thoughts on this? @juergba or anybody else? I would happily work on this as it is really pain-point for me 😒

[juergba]

@AuHau Could this issue be related to #4183?
Can you briefly outline how to cache and filter those tests?

I don't really like the idea to implement this issue using huge grep expressions. As mocha-broken does?

[AuHau]

Yeah, the #4183 would be definitely beneficial for this idea as I don't like the "huge grep" approach either.

I am not yet familiar how mocha internally identifies each test-case, but the general outline could be like this:

For every test run

1. successful finished test run (eg. ignoring early-terminated runs using, for example, SIGTERM/SIGKILL)
2. save the result (success/failure/skipped) for each test-case to cache

Last failed / first failed mocha run

1. when user requests either last failed (using flag --last-failed) or first failed (using flag --failed-first) run
2. load the previous results of test cases from cache
3. manipulating the test-case run
   3.1 for last-failed case: filtering out tests based on the status
   3.2 for first-failed case: reordering the run queue

Cache

I would propose to follow the direction of projects like nyc or babel and use node_modules/.cache/... placement of the config using find-cache-dir

The cache can be a simple JSON object with schema:

{
  "lastRun": <<timestamp of last run>>,
  "results": {
    "<<ID of testcase>>": <<result>>
  }
}

where <<result>> could be Enum:

const ResultEnum = {
  SUCCESS: 'success',
  FAILURE: 'failure',
  SKIPPED: 'skipped'
}

Issues

Reordering

I assume filtering out test-cases should be fairly straight-forward, especially with the #4183, yet I am not sure how reordering of the test-cases is feasible.

But since there is no need for very complex reordering this could be for example solved by two internal test runs. The first test run would consist of filtered failed test cases and the second test run would consist of the rest test cases.

@juergba your thoughts?

[juergba]

@AuHau thank you.

First I need to know what the author of #4183 exactly wants.

• the test identifier would probably be fullTitle
• last-failed: seems ok so far, to run only the failed tests without any re-ordering.
• first-failed: I don't think we can split our describe blocks into two parts, the failing tests at the beginning and then later the passing tests. This would break our hook pattern (before / after) and I also don't know wether our reporters could handle this situation. Maybe two internal test runs could be a solution, maybe we don't need this feature.
• cache: reminds me on our JSON reporter. I will have a look at find-cache-dir to learn more about its benefits.
• default should be not to save the cache.

[juergba]

There is one additional problem, see #1955.
Currently when a hook fails, Mocha skips the corresponding tests. The skipped tests just disappear, don't get any state as eg. 'skipped' and aren't listed anywhere. So these tests can't be stored in a cache for re-running.

[jedwards1211]

@AuHau I think a watch mode that reruns only failed tests would be even more efficient, since you would only need to run a single command, instead of an initial run and then a separate --last-failed run. Can you look at what I proposed in #4762 and see if you think it would be even more convenient for your workflow than what you're proposing here?

[AuHau]

@jedwards1211 hmm while I understand your workflow and its value, I don't think that it should be "one feature" as you propose it, but rather separate ones that are interoperable. So you can do something like --watch --last-failed.

I can see use-cases and workflows that does not need or want the watch mode to be part of this (for example IDEs could support it natively and there the "Watch mode" does not really make sense).

[jedwards1211]

I see, well rerunning failed without watch mode would be a more complicated change since Mocha would have to write failed tests to some file, meaning people have to edit their gitignore etc. With watch mode it could just remember them in memory

[AuHau]

Well, that is true, but IMHO it is not such a big deal as there are solutions already in widely used projects that worked around needing "git-ignoring" and other problems, with the solution that I have described earlier using the find-cache-dir, that places these information into node_modules/.cache folder...

[jedwards1211]

Oh yeah, that's true, that would work

[JoshuaKGoldberg]

Duplicate of #1690.