Support error logging before jest retry

[zackasaurus]

Summary

Currently, to my knowledge Jest does not support logging errors before a retry event happens. This makes it difficult to debug why a test failed, especially if the code the test is testing is not deterministic (it will fail once, but will pass during the first retry), and thus no error will be visible as why it failed during the first time.

The need for visibility is a big driver here as I'd like to know if the reason that the retry occurred was due to a Jest.setTimeout error or an error that occurred in the test itself.

Test plan

Visuals

[facebook-github-bot]

Hi @zackasaurus!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[SimenB]

Thanks for the PR! I like the idea.

can you update the documentation as well?

[SimenB]

Suggested change

	jest.retryTimes(3);
	jest.setLogTestErrorsBeforeRetry(true);
	jest.retryTimes(3, {logErrorsBeforeRetry: true});=

I'd rather we just add an options bag to the existent call than adding a new method entirely

[zackasaurus]

Makes sense, I've updated it. Do we want to handle the Symbol differently as well? Right now, there is still an additional symbol for the logErrorsBeforeRetry.

[codecov-commenter]

Codecov Report

Merging #12201 (521dd6c) into main (2953433) will decrease coverage by 0.01%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##             main   #12201      +/-   ##
==========================================
- Coverage   66.93%   66.92%   -0.02%     
==========================================
  Files         329      329              
  Lines       17344    17352       +8     
  Branches     5061     5063       +2     
==========================================
+ Hits        11610    11612       +2     
- Misses       5702     5708       +6     
  Partials       32       32              

Impacted Files	Coverage Δ
packages/jest-circus/src/run.ts	5.81% <0.00%> (-0.22%)	⬇️
packages/jest-types/__typetests__/jest.test.ts	0.00% <0.00%> (ø)
packages/jest-runtime/src/index.ts	55.25% <50.00%> (-0.08%)	⬇️
packages/jest-circus/src/types.ts	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2953433...521dd6c. Read the comment docs.

[SimenB]

@zackasaurus ping 🙂

[zackasaurus]

not a fan of using console here. Can we update the result object and reporter to include the information instead?
@zackasaurus ping 🙂

Hey @SimenB !
Just making sure I'm on the right track here, but basically add something like this in packages/jest-core/src/ReporterDispatcher.ts =>

async onTestRetry(
    results: AggregatedResult,
    options: ReporterOnTestRetryOptions,
  ): Promise<void> {
    for (const reporter of this._reporters) {
      reporter.onTestRetry && (await reporter.onTestRetry(results, options));
    }
  }

and have packages/jest-circus/src/run.ts dispatch that method?

 while (numRetriesAvailable > 0 && test.errors.length > 0) {
      if (logErrorsBeforeRetry) {
        // dispatch test retry reporter method ?
      }
      // Clear errors so retries occur
      await dispatch({name: 'test_retry', test});

      await _runTest(test, isSkipped);
      numRetriesAvailable--;
    }

[SimenB]

I think we should just add something to the test result that allows the reporter to print some context, which the reporter can then print after the tests is done. I don't think the reporter needs to be told when it happens? So essentially, look at where circus currently handles the test_retry event (it just clears errors) and have it push those errors into some retryReasons array or some such. Then when the test is done, you can ensure it's on the test result. Then in the reporter when it's printing the results it can inspect this new array and then print something.

Makes sense?

[zackasaurus]

@SimenB I've yet to update the tests for this feature, but do the newest changes reflect your vision? I could only get the errors to show up correctly when using jest-serializer, so I'm passing an array of buffers to the reporter through the test result, and had to modify the deepCyclicCopy so the buffer data type would not change. I also included a visual of what the reporter displays when running a test. Let me know what you think! Thanks for all your help. :)

[SimenB]

@zackasaurus I need to re-process this PR, might be a few days. Gut feeling is that this solution is more complex than it needs to be, but I need to dig into it.

I'll add it to the milestone so I don't forget 🙂

[SimenB]

@zackasaurus I tweaked the logging a bit, thoughts?

[SimenB]

I'm happy with the current state of this 👍

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.