[Feature]: Add handler when tests are being retried

[ryana]

🚀 Feature Proposal

I would like to know when tests are being retried. The retryTimes feature is great, but it can effectively hide the fact that a test is flakey. The ideal solution would allow for providing a handler that gets called whenever a test is retried, and would be passed the test name.

Note: I tried to use addEventHandler from state.js and watch for the test_retry event but I couldn't get it working after ~30 minutes of playing around.

Motivation

retryTimes is necessary when running E2E tests on larger apps, but it can hide flakey tests. On large teams running scores of builds in CI a day, being able to centralize recording those flakey tests can give valuable insight to help stabilize test suites.

Example

We could optionally augment retryTimes with a single handler:

jest.retryTimes(3, (testName) => console.log(`Retrying ${test name}`))

Or we could provide a list of handlers:

jest.retryTimes(3)
jest.addRetryHandler((testName) => console.log(`Retrying ${testName}`))
jest.addRetryHandler((testName) => metricsService.recordMetric(`test/retry/${testname}`, 1))

Pitch

Because I don't want to have to write my own runner to do this :) I want to keep using the default jest runner.

Also I don't know if this goes without saying, but I'm happy to do all the work here.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[SimenB]

We just landed #12201 yesterday, which adds optional logging, does that cover your use case? Note that this means you can use a custom reporter to do your recordMetric as the retry reasons are included in the test results.

That said, I think we now only pass the reasons if logging is enabled, that's probably the wrong way to do it... Might make more sense to always pass them, and just add some config to the reporter or something. Thoughts? (this is not shipped yet, but you can build the repo and play with it locally)

[mrazauskas]

What if these retry errors would be logged as warnings by default? Also always passed to reporters. The option of jest.retryTimes() could disable the logging. All just the opposite to current implementation, which was authored as non-breaking. But isn't it that the opposite behaviour is more useful?

Still good time to do this. I can work, if the idea is acceptable.

[SimenB]

I don't think we should log by default, but we should always pass the reasons in the test results. Not sure how to communicate logging being on or off, tho.

Separately, we should make sure test_retry event works

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.