Retry flaky specs automatically #4893
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This helper relies on rspec-retry, which will retry flagged specs when they fail. We can use this helper when we have flaky specs that we are not able to solve. The retry is only enabled in the CI environment.
github-actions
bot
added
changelog:repository
kennyadsl
removed
changelog:solidus_backend
kennyadsl
force-pushed
the
kennyadsl/flacky-retry
branch
from
d96e2d7 to
a652113
Compare
github-actions
bot
added
changelog:solidus_backend
adammathys
approved these changes
Jan 31, 2023
kennyadsl
force-pushed
the
kennyadsl/flacky-retry
branch
from
a652113 to
24eec8d
Compare
This will trigger a retry in the CI when they fail.
kennyadsl
force-pushed
the
kennyadsl/flacky-retry
branch
from
24eec8d to
ee52b9f
Compare
elia
approved these changes
Feb 1, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Increasing our testing matrix on the CI, we are being hit more often by the flaky specs we have around. Until we have time and will to fix them, we can now explicitly mark those specs as flaky and retry them multiple times automatically leveraging the rspec-retry library.
This will have the benefit of saving CI time along with maintainers' time, who have to continuously open CircleCI and rerun the failing jobs, which in turn will rerun the whole test suite.
This PR adds a wrapping testing support helper for flaky specs. This helper relies on
rspec-retry, which will retry flagged specs when they fail. We can use this helper when we have flaky specs that we cannot solve.The retry mechanism is only enabled in the CI environment.
Checklist
Check out our PR guidelines for more details.
The following are mandatory for all PRs:
The following are not always needed (
cross them outif they are not):