[RB] Handle retries in the CLI rather than server-side #8195
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maggie-lou
force-pushed
the
rb_reenq_cli
branch
2 times, most recently
from
99f87cd to
40dcab6
Compare
maggie-lou
force-pushed
the
rb_reenq
branch
2 times, most recently
from
8db56b9 to
1688dae
Compare
maggie-lou
force-pushed
the
rb_reenq_cli
branch
from
40dcab6 to
23fdae1
Compare
dan-stowell
approved these changes
Jan 22, 2025
Not the cheapest, but I tested by hard coding a local executor to return a retryable error 20% of the time and verified by the remote bazel logs that the CLI was successfully retrying |
bduffany
reviewed
Jan 27, 2025
maggie-lou
force-pushed
the
rb_reenq_cli
branch
3 times, most recently
from
8774574 to
b16cc78
Compare
bduffany
approved these changes
Jan 28, 2025
maggie-lou
force-pushed
the
rb_reenq
branch
2 times, most recently
from
fc2dbba to
bea6602
Compare
maggie-lou
force-pushed
the
rb_reenq_cli
branch
from
b16cc78 to
35c7803
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Extracted CLI code from #8172
This PR disables automatic retries when using the remote bazel CLI. The CLI then implements retry behavior itself so it can detect when a retry is happening
This fixes a bug where if the remote run fails due to a transient error, the CLI will report a failure. However the scheduler may retry the run in the background and it may eventually succeed. When users go to the invocation link, they'll see that it succeeded despite expecting a failure
Slack thread (Issue 2): https://buildbuddy-corp.slack.com/archives/C07GMM2VBLY/p1734595804049229?thread_ts=1734595606.837269&cid=C07GMM2VBLY
Should be merged after #8172 is rolled out