Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remote: Fixes an issue when --experimental_remote_cache_async encounter flaky tests. #14220

Closed
wants to merge 2 commits into from

Conversation

coeuvre
Copy link
Member

@coeuvre coeuvre commented Nov 3, 2021

When --experimental_remote_cache_async is set, the uploads happened in the background -- usually after spawn execution.

When the test is failed and there is another test attempt, the outputs of previous test attempt are moved to other places immediately after spawn execution. This fine when combining --experimental_remote_cache_async because outputs of failed action don't get uploaded.

However, there is an exception that test.xml is generated with another spawn before the "move" happens. The result of the spawn used to generate test.xml is usually "succeed" which means Bazel will attempt upload test.xml even if the test itself is failed.

This PR makes the test.xml generation spawn ignores remote cache if the test itself is failed.

Fixes #14008.

@coeuvre coeuvre requested a review from a team as a code owner November 3, 2021 11:08
@google-cla google-cla bot added the cla: yes label Nov 3, 2021
@coeuvre coeuvre changed the title Remote: Fix issue when --experimental_remote_cache_async encountered flaky tests. Remote: Fixes an issue when --experimental_remote_cache_async encounter flaky tests. Nov 3, 2021
@coeuvre coeuvre added this to the Bazel 5.0 Release Blockers milestone Nov 4, 2021
Copy link
Member

@meteorcloudy meteorcloudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@bazel-io bazel-io closed this in f5cf8b0 Nov 8, 2021
coeuvre added a commit to coeuvre/bazel that referenced this pull request Nov 8, 2021
…er flaky tests.

When `--experimental_remote_cache_async` is set, the uploads happened in the background -- usually after spawn execution.

When the test is failed and there is another test attempt, the outputs of previous test attempt are moved to other places immediately after spawn execution. This fine when combining `--experimental_remote_cache_async`  because outputs of failed action don't get uploaded.

However, there is an exception that `test.xml` is generated with another spawn before the "move" happens. The result of the spawn used to generate `test.xml` is usually "succeed" which means Bazel will attempt upload `test.xml` even if the test itself is failed.

This PR makes the `test.xml` generation spawn ignores remote cache if the test itself is failed.

Fixes bazelbuild#14008.

Closes bazelbuild#14220.

PiperOrigin-RevId: 408237437
coeuvre added a commit to coeuvre/bazel that referenced this pull request Nov 8, 2021
…er flaky tests.

When `--experimental_remote_cache_async` is set, the uploads happened in the background -- usually after spawn execution.

When the test is failed and there is another test attempt, the outputs of previous test attempt are moved to other places immediately after spawn execution. This fine when combining `--experimental_remote_cache_async`  because outputs of failed action don't get uploaded.

However, there is an exception that `test.xml` is generated with another spawn before the "move" happens. The result of the spawn used to generate `test.xml` is usually "succeed" which means Bazel will attempt upload `test.xml` even if the test itself is failed.

This PR makes the `test.xml` generation spawn ignores remote cache if the test itself is failed.

Fixes bazelbuild#14008.

Closes bazelbuild#14220.

PiperOrigin-RevId: 408237437
Wyverald pushed a commit that referenced this pull request Nov 8, 2021
…er flaky tests. (#14241)

When `--experimental_remote_cache_async` is set, the uploads happened in the background -- usually after spawn execution.

When the test is failed and there is another test attempt, the outputs of previous test attempt are moved to other places immediately after spawn execution. This fine when combining `--experimental_remote_cache_async`  because outputs of failed action don't get uploaded.

However, there is an exception that `test.xml` is generated with another spawn before the "move" happens. The result of the spawn used to generate `test.xml` is usually "succeed" which means Bazel will attempt upload `test.xml` even if the test itself is failed.

This PR makes the `test.xml` generation spawn ignores remote cache if the test itself is failed.

Fixes #14008.

Closes #14220.

PiperOrigin-RevId: 408237437
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upload failure using --experimental_remote_cache_async and flaky test retries
2 participants