Store TTL in RemoteFileArtifactValue #17639

coeuvre · 2023-03-01T15:54:42Z

When building without the bytes, Bazel stores RemoteFileArtifactValue in skyframe (inmemory) and in local action cache which represents a file that is stored remotely. Bazel assumes that the remote file will never expire which is wrong. In practice, remote cache often evict files due to space constraint, and when it happens, the builds could fail.

This PR introduces flag --experimental_remote_cache_ttl which tells Bazel at least how long the remote cache could store a file after returning a reference of it to Bazel. Bazel calculates the TTL of the file and store it in the RemoteFileArtifactValue. In an incremental build, Bazel will discard the RemoteFileArtifactValue and rerun the generating actions if it finds out that the RemoteFileArtifactValue is expired. The new field expireAtEpochMilli replaces actionId (deleted by f62a8b9), so there shouldn't be memory regression.

There are two places Bazel checks the TTL:

If the skyframe has in-memory state about previous builds (e.g. incremental builds), the SkyValues are marked as dirty if the RemoteFileArtifactValue is expired.
When checking local action cache, if the RemoteFileArtifactValue is expired, the cache entry is ignored.

So that the generating actions can be re-executed.

Part of #16660.

src/main/java/com/google/devtools/build/lib/actions/ActionCacheChecker.java

tjgq · 2023-03-02T08:46:10Z

src/main/java/com/google/devtools/build/lib/remote/options/CommonRemoteOptions.java

+
+  @Option(
+      name = "experimental_remote_cache_ttl",
+      defaultValue = "3h",


Can we go with a larger default, say 8h or even 24h? I'm mostly worried about this throwing off long-running benchmarks.

We should also add this to the relnotes.

Higher value by default means higher space requirement on remote cas. I picked 3h randomly because it's on par with the default value for --max_idle_secs.

The TTL only affects incremental builds. If, for example, a long-running invocation lasts more than 3h, nothing goes wrong during this invocation. It's just for next incremental build, all remote metadata are discarded. Also, the plan is to have a background thread refresh the lease for remote metadata. So for long-running invocations, it doesn't matter what the value of --experimental_remote_cache_ttl is set to. The frequency of the refresh is based on --experimental_remote_cache_ttl, though.

That being said, I am open to give it a higher default value.

@brentleyjones, you commented on this flag in the prototype, do you have any suggestions here?

I don't like it being as high as 24h, because I know there are some remote cache instances that can't reach that level of guarantee. And like you said, this doesn't impact long-running builds, only incremental builds. I'm good with 3h, but I don't think it should be larger than 8h.

src/main/java/com/google/devtools/build/lib/skyframe/FilesystemValueChecker.java

src/test/java/com/google/devtools/build/lib/actions/ActionCacheCheckerTest.java

src/test/java/com/google/devtools/build/lib/skyframe/FilesystemValueCheckerTest.java

With #17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With #17462 and #17747, Bazel is able to continue the build without bazel clean or bazel shutdown. However, even with #17639 and following changes to extend the lease, remote cache can still evict blobs in some rare cases. Based on above changes, this PR makes bazel retry the invocation if it encountered the remote cache eviction error during previous invocation if `--experimental_remote_cache_eviction_retries` is set, or **build rewinding**. ``` $ bazel build --experimental_remote_cache_eviction_retries=5 ... INFO: Invocation ID: b7348bfa-9446-4c72-a888-0a0ad012f225 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: .../workspace/a/BUILD:8:8: Executing genrule //a:bar failed: Failed to fetch blobs because they do not exist remotely: Missing digest: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/4 Target //a:bar failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.447s, Critical Path: 0.05s INFO: 2 processes: 2 internal. ERROR: Build did NOT complete successfully Found remote cache eviction error, retrying the build... INFO: Invocation ID: 983f60dc-8bb9-4b82-aa33-a378469ce140 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt Target //a:bar up-to-date: bazel-bin/a/bar.out INFO: Elapsed time: 0.866s, Critical Path: 0.35s INFO: 3 processes: 1 internal, 1 processwrapper-sandbox, 1 remote. INFO: Build completed successfully, 3 total actions $ ``` Part of #16660. Closes #17711. PiperOrigin-RevId: 520610524 Change-Id: I20d43d1968767a03250b9c8f8a6dda4e056d4f52

With bazelbuild#17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With bazelbuild#17462 and bazelbuild#17747, Bazel is able to continue the build without bazel clean or bazel shutdown. However, even with bazelbuild#17639 and following changes to extend the lease, remote cache can still evict blobs in some rare cases. Based on above changes, this PR makes bazel retry the invocation if it encountered the remote cache eviction error during previous invocation if `--experimental_remote_cache_eviction_retries` is set, or **build rewinding**. ``` $ bazel build --experimental_remote_cache_eviction_retries=5 ... INFO: Invocation ID: b7348bfa-9446-4c72-a888-0a0ad012f225 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: .../workspace/a/BUILD:8:8: Executing genrule //a:bar failed: Failed to fetch blobs because they do not exist remotely: Missing digest: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/4 Target //a:bar failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.447s, Critical Path: 0.05s INFO: 2 processes: 2 internal. ERROR: Build did NOT complete successfully Found remote cache eviction error, retrying the build... INFO: Invocation ID: 983f60dc-8bb9-4b82-aa33-a378469ce140 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt Target //a:bar up-to-date: bazel-bin/a/bar.out INFO: Elapsed time: 0.866s, Critical Path: 0.35s INFO: 3 processes: 1 internal, 1 processwrapper-sandbox, 1 remote. INFO: Build completed successfully, 3 total actions $ ``` Part of bazelbuild#16660. Closes bazelbuild#17711. PiperOrigin-RevId: 520610524 Change-Id: I20d43d1968767a03250b9c8f8a6dda4e056d4f52

…ror (#18171) With #17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With #17462 and #17747, Bazel is able to continue the build without bazel clean or bazel shutdown. However, even with #17639 and following changes to extend the lease, remote cache can still evict blobs in some rare cases. Based on above changes, this PR makes bazel retry the invocation if it encountered the remote cache eviction error during previous invocation if `--experimental_remote_cache_eviction_retries` is set, or **build rewinding**. ``` $ bazel build --experimental_remote_cache_eviction_retries=5 ... INFO: Invocation ID: b7348bfa-9446-4c72-a888-0a0ad012f225 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: .../workspace/a/BUILD:8:8: Executing genrule //a:bar failed: Failed to fetch blobs because they do not exist remotely: Missing digest: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/4 Target //a:bar failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.447s, Critical Path: 0.05s INFO: 2 processes: 2 internal. ERROR: Build did NOT complete successfully Found remote cache eviction error, retrying the build... INFO: Invocation ID: 983f60dc-8bb9-4b82-aa33-a378469ce140 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt Target //a:bar up-to-date: bazel-bin/a/bar.out INFO: Elapsed time: 0.866s, Critical Path: 0.35s INFO: 3 processes: 1 internal, 1 processwrapper-sandbox, 1 remote. INFO: Build completed successfully, 3 total actions $ ``` Part of #16660. Closes #17711. PiperOrigin-RevId: 520610524 Change-Id: I20d43d1968767a03250b9c8f8a6dda4e056d4f52

When building without the bytes, Bazel stores `RemoteFileArtifactValue` in skyframe (inmemory) and in local action cache which represents a file that is stored remotely. Bazel assumes that the remote file will never expire which is wrong. In practice, remote cache often evict files due to space constraint, and when it happens, the builds could fail. This PR introduces flag `--experimental_remote_cache_ttl` which tells Bazel at least how long the remote cache could store a file after returning a reference of it to Bazel. Bazel calculates the TTL of the file and store it in the `RemoteFileArtifactValue`. In an incremental build, Bazel will discard the `RemoteFileArtifactValue` and rerun the generating actions if it finds out that the `RemoteFileArtifactValue` is expired. The new field `expireAtEpochMilli` replaces `actionId` (deleted by f62a8b9), so there shouldn't be memory regression. There are two places Bazel checks the TTL: 1. If the skyframe has in-memory state about previous builds (e.g. incremental builds), the `SkyValue`s are marked as dirty if the `RemoteFileArtifactValue` is expired. 2. When checking local action cache, if the `RemoteFileArtifactValue` is expired, the cache entry is ignored. So that the generating actions can be re-executed. Part of bazelbuild#16660. Closes bazelbuild#17639. RELNOTES: Add flag `--experimental_remote_cache_ttl` and set the default value to 3 hours. PiperOrigin-RevId: 513819724 Change-Id: I9c9813621d04d5b1b94312be39384962feae2f7b

With bazelbuild#17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With bazelbuild#17462 and bazelbuild#17747, Bazel is able to continue the build without bazel clean or bazel shutdown. However, even with bazelbuild#17639 and following changes to extend the lease, remote cache can still evict blobs in some rare cases. Based on above changes, this PR makes bazel retry the invocation if it encountered the remote cache eviction error during previous invocation if `--experimental_remote_cache_eviction_retries` is set, or **build rewinding**. ``` $ bazel build --experimental_remote_cache_eviction_retries=5 ... INFO: Invocation ID: b7348bfa-9446-4c72-a888-0a0ad012f225 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: .../workspace/a/BUILD:8:8: Executing genrule //a:bar failed: Failed to fetch blobs because they do not exist remotely: Missing digest: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/4 Target //a:bar failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.447s, Critical Path: 0.05s INFO: 2 processes: 2 internal. ERROR: Build did NOT complete successfully Found remote cache eviction error, retrying the build... INFO: Invocation ID: 983f60dc-8bb9-4b82-aa33-a378469ce140 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt Target //a:bar up-to-date: bazel-bin/a/bar.out INFO: Elapsed time: 0.866s, Critical Path: 0.35s INFO: 3 processes: 1 internal, 1 processwrapper-sandbox, 1 remote. INFO: Build completed successfully, 3 total actions $ ``` Part of bazelbuild#16660. Closes bazelbuild#17711. PiperOrigin-RevId: 520610524 Change-Id: I20d43d1968767a03250b9c8f8a6dda4e056d4f52

coeuvre requested a review from a team as a code owner March 1, 2023 15:54

coeuvre force-pushed the remote-cache-ttl branch 4 times, most recently from 336958d to 35fd42d Compare March 1, 2023 16:15

sgowroji added team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-review PR is awaiting review from an assigned reviewer labels Mar 1, 2023

Store TTL in RemoteFileArtifactValue.

61711f1

coeuvre force-pushed the remote-cache-ttl branch from 35fd42d to 61711f1 Compare March 1, 2023 16:37

tjgq reviewed Mar 2, 2023

View reviewed changes

Add more tests

de37b24

tjgq approved these changes Mar 2, 2023

View reviewed changes

copybara-service bot closed this in 1ebb04b Mar 3, 2023

coeuvre deleted the remote-cache-ttl branch March 6, 2023 10:42

coeuvre mentioned this pull request Mar 9, 2023

Automatically retry the build if encountered remote cache eviction error #17711

Closed

ShreeM01 mentioned this pull request Mar 30, 2023

[6.2.0]Automatically retry the build if encountered remote cache eviction error #17938

Closed

ShreeM01 mentioned this pull request Apr 3, 2023

[6.2.0]Automatically retry the build if encountered remote cache eviction error #17953

Closed

coeuvre mentioned this pull request Apr 21, 2023

[6.2.0] Automatically retry the build if encountered remote cache eviction error #18171

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store TTL in RemoteFileArtifactValue #17639

Store TTL in RemoteFileArtifactValue #17639

coeuvre commented Mar 1, 2023 •

edited

Loading

tjgq Mar 2, 2023

coeuvre Mar 2, 2023

brentleyjones Mar 2, 2023

Store TTL in RemoteFileArtifactValue #17639

Store TTL in RemoteFileArtifactValue #17639

Conversation

coeuvre commented Mar 1, 2023 • edited Loading

tjgq Mar 2, 2023

Choose a reason for hiding this comment

coeuvre Mar 2, 2023

Choose a reason for hiding this comment

brentleyjones Mar 2, 2023

Choose a reason for hiding this comment

coeuvre commented Mar 1, 2023 •

edited

Loading