Report background download for BwoB #17604
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sgowroji
added
the
team-Remote-Exec
|
@coeuvre is this cherry-pickable? Nice change! |
|
Thanks! I think it's possible to include it in 6.1. |
|
@bazel-io flag |
bazel-io
added
the
potential release blocker
coeuvre
added a commit
to coeuvre/bazel
that referenced
this pull request
Feb 28, 2023
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.
This PR makes output downloads of toplevel targets visible to the UI.
For downloads during the build, the UI looks like:
```
[2 / 5] 2 actions, 1 running
[Prepa] Executing genrule //:foo
Executing genrule //:bar; 2s
Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```
For downloads after build is completed, the UI looks like:
```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```
Fixes bazelbuild#17417.
Closes bazelbuild#17604.
PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
ShreeM01
pushed a commit
that referenced
this pull request
Feb 28, 2023
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.
This PR makes output downloads of toplevel targets visible to the UI.
For downloads during the build, the UI looks like:
```
[2 / 5] 2 actions, 1 running
[Prepa] Executing genrule //:foo
Executing genrule //:bar; 2s
Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```
For downloads after build is completed, the UI looks like:
```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```
Fixes #17417.
Closes #17604.
PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
ShreeM01
removed
the
potential release blocker
fweikert
pushed a commit
to fweikert/bazel
that referenced
this pull request
May 25, 2023
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.
This PR makes output downloads of toplevel targets visible to the UI.
For downloads during the build, the UI looks like:
```
[2 / 5] 2 actions, 1 running
[Prepa] Executing genrule //:foo
Executing genrule //:bar; 2s
Fetching bazel-out/darwin-fastbuild/bin/baz; 50.8 MiB / 1.0 GiB; 4s
```
For downloads after build is completed, the UI looks like:
```
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions
Fetching bazel-out/darwin-fastbuild/bin/baz; 48.8 MiB / 1.0 GiB
```
Fixes bazelbuild#17417.
Closes bazelbuild#17604.
PiperOrigin-RevId: 512934756
Change-Id: I502489420ab9b84efb74ceabbe3dd4cb4a7a62e2
copybara-service bot
pushed a commit
that referenced
this pull request
Jun 12, 2023
…r#prefetchFiles`. Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`. The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example: 1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. #17724 (comment) 2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. #17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes. PiperOrigin-RevId: 539635790 Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f
coeuvre
added a commit
to coeuvre/bazel
that referenced
this pull request
Jun 13, 2023
…r#prefetchFiles`. Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`. The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example: 1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. bazelbuild#17724 (comment) 2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. bazelbuild#17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes. PiperOrigin-RevId: 539635790 Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f
coeuvre
added a commit
to coeuvre/bazel
that referenced
this pull request
Jun 13, 2023
…r#prefetchFiles`. Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`. The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example: 1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. bazelbuild#17724 (comment) 2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. bazelbuild#17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes. PiperOrigin-RevId: 539635790 Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f
iancha1992
added a commit
that referenced
this pull request
Jun 13, 2023
…r#prefetchFiles`. (#18656) Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`. The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example: 1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. #17724 (comment) 2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. #17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes. PiperOrigin-RevId: 539635790 Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f Co-authored-by: Ian (Hee) Cha <heec@google.com>
traversaro
pushed a commit
to traversaro/bazel
that referenced
this pull request
Jun 24, 2023
…r#prefetchFiles`. Due to recent overhual of `prefetchFiles`, the call sites now always have the reference to `ActionExecutionMetadata`, which makes it possible to pass it to `prefetchFiles`. The reason why we want to access `ActionExecutionMetadata` inside `prefetchFiles` is to associate downloads with actions. For example: 1. When downloading from remote cache, we want to provide action metadata to the RPCs so that remote server can associate them with actions. `actionId` was removed from remote file metadata by f62a8b9 because it is not the right action to associate with for downloads inside this context. We left the action metadata empty for RPCs until this change. bazelbuild#17724 (comment) 2. When building with the bytes, we are able to report download progress to UI. However, `prefetchFiles` is the major way to download when building without the bytes and we can't report download progress on action basis. bazelbuild#17604 added the download report for background downloads, but now, we no longer download toplevel artifacts in background (a5dde12). With this change, we are able to report download progress for each action, either when they download outputs, or prefetch inputs for building without the bytes. PiperOrigin-RevId: 539635790 Change-Id: Ic9cc1024f5d3560ac46bc462bd549dd81712d92f
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When building without the bytes, downloads can happen when Bazel need to fetch inputs for local actions, download outputs of toplevel targets, and etc. The downloads happen silently at the background.
This PR makes output downloads of toplevel targets visible to the UI.
For downloads during the build, the UI looks like:
For downloads after build is completed, the UI looks like:
Fixes #17417.