Remote Output Service: initial proto definition #18775
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
github-actions
bot
added
the
awaiting-review
sgowroji
added
the
team-Remote-Exec
stagnation
force-pushed
the
feature/ROS-proto-defintion
branch
from
0dc47c9
to
7db94f9
Compare
| message File { | ||
| // The hash and size of the file. This field is only set when | ||
| // BatchStatRequest.include_file_digest is set. | ||
| build.bazel.remote.execution.v2.Digest digest = 1; |
There was a problem hiding this comment.
Some more experience using this protocol: my recommendation would be to allow the server to not report this value if it prefers.
Namely, I have observed Bazel sometimes prefers to obtain the digest of a file for which it still has one or more open file descriptors. If the FUSE file system has write-back caching enabled, this may cause dirty pages to remain in the page cache.
In that case it would be desirable for bb_clientd or any other implementation to omit this field, instructing Bazel to compute the digest itself.
// The hash and size of the file. This field is only set when
// BatchStatRequest.include_file_digest is set.
+ //
+ // This field may also be omitted if the remote output service is
+ // unable to compute it accurately. For example, when a file is
+ // opened for writing, the kernel may buffer data to be written.
+ // When absent, the caller should fall back to computing the digest
+ // manually.
build.bazel.remote.execution.v2.Digest digest = 1;
There was a problem hiding this comment.
I agree. From Bazel's perspective, this is a best-effort/fast approach to get digest if available. Bazel can always fallback to manually compute the digest.
There was a problem hiding this comment.
Good :) I will update the comment
stagnation
force-pushed
the
feature/ROS-proto-defintion
branch
from
7db94f9
to
ba3336c
Compare
Introduce a .proto file for the proposed Remote Output Service. This commit is part of Ed Schouten's ideas and demonstration of feasibility described in the linked pull request below, this commit is just clerical. The Remote Output Service can speed up build with large output files by avoiding the downloads. Combined with an on-demand filesystem solution clients can download just the files they need. The big picture is described and tracked in bazelbuild#12823 . This is the first step in a rough process: 1 Review and merge the remote_output_service.proto definition first. 2 Implement the Bazel side from scratch, so we only have one implementation of OutputService. 3 The community will implement and maintain the server part (local daemon) outside of the Bazel source tree. 4 Eventually, remote_output_service.proto will move out of the Bazel source tree and be maintained by the community, similar to the REAPI spec today.
stagnation
force-pushed
the
feature/ROS-proto-defintion
branch
from
ba3336c
to
f128e40
Compare
|
@stagnation I think this can be closed now, right? |
github-actions
bot
removed
the
awaiting-review
Introduce a .proto file for the proposed Remote Output Service. This commit is part of Ed Scouten's ideas and demonstration of feasibility described in the linked pull request below, my contribution is just clerical. The Remote Output Service can speed up build with large output files by avoiding the downloads. Combined with an on-demand filesystem solution clients can download just the files they need. The big picture is described and tracked in
#12823 .
This is the first step in a rough process: