Remote Output Service: place bazel-out/ on a FUSE file system #12823

EdSchouten · 2021-01-13T15:28:16Z

Builds that yield large output files can generate lots of network
bandwidth when remote execution is used. To combat this, we have flags
such as --remote_download_minimal to disable downloading of output
files. Unfortunately, this makes it hard to perform ad hoc exploration
of build outputs.

In an attempt to solve this, this change adds an option to Bazel named
--remote_output_service. When enabled, Bazel effectively gives up the
responsibility of managing a bazel-out/ directory. Instead, it calls
into a gRPC service to request a directory and creates a symlink that
points to it.

Smart implementations of this gRPC service may use things like FUSE to
let this replacement bazel-out/ directory be lazy-loading, thereby
reducing network bandwidth significantly. In order to create
lazy-loading files and directories, Bazel can call into a BatchCreate()
RPC that takes a list of Output{File,Directory,Symlink} messages,
similar to REv2's ActionResult. This call is also used to create
runfiles directories by providing fictive instances of OutputSymlink.

To prevent Bazel from reading the contents of files stored in the FUSE
file system (which would cause network I/O), the protocol offers a
BatchStat() call that can return information such as the REv2 digest.
Though this is redundant with --unix_digest_hash_attribute_name, there
are a couple of reasons why I added this feature:

For non-Linux operating systems, it may make more sense to use NFSv4
instead of FUSE (i.e., running a virtual NFS daemon on localhost).
Even though RFC 8276 adds support for extended attributes to NFSv4,
not all operating systems implement it.
It addresses the security/hermeticity concerns that people raised
when this feature was added. There is no way to add extended
attributes to files that can't be tampered with (as a non-root user),
while using gRPC solves that problem inherently.
Callers of Bazel's BatchStat.batchStat() may generate many system
calls successively. This causes a large number of context switches
between Bazel and the FUSE daemon. Using gRPC seems to be cheaper.

By requiring that the output path returned by the gRPC service is
writable, no-remote actions can still run as before, both with
sandboxing enabled and disabled. The only difference is that it will use
space on the gRPC service side, as opposed to the user's home directory
(though the gRPC service may continue to store data in the user's home
directory.

I have a server implementation is written in Go on top of Buildbarn's
storage and file system layer. My plan is to release the code for this
service as soon as I've got a 'thumbs up' on the overall approach.

EdSchouten · 2021-01-13T15:32:21Z

Initially my plan was to try to get the protocol that Bazel will speak with the FUSE daemon be standardized as part of the remote-apis project. Mailing list discussion:

https://groups.google.com/g/remote-execution-apis/c/qOSWWwBLPzo

After discussions that took place during the monthly meeting, and after taking another look at some of the (remotely) similar protocols in this space, such as Buildgrid's local_casd protocol, I think it would make more sense to keep a protocol like this in the Bazel tree, so that the protocol remains minimal.

I have not yet invested time in adding unit/integration testing coverage for this code. This is something that I'd rather do after first having some discussion here. For example, what do you think of the overall approach? What about the protocol? I'd love to get your feedback!

Cc @janakdr @bergsieker @ulfjack

janakdr · 2021-01-13T16:26:20Z

cc @susinmotion @alexjski

rupertks · 2021-01-13T23:57:29Z

cc @tetromino

janakdr

Looks cool!

src/main/java/com/google/devtools/build/lib/remote/GrpcRemoteOutputService.java

src/main/protobuf/remote_output_service.proto

meisterT · 2021-01-14T07:35:22Z

cc @coeuvre

jin · 2021-01-15T05:15:03Z

@lberki if needed, please reassign to someone that could shepherd this PR into Google?

janakdr

@jin we may need to get a few more eyes to give general approval of the idea. When it comes time for a review, @alexjski should be one of the reviewers.

src/main/protobuf/remote_output_service.proto

In the nearby future, I want to release a client-side FUSE daemon that people can use to speed up builds that use remote execution. More details can be found here: bazelbuild/bazel#12823 This daemon is going to make extensive use of the path resolution library that we added some time. This commit contains all of the components that the FUSE daemon is going to need in addition to what's already there: - AbsoluteScopeWalker: To enforce that paths are absolute. - Builder: To construct normalized pathname strings. - LoopDetectingScopeWalker: To prevent infinite traversal of symlinks. - VirtualRootScopeWalkerFactory: To simulate resolution of absolute paths that start with a prefix that needs to be trimmed. - Void{Component,Scope}Walker: To perform path resolution without making any assumptions about the underlying file system.

This protocol is also part of this PR against Bazel: bazelbuild/bazel#12823 Let's add it to our tree, so that we can already start using it, even though it's not part of Bazel properly. This is the protocol that Bazel will use to communicate with the client-side FUSE daemon.

This change extends the pathname structure offered by bb_clientd. Instead of letting the top-level directory provide immediate access to the CAS, we move this feature to a subdirectory named "cas". Two new top-level directories are added: - "outputs", which provides an implementation of the Bazel Remote Output Service. Details: bazelbuild/bazel#12823 - "scratch", which provides a simple scratch space to test the behaviour of the FUSE file system without requiring Bazel to create an output directory for you.

EdSchouten · 2021-02-03T15:36:17Z

I've just rebased this change on top of latest master, and made CI happy once again. Earlier today I released my implementation of the FUSE daemon on GitHub as well:

https://github.com/buildbarn/bb-clientd

🎉 🎉 🎉

Be sure to let me know what I can do from my end to get this change polished up, upstreamed, etc. etc. etc.

janakdr · 2021-02-03T15:38:47Z

@alexjski could you take the lead on this? Thanks!

lberki · 2021-02-03T15:51:25Z

/cc @philwo

src/main/java/com/google/devtools/build/lib/remote/GrpcRemoteOutputService.java

sohaibiftikhar · 2021-02-04T20:14:55Z

This is amazing 🥳

lberki · 2021-02-08T08:38:58Z

As a general comment, a design doc would be useful so that everyone can understand more easily what's going on. We even have a separate repository for those:

https://github.com/bazelbuild/proposals

Apart from that procedural note, my main worry is that this adds Yet Another Way for Bazel to handle the metadata pertaining to action outputs, of which we already have too many (objfsd, the internal equivalent of this FUSE service, ActionFS, the output tree under bazel-out/, the local action cache, the disk cache, etc.

@philwo is probably the most competent person to comment on all these things. I personally don't feel confident saying anything about this change other than "pls no I don't need more abstractions", which is not exactly helpful. At the very least, it should be written down somewhere how this proposal interacts with these other metadata handling systems.

@lberki

As requested by @lberki in bazelbuild/bazel#12823, I'm hereby sending out a proposal for adding support for a client-side FUSE daemon to Bazel. Suggestions are welcome!

stagnation · 2023-08-14T09:02:16Z

Back from vacation now, I posted the proto in this PR: #18775 . Happy to address any comments.

When doing a regular build without passing in --remote_download_*, ExecutionTool will automatically clean up any traces of an OutputService that replaces bazel-out/ with a symlink pointing to a remote file system. The --remote_download_* option is implemented by installing an OutputService of its own, meaning that the regular logic in ExecutionTool is skipped. The rationale being that an OutputService essentially takes ownership of bazel-out/. This change extends RemoteOutputService to be a more complete implementation of OutputService that ensures that bazel-out/ is set up properly. This improves interoperability with changes such as the gRPC based Remote Output Service protocol (see bazelbuild#12823).

In case DirectoryContext.createOrSetWritable() is called on a directory inside of a Tree, it already does a nice forward sweep along the path to create missing intermediate directories. When creating a bare output file in a nested directory, it does not. It implicitly assumes that the output directory was already created at a previous point in time. This doesn't cause issues whenever we do a fully clean build, as some earlier phase (the analysis phase?) creates the missing parent directories for us. However, if we remove the nested directory structure from bazel-bin/ manually and rerun the test, this causes failures along the lines: java.io.FileNotFoundException: /REDACTED/execroot/main/bazel-out/darwin_x86_64-fastbuild/bin/some/very/very/very/deeply/nested (No such file or directory) Solve this issue by properly catching FileNotFoundExceptions that are generated when we call Path.isWritable() on the parent directory of the output file. In that case we resort to calling Path.createDirectoryAndParents(). I think this is likely the most efficient strategy, as it means we don't need to do a full forward scan of the path in the common case where the parent directory does exist. This issue can be noticed in practice when switching from --remote_output_service (PR bazelbuild#12823) to --remote_download_minimal.

When doing a regular build without passing in --remote_download_*, ExecutionTool will automatically clean up any traces of an OutputService that replaces bazel-out/ with a symlink pointing to a remote file system. The --remote_download_* option is implemented by installing an OutputService of its own, meaning that the regular logic in ExecutionTool is skipped. The rationale being that an OutputService essentially takes ownership of bazel-out/. This change extends RemoteOutputService to be a more complete implementation of OutputService that ensures that bazel-out/ is set up properly. This improves interoperability with changes such as the gRPC based Remote Output Service protocol (see bazelbuild#12823).

When doing a regular build without passing in --remote_download_*, ExecutionTool will automatically clean up any traces of an OutputService that replaces bazel-out/ with a symlink pointing to a remote file system. The --remote_download_* option is implemented by installing an OutputService of its own, meaning that the regular logic in ExecutionTool is skipped. The rationale being that an OutputService essentially takes ownership of bazel-out/. This change extends RemoteOutputService to be a more complete implementation of OutputService that ensures that bazel-out/ is set up properly. This improves interoperability with changes such as the gRPC based Remote Output Service protocol (see bazelbuild#12823). Change-Id: I5bedea2895785f3b8836fe8c5f8ae97ca2689233

When doing a regular build without passing in --remote_download_*, ExecutionTool will automatically clean up any traces of an OutputService that replaces bazel-out/ with a symlink pointing to a remote file system. The --remote_download_* option is implemented by installing an OutputService of its own, meaning that the regular logic in ExecutionTool is skipped. The rationale being that an OutputService essentially takes ownership of bazel-out/. This change extends RemoteOutputService to be a more complete implementation of OutputService that ensures that bazel-out/ is set up properly. This improves interoperability with changes such as the gRPC based Remote Output Service protocol (see bazelbuild#12823).

Builds that yield large output files can generate lots of network bandwidth when remote execution is used. To combat this, we have flags such as --remote_download_minimal to disable downloading of output files. Unfortunately, this makes it hard to perform ad hoc exploration of build outputs. In an attempt to solve this, this change adds an option to Bazel named --remote_output_service. When enabled, Bazel effectively gives up the responsibility of managing a bazel-out/ directory. Instead, it calls into a gRPC service to request a directory and creates a symlink that points to it. Smart implementations of this gRPC service may use things like FUSE to let this replacement bazel-out/ directory be lazy-loading, thereby reducing network bandwidth significantly. In order to create lazy-loading files and directories, Bazel can call into a BatchCreate() RPC that takes a list of Output{File,Directory,Symlink} messages, similar to REv2's ActionResult. This call is also used to create runfiles directories by providing fictive instances of OutputSymlink. To prevent Bazel from reading the contents of files stored in the FUSE file system (which would cause network I/O), the protocol offers a BatchStat() call that can return information such as the REv2 digest. Though this is redundant with --unix_digest_hash_attribute_name, there are a couple of reasons why I added this feature: 1. For non-Linux operating systems, it may make more sense to use NFSv4 instead of FUSE (i.e., running a virtual NFS daemon on localhost). Even though RFC 8276 adds support for extended attributes to NFSv4, not all operating systems implement it. 2. It addresses the security/hermeticity concerns that people raised when this feature was added. There is no way to add extended attributes to files that can't be tampered with (as a non-root user), while using gRPC solves that problem inherently. 3. Callers of Bazel's BatchStat.batchStat() may generate many system calls successively. This causes a large number of context switches between Bazel and the FUSE daemon. Using gRPC seems to be cheaper. By requiring that the output path returned by the gRPC service is writable, no-remote actions can still run as before, both with sandboxing enabled and disabled. The only difference is that it will use space on the gRPC service side, as opposed to the user's home directory (though the gRPC service may continue to store data in the user's home directory. I have a server implementation is written in Go on top of Buildbarn's storage and file system layer. My plan is to release the code for this service as soon as I've got a 'thumbs up' on the overall approach.

When doing a regular build without passing in --remote_download_*, ExecutionTool will automatically clean up any traces of an OutputService that replaces bazel-out/ with a symlink pointing to a remote file system. The --remote_download_* option is implemented by installing an OutputService of its own, meaning that the regular logic in ExecutionTool is skipped. The rationale being that an OutputService essentially takes ownership of bazel-out/. This change extends RemoteOutputService to be a more complete implementation of OutputService that ensures that bazel-out/ is set up properly. This improves interoperability with changes such as the gRPC based Remote Output Service protocol (see #12823). Closes #19377. PiperOrigin-RevId: 562482812 Change-Id: I6352838c24d7fe8b3192ebe064647a140fad5c8c

coeuvre · 2023-09-28T09:29:43Z

Quick update: @tjgq and I had a in-depth review for the protocol and we want to make some changes. I am working on a proof of concept naive server implementation (which will also be used in the integration tests) to make sure our changes make sense before we share the feedbacks.

Introduce a .proto file for the proposed Remote Output Service. This commit is part of Ed Schouten's ideas and demonstration of feasibility described in the linked pull request below, this commit is just clerical. The Remote Output Service can speed up build with large output files by avoiding the downloads. Combined with an on-demand filesystem solution clients can download just the files they need. The big picture is described and tracked in bazelbuild#12823 . This is the first step in a rough process: 1 Review and merge the remote_output_service.proto definition first. 2 Implement the Bazel side from scratch, so we only have one implementation of OutputService. 3 The community will implement and maintain the server part (local daemon) outside of the Bazel source tree. 4 Eventually, remote_output_service.proto will move out of the Bazel source tree and be maintained by the community, similar to the REAPI spec today.

This protocol is also part of this PR against Bazel: bazelbuild/bazel#12823 Let's add it to our tree, so that we can already start using it, even though it's not part of Bazel properly. This is the protocol that Bazel will use to communicate with the client-side FUSE daemon.

moroten · 2023-10-23T13:38:44Z

Bazel builds with this PR are available at https://github.com/meroton/bazel/releases

coeuvre · 2023-12-15T05:37:51Z

Prototype is done. Working on the design doc and will start checking in the code soon.

EdSchouten · 2023-12-21T15:22:26Z

At BazelCon you folks mentioned you wanted to make some protocol changes. Any chance we can already discuss these ahead of things getting merged?

coeuvre · 2024-01-02T08:41:07Z

Yes, I will include the changes in the design doc and have it reviewed by the community before merging the code.

coeuvre · 2024-01-18T03:52:23Z

The design doc is out! Please check #20933.

EdSchouten · 2024-03-06T19:01:13Z

Closing this, as #21140 added the new protocol for which @coeuvre is going to add support to Bazel. The latest version of bb_clientd implements the new protocol as well.

google-cla bot added the cla: yes label Jan 13, 2021

This was referenced Jan 13, 2021

Allow the exec root to be placed outside the output base #12558

Closed

Let --remote_download_symlink_template use hard links #12566

Closed

EdSchouten force-pushed the eschouten/20210115-remote-output-service-integration branch from a7feeb1 to b1992ca Compare January 13, 2021 15:34

EdSchouten force-pushed the eschouten/20210115-remote-output-service-integration branch from b1992ca to 214a642 Compare January 13, 2021 17:22

janakdr reviewed Jan 14, 2021

View reviewed changes

src/main/java/com/google/devtools/build/lib/remote/GrpcRemoteOutputService.java Show resolved Hide resolved

src/main/java/com/google/devtools/build/lib/remote/GrpcRemoteOutputService.java Show resolved Hide resolved

src/main/protobuf/remote_output_service.proto Show resolved Hide resolved

jin added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Jan 15, 2021

jin assigned lberki Jan 15, 2021

janakdr reviewed Jan 15, 2021

View reviewed changes

src/main/protobuf/remote_output_service.proto Show resolved Hide resolved

EdSchouten force-pushed the eschouten/20210115-remote-output-service-integration branch 2 times, most recently from dc9e493 to 15d2bb3 Compare February 3, 2021 14:51

janakdr requested review from alexjski and janakdr and removed request for janakdr February 3, 2021 15:37

EdSchouten commented Feb 4, 2021

View reviewed changes

src/main/java/com/google/devtools/build/lib/remote/GrpcRemoteOutputService.java Show resolved Hide resolved

lberki assigned philwo Feb 8, 2021

sluongng mentioned this pull request Jul 31, 2023

[Experimental] Project Skymeld: Merging Analysis and Execution Phases of Skyframe #14057

Open

tjgq mentioned this pull request Aug 4, 2023

Runfiles: support paths with spaces #4327

Open

EdSchouten mentioned this pull request Aug 31, 2023

Let RemoteOutputService ensure bazel-out/ is not a symlink #19377

Closed

EdSchouten mentioned this pull request Aug 31, 2023

Ensure DirectoryContext.createOrSetWritable() always creates directories #19381

Closed

EdSchouten force-pushed the eschouten/20210115-remote-output-service-integration branch from 3925745 to 5e3f6ae Compare September 1, 2023 12:14

EdSchouten force-pushed the eschouten/20210115-remote-output-service-integration branch from 5e3f6ae to b9bfcf0 Compare September 1, 2023 12:23

coeuvre mentioned this pull request Sep 7, 2023

Find out how many users want lazy downloading of outputs, but can't use a daemon or FUSE / NFS #13669

Closed

coeuvre mentioned this pull request Nov 9, 2023

Allow disabling fetching of test outputs of tests running remotely #20068

Open

tetromino removed their request for review March 6, 2024 14:54

EdSchouten closed this Mar 6, 2024

github-actions bot removed the awaiting-review PR is awaiting review from an assigned reviewer label Mar 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remote Output Service: place bazel-out/ on a FUSE file system #12823

Remote Output Service: place bazel-out/ on a FUSE file system #12823

EdSchouten commented Jan 13, 2021

EdSchouten commented Jan 13, 2021 •

edited

Loading

janakdr commented Jan 13, 2021

rupertks commented Jan 13, 2021

janakdr left a comment

meisterT commented Jan 14, 2021

jin commented Jan 15, 2021

janakdr left a comment

EdSchouten commented Feb 3, 2021

janakdr commented Feb 3, 2021

lberki commented Feb 3, 2021

sohaibiftikhar commented Feb 4, 2021

lberki commented Feb 8, 2021

stagnation commented Aug 14, 2023

coeuvre commented Sep 28, 2023

moroten commented Oct 23, 2023

coeuvre commented Dec 15, 2023

EdSchouten commented Dec 21, 2023

coeuvre commented Jan 2, 2024

coeuvre commented Jan 18, 2024

EdSchouten commented Mar 6, 2024 •

edited

Loading

Remote Output Service: place bazel-out/ on a FUSE file system #12823

Remote Output Service: place bazel-out/ on a FUSE file system #12823

Conversation

EdSchouten commented Jan 13, 2021

EdSchouten commented Jan 13, 2021 • edited Loading

janakdr commented Jan 13, 2021

rupertks commented Jan 13, 2021

janakdr left a comment

Choose a reason for hiding this comment

meisterT commented Jan 14, 2021

jin commented Jan 15, 2021

janakdr left a comment

Choose a reason for hiding this comment

EdSchouten commented Feb 3, 2021

janakdr commented Feb 3, 2021

lberki commented Feb 3, 2021

sohaibiftikhar commented Feb 4, 2021

lberki commented Feb 8, 2021

stagnation commented Aug 14, 2023

coeuvre commented Sep 28, 2023

moroten commented Oct 23, 2023

coeuvre commented Dec 15, 2023

EdSchouten commented Dec 21, 2023

coeuvre commented Jan 2, 2024

coeuvre commented Jan 18, 2024

EdSchouten commented Mar 6, 2024 • edited Loading

EdSchouten commented Jan 13, 2021 •

edited

Loading

EdSchouten commented Mar 6, 2024 •

edited

Loading