Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let --remote_download_symlink_template use hard links #12566

Closed

Conversation

EdSchouten
Copy link
Contributor

In an attempt to achieve 'Builds without the Bytes' without losing
access to build outputs, I am experimenting with a FUSE file system that
gives access to objects stored in the CAS. In PR #11703, I added a
command line flag to let Bazel emit symbolic links pointing into this
FUSE file system, as opposed to downloading files into the exec root.

Though this change has allowed me to get quite a lot of stuff working,
there are also many build actions that break. For example, Python calls
realpath(argv[0]) to figure out its installation path. Because the FUSE
file system does not mimic the execroot, Python won't be able to find
its site packages. Similar problems hold with shared library resolution
in general.

This is why I think the only proper way we can get this to work is by
using hard links instead of symbolic links. That way the usual file
hierarchy remains intact. This change renames the
--remote_download_symlink_template flag to
--remote_download_hard_link_template and changes the code to create hard
links instead.

When used in combination with --exec_root_base (#12558), it's now
possible to let Bazel construct an exec root that does not have any
additional indirection through symbolic links, thereby keeping programs
that do symlink expansion happy.

@google-cla google-cla bot added the cla: yes label Nov 26, 2020
@EdSchouten
Copy link
Contributor Author

Cc @philwo @coeuvre

@Yannic
Copy link
Contributor

Yannic commented Nov 26, 2020

Do hardlinks work if source/destination are on different filesystems?

@EdSchouten
Copy link
Contributor Author

Do hardlinks work if source/destination are on different filesystems?

They do not. That's why this flag now needs to be used in combination with --exec_root_base, which is being introduced in #12558. You thus need a FUSE file system that both provides access to raw CAS objects, and a scratch space where the exec root may be placed.

In an attempt to achieve 'Builds without the Bytes' without losing
access to build outputs, I am experimenting with a FUSE file system that
gives access to objects stored in the CAS. In PR bazelbuild#11703, I added a
command line flag to let Bazel emit symbolic links pointing into this
FUSE file system, as opposed to downloading files into the exec root.

Though this change has allowed me to get quite a lot of stuff working,
there are also many build actions that break. For example, Python calls
realpath(argv[0]) to figure out its installation path. Because the FUSE
file system does not mimic the execroot, Python won't be able to find
its site packages. Similar problems hold with shared library resolution
in general.

This is why I think the only proper way we can get this to work is by
using hard links instead of symbolic links. That way the usual file
hierarchy remains intact. This change renames the
--remote_download_symlink_template flag to
--remote_download_hard_link_template and changes the code to create hard
links instead.

When used in combination with --exec_root_base (bazelbuild#12558), it's now
possible to let Bazel construct an exec root that does not have any
additional indirection through symbolic links, thereby keeping programs
that do symlink expansion happy.
@aiuto aiuto added the team-Remote-Exec Issues and PRs for the Execution (Remote) team label Nov 30, 2020
@jin jin assigned coeuvre and philwo Dec 4, 2020
@EdSchouten
Copy link
Contributor Author

Superseded by #12823.

@EdSchouten EdSchouten closed this Jan 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes team-Remote-Exec Issues and PRs for the Execution (Remote) team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants