Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"failed to delete output files before executing action" with tree artifacts and --remote_download_minimal #12678

Closed
scele opened this issue Dec 10, 2020 · 1 comment
Assignees
Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@scele
Copy link
Contributor

scele commented Dec 10, 2020

Description of the problem / feature request:

Getting this error when doing an incremental build with --remote_download_minimal.

Mnemonic target [for host] failed: failed to delete output files before executing action

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ cat BUILD 
load("rule.bzl", "foo")

foo(
    name = "foo",
    srcs = ["temp.txt"],
)
$ cat rule.bzl 
def _foo_impl(ctx):
    d = ctx.actions.declare_directory(ctx.attr.name)
    ctx.actions.run_shell(
        command = "mkdir {d}/subdir && echo foo > {d}/subdir/foo.txt".format(d = d.path),
        inputs = ctx.files.srcs,
        outputs = [d],
    )
    return [DefaultInfo(files = depset([d]), runfiles = ctx.runfiles([d]))]

foo = rule(
    implementation = _foo_impl,
    attrs = {"srcs": attr.label_list(allow_files = True)},
)
$ cat temp.txt 
foo
$ cat repro.sh 
#!/bin/bash
BAZEL=${1:-bazel}
bazel version
rm -rf $PWD/disk-cache
$BAZEL clean
$BAZEL build :foo
echo foo >> temp.txt
$BAZEL build :foo --disk_cache=$PWD/disk-cache --remote_download_minimal

$ USE_BAZEL_VERSION=3.7.1 ./repro.sh
Bazelisk version: development
Starting local Bazel server and connecting to it... 
Build label: 3.7.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Nov 24 17:38:30 2020 (1606239510)
Build timestamp: 1606239510
Build timestamp as int: 1606239510
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed target //:foo (4 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
  bazel-bin/foo
INFO: Elapsed time: 0.819s, Critical Path: 0.06s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Invocation ID: 1829a11b-6583-4511-9a27-13eb559eceb6
INFO: Build options --build_runfile_links, --experimental_inmemory_dotd_files, and --experimental_inmemory_jdeps_files have changed, discarding analysis cache.
INFO: Analyzed target //:foo (0 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
  bazel-bin/foo
INFO: Elapsed time: 0.243s, Critical Path: 0.10s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions

$ USE_BAZEL_VERSION=4.0.0rc4 ./repro.sh 
Bazelisk version: development
Starting local Bazel server and connecting to it...
Build label: 4.0.0rc4
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Dec 7 18:32:14 2020 (1607365934)
Build timestamp: 1607365934
Build timestamp as int: 1607365934
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
INFO: Analyzed target //:foo (4 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:foo up-to-date:
  bazel-bin/foo
INFO: Elapsed time: 0.764s, Critical Path: 0.07s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Invocation ID: 661bf76e-7634-4dec-86f4-814701634b05
INFO: Build options --build_runfile_links, --experimental_inmemory_dotd_files, and --experimental_inmemory_jdeps_files have changed, discarding analysis cache.
INFO: Analyzed target //:foo (0 packages loaded, 7 targets configured).
INFO: Found 1 target...
ERROR: /home/lpeltonen/tmp/bazel-repro2/BUILD:3:4: Action foo failed: failed to delete output files before executing action
Target //:foo failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.133s, Critical Path: 0.00s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

What operating system are you running Bazel on?

Ubuntu 18.04.3 LTS

What's the output of bazel info release?

release 4.0.0rc4

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

Looks like the exception is thrown from this branch in src/main/java/com/google/devtools/build/lib/actions/AbstractAction.java:deleteOutput

      Root outputRoot = root.getRoot();
      if (!outputRoot.contains(path)) {
        throw new IOException(e);
      }

Reverting 4009b17 seems to fix this.

@scele scele mentioned this issue Dec 10, 2020
10 tasks
benjaminp added a commit to benjaminp/bazel that referenced this issue Dec 10, 2020
4009b17 resolved output paths but not the related roots.

Fixes bazelbuild#12678.
@philwo philwo added P1 I'll work on this now. (Assignee required) release blocker team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug labels Dec 11, 2020
@philwo philwo self-assigned this Dec 11, 2020
@philwo
Copy link
Member

philwo commented Dec 11, 2020

@benjaminp Thank you for the fix!

meisterT pushed a commit that referenced this issue Dec 15, 2020
4009b17 resolved output paths but not the related roots.

Fixes #12678.

Closes #12634.

PiperOrigin-RevId: 346975821
ulfjack pushed a commit to EngFlow/bazel that referenced this issue Mar 5, 2021
4009b17 resolved output paths but not the related roots.

Fixes bazelbuild#12678.

Closes bazelbuild#12634.

PiperOrigin-RevId: 346975821
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants