remote execution fails with a garbage collecting remote cache

[buchgr]

Using bazel 0.5.3rc3 for local execution. The remote cache is nginx. We have a python script running that enforces an upper size bound and deletes files based on their atime.

We get errors like

/usr/bin/ar: bazel-out/local-fastbuild/bin/third_party/zlib/libzlib.a: File format not recognized

The problem being that the file is empty.

$ ll bazel-out/local-fastbuild/bin/third_party/zlib/libzlib.a
-rw-r----- 1 buchgr eng 0 Jul 26 10:43 bazel-out/local-fastbuild/bin/third_party/zlib/libzlib.a

cc: @damienmg @ulfjack @ola-rozenfeld

[ulfjack]

That is ... not ideal. Any ideas?

[buchgr]

I don't know the cause yet. I am debugging it currently. The good thing is that I can reproduce it 100%.

[buchgr]

We somehow seem to upload failed action execution to the action cache. I managed to get the CAS hash of the ActionKey which is 9456920373db4ff976e09c3d1446cba8bc301e66.

I then logged on to the remote cache machine and disassembled the blob (which is of type message ActionResult)

protoc --decode_raw < 9456920373db4ff976e09c3d1446cba8bc301e66
2 {
  1: "bazel-out/local-fastbuild/bin/third_party/ijar/libplatform_utils.a"
  2 {
    1: "da39a3ee5e6b4b0d3255bfef95601890afd80709"
  }
}
6 {
  1: "da39a3ee5e6b4b0d3255bfef95601890afd80709"
}
8 {
  1: "9026d157afa41c8daa19757d490a42cc77de382e"
  2: 108
}

Tag 2 is output files, 6 is stdout digest, 8 is stderr digest.

da39a3ee5e6b4b0d3255bfef95601890afd80709 is the hash of the zero byte blob. So the output file and stdout is empty. However, there is also stderr 9026d157afa41c8daa19757d490a42cc77de382e and that is 108 bytes long.

$ cat 9026d157afa41c8daa19757d490a42cc77de382e
/usr/bin/ar: bazel-out/local-fastbuild/bin/third_party/ijar/libplatform_utils.a: File format not recognized

So we seem to upload failed action execution. That means we have two bugs:

1. If we purge the remote cache, bazel errors.
2. Bazel uploads these errors to the remote cache.

😄 🍾 🎆

[buchgr]

I fixed the upload of failed action results, and now things are better but we are still failing in weird ways. Looking...

[buchgr]

We seem to have a problem with output files of cached actions having been deleted from the cache. So the action is still cached, but one or more of the output files has been deleted.

[buchgr]

Ok we found the problem. If the remote download of an output file fails, it still creates an empty output file ... and then the local fallback execution fails because the output file already exists, as we are not using sandboxing as a fallback.

Solution seems to be to delete all output files if the remote download fails.

[ola-rozenfeld]

Right, I thought we should delete the partially created file as part of the fail/cleanup, but Ulf just explained this will not be enough for some tools. You're right, deleting everything is the way to go.

[buchgr]

Ok got no more failures, even with aggressive purging ... working on fixes ...