Enable --remote_download_minimal

[jwnimmer-tri]

Baseline from master:

• https://drake-jenkins.csail.mit.edu/job/linux-jammy-gcc-bazel-experimental-debug/536/consoleFull
  • 333 seconds overall.
    • 288 seconds in bazel.
    • 45 seconds in prep / cleanup.
• https://drake-jenkins.csail.mit.edu/job/mac-arm-monterey-clang-bazel-experimental-debug/9/consoleFull
  • ??? seconds overall.
    • ??? seconds in bazel.
    • ??? seconds in prep / cleanup.

Canary using this pull request:

• https://drake-jenkins.csail.mit.edu/job/linux-jammy-gcc-bazel-experimental-debug/535/consoleFull
  • 69 seconds overall.
    • 34 seconds in bazel.
    • 35 seconds in prep / cleanup.
• https://drake-jenkins.csail.mit.edu/job/mac-arm-monterey-clang-bazel-experimental-debug/10/consoleFull
  • ??? seconds overall.
    • ??? seconds in bazel.
    • ??? seconds in prep / cleanup.

This risks bazelbuild/bazel#8250 if our cache TTL is too low, but I am thinking we're probably safe.

---

This change is 

[svenevs]

I added the equivalent mac-arm jobs since those are cached, this PR build is already finished in 3 minutes and 43 seconds. Basline master is 15 minutes and 33 seconds. I wasn't sure where you were looking to determine where seconds in bazel and seconds in prep / cleanup go but it seems like we should let this run and revert if we start encountering problems.

I didn't fully understand some of the issues contributors on the bazel issue were describing, but it seemed like you can only run into problems if things start getting deleted from the cache somehow. I don't think we will encounter that since nightly and continuous just overwrite everything? We just use --remote_upload_local_results=yes, there doesn't appear to be anything like "remote upload minimal" at least.

And just tell the buildcops on slack if they start seeing errors like java.lang.IllegalStateException: Unexpected inconsistency: ActionLookupData or java.io.IOException: Failed to fetch file with hash then we'll probably want to revert this.

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @jwnimmer-tri)

[BetsyMcPhail]

Reviewed 1 of 1 files at r1, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @jwnimmer-tri)

[jwnimmer-tri]

... it seemed like you can only run into problems if things start getting deleted from the cache somehow. I don't think we will encounter that since nightly and continuous just overwrite everything?

Maybe. It's not so much "overwrite" as "repopulate" though.

I think the risk is that "access time" is no longer a sound measure of what is still relevant in the cache, so our "evict-after-a-few-days-without-access" script rule might delete a build output, but not delete the ac entry that refers to that output.

I also realized -- if we see trouble here, we might need to not only revert this patch but also bump the cache key version from v2 => v3 to force a re-population.

drake-ci/driver/configurations/cache.cmake

Line 493 in fa06c03

set(DASHBOARD_REMOTE_CACHE_KEY_VERSION "v2")