Permission denied on Windows tests (flaky)

[laurentlb]

I see lots of Windows failures on this pipeline: https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/579
For example, I see 7 failures for the flag incompatible_string_replace_count, all of them seem flaky.

Examples of error messages:

(03:05:57) ERROR: C:/b/bk-windows-drwz/bazel-downstream-projects/buildtools/buildifier2/BUILD.bazel:3:11: no such package '@skylark_syntax//syntax': no such package '@bazel_gazelle_go_repository_cache//': C:/b/pmtr6clt/external/bazel_gazelle_go_repository_cache/pkg/mod/github.com/chzyer/readline@v0.0.0-20180603132655-2972be24d48e/doc (Permission denied) and referenced by '//buildifier2:go_default_library'

(03:13:11) ERROR: C:/b/hhtk5fpz/external/bazel_tools/tools/jdk/BUILD:383:6: @bazel_tools//tools/jdk:remote_toolchain depends on @remote_java_tools_windows//:toolchain in repository @remote_java_tools_windows which failed to fetch. no such package '@remote_java_tools_windows//': C:/b/hhtk5fpz/external/remote_java_tools_windows/java_tools/JavaBuilder_deploy.jar (Permission denied)

(03:05:33) ERROR: C:/b/bk-windows-601l/bazel-downstream-projects/bazelisk/BUILD:41:11: no such package '@com_github_mitchellh_go_homedir//': no such package '@bazel_gazelle_go_repository_cache//': C:/b/5t7h6vom/external/bazel_gazelle_go_repository_cache/pkg/mod/github.com/hashicorp/go-version@v1.2.0 (Permission denied) and referenced by '//:go_default_library'

[meteorcloudy]

I can reproduce this issue locally, will look into it soon!

[meteorcloudy]

It looks like we are hitting something similar to bazelbuild/bazel#7458

I haven't figured out exactly why, but enabling symlink support via --windows_enable_symlink seems to fix the problem.

[meteorcloudy]

OK, the reason of this failure is because the Java worker will hold the file handle of C:\src\tmp\cbtx3svz\external\remote_java_tools_windows\java_tools\JavaBuilder_deploy.jar.
When we change an incompatible flag that will cause Bazel to refetch the java tool (eg. --incompatible_disable_depset_items), Bazel will fail the clean up the directory of C:\src\tmp\cbtx3svz\external\remote_java_tools_windows, because JavaBuilder_deploy.jar is still open.

To workaround this issue on CI, we can use Bazelisk's BAZELISK_SHUTDOWN feature, which will shut down bazel between builds and ensure the file handles are released by the Java workers.

[laurentlb]

It still fails: https://buildkite.com/bazel/bazelisk-plus-incompatible-flags/builds/592#83a2bd3b-a04d-4dea-8aa1-3812ec39c251

(03:03:01) ERROR: C:/b/bk-windows-drwz/bazel-downstream-projects/bazel-skylib/gazelle/BUILD:6:11: no such package '@com_github_bazelbuild_buildtools//build': no such package '@bazel_gazelle_go_repository_cache//': C:/b/zbugc6cn/external/bazel_gazelle_go_repository_cache/pkg/mod/github.com/!burnt!sushi/toml@v0.3.1/cmd/toml-test-decoder (Permission denied) and referenced by '//gazelle:go_default_library'

[laurentlb]

Can we reopen this bug? Or I can file a new one.

[meteorcloudy]

Oh sorry, we can re-open this. Looks like my change only fixed the Bazel Federation case, but not for this one.

[meteorcloudy]

The bug is on Bazel side. Can you please help review the fix bazelbuild/bazel#11982?

[meteorcloudy]

Let's keep this open until we are sure the problem is fixed. We'll have to wait for the next month's Bazel release.

[tetromino]

FYI, I'm hitting this in Stardoc today with Bazel 6.3.2 as well as 7.0 pre: bazelbuild/stardoc#179 -> https://buildkite.com/bazel/stardoc/builds/1081

[rickeylev]

FYI: I've been seeing this lately. Mostly with rules_python, but I also so it in a BCR PR today.

Not a lot, maybe like once or twice a week / 50-100 CI runs. Few enough that pressing retry isn't a big inconvenience, but often enough that retry doesn't always fix it, and enough to mention it.