Migrate to an incremental format for MODULE.bazel.lock

[fmeum]

The old lockfile fields and all code related to it as well as the workarounds for local path inclusions are removed.

The distribution archive lockfile needs to be checked in temporarily as the main MODULE.bazel.lock file does not contain the hashes for the registry files.

Fixes #19621
Fixes #20369

RELNOTES: The format for MODULE.bazel.lock is now less likely to result in merge conflicts and is updated incrementally, with only new files downloaded from registries and existing ones taken from the repository cache (if configured).

[fmeum]

@Wyverald This is stacked on #21901, but otherwise ready for a first review.

[fmeum]

I rebased and fixed the remaining tests. This got a bit larger as I reverted some commits instead of adapting their code and removing it in a follow-up PR, but I kept all their tests around that still apply.

As a follow-up, I could relatively easily add a new lockfile mode if that's our preferred way of allowing users to refetch mutable registry content.

@SalmaSamy I'm not entirely sure how this interacts with bazel vendor and airgapped builds. Do you have an integration test that models the expected behavior? Then I could make sure that the new format doesn't break it.

[fmeum]

Forgot to update the test lockfile, hope this is green now.

[fmeum]

@Wyverald The --batch tests in starlark_repository_test appear to time out pretty reliably. Do you know whether --batch messes with repo cache or lockfile usage in any way? I'm not sure why there would be more BCR connections now, although perhaps they are just timed differently and that tickles the RBE setup in the wrong way.

[fmeum]

I noticed that this patch causes the verify_default_lockfile test to fail, which I wouldn't expect if the test repo cache is populated correctly (with the new lockfile format):

diff --git a/src/test/tools/bzlmod/BUILD b/src/test/tools/bzlmod/BUILD
index ccaa79a657..a0bbf6bc15 100644
--- a/src/test/tools/bzlmod/BUILD
+++ b/src/test/tools/bzlmod/BUILD
@@ -23,6 +23,5 @@ sh_test(
         "//src:bazel",
         "//src/test/shell/bazel:test-deps",
     ],
-    tags = ["requires-network"],
     deps = ["@bazel_tools//tools/bash/runfiles"],
 )

Do you know whether I need to do anything more to ensure all tests have access to a repo cache containing the BCR files? It's surprising that no existing test is failing.

Edit: Nvm, the repo cache is only used in CI, not locally.

[Wyverald]

nice. I like this new approach and the comments make it much easier to understand too. Let's ship it!!

[Wyverald]

@bazel-io fork 7.2.0