new gazelle v0.36.0 fails with Go sum mismatches

[tyler-french]

What version of gazelle are you using?

https://github.com/bazelbuild/bazel-gazelle/releases/tag/v0.36.0

What version of rules_go are you using?

https://github.com/bazelbuild/rules_go/releases/tag/v0.46.0

What version of Bazel are you using?

7.1.1

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Linux AMD 64

What did you do?

Darwin ARM 64

What did you expect to see?

Successful gazelle run

What did you see instead?

ERROR: An error occurred during the fetch of repository 'com_github_procyon_projects_chrono':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_johankj/cf455b47f50dd837b297aa6fcdb2d775/external/bazel_gazelle/internal/go_repository.bzl", line 263, column 17, in _go_repository_impl
		fail("failed to fetch %s: %s" % (ctx.name, result.stderr))
Error in fail: failed to fetch com_github_procyon_projects_chrono: fetch_repo: resulting module with sum h1:l0VwRF053FgTUXw2dMwpHYBY9d0SuZOAmjSBOvdeO1Y=; expected sum h1:edOeKOW2eGP8j3gI6wQWzyoUM0njjuxOlbtpguIkXqQ=
ERROR: /Users/johankj/Uber/go-code-odin/WORKSPACE:68:26: fetching go_repository rule //external:com_github_procyon_projects_chrono: Traceback (most recent call last):
	File "/private/var/tmp/_bazel_johankj/cf455b47f50dd837b297aa6fcdb2d775/external/bazel_gazelle/internal/go_repository.bzl", line 263, column 17, in _go_repository_impl
		fail("failed to fetch %s: %s" % (ctx.name, result.stderr))
Error in fail: failed to fetch com_github_procyon_projects_chrono: fetch_repo: resulting module with sum h1:l0VwRF053FgTUXw2dMwpHYBY9d0SuZOAmjSBOvdeO1Y=; expected sum h1:edOeKOW2eGP8j3gI6wQWzyoUM0njjuxOlbtpguIkXqQ=
Analyzing: target //src/code.uber.internal/compute/foosbot:foosbot (462 packages loaded, 2947 targets configured)

However this is a pretty normal go_repository rule:

 go_repository(
        name = "com_github_procyon_projects_chrono",
        build_extra_args = ["-go_naming_convention_external=go_default_library"],
        build_file_generation = "on",
        build_file_proto_mode = "disable",
        importpath = "github.com/procyon-projects/chrono",
        sum = "h1:edOeKOW2eGP8j3gI6wQWzyoUM0njjuxOlbtpguIkXqQ=",
        version = "v1.0.0",
    )

[tyler-french]

Might be related to: #1755

[fmeum]

@scaiper Do you happen to see why this particular module would result in a different hash?

[scaiper]

I've tried this exact version with sum = "h1:edOeKOW2eGP8j3gI6wQWzyoUM0njjuxOlbtpguIkXqQ=" and it worked. But we use Bzlmod, maybe there are some differences.
Also this project changed module name, in recent versions it is codnect.io/chrono, maybe it is somehow relevant.
I'll try to look at this issue tomorrow.

[scaiper]

Can you confirm that this issue still occurs after bazel clean --expunge?

[scaiper]

Are you using either GO_REPOSITORY_USE_HOST_CACHE or GO_REPOSITORY_USE_HOST_MODCACHE options? If you do, could you also try cleaning host module cache? It could be modified.

[tyler-french]

We're only noticing this in a very specific environment, so we may be doing something weird. Will follow-up

[tyler-french]

(cc @linzhp) It looks like the reason this is happening is because of our "sparse checkout" tooling. We have a feature to be able to checkout only a small part of our Monorepo (and all it's required deps, including external deps), which we copy over from a "full" work tree.

In this case, it looks like the sum of the downloaded archive matches the SUM in the go.sum file and go_repository rule, but at build time, the external directory's sum might be slightly different, as it only has the required packages for the current targets.

[scaiper]

To be precise, it's not a build time verification. Verification is done in repository rule right after downloading and coping module to final location.
It should only differ if host module cache is in use and this cache is modified.

[tyler-french]

Hmm, it does look like we are using a local GOMODCACHE. I am curious if/why this would be getting corrupted.

[tyler-french]

Created this PR: #1782
Some users have go mod cache's that are years old, and we have never verified them

If we start verifying them, I think it would be good to let them know that this might be the failure cause.