bzlmod: Isolation between two different modules using rules_go/gazelle

[zecke]

What version of gazelle are you using?

gazelle = 0.30.0

What version of rules_go are you using?

rules_go = 0.39.1

What version of Bazel are you using?

bazel = 6.2.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

MacOS/AARCH64

What did you do?

Imagine we have a Bazel module that contains tools we use during building. These tools are implemented in Go and might make use of a variety of dependencies where the specific version does matter. Common examples might be modules not following a Go1 promise or where the generated output changes in non-backward compatible ways. An example of this might be github.com/prometheus/common and github.com/prometheus/prometheus.

In the main application I am importing this module but also have dependencies the same Go modules. A very basic example of this might be:

$ cat project1/MODULE.bazel
module(name = "project1", version = "1.0")

bazel_dep(name = "rules_go", version = "0.39.1", repo_name="rules_go")
bazel_dep(name = "gazelle", version = "0.30.0", repo_name="gazelle")

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(version = "1.20.3")

go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//:go.mod")

use_repo(
    go_deps,
    "com_github_prometheus_prometheus",
)

go_deps.gazelle_override(
    directives = [
        "gazelle:proto disable",
    ],
    path = "github.com/prometheus/prometheus",
)

$ cat project1/go.mod
...
require github.com/prometheus/prometheus v0.44.0
...

$ cat project2/MODULE.bazel
module(name = "project2", version = "1.0")

bazel_dep(name = "rules_go", version = "0.39.1")
bazel_dep(name = "gazelle", version = "0.30.0")

go_sdk = use_extension("@rules_go//go:extensions.bzl", "go_sdk")
go_sdk.download(version = "1.20.3")

go_deps = use_extension("@gazelle//:extensions.bzl", "go_deps")
go_deps.from_file(go_mod = "//:go.mod")

use_repo(
    go_deps,
    "com_github_prometheus_prometheus",
)

go_deps.gazelle_override(
    directives = [
        "gazelle:proto disable",
    ],
    path = "github.com/prometheus/prometheus",
)

$ cat project2/go.mod
...
require github.com/prometheus/prometheus v0.43.0
...

$ cat project_combine/MODULE.bazel
module(name = "foo", version = "1.0")

local_path_override(module_name = "project1", path = "../project1")
local_path_override(module_name = "project2", path = "../project2")

bazel_dep(name = "project1", version = "1.0")
bazel_dep(name = "project2", version = "1.0")

$ bazel build  --enable_bzlmod @project1//:project @project2//:project

What did you expect to see?

I expect to build both versions and one being linked against one version of prometheus and one against the other version.

What did you see instead?

ERROR: Traceback (most recent call last):
        File "/private/var/tmp/_bazel_ich/586a70483d7911528e745954ae0c71df/external/gazelle~0.30.0/internal/bzlmod/go_deps.bzl", line 161, column 36, in _go_deps_impl
                _fail_on_non_root_overrides(module, "gazelle_override")
        File "/private/var/tmp/_bazel_ich/586a70483d7911528e745954ae0c71df/external/gazelle~0.30.0/internal/bzlmod/go_deps.bzl", line 71, column 17, in _fail_on_non_root_overrides
                fail(_report_forbidden_override(module, tag_class))
Error in fail: Using the "go_deps.gazelle_override" tag in a non-root Bazel module is forbidden, but module "project1" requests it.

If you need this override for a Bazel module that will be available in a public registry (such as the Bazel Central Registry), please file an issue at https://github.com/bazelbuild/bazel-gazelle/issues/new or submit a PR adding the required directives to the "directives.bzl" file at https://github.com/bazelbuild/bazel-gazelle/tree/master/internal/bzlmod/directives.bzl.

It fails as neither project is allowed to apply the necessary directives. Even after removing several of these checks, it appears to define a single repository and is linking against it.

My understanding of the visibility of repositories within a module sounded that something like this ought to be possible. The hackiest approach might be to produce a suffix for each repo dependency within a module?

[fmeum]

This issue consists of two independent problems, one of which we have already solved: Could you add the Gazelle directives to https://github.com/bazelbuild/bazel-gazelle/blob/3a1235c298b7dc11fa7ba994db4f92587c1aca79/internal/bzlmod/default_gazelle_overrides.bzl#L7? That will allow you to drop the overrides and allows anyone to use the Go module in a non-root Bazel module.

To fully solve the issue, we also need a way to have a Bazel module depend on a pinned version of a Go dependency. I have some ideas on how that could look like, but need to prototype this first to verify that they actually work.

[zecke]

We might need to pin not only the dependency but the entire set of dependencies. In a hacky way something like adding a suffix to all defined repos in a module and using that in the BUILD file.

Ideally this would allow to patch the repository too (but let's say only for our build tools but not the app).

[fmeum]

This is possible with Bazel 6.3.0, soon to be released, via the new isolate parameter of use_extension. We should allow overrides in isolated extension usages, which is the missing piece.