Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using maven as the repo name causes duplicate warnings when using bzlmod #16839

Closed
jschaf opened this issue May 13, 2024 · 8 comments
Closed

Comments

@jschaf
Copy link
Contributor

jschaf commented May 13, 2024

Downstream bug in rules_jvm_external: bazel-contrib/rules_jvm_external#916

What version of protobuf and what language are you using?

v23.1

What did you expect to see

A clean build when using bzlmod.

What did you see instead?

DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
The maven repository 'maven' is used in two different bazel modules,
originally in '<my_workspace>' and now in 'protobuf'

Summarizing @shs96c in bazel-contrib/rules_jvm_external#916 (comment)

The common maven repo name allows rulesets to contribute to the user's JARs. However, this implies that maven is for the end user, not for transitive dependencies. If a ruleset needs private dependencies, it should use a custom namespace rather than the maven namespace.

Proposed fixes

There are a few ways to fix the logline:

  1. Patch protobuf in the Bazel Central Registry rewrite maven to protobuf_maven.
  2. Modify the protobuf repo to rewrite maven to protobuf_maven.
  3. Fetch the JARs manually without using Maven or rules_jvm_external. Trims the dependency graph a fair bit.

I think 2 is the best short-term option. If you're open to it, I can send a PR.

@jschaf jschaf added the untriaged auto added to all issues by default when created. label May 13, 2024
@bnordli
Copy link

bnordli commented Jun 25, 2024

Can this please be fixed in both the v3 and v4 protobuf libraries in Bazel Central?

I cannot find a way to rename the maven repository in our modules, because there needs to be a maven repository defined (somewhere), and I cannot see there are support for multiple maven repositories when using bzlmod (https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/bzlmod.md).

@shs96c
Copy link

shs96c commented Jun 25, 2024

@bnordli, assuming you're using bzlmod, the examples list by @jschaf show you how to create a "private" set of dependencies for protobuf. If you were to do this, you would require consumers of this ruleset to add the com.google.protobuf:protobuf-java version they need to their own maven_install or install.

use_extension also supports a new isolated parameter, which may also be useful if you're working on a root project.

mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 2, 2024
Resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 2, 2024
Resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190

Signed-off-by: Mike Bland <mbland@engflow.com>
mbland added a commit to EngFlow/bazel_invocation_analyzer that referenced this issue Jul 3, 2024
Updates as much as possible except rules_jvm_external.

The latest rules_jvm_external v6.1 breaks this project, both under the
previous Bazel version (7.0.2) and the new one (7.2.1). I've filed
bazel-contrib/rules_jvm_external#1189, which uses this repo and this PR as
an example.

See the last section below for details.

---

Bazel update:
- Bazel v7.2.1
  https://github.com/bazelbuild/bazel/releases/tag/7.2.1

Bazel module updates:
- bazel-skylib v1.7.1
  https://github.com/bazelbuild/bazel-skylib/releases/tag/1.7.1
- platforms v0.0.10
  https://github.com/bazelbuild/platforms/releases/tag/0.0.10
- rules_proto v6.0.2
  https://github.com/bazelbuild/rules_proto/releases/tag/6.0.2

JAR updates:
- com.google.code.gson:gson:2.11.0
- com.google.guava:guava:33.2.1-jre
- commons-cli:commons-cli:1.8.0

Test JAR updates:
- com.google.googlejavaformat:google-java-format:1.22.0
- com.google.truth:truth:1.4.3
- com.google.truth.extensions:truth-java8-extension:1.4.3
- org.mockito:mockito-core:5.12.0

---

Added explicit module spec and repinned the maven deps for:
- protobuf v27.2
  https://github.com/protocolbuffers/protobuf/releases/tag/v27.2

Adding protobuf explicitly resolves these warnings:

```txt
DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
  The maven repository 'maven' is used in two different bazel modules,
  originally in 'com_engflow_bazel_invocation_analyzer'
  and now in 'protobuf'

DEBUG: .../external/rules_jvm_external~/coursier.bzl:593:18:
    Found duplicate artifact versions
    com.google.code.gson:gson has multiple versions 2.11.0, 2.8.9
    com.google.guava:guava has multiple versions 33.2.1-jre, 31.1-jre
    com.google.truth:truth has multiple versions 1.4.3, 1.1.2
    org.mockito:mockito-core has multiple versions 5.12.0, 4.3.1
Please remove duplicate artifacts from the artifact list so you do not
get unexpected artifact versions
```

See also:

- Duplicate maven repositories when importing bazel_deps that use
  maven.install
  bazel-contrib/rules_jvm_external#916

- Using maven as the repo name causes duplicate warnings when using
  bzlmod
  protocolbuffers/protobuf#16839

- MODULE.bazel doesn't define @maven repository
  protocolbuffers/protobuf#17176

- Stop including extra artifacts on maven.install()
  bazel-contrib/rules_jvm_external#1168

- Use a custom name for the maven repository (maven_protobuf)
  protocolbuffers/protobuf#17190

---

rules_jvm_external v6.1 somehow creates duplicate `jvm_import` rules for
binary and source jars, instead of using the `srcjar` attribute:

```txt
ERROR: Traceback (most recent call last):
  File ".../external/rules_jvm_external~~maven~maven/BUILD",
    line 34, column 11, in <toplevel>
      jvm_import(

Error in jvm_import: jvm_import rule
  'com_google_auto_value_auto_value_annotations' in package ''
  conflicts with existing jvm_import rule, defined at
  .../external/rules_jvm_external~~maven~maven/BUILD:9:11
```

The content of rules_jvm_external~~maven~maven/BUILD at lines 9 and 34:

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"],
```

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar"],
```

This pattern repeats for all the JAR targets.

The BUILD contents from v5.3, which builds successfully both before and
after applying the PR changes:

```bzl
jvm_import(
  name = "com_google_auto_value_auto_value_annotations",
  jars = ["com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1.jar"],
  srcjar = "com/google/auto/value/auto-value-annotations/1.10.1/auto-value-annotations-1.10.1-sources.jar",
```

---------

Signed-off-by: Mike Bland <mbland@engflow.com>
@mpatou-openai
Copy link

It seems latest version of protobuf in bcr are not using maven anymore in bzlmod(I can still see it in the WORKSPACE) and it leads to other set of issues:

no such package '@@[unknown repo 'maven' requested from @@protobuf~]//': The repository '@@[unknown repo 'maven' requested from @@protobuf~]' could not be resolved: No repository visible as '@maven' from repository '@@protobuf~'

@JasonLunn
Copy link
Contributor

It seems latest version of protobuf in bcr are not using maven anymore in bzlmod(I can still see it in the WORKSPACE) and it leads to other set of issues:

no such package '@@[unknown repo 'maven' requested from @@protobuf~]//': The repository '@@[unknown repo 'maven' requested from @@protobuf~]' could not be resolved: No repository visible as '@maven' from repository '@@protobuf~'

Can you open a separate issue with details?

@JasonLunn
Copy link
Contributor

@bnordli, assuming you're using bzlmod, the examples list by @jschaf show you how to create a "private" set of dependencies for protobuf. If you were to do this, you would require consumers of this ruleset to add the com.google.protobuf:protobuf-java version they need to their own maven_install or install.

use_extension also supports a new isolated parameter, which may also be useful if you're working on a root project.

@jschaf / @bnordli - did the suggestion from @shs96c work for you?

@JasonLunn JasonLunn added java bazel wait for user action and removed untriaged auto added to all issues by default when created. labels Oct 3, 2024
@jschaf
Copy link
Contributor Author

jschaf commented Oct 3, 2024

The isolate parameter is marked experimental with a strong warning not to depend on it. I think using one of the alternatives in the first comment is a better approach.

Experimental. This parameter is experimental and may change at any time. Please do not depend on it. It may be enabled on an experimental basis by setting --experimental_isolated_extension_usages

@JasonLunn
Copy link
Contributor

If you send a PR I can get it routed to the right reviewers.

jschaf added a commit to jschaf/protobuf that referenced this issue Oct 7, 2024
The default name for the rules_jvm_external.maven rule is "maven". When not set,
it defaults to "maven". For root modules also using rules_jvm_external, the name
clash causes a warning:

```
DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
The maven repository 'maven' is used in two different bazel modules,
originally in '<my_workspace>' and now in 'protobuf'
```

Summarizing @shs96c in [1]:

> The common maven repo name allows rulesets to contribute to the user's JARs.
> However, this implies that maven is for the end user, not for transitive
> dependencies. If a ruleset needs private dependencies, it should use a custom
> namespace rather than the maven namespace.

Since protobuf is not contributing to user's JARs, we'll use a custom namespace.
There's precedent for using a custom namespace for library modules:

- rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`.
- rules_kotlin uses `kotlin_rules_maven` instead of `maven`.

[1]: bazel-contrib/rules_jvm_external#916 (comment)

Fixes protocolbuffers#16839.
@jschaf
Copy link
Contributor Author

jschaf commented Oct 7, 2024

@JasonLunn #18641

zhangskz pushed a commit that referenced this issue Dec 2, 2024
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning:

```
DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
The maven repository 'maven' is used in two different bazel modules,
originally in '<my_workspace>' and now in 'protobuf'
```

Summarizing @shs96c in [1]:

> The common maven repo name allows rulesets to contribute to the user's JARs.
> However, this implies that maven is for the end user, not for transitive
> dependencies. If a ruleset needs private dependencies, it should use a custom
> namespace rather than the maven namespace.

Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules:

- rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`.
- rules_kotlin uses `kotlin_rules_maven` instead of `maven`.

[1]: bazel-contrib/rules_jvm_external#916 (comment)

Fixes #16839.

Closes #18641

COPYBARA_INTEGRATE_REVIEW=#18641 from jschaf:joe/protobuf-maven bd2c62f
PiperOrigin-RevId: 684625084
zhangskz added a commit that referenced this issue Dec 2, 2024
The default name for the rules_jvm_external.maven rule is "maven". When not set, it defaults to "maven". For root modules also using rules_jvm_external, the name clash causes a warning:

```
DEBUG: $TMP/external/rules_jvm_external~/private/extensions/maven.bzl:154:14:
The maven repository 'maven' is used in two different bazel modules,
originally in '<my_workspace>' and now in 'protobuf'
```

Summarizing @shs96c in [1]:

> The common maven repo name allows rulesets to contribute to the user's JARs.
> However, this implies that maven is for the end user, not for transitive
> dependencies. If a ruleset needs private dependencies, it should use a custom
> namespace rather than the maven namespace.

Since protobuf is not contributing to user's JARs, we'll use a custom namespace. There's precedent for using a custom namespace for library modules:

- rules_jvm_external uses `rules_jvm_external_deps` instead of `maven`.
- rules_kotlin uses `kotlin_rules_maven` instead of `maven`.

[1]: bazel-contrib/rules_jvm_external#916 (comment)

Fixes #16839.

Closes #18641

COPYBARA_INTEGRATE_REVIEW=#18641 from jschaf:joe/protobuf-maven bd2c62f
PiperOrigin-RevId: 684625084

Co-authored-by: Joe Schafer <joe.schafer@delta46.us>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants