pip.parse does not accept lock files with duplicated dependencies

[AndrewGuenther]

🐞 bug report

Affected Rule

pip.parse

Is this a regression?

Yes, this behavior is only present when using bzlmod

Description

When using pip.parse, if a dependency is repeated in your requirements lock, you'll get an error. This is handled just fine by pip and is necessary for tools like poetry to export this way because it's possible for the same dependency to have different constraints in different parts of the dependency tree.

Ref: python-poetry/poetry#5688

🔬 Minimal Reproduction

Repo with repro here: https://github.com/AndrewGuenther/rules_python_1615_repro

The main branch uses bzlmod and workspace uses WORKSPACE. The workspace branch builds fine, but main does not. Same requirements on both branches, the only difference is bzlmod.

🔥 Exception or Error

If you try and build the example above, you'll get an error to the following effect:

ERROR: Traceback (most recent call last):
        File "/home/andrew/.cache/bazel/_bazel_andrew/9ca33225fba41f1be5682708dd655aad/external/rules_python~0.27.1/python/private/bzlmod/pip.bzl", line 298, column 30, in _pip_impl
                _create_whl_repos(module_ctx, pip_attr, hub_whl_map, whl_overrides)
        File "/home/andrew/.cache/bazel/_bazel_andrew/9ca33225fba41f1be5682708dd655aad/external/rules_python~0.27.1/python/private/bzlmod/pip.bzl", line 129, column 20, in _create_whl_repos
                whl_library(
Error in repository_rule: A repo named pip_38_pyjwt is already generated by this module extension at /home/andrew/.cache/bazel/_bazel_andrew/9ca33225fba41f1be5682708dd655aad/external/rules_python~0.27.1/python/private/bzlmod/pip.bzl:129:20
ERROR: error evaluating module extension pip in @@rules_python~0.27.1//python/extensions:pip.bzl
INFO: Elapsed time: 1.071s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
FAILED: 
    Fetching module extension pip in @@rules_python~0.27.1//python/extensions:pip.bzl; starting

🌍 Your Environment

Operating System:

  
Ubuntu
  

Output of bazel version:

  
6.4.0
  

Rules_python version:

  
0.27.1
  

Anything else relevant?

[AndrewGuenther]

So this appears to be a bzlmod exclusive behavior? We can see this issue with bzlmod starting from 0.15.0, but if we use pip_parse in our WORKSPACE pointing to the same version it works just fine. Are these implementations different between bzlmod and non-bzlmod?

[AndrewGuenther]

Alright, I've now created a repo which shows this is exclusively an issue with bzlmod: https://github.com/AndrewGuenther/rules_python_1615_repro

The main branch uses bzlmod and the workspace branch is the equivalent. main fails to build, but workspace builds no problem.

[rickeylev]

That behavior differs between bzlmod and workspace here is a bit strange. The two implementation re-use a lot of the lower level pieces.

The thing that sticks out to me is both use the same function to parse the requirements. That parsing ignores the [extras] part of a requirements line, e.g. foo ... and foo[bar] ... gets returned as [foo, foo] in the parsed result. So, something in the workspace code must be doing...something...to prevent a duplicate call to whl_library(name="foo"), but I don't see where that is happening.

[rickeylev]

Uhhh, huh.

So the workspace code path isn't de-duplicating the names. It is calling whl_library(name="pip_pyjwt") twice, but the repository rule implementation is only invoked once. I have no idea how this isn't an error. I don't see any maybe wrappers being used.

In this case, the last one wins. The last one is the one whose requirement line sorts last (there's an earlier sorted() call on the list of requirement lines).

In practice, I think this works out OK? Because the requirements file expands out extras into their actual packages? I'm not too familiar with how extras work. If so, then an earlier line being ignored doesn't matter.

It should be easy to mimic this behavior in bzlmod. We just have to dedupe the names, giving preference to the last one.

[AndrewGuenther]

@rickeylev I'm happy to submit a PR if you can point me in a direction. Unless this is one of those "it'd take the same time to explain as it would to implement" sort of things.

[rickeylev]

I've got a fix, so now its just about constructing a test case.

Is there a trick to generating a requirements file with both pyjwt and pyjwt[crypto] entries? When I do it myself, it just has pyjwt[crypto].

[AndrewGuenther]

It's a quirk with poetry and it's export function. If you have a pyproject.toml like this:

[tool.poetry]
name = "example"
version = "0.1.0"
description = "Example"

[tool.poetry.dependencies]
python = "~3.8"
djangorestframework-simplejwt = "^5.2.2"
PyJWT = {extras = ["crypto"], version = "^2.1.0"}

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

and run poetry export you'll get two instances of pyjwt in the requirements output.

[AndrewGuenther]

The relevant poetry behavior is discussed a bit more in this issue: python-poetry/poetry#5688

It's not so much an issue with extras as it is with constraints in general.