Cannot upload Python wheels with direct references in the Requires-Dist field

[geryogam]

The Issue

At work we are trying to upload a Python wheel with a direct reference (URI) in its {package}.dist-info/METADATA file (Requires-Dist field) to a private index server using twine:

Requires-Dist: keras-contrib @ git+https://www.github.com/keras-team/keras-contrib.git

But during the upload:

$ twine upload --repository {...} --config-file {...}/.pypirc {...}/dist/*

we get the following error:

HTTPError: 400 Client Error: Bad Request - Packages with direct (URL) references in Requires-Dist are not allowed.

A similar issue was opened on the Github repository of pip, but this time the user could not upload its Python wheel to a public index server (PyPI) instead of a private one: pypa/pip#6301

PEP 566 has been revised to make it explicit that all Python tools should support direct references in the Requires-Dist field (only "public index servers MAY prohibit strict version matching clauses or direct references in this field"), so it seems that twine is not compliant yet: python/peps#763

@dstufft, is this feature planned?

Environment

• Windows Server 2016
• Python 3.6.8
• Twine 1.13.0
• Pip 19.2.1
• Setuptools 40.6.2

PKG-INFO file:

Metadata-Version: 2.1
Name: {...}
Version: 0.0.1
Summary: {...}
Home-page: https://dev.azure.com/{...}/{...}/_git/{...}
Author: {...}
Author-email: {...}
License: UNKNOWN
Description: {...}

Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: Microsoft :: Windows
Description-Content-Type: text/x-rst

.pypirc file:

[distutils]
Index-servers =
pypi
{...}

[{...}]
Repository = https://pkgs.dev.azure.com/{...}/_packaging/{...}/pypi/upload

[jamadden]

I see that PyPI itself enforces that.

I'm not sure what repository software you're using, but I suspect it's also enforcing that, so there wouldn't be anything twine could do.

[sigmavirus24]

To further explain what @jamadden said, you're seeing HTTPError: 400 Client Error: Bad Request. That means we sent your wheel to your repository server and the repository server responded to our HTTP request with a 400 Bad Request response and the body was Packages with direct (URL) references in Requires-Dist are not allowed. Your repository software is the problem, not twine. Twine doesn't introspect that field in the first place.

[geryogam]

@jamadden @sigmavirus24 Thank you for the analysis. We are using Azure DevOps pipelines.

[tuomastik]

I had the same setup and error as @maggyero and managed to solve the error by removing dependencies with direct URL references right before building the wheel:

sed -i '/ssh:/d' requirements_dev.txt

In my case that was ok since the dependencies with direct URL references were the ones that are only needed by the developers and not during deployment.

[jasonwbarnett]

I'd like to note that this came up for me when trying to build a poetry package that had references to other local packages in a monorepo.

[tool.poetry.dependencies]
local-package-name = { path = "../local-package-path", develop = true }

The generated PKG-INFO contained:

Requires-Dist: local-package-name @ file:///Users/user.name/work/monorepo/packages/local-package-name

The fix was to pin the package to a specific version to use poetry add before running poetry build, i.e.

$ poetry add local-package-name@1.0.0
$ poetry build
$ twine upload

[jzazo]

I installed this poetry plugin instead of manually pinning the local package. Thanks @jasonwbarnett for the tip!
https://github.com/TechnologyBrewery/poetry-monorepo-dependency-plugin

[jasonwbarnett]

@jzazo the internet is an amazing place. Thanks for returning the favor. I may end up using that plugin ❤️