ENH: add support for PEP 639

[dnicolodi]

Closes gh-270

[henryiii]

Looks pretty good from a quick review. I'll keep this in mind when I implement it (very soon) in scikit-build-core.

[dnicolodi]

Looks pretty good from a quick review. I'll keep this in mind when I implement it (very soon) in scikit-build-core.

It is a bit uglier than what I would like because I wanted to keep compatibility with older pyproject-metadata. meson-python supports anything back to version 0.7.1.

[QuLogic]

I gave this a shot in matplotlib/matplotlib#28982, and License-Expression is correctly pulled from meson.build, as are License-File from the main project, but what I'd be missing there is License-File from the subprojects.

[dnicolodi]

I'm not very well versed in licensing issues, but I don't think the way PEP 639 supports declaring package licenses works well for this use case. The main problem is that it expects the license of the package's sdist and wheel to be exactly the same. For packages like matplotlib that link statically with some libraries, this is not the case: the license of the source code is the matplotlib license, the license of the wheel is the union of the matplotlib license with the license of all the libraries linked to it.

Concretely, the issue is that in the most common case the subprojects directory does not contain the source code of the subprojects but only references to them (in the matplotlib case in the form of meson wrap files). However, the License-File fields need to point to actual files distributed in the sdist and in the wheel. The sdist in these cases does not contain the license files, thus the License-File fields for the subprojects cannot be added. The metadata information for the sdist and the wheel need to be the same, thus the fields cannot be added to the wheel metadata either.

I don't know how the problem could be solved. The most formally right solution would to allow source and binary distributions to have different licenses. But this is a quite significant change in behavior for the package metadata system.

The pragmatic solution is to add the subprojects' license files to the source distribution and accept that the sdist as a whole might have a more restrictive license than what it could have.

[eli-schwartz]

However, the License-File fields need to point to actual files distributed in the sdist and in the wheel. The sdist in these cases does not contain the license files, thus the License-File fields for the subprojects cannot be added. The metadata information for the sdist and the wheel need to be the same, thus the fields cannot be added to the wheel metadata either.

This is not true. Metadata can be dynamic and vary between the sdist and the wheel. It's already dynamic as a result of being calculated from meson introspection, so you just need to mark that in the sdist as well.

The decision to make sdist metadata static and forbidden to change is... definitely a decision of all time. But the good news is it comes with an escape hatch. This escape hatch is needed since the spec also says that it's illegal per the spec to apply open source patches to an sdist metadata if the fields are marked as static. 🤷

[dnicolodi]

Also, noticed there's no quick and easy link to the source from readthedocs

I'm not sure about what you mean with this. meson-python documentation is not on readthedocs and we have a "Source Code" link in the left sidebar (and typing this made me realize that the capitalization of the title of the link is inconsistent with the other titles).

[dnicolodi]

packaging 24.2 is out!

Thanks. I'll add tests making sure the validation of the license field works as expected.

[henryiii]

By readthedocs I meant https://mesonbuild.com/meson-python/. Okay, I was looking for a GitHub icon on the bottom or top or similar, but that's not bad.

[dnicolodi]

Anyway I think people should have the right to use dynamic metadata if they want to, and use pyproject.toml metadata if they want to.

With that in mind, the ability to have different metadata for sdist vs. wheel by making use of "Dynamic" entries in current versions of the Core Metadata, is a powerful escape hatch for those who choose to do so.

Dynamic metadata is seen as such a nice feature that there is a discussion about eliminating it:
https://discuss.python.org/t/brainstorming-elliminating-dynamic-metadata/71405/1

I don't have the time to read these long threads on Python discourse (even more so because discourse is horrible) let alone contribute any opinions. But I thought that some here may be interested...

[eli-schwartz]

Dynamic metadata is seen as such a nice feature that there is a discussion about eliminating it: https://discuss.python.org/t/brainstorming-elliminating-dynamic-metadata/71405/1

I don't have the time to read these long threads on Python discourse (even more so because discourse is horrible) let alone contribute any opinions. But I thought that some here may be interested...

Agreed on both counts about reading the Python discourse.

But from a quick look at the initial post before the site became impossible to read, it appears to make a few basic arguments about why dynamic metadata is "bad".

• users of editable installs have inaccurate versions from git describe
• poetry and uv rerun metadata generation constantly and would rather not have to execute a build backend hook to find out things like the version or dependencies, as it is "slow"

I find both arguments weak enough that I suspect the dislike of dynamic metadata came first and the supporting arguments were created to fit the narrative. None of this matters for sdists or wheels because both contain static metadata explicitly intended to be a cache, and editable installs are an extremely specific use case that users have already chosen to make some sacrifices in order to use -- plainly they are not worried about the version number or if they care they are using myprog --version which isn't actually covered by pypa standards about dynamic vs static metadata.

Lockfiles can just avoid self-inflicted wounds by generating them when asked, but it doesn't actually matter either way since lockfiles only have to do ANY of this on bare source trees, not dependencies obtained via sdist/ wheel, which means that users who use poetry / uv can choose to avoid dynamic metadata and users who prefer dynamic metadata can generate lockfiles less frequently.

This is yet another example of people who think the point of a standard is to hard-line forbid people from doing perfectly reasonable things that are optional, because the people doing the forbidding don't personally value that thing, and cannot abide the idea that anyone else would value it.

This contrasts with the fact that the standard currently allows all those perfectly reasonable things precisely because they are perfectly reasonable, and forbidding them would upset the people who are perfectly justified in doing perfectly reasonable things.

The whole argument is undoubtedly just perl code. They'll get tired of it after a while and nothing will change, not even opinions.

[dnicolodi]

Ready to merge I think - but I'd hold off until packaging and PyPI have support.

packaging releases a version with the license validation and normalization. PyPI is ironing out the kinks in their support for PEP 639. I've tried to upload to PyPI a package using PRP 639 fields, but neither twine nor uv publish support Metadata 2.4 at the moment. I don't know of anyone working on adding support to twine. I wonder what is the recommended way to upload to PyPI nowadays...

[dnicolodi]

I've looked into what it would take to add support for metadata 2.4 to twine, and the answer is worst than what I expected: it would take a miracle to have it any time soon. twine uses pkginfo to parse the package metadata. So adding support for metadata 2.4 requires a new release of pkginfo. However, twine is not compatible with the latest pkginfo release. There is a pull request to make twine compatible with the latest pkginfo but it is open since June and it stalled.

I don't know why twine uses pkginfo and not packaging to parse the metadata. warehouse uses packaging thus I would have expected twine to use it too. pkginfo looks to be a one man show maintained on Launchpad in a Bazar repository https://code.launchpad.net/pkginfo

[agriyakhetarpal]

In the meantime, it is possible to install pkginfo>=1.11 after you install twine (i.e., by breaking the dependency tree) and twine check --strict should then work with wheels with the Metadata-Version set as 2.4. PyPI now accepts those wheels, too, if uploaded manually – zig-pypi (https://pypi.org/project/ziglang/0.13.0.post1/#files) is an example.

[dnicolodi]

@agriyakhetarpal The dependecy problem is not a problem in itself. It is only a problem because implementing metadatya 2.4 support in twine requires adding support to pkginfo and make twine consume the added metadata fields. If there is no maintainer interest in supporting a version of pkginfo release a long while back, I don't see how there could be interest in merging patches to work with a future release. It is my understanding that, without support in the upload tool, the license information recorded in the License-Expression metadata field is not picked up by PyPI. For example, the ziglang release you link to does not show any license information on the PyPI page. Getting accurate information shown on PyPI is the purpose of this whole exercise, thus I don't think that what you show is a successful example.

[dnicolodi]

For what is worth, uv publish can upload metadata 2.4 distributions, but the lack of explicit metadata 2.4 support means that the license is not recorded in the PyPI index and not shown in the package page on PyPI.

[rgommers]

twine is still the PyPA-recommended tool, but I expect that uv publish will replace it soon enough if twine is too slow to add support for new features like PEP 639. The uv issue with license metadata has an issue for it (unblocked, the linked bug in Warehouse is fixed already): astral-sh/uv#9204

[dnicolodi]

I've opened a bug offering to port twine from pkginfo to packaging to parse metadata and offering to do the work pypa/twine#1178 That should make supporting new metadata versions much easier. Let's see how that is received

[agriyakhetarpal]

For what is worth, uv publish can upload metadata 2.4 distributions, but the lack of explicit metadata 2.4 support means that the license is not recorded in the PyPI index and not shown in the package page on PyPI.

I understand the state of things better now, thanks for the context!