Prefer binary wheel if using manylinux1 platform and newer wheel only available for manylinux2010

[jcfr]

What's the problem this feature will solve?

With the transition to manylinux2010, build relying on binary wheels will now attempt to build project from source instead of installing the latest binary wheel available for manylinux1.

Describe the solution you'd like

If (1) current system is manylinux1 compatible and (2) the latest release of a python package is manylinux2010 compatible:

• Approach 1: Prefer binary wheel and report a warning suggesting the user to update their platform could be more helpful and would provide a path forward. A --no-prefer-binary would allow to disable this.

• Approach 2: Continue building the sdist (assuming --prefer-binary was not passed) but display a warning indicating that no manylinux2010 wheel was available and this why the project is built from source.

• Approach 3: Update pep-345 to allow specifying that a sdist requiring a specific version of manylinux ?

This would streamline the user experience while providing a path forward.

Example of real-world issues are documented here:

• Failure to install manylinux1 wheel from manylinux1 docker image (Centos5) when a newer manylinux2010 wheel is available #6547 (comment)
• ci: Fix build of linux wheel requiring cmake<3.14 equinor/dlisio#116
• 3.14, "Problem with the CMake installation, aborting build" scikit-build/cmake-python-distributions#76

Alternative Solutions

Systematically passing --prefer-binary alleviate the problem but is not trivial to understand for user of your project.

Additional context

See #6547 (comment)

[cjerdonek]

@njsmith Do you have any thoughts on this? I figured you'd be one good person to ask.

[njsmith]

So if I understand correctly, the situation is:

• Someone is running CentOS 5 or similar
• somepackage v1 has an sdist and a manylinux1 wheel
• somepackage v2 has an sdist and a manylinux2010 wheel
• pip install somepackage sees that a v2 package is available, and can't use the manylinux2010 wheel, because this is an ancient system, and tries to build from sdist instead
• You would like it to automagically install somepackage v1 instead

I doubt the pip devs are going to go for any solution where pip automatically ignores a new release and installs and old one instead, without any user intervention. They chose their current behavior intentionally many years ago, for good reasons. Ignoring new package updates is a serious problem for security and stability (new versions have fixes!). It would also mean that if some maintainers ever published a wheel for a given platform, they are forced to keep publishing wheels for that platform forever, or else their users will get trapped on some old unsupported version.

I don't see how the manylinux1 → manylinux2010 transition is any different from any other situation where a package stops providing wheels for some platform. And you really really really really should not be using centos 5. I'm guessing that the real problem here is that you aren't really using centos5, you're just trying to use the manylinux1 build image to build manylinux1 wheels, and some of your dependencies are dropping support for manylinux1? I don't think there's anything better to do there than to use --prefer-binary, and migrate to manylinux2010 yourself...

[pfmoore]

@njsmith Thanks for the analysis. I concur that --prefer-binary is the intended way of dealing with this situation, and I'd be very uncomfortable with any solution that didn't require the user to explicitly opt in to ignoring newer releases.

[jcfr]

Thanks for looking into this.

It makes sense, passing --prefer-binary is the way to go.

you're just trying to use the manylinux1 build image to build manylinux1 wheels, and some of your dependencies are dropping support for manylinux1?

To be specific:

• the latest release of the cmake wheels is done using manylinux2010
• maintainer of python project that are still using manylinux1 saw their build failing because their project started to try build cmake wheel from source.
• documenting the problem and suggestion to use --prefer-binary is the way to go

I am also wondering if I should just deleted the cmake source distribution from pypi, that way it should nicely fall back without requiring user to pass --prefer-binary.

[jcfr]

should just deleted the cmake source distribution from pypi

I removed the source distribution and the install fallback to the binary wheel without requiring the --prefer-binary option.

[pfmoore]

I am also wondering if I should just deleted the cmake source distribution from pypi

This sounds like an unfortunate choice. Having binary-only distributions on PyPI is not ideal, as it blocks usage of the projects by other users who use a platform for which wheels are not provided, but who can build from source.

I feel sufficiently strongly that projects should distribute source, that I'd be willing to look at alternative options here. I just don't know what they would be (given that we definitely don't want to prefer older binaries for pure Python projects, which can be easily built on all platforms).

[jcfr]

Having binary-only distributions on PyPI is not ideal

Generally speaking, I totally agree. Project should upload their sdist 👍

This is a on-off solution to ensure existing project using both cmake (and manylinux1 for building wheel) can download a valid cmake wheel.

The idea is to avoid this issue: equinor/dlisio#116

it blocks usage of the projects by other users who use a platform for which wheels are not provided

Worth noting that we are now distributing the CMake wheels as "platform generic wheels" (i.e ABI is none, and python implementation is generic). This should cover a lot of cases.