Require minimum pip version?

[dstufft]

With #3691 projects can take control over every version of tooling they interact with from build tools to runtime dependencies. The one thing they can't control is what version of thing they are installed with. This makes a lot of sense both from a practical standpoint (you can't run N versions of pip at once for each thing you're installing) and from a purity standpoint (our standards are not written solely for pip, they're written so any installer can function instead of pip if desired).

That being said, it's still useful for projects to require a specific version of pip to install their thing, particularly when installing from sdist as wheels have version numbers and the like to handle compatible/incompatible changes to that spec and there isn't really much place for logic to happen that people might depend on something pip is providing (or not). However, with sdists there is a lot of feature detection that is going to be happening (does pip support pyproject.toml or not? What about a future abstract build system? what about other, future things we're not even currently thinking of?). We could just shrug our shoulders and ignore it for now and say that someday we'll make a sdist 2.0 that also bakes in a version number like wheel does. That is a good solution but will require a fair amount of effort to get done, but as a stop gap we could do something like using the new pyproject.toml file and add our own section to it. Something like:

[tool.pip]
minimum-version = "10"

With this pip could look and if this is declared, it can fail out saying that the project requires pip X.Y in order to build. This of course won't help all of the old versions of pip, but going forward it could be a useful escape hatch for people.

Thoughts @pypa/pip-committers ?

[xavfernandez]

If pip is looking at a pyproject.toml file, it means it will have to build the package, ending up with some wheel XXX.whl.

To build the wheel, I don't see how pip should be involved outside of setting up the proper build environment so I don't see why a specific version of pip would be required...

The only reason I see would be that the pyproject.toml file has gained some new meaning, but then it should maybe just have its semantics-version added ?

If the built wheel is incompatible with older version of pip, pip already has a mecanism to recognize if it is able to properly install a wheel via its wheel version (cf https://github.com/pypa/pip/blob/b559221/pip/wheel.py#L550-L576).

[pfmoore]

I'm with @xavfernandez here. I can't see why a project would need to require a given version of pip. For wheels, the wheel version is enough. For sdists, there's either pyproject.toml versioning, or if the project still uses setup.py, then any version of pip will do.

So I'd claim YAGNI for this.

[dstufft]

pyproject.toml does not support versioning itself currently and even if we did I don't think it's directly applicable for this use case. For instance, at some point we're going to get support for build dependencies via pyproject.toml, that's great but how do developers present a good user experience to people who are still using an old version of pip if they want to use them? For that matter, once we add the abstract build system we're back in the same scenario again, old versions of pip won't know how to handle the new input correctly and authors are left with no real recourse except either don't use the new feature or put up with a pretty crummy error message.

[pfmoore]

Maybe I missed something. I thought the pyproject.toml specification included a version number? If so, I'd expect pip to either be able to cope with what's in the file, or fail with an error along the lines of "Cannot handle pyproject.toml format version X, you need a newer version of pip". Which I think is perfectly fine as an error.

Project authors shouldn't need to be responsible for tracking which precise version of pip supports the features they use. As long as they are using a defined version of the file format, they should be fine.

Of course, unless I'm mistaken, the pyproject.toml format isn't implemented/supported in pip yet, so why don't we wait till it is, and see if there's an actual problem to be solved here, before rushing to solve it? (Or to put it another way, still seems like YAGNI to me).

[xavfernandez]

My main point is that I don't see why we should have a pip specific semantic-version which is basically what you are suggesting with:

[tool.pip]
minimum-version = "10"

Either we declare it useless (as it was chosen to do in https://www.python.org/dev/peps/pep-0518/#a-semantic-version-key , with which I don't agree ^^) or we bite the bullet and declare a full-fledged semantic-version.

I'd hate it to see other tool than pip use the tool.pip.minimum-version to check if they are able to deal with it.

[dstufft]

The version number in pyproject.toml was explicitly excluded, but even if it hadn't been it wouldn't have been useful for this (or anything IMO). That version number was only there so that we could theoretically break compatibility, not add a new feature in a compatible way. So, for example, the abstract build system would not have increased the version number at all. If we had wanted to sue it for that it would have needed to become more complicated by instead of making it an integer, making it a decimal and bump the second digit on every change to pyproject.toml... but I think that is a pretty crummy experience because it's a mandatory field then. You can't add in something that is only a progressive enchancement (for example, the build dependencies, you can add them to pyproject.toml AND to setup_requires and have things work with old and the new).

Beyond that though, this key isn't just useful for pyproject.toml. Examples in the past when it would have been useful:

• Projects wanting to use environment markers in a sdist that wanted to error out on pip < 6 (which did not support them).
• Projects wanting to use the new Requires-Python support, who did not want to fail open, but rather wanted to fail closed.
• Projects which do not want people to automatically fall back to the sdist when a wheel is available, because building the project is non trivial and unlikely to work by default, but either their pip is too old for wheel, or their pip is too old to support the kind of wheel they need (manylinux1, etc).

And those are just off the top of my head. That gives two key benefits to this over simply versioning the pyproject.toml:

• The desire for this isn't specific to pyproject.toml, it just happens to be the next thing that i can see it being useful on.
• This is opt-in rather than mandatory, it lets authors decide when they would rather use some new feature in a progressively enhanced way or when they would rather hard fail when pip doesn't support it.

[pfmoore]

Hmm, OK. Thanks for the examples. I'm still not clear, though.

Pip's job here is to do the install, which means unpacking and installing a wheel. We also orchestrate the building of wheels, if there is only a sdist available, but we do that currently by calling out to setuptools, and in future by calling out to an arbitrary build tool. So surely it's the build tool that has to be the right version to support these features, not pip? In the case of environment markers or requires-python, I can see the project wanting to ensure that the build tool has support for those features - but why would they care what version of pip is involved?

The only thing I can think of is if we're thinking of wheels having metadata that older versions of pip don't respect. That seems to me to be a problem with either pip or the metadata spec - if "ignore unknown metadata" isn't appropriate for a given piece of metadata, then adding that metadata should have required a metadata version bump (so that installers such as pip can say "don't know what to do with this version"). Of course, historically we may not have done that, but going forward maybe we should? I'd see that as a better solution than allowing projects to say "you can't use pip version X for my project" (after all, users will still be able to use pip 9 or earlier, because those versions won't respect the "minimum pip version" metadata!)

As for "don't fall back to sdist if there's no wheel", isn't that what --only-binary is for? Are you saying you want projects to be able to force that choice on the user? I'm not sure I agree with that.

I'd really rather see an example of a project saying "we have a problem with people using old versions of pip to install our code, and it's causing this specific issue" before we try to design a general solution that may get over-used.

[dstufft]

@pfmoore In the case of all of the examples listed, the problem was in pip not in setuptools, furthermore, once we have build requirements setup via pyproject.toml in pip, then people can pretty easily assert the version of the build tool they require and have it handle that just fine.

They care about what version of pip is involved, because if pip doesn't support them then their project is either going to be silently broken or get bad behavior.

In the case of the environment markers for instance, pip was not interpreting environment markers from a sdist (even though setuptools had them and was writing them) until pip 6.0. So projects who want to support pip < 6 need to replicate the environment markers using python logic inside of their setup.py. There is an open issue about this already at #3111, where we were asked to export a PIP_VERSION environment variable so that projects could do different things in their setup.py based upon pip version. I was against that because I didn't want projects to be able to do conditional logic based upon the pip version that is installing them. However, this proposal solves the same problem without exposing us to people doing conditional logic in their setup.py based upon pip version.

I opened this issue right now because another person had reached out asking how they can tell what version of pip was being used to install their project, because their project is extremely complex to build (moreso than numpy) and so they publish wheels for all of the various platforms. However people with old pips fall back to downloading the sdist by default and then get a very confusing error and start coming to them asking for help, when they would be fine if they simply upgraded their version of pip.

As I was thinking about this feature, it also occurred to me that if we added it, then projects could use it when we add other features in the future that they depend on if they want to hard fail instead of trying to maintain a compatibility shim. Such as with the abstract build system.

[pfmoore]

OK. Thanks for the explanation.

But am I right in thinking that a specification of "requires pip 12+" (for example) would actually mean "requires pip 12+, or pip 9 or earlier", because pip 9 will ignore this new metadata? So while this may address issues like this, it won't be a complete solution (not that anything could be) until pip 9 is "too old to be worth worrying about"?

I still don't really like this option, as it seems like it's too blunt an instrument, but if no-one's offering more nuanced solutions, I'm not going to block this as a "better than nothing" option.

However, I would like to consider the option of making the message a warning, rather than an error. A warning along the lines of "This project is designed to be installed with pip version X or later. You are using an older version - if you get errors you should upgrade pip" would have much the same effect, but without blocking out cases that would work (maybe the user has just put a lot of effort into setting up the right build environment, because he can't upgrade pip for whatever reason).

[dstufft]

@pfmoore Yes that is correct, it would effectively be >=12 OR <9.1 because this hypothetical feature did not exist prior to that. So this feature itself is not entirely useful until 9.1 becomes the bare minimum. Unfortunately we can't go back in time to add it, so it's kind of unavoidable.

It's a bit of a chicken and egg scenario of course, where the longer we wait to add it, the window of < 9.1 just keeps moving up for each release we do. Of course we can decide that that gap means the solution isn't "good enough" to pass muster because there's always going to be some older pips that don't support it.

Making it a warning, either just by always making it a warning or by making it an error with a flag to downgrade it to a warning seems fine to me.

To be clear, I don't know exactly how I feel about it. I think it could be a useful option to have, but I also don't want to be in a palce where every project is just sort of expected to have some sort of minimum pip version chucked in a file alongside things.

[xavfernandez]

I thought one of the point of PEP 518 was to help differentiate the roles of the installer (pip) and the builder. I like this separation of concerns.

But I guess we are not there yet and pip's version is still important as of the capabilities of the installer: does it undestand such metadata ? will it correctly resolve dependencies ? does it understand pure pyproject.toml (without setup.py like in PEP-517) ?

I guess what is bothering me with this is that it will encourage project maintainers to specify their installer requirements via a pip minimum version.
This seems to be the contrary to what we are aiming, i.e. moving away from implementation specific behaviors.

[dstufft]

@xavfernandez There is still a separation of concerns, but no matter what people are going to be dependent on which feature the installer that their end users are using have implemented. We've successfully documented and standardized things as we've gone alone which makes it so that people can write their own installer if they so wish and it can slot in just as pip does now.

If there becomes a second popular installer, I could imagine it getting a similar option itself. Importantly, this is not something that is getting added to the standard metadata and it does not influence anything about how the actual installation happens. It merely states whether or not pip is capable of using the new, standard features that this project relies on.

[xavfernandez]

@dstufft I think you've convinced me of the validity of the need, thanks for the clarification/explanation 👍

[pradyunsg]

@pfmoore Is there still a concern regarding this?

@pypa/pip-committers Do we want to go ahead with this?

[pfmoore]

I don't object to it

[pfmoore]

I'm assuming this will not happen in pip 10, so we'll move it to the "10.1" milestone once that is created.

[pradyunsg]

Pushing this to the next release since I don't think we'll be ready for this by pip 18.1.

[hugovk]

An example use case for this:

• pip 19.2 or later is needed to install Pillow wheels on Python 3.8, and falling back to sdist compilation on Windows rarely works

• Python 3.8 Pillow binaries not found python-pillow/Pillow#4204

[chrahunt]

@hugovk, are you suggesting that there be an option for matching Python version as well, or is always displaying a warning that the user may need to upgrade sufficient?

[hugovk]

I think it's fine to always display it, that we could simply require (or warn about) pip >= 19.2 for all our supported Pythons (3.5-3.8). It's good for people to keep pip up-to-date.

[chrahunt]

After thinking about it more, I'm not sure there's anything here that couldn't be handled by a build backend, assuming we're talking about sdists. To me that would be a good enough reason not to implement it in pip.

First, this wouldn't need to be implemented in every backend since a backend wrapper can itself support a tools.wrapper.build-backend key pointing to the "real" backend to be invoked.

Then, if someone did want to explicitly warn or fail based on pip version, the wrapper could recurse up the process hierarchy until it found pip, invoke the equivalent of pip --version, and then do a comparison with tools.wrapper.minimum-pip-version, and fail or trace a warning before invoking tools.wrapper.build-backend. Alternatively I think pip exposes itself in the build environment for installs, so a {sys.executable} -m pip --version might do.

This handles several concerns expressed above:

1. "only pips after this is implemented will understand the option and provide a warning" - as a build backend it would work with any PEP 517 capable pip
2. YAGNI - if someone needs it they can make it
3. fail vs warn - leave it to the backend implementer to decide and if someone disagrees they can make their own backend

[pradyunsg]

YAGNI - if someone needs it they can make it

I'm gonna apply this logic and take the liberty of closing this issue.