pip wheels (from ensurepip) show nonexistant entry_points in importlib.metadata

[puetzk]

Description

I was working on some code (a conan generator to write cmake add_executable(... IMPORTED) bindings based on importlib.metadata.entry_points), and noticed something odd - there was an entry point for pip3.8 showing up that didn't actually have a corresponding launcher pip3.8.exe (since I was/am stuck using python 3.7 to avoid #8649 until a cpython release picks up pip 21.2, the actual launcher was pip3.7.exe).

Expected behavior

importlib.metadata.Distribution.entry_points will correspond to the entry_point launchers that actually exist.

pip version

20.2.3-21.2.4

Python version

3.9.4

OS

Windows

How to Reproduce

from importlib.metadata import distribution
distribution("pip").entry_points

note the presence of EntryPoint(name='pip3.8', value='pip._internal.cli.main:main', group='console_scripts'), even though this is python 3.9, not 3.8.

where pip3.8
INFO: Could not find files for the given pattern(s).
>where pip3.9
C:\Program Files\Python39\Scripts\pip3.9.exe

Output

[EntryPoint(name='pip', value='pip._internal.cli.main:main', group='console_scripts'), EntryPoint(name='pip3', value='pip._internal.cli.main:main', group='console_scripts'), EntryPoint(name='pip3.8', value='pip._internal.cli.main:main', group='console_scripts')]

entry_points.txt

[console_scripts]
pip = pip._internal.cli.main:main
pip3 = pip._internal.cli.main:main
pip3.8 = pip._internal.cli.main:main

The wheel for 21.2.4 still seems to show the same sort of thing going on: https://files.pythonhosted.org/packages/ca/31/b88ef447d595963c01060998cb329251648acf4a067721b0452c45527eb8/pip-21.2.4-py3-none-any.whl (though there it's pip3.9 that is present)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[puetzk]

The problem is of course that setup.py is choosing its list of console_scripts dynamically based on the python version in which it is run:

pip/setup.py

Lines 70 to 73 in a974526

	"console_scripts": [
	"pip=pip._internal.cli.main:main",
	"pip{}=pip._internal.cli.main:main".format(sys.version_info[0]),
	"pip{}.{}=pip._internal.cli.main:main".format(*sys.version_info[:2]),

So the scripts created at install time and the metadata recorded in the wheel (pip-*.dist-info\entry_points.txt) do not necessarily agree.

I'm not sure what the best resolution is (create a new entry_points.txt when actually doing the install/creating the entry_point launchers? somehow skip the version specific pip3x entry_points when building a "generic" wheel, i.e. just "pip" and "pip3" for a .py3. wheel, and just "pip" for a .py2.py3 wheel (which isn't done anymore, so whatever)?

I don't know enough about how the tags for the wheel get chosen (I assume it's related to the classifiers") to know how setup.py could know which ones to mention when creating a wheel...

Or maybe the wrong metadata is not that important, and this can just be a WONTFIX. But it seemed weird enough to be worth mentioning.

[uranusjr]

Pip has special treatment to itself and actually does not respect the listed entry points, but creates pipX and pipX.Y on its own depending on what version of Python it is installed under, which is of course only possible at runtime and not listable in a wheel.

IIRC there was talk we should probably remove the confusing hard-coded pipX and pipX.Y entry points from the wheel, but it’s not trivial to do since we don’t want to loose them if the user builds pip from source, which Linux distributions do to get the right executables, and none of the maintainers think this is that big an issue anyway, so nothing came of that discussion. If this bothers you, feel free to work on a solution; a PR would certainly be welcomed, at least by me.

[puetzk]

it’s not trivial to do since we don’t want to loose them if the user builds pip from source

I don't quite understand what you mean, since pip install . these days builds a (temporary) wheel too, no? Or do you mean the classic python setup.py install?

If the latter, one could just do a hack like

    entry_points={
        "console_scripts": [
            "pip=pip._internal.cli.main:main",
        ] + ([] if 'bdist_wheel' in sys.argv else [
            "pip{}=pip._internal.cli.main:main".format(sys.version_info[0]),
            "pip{}.{}=pip._internal.cli.main:main".format(*sys.version_info[:2]),
        ])
    },

Admittedly not a thing of beauty, but it does install all 3 for python setup.py install but only pip gets put in the wheel. Is that the kind of thing you were saying would be welcomed?

[uranusjr]

it’s not trivial to do since we don’t want to loose them if the user builds pip from source

I don't quite understand what you mean, since pip install . these days builds a (temporary) wheel too, no?

Yes it does. What I meant was if we’re going to remove anything, it’d be from the wheels we build for distribution (e.g. to PyPI). But in the mean time, if the user builds from source (e.g. pip install .), the temporary wheel should still contain those versioned entry points. Which is the non-trivial part.

I think one way to do it is instead of producing version-independent wheels (none-any), pip needs to produce version-sensitive wheels by default (e.g. none-py38) that contain versioned entry points. This means wheels built on-demand will continue to work like they do now, and the versioned wheel tag ensures pipX.Y is only ever installed on pyXY. But when we’re doing distribution (e.g. running nox -s build-release), we’d pass in a special flag to build a version-indepenent wheel instead to upload to PyPI and bundle in CPython.

If this sounds complex, it indeed is. And I’m not sure if it’s even possible without either hacks in setup.py (which everyone hates so we should avoid) or a home-brewed PEP 517 backend (which a lot of people will hate because they insist on using vanilla setuptools). So yeah, it’s not a good or easy situation.

[pfmoore]

I think if we want to solve this issue "properly", we have to switch to just having a pip entry point. I'm not sure why the special-casing of pip in get_console_script_specs (from pip._internal.operations.install.wheel) isn't enough to then ensure that the right executables get written. I guess it's because people are building a pip wheel and then using a mechanism other than pip to install it. In that case, they really need to replicate the special casing from pip if they want the versioned executables. But that is apparently not happening at the moment (and the extra entry points are there to allow them to ignore the issue) - so this would be an incompatible change, and need publicising and a suitable transition period, but that's fine, we have a process for that.

But I don't want to pile more hacks on to cater for this - the existing approach is a hack that delivers something that (as I understand it) our existing downstream consumers can work with. If new consumers, like the OP, cannot work around the existing hack, we can remove it (and inform our downstream that they will now have to handle this themselves) or retain it (and inform the new consumers that it's needed and they'll have to deal with it). Making ever more complex hacks each time someone new comes along with a use case simply isn't sustainable IMO.

But as @uranusjr says, none of the pip maintainers think this is a particularly important issue, so it'll be up to someone else to make a PR for this, ensure that any solution is acceptable both to pip and to downstream, etc.

[pradyunsg]

Wait, we are generating the correct scripts when installing. OP's concern is that those can mismatch with what's in entry_points.txt.

[pfmoore]

Yes, that's because we generate versioned values in entry_points.txt which are not needed when we generate the scripts, but (as I understand it from @uranusjr's comment) are used by Linux distributions, who build the wheels, but don't use pip to install them, but rather extract the wheel using their own process (which presumably relies on the entry points).

I'm saying that to be correct, our entry point should just be pip (because entry points cann't be dependent on the installation environment), but if we did that those Linux distributions would have to make new arrangements to generate the right scripts.

[puetzk]

(as I understand it from @uranusjr's comment) are used by Linux distributions, who build the wheels, but don't use pip to install them, but rather extract the wheel using their own process (which presumably relies on the entry points).

If that's true, it sure looks like they'd already break frequently already. Sampling the assortment of versions I happen to have handy, we appear to be 1/5

• CPython 3.7.9 (win32) ensurepip bundled pip-20.1.1-py2.py3-none-any, which has a pip3.8 entry_point (not pip3.7)
• CPython 3.9.4 (win32) ensurepip bundled pip-20.1.1-py2.py3-none-any, which has a pip3.8 entry_point (not pip3.9)
• CPython 3.8.5 (Ubuntu 20.04) bundled pip-20.0.2-py2.py3-none-any.whl, which has a pip3.8 entry_point (this one matches!)
• CPython 3.10 branch and master (3.11?) currently have pip-21.2.3-py3-none-any.whl, which has a pip3.8 entry_point (not pip3.10 or pip3.11)

It appears @uranusjr updated to 3.9 some time in the last few days, because the very latest pip-21.2.4-py3-none-any.whl on https://pypi.org/project/pip/21.2.4/ now has pip3.9 (so when/if 21.2.4 gets into the 3.9 branch, maybe 3.9.7 will match up). But in 3.10 it won't.

Now, it seems that ensurepip uses pip to install itself (by adding pip's wheel to sys.path, in order to invoke the not-yet-installed version to install itself). So that ought to (and seems to) get the special-casing and produce the right scripts. But if someone else is following the entry_point.txt, they'll get some kind of versioned scripts, but seemingly more often than not the wrong ones (even when using "matching" cpython and pip versions).

That said, this issues's not that critical to me either. I just stumbled across it because I was converting this cmake-import generator from pkg_resources to importlib.metadata (preparing for the day coming soon when we'll be able to upgrade our CI past python >= 3.8). I normally only use this generator to import a couple packages (pip not among them), but I opened the floodgates to process everything I had installed, just to see if anything interesting broke. And then I stumbled on this stray pip3.8 entry_point where it couldn't find the matching script. But it wouldn't have actually hurt my "real" usage (which is importing things like sphinx that I want to call from CMake targets). And I didn't find any similar issue ever discussed. Sometimes those loose threads unravel larger mysteries. But if the answer is "the maintainers have thought about it, it's a hack, but it's the best compromise we can come up with", then fine, feel free to just close the issue. I don't have any immediate ideas, or any idea where to even start looking for for folks installing pip's wheel by unusual means.

--

What triggered me to even be messing with all this was that pip 21.2 is now released (with packaging.tags 21.0), and both are merged to the python 3.10 branch, bpo-38989 is solved, which means there will soon be a python > 3.7.9 that again works if called in an MSVC cross-compiling environment again. Hooray 🎈 🎉 🍰!

[pradyunsg]

Cool. Given that everyone on this thread doesn't think this is important or worth fixing, let's close this?

[uranusjr]

If that's true, it sure looks like they'd already break frequently already. Sampling the assortment of versions I happen to have handy, we appear to be 1/5

From my understanding, Linux distributions don’t use ensurepip to install pip, but build pip from source. This is usually for policy reasons (most of them can’t distribute binary packages, and wheels are categorised as such), but as a result they are never affected by those entry points.

It appears @uranusjr updated to 3.9 some time in the last few days, because the very latest pip-21.2.4-py3-none-any.whl on pypi.org/project/pip/21.2.4 now has pip3.9 (so when/if 21.2.4 gets into the 3.9 branch, maybe 3.9.7 will match up). But in 3.10 it won't.

Actually I was just building on a different computer 🙂

[pfmoore]

but as a result they are never affected by those entry points.

I've already given a "thumbs up" to @pradyunsg's comment suggesting we just close this, but I'm curious. If that's the case, why would it make any difference to anyone if we just removed the versioned entry points altogether from pip's setup.py?

[uranusjr]

Because when the distributors build pip on their own (either by wheel or setup.py install, they still rely on the entry pointe declared in setup.py. The difference is, since they build pip against the same Python they are going to put in the distribution, the version suffixes from setup.py are guaranteed to match what would’ve been generated in code.

[pfmoore]

OK, cool. If anyone cared enough, I guess we could change setup.py so the extra entry points are not generated if an environment variable is set, and update our automation to set that variable when building. (It would be nicer to have the entry points only generated when the variable was set, not the other way round, but that's backward incompatible). I'm still not sure it's worth the effort, though.

[uranusjr]

Yeah, I’m definitely not going to work on it myself, but also not going to stop anyone if they feel strong enough about this.

[puetzk]

I guess we could change setup.py so the extra entry points are not generated if an environment variable is set, and update our automation to set that variable when building.

Actually, with one more tweak that sounds like it could be a pretty clean answer. The tweak would be to also tie the [bdist_wheel]python_tag to the same option/envvar/whatever, so that a default build from source includes the versioned entry-points, but also produced a -py39-* (or whatever) version-dependent wheel (if you're building from source such that it matches the python you want to use it with, this won't be any problem). Or in your release automation, set something such that it omits the versioned entry-points and thus produces a universal -py3-none-any wheel.

I'm just not sure how to control [bdist_wheel]python_tag dynamically when running pip wheel. One can put it into setup.cfg

[bdist_wheel]
python_tag = py39

and this works, but I haven't figured out how I can substitute a dynamic value like sysconfig.get_config_var("py_version_nodot"). I think this sort of thing is supposed to be possible using the attr: syntax, but somehow I'm not getting it right. Maybe I can figure it out and propose something reasonably tidy.

[pfmoore]

I'm -1 on setting the python tag. There's nothing about the resulting wheel that's specific to the Python version - an entry point of pip3.9 is completely valid in a wheel built and used on Python 3.8 - it may be a confusing name to use, but it's entirely legitimate. I'd consider setting the Python tag to be a misuse of the tag system, TBH.

[pradyunsg]

Is there an actual user-facing problem, originating from the mismatched entry points declaration?

[puetzk]

an entry point of pip3.9 is completely valid in a wheel built and used on Python 3.8 - it may be a confusing name to use, but it's entirely legitimate.

Well, half-legitimate - it would be legitimate to create, but it doesn't actually get created because pip special-cases itself. So it depends on whether you're using the metadata to know what to create (it would be fine, if odd) or if you're using it to know what to look for/copy (it refers to a file that isn't there).

Is there an actual user-facing problem, originating from the mismatched entry points declaration?

my (custom) conan generator to enumerate the package entry_points for python tools (things like sphinx) which it has installed in a venv, and write corresponding CMake add_executable(IMPORTED) targets that can be used to call easily them during the build or tests failed. This is just because it threw a FileNotFound exception while trying to get an absolute path for the corresponding executable (because there wasn't one for the wrongly-versioned name).

I have already fixed said generator to just catch that exception, print a warning about the nonexistant script, and continue with any entry_points that do exist. So... yes, but not a very serious one, and it's worked-around now.

[chipx86]

I know this is an old issue, but after a long investigation as to why my pip3.8 symlinks on some older build and dev environments and my pip3.10 symlinks on some newer ones kept getting deleted, I finally worked my way through the code and then eventually to this issue.

The inclusion of the pipX.Y entrypoint can indeed cause real-world issues for users, any time pip is upgraded in an environment where pipX.Y binaries share the same directory. I've encountered this on some special dev environments we have set up that work like/wrap around virtualenv and provide a unified multi-version Python environment, as well as on our CI Docker images that provide multiple versions of Python, and on macOS with Homebrew.

I'll demonstrate the bug using pip 22.3.1 (which has pip3.10 in entrypoints.txt), and both Python 3.9 and Python 3.10 installed (any non-3.10 version of Python will reproduce this with 22.3.1). I'll use homebrew for macOS for this particular test, but I've reproduced it in Linux Docker containers that ship multiple versions of Python:

$ which pip3.10
/opt/homebrew/bin/pip3.10
$ which pip3.9
/opt/homebrew/bin/pip3.9
$ pip3.9 install -U --no-cache --force-reinstall pip
Collecting pip
  Downloading pip-22.3.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 24.2 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.3.1
    Uninstalling pip-22.3.1:
      Successfully uninstalled pip-22.3.1
Successfully installed pip-22.3.1
$ ls -l /opt/homebrew/bin/pip3.10
ls: /opt/homebrew/bin/pip3.10: No such file or directory

This is happening in the uninstallation phase (which happens on upgrade as well). We can verify that with:

$ pip3.11 uninstall pip
Found existing installation: pip 22.3.1
Uninstalling pip-22.3.1:
  Would remove:
    /tmp/delme/bin/pip
    /tmp/delme/bin/pip3
    /tmp/delme/bin/pip3.10    # <----
    /tmp/delme/bin/pip3.11
    /tmp/delme/lib/python3.11/site-packages/pip-22.3.1.dist-info/*
    /tmp/delme/lib/python3.11/site-packages/pip/*
Proceed (Y/n)?

This seems to me like a pretty big unintentional consequence of that extra entry.

If it were just our CI setup, fine, but it's a problem on system installs as well. One can argue that virtualenv should always be used and you should never pip upgrade a system install, but regardless, this behavior is probably not ideal.

[pfmoore]

Ultimately, the solution to this is #3164 (deprecate the versioned entry points).

[pradyunsg]

It looks like we can just safely remove the version-specific console_scripts of pip today FWIW. The entry point that triggers the special case handling is pip.

pip/src/pip/_internal/operations/install/wheel.py

Lines 316 to 346 in 90f51db

	pip_script = console.pop("pip", None)
	if pip_script:
	if "ENSUREPIP_OPTIONS" not in os.environ:
	scripts_to_generate.append("pip = " + pip_script)
	if os.environ.get("ENSUREPIP_OPTIONS", "") != "altinstall":
	scripts_to_generate.append(
	"pip{} = {}".format(sys.version_info[0], pip_script)
	)
	scripts_to_generate.append(f"pip{get_major_minor_version()} = {pip_script}")
	# Delete any other versioned pip entry points
	pip_ep = [k for k in console if re.match(r"pip(\d+(\.\d+)?)?$", k)]
	for k in pip_ep:
	del console[k]
	easy_install_script = console.pop("easy_install", None)
	if easy_install_script:
	if "ENSUREPIP_OPTIONS" not in os.environ:
	scripts_to_generate.append("easy_install = " + easy_install_script)
	scripts_to_generate.append(
	"easy_install-{} = {}".format(
	get_major_minor_version(), easy_install_script
	)
	)
	# Delete any other versioned easy_install entry points
	easy_install_ep = [
	k for k in console if re.match(r"easy_install(-\d+\.\d+)?$", k)
	]
	for k in easy_install_ep:
	del console[k]

[pradyunsg]

    /tmp/delme/bin/pip3.10    # <----

Yea, that's a bug due to the del calls above only filtering out \d(\.\d)? which doesn't match pip3.10.

[uranusjr]

I wonder if we should just get rid of them all; we’ve always wanted to delete them anyway, are there reasons we can’t change the detection to “we are installing a package called pip” instead?

[chipx86]

    /tmp/delme/bin/pip3.10    # <----

Yea, that's a bug due to the del calls above only filtering out \d(\.\d)? which doesn't match pip3.10.

I'm running the version with that patch, and have even hard-coded removing it there. It doesn't seem to be at that stage (and not limited to pipX.Y).

[chipx86]

The code calculating the files to remove is pip/_internal/req/req_uninstall.py in UninstallPathSet.from_dist.iter_scripts_to_remove(). This just iterates through entrypoints on the distribution and schedules them for removal. I can print that result and see pip3.10 show up. UninstallPathSet.remove() is handling the actual deletion of the files. None of these have pip-specific logic in them. They just trust the contents of entry_points.txt.