Fix the install scheme to posix_prefix when --prefix is passed

[frostming]

What's the problem this feature will solve?

On Debian systems, due to the sysconfig patch, sysconfig.get_preferred_scheme('prefix') will return posix_local, which is also the default scheme. This causes problem when using --prefix <path> with pip install, where packages are installed to <prefix_path>/local/lib/pythonX.Y/dist-packages. This is apparently wrong.

Describe the solution you'd like

Add a special case to pip._internal.locations._sysconfig:get_scheme(), when posix_local is found in INSTALL_SCHEMES and prefix is not None, rewrite the scheme as posix_prefix.

Alternative Solutions

I am not sure if it is better to report it to Debian, but IMO it makes more sense to fix it in pip to be more robust.

Additional context

• https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/patches/sysconfig-debian-schemes.diff
• python3.10+: global pip install layout does not match debian's due to sysconfig changes deadsnakes/issues#182
• Debian bug report

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

What is the behaviour of the distutils backend?

[pradyunsg]

I am not sure if it is better to report it to Debian, but IMO it makes more sense to fix it in pip to be more robust.

If this behaviour originates due to a Debian patch, it is usually a good idea to file an issue with them. Please file a bug with Debian, with reportbug python3-pip (their reporting documentation). You can link to this issue in your bug report.

[frostming]

What is the behaviour of the distutils backend?

They are different:

>>> from pip._internal.locations import _distutils, _sysconfig
>>> _sysconfig.get_scheme('click', prefix='myprefix').purelib
'myprefix/local/lib/python3.10/dist-packages'
>>> _distutils.get_scheme('click', prefix='myprefix').purelib
'myprefix/lib/python3.10/site-packages'

If this behaviour originates due to a Debian patch, it is usually a good idea to file an issue with them. Please file a bug with Debian, with reportbug python3-pip (their reporting documentation). You can link to this issue in your bug report.

Thanks for the guidance but I am not able to do so now(some problem with reportbug).

[uranusjr]

Hmm yeah at least they should match. I don’t think Debian can handle this though; the --prefix value is not passed to sysconfig so they have no way to magically choose a scheme based on it. We’ll likely need to somehow handle this in pip, or raise an issue to CPython to further enhance the sysconfig API.

[speth]

I'm seeing a similar problem on Fedora 37 (Rawhide) while trying to package a Python module (see Cantera/cantera#1233), using both the Fedora-packaged version of pip or pip installed from PyPI, so I don't think that this is only a Debian problem. In both cases this is using pip 22.0.4. There are some differences, though, compared to the behavior reported on Debian -- I see the local directory being incorrectly inserted through both routes:

>>> from pip._internal.locations import _distutils, _sysconfig
>>> _sysconfig.get_scheme('click', prefix='myprefix').purelib
'myprefix/local/lib/python3.10/site-packages'
>>> _distutils.get_scheme('click', prefix='myprefix').purelib
'myprefix/local/lib/python3.10/site-packages'

[uranusjr]

If it's injected for both implementations, this is not a bug but an intended behaviour. You need to check with Red Hat to understand their intention behind this and resolve the issue there, pip cannot change it here.

[henryptung]

Not to disrupt the conversation, but isn't the prefix install scheme fairly-explicitly specified in https://docs.python.org/3/install/index.html#alternate-installation-unix-the-prefix-scheme? Is this behavior changing, and is that documented somewhere?

[uranusjr]

It’s not the install prefix that’s the problem, but Debian changes how it uses the install prefix, and we need their confirmation why they changed one place but not the other (and if one of them is wrong, which is the right one).

[adamjstewart]

We also ran into this same issue in Spack where we use pip to install packages to unique installation prefixes. Thanks @trws for finding this bug report.

[pradyunsg]

FWIW, Fedora also has a similarly breaking patch (https://github.com/fedora-python/cpython/blob/f52f499bb9ac03d7793fa2e1ccaaab8a3169685e/Lib/sysconfig.py#L77) and they directly modify posix_prefix.

I suggest we move further discussion to https://discuss.python.org/t/18240, mostly because that's a better forum for the sort of cross-project discussion that this is gonna need.

[simonvanderveldt]

This causes problem when using --prefix <path> with pip install, where packages are installed to <prefix_path>/local/lib/pythonX.Y/dist-packages. This is apparently wrong.

Not sure if I'm misunderstanding the issue description but for me it seems to have a different/the opposite behavior with regards to site/dist-packages.

Installing a Python package using pip3 in Debian without --prefix installs the package in /usr/local/lib/python3.7/dist-packages (so in dist-packages in local) whereas when using --prefix it gets installed in <prefix>/lib/python3.7/site-packages (so in site-packages not in local). So for some reason they are not installed to dist-packages when using --prefix.
This is causing some issues for me. I'm not sure what the intended behavior is?

This was tested on Debian Buster as well as Debian Bullseye which has

root@b95b3bc69071:/# pip --version
pip 20.3.4 from /usr/lib/python3/dist-packages/pip (python 3.9)

[martinpitt]

The corresponding Debian bug report is https://bugs.debian.org/1035546 . Perhaps you can edit the description to make it easier to find? Thanks!