Prefer included distutils even without importing setuptools. Closes #2259.

[jaraco]

Summary of changes

In #2259, Paul outlines the problem - users that import distutils before importing setuptools, a common if not recommended practice, will receive a warning and will end up with objects from different implementations of distutils.

When enabled, this change installs a .pth file (except for editable installs of setuptools) to the user's environment to ensure that the setuptools-provided distutils is preferred before setuptools has been imported (and even if setuptools is never imported).

This feature relies on the same SETUPTOOLS_USE_DISTUTILS flag for opt-in now and opt-out in the future (set to "local" for opt-in and "stdlib" for opt-out).

This patch was originally submitted as #2260 but backs out the refactoring that left the implementation in an unsuitable state.

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d. See documentation for details

[pganssle]

Per my comments in #2260, I strongly disagree with this course of action. I don't think we should move forward with this change without testing, and even if we did, I think we should at least clean up the history so it's easier to revert specific subsets of the changes if necessary.

[jaraco]

I very much respect your opinion @pganssle, so I'm going to take another stab at writing a suite of tests for the distutils expectation.

[jaraco]

In #2315, I've written three tests that capture the user-facing behavior, including xfailing on the limitation identified in #2259. I expect that test to start xpassing with this change.

[jaraco]

I observe that the tests are failing on Windows, even though I thought they were passing in the previous PR. The test failures are in test_virtualenv.test_pip_upgrade_from_source. It's possible I missed this failure before because I mis-recognized the failure in appveyor as a "fail fast" (where it stops running and marks as fail if a newer version is pushed). Regardless, we'll need to investigate that issue too.

[jaraco]

The error occurs when it tries to run sdist and fails with this error:

error: could not create 'build\bdist.win-amd64\wheel\.\import os; enabled = os.environ.get('SETUPTOOLS_USE_DISTUTILS') == 'local'; enabled and __import__('_distutils_hack').add_shim(); \easy_install.py': No such file or directory

It seems the _restore_install_lib method isn't working properly under Windows.

[jaraco]

Here are the values of install_lib and install_libbase during the execution:

{'install_lib': 'build\\bdist.win-amd64\\wheel\\.\\import os; enabled = '
                "os.environ.get('SETUPTOOLS_USE_DISTUTILS') == 'local'; "
                "enabled and __import__('_distutils_hack').add_shim(); ",
 'install_libbase': 'build\\bdist.win-amd64\\wheel\\.'}

[jaraco]

Weird - I notice that suffix has no space at the end:

'suffix': "import os; enabled = os.environ.get('SETUPTOOLS_USE_DISTUTILS') == 'local'; enabled and __import__('_distutils_hack').add_shim();"

But when I run it locally, I get a different result:

>>> install_lib = "build\\bdist.win-amd64\\wheel\\.\\import os; enabled = os.environ.get('SETUPTOOLS_USE_DISTUTILS') == 'local'; enabled and __import__('_distutils_hack').add_shim(); "
>>> install_libbase = 'build\\bdist.win-amd64\\wheel\\.'
>>> ntpath.relpath(install_lib, install_libbase)
"import os; enabled = os.environ.get('SETUPTOOLS_USE_DISTUTILS') == 'local'; enabled and __import__('_distutils_hack').add_shim(); "

[jaraco]

Well, that's annoying. The pypy3 tests on macOS only are failing the new test. I guess that's expected, since we know pypy imports stdlib distutils on startup.

[pganssle]

Hmm, is that because it imports distutils before the pth file is executed?

[jaraco]

Hmm, is that because it imports distutils before the pth file is executed?

That's right.

$ pypy3 -V
Python 3.6.9 (?, Apr 18 2020, 02:46:07)
[PyPy 7.3.1 with GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.32.59)]
$ pypy3 -S -c "import sys; print('distutils' in sys.modules)"
False
$ pypy3 -c "import sys; print('distutils' in sys.modules)"
True

If I recall correctly (low confidence), pypy for Python 3.7 no longer has this issue.

But here's something weird:

$ pypy3 -S -c "import site, sys; print('distutils' in sys.modules)"
False

I thought the only thing -S did was skip import site.

[jaraco]

TIL:

-S: Disable the import of the module site and the site-dependent manipulations of sys.path that it entails. Also disable these manipulations if site is explicitly imported later (call site.main() if you want them to be triggered).

[jaraco]

~ $ pypy3 -S -c "import site, sys; site.main(); print('distutils' in sys.modules)"
True

[jaraco]

It's the calling of sysconfig.getconfigvar that causes the import of distutils:

$ pypy3 -S -c "import sys, sysconfig; print('distutils' in sys.modules); sysconfig.get_config_var('foo'); print('distutils' in sys.modules)"
False
True

[jaraco]

Right - now I remember:

$ pypy3 -S -c "import sys, _sysconfigdata; print('distutils' in sys.modules)"
True

[jaraco]

It does appear that even on the unreleased PyPy for Python 3.7 still imports distutils early.

[jaraco]

Perhaps there are some optimizations or tweaks we can make for PyPy, but for now, I'd prefer to push forward with this approach.

[jaraco]

Released as 49.3.0

[gaborbernat]

This PR shouldn't have closed the issue. The issue is not fixed and manifests just as it did before. It's now only able to be bypassed by setting an environment variable. So to fix it still requires me an arbitrary environment variable change in my code...

[jaraco]

Thanks Bernát, though I'm not sure I follow. #2259 reported an issue that when the new distutils is included, it only works when setuptools is imported prior to importing distutils. This patch fixed that condition. #2259 was not about ensuring that distutils was preferred in all environments, though that's still a goal.

I have added new issues to the milestone to capture ongoing efforts. If that doesn't capture the missed use-cases, please elaborate.