WIP direct module load shims

[nitzmahone]

Summary of changes

• .pth uses direct shim module load from sibling setuptools dir instead of import
• import shims attempt to remove themselves/others when it's clear they're not needed (eg, setuptools path mismatch from .pth)
• removes _distutils_hack top-level package

Closes #2957

Pull Request Checklist

• Changes have tests
• News fragment added in changelog.d/.
  (See documentation for details)

[jaraco]

Introducing imperative code ahead of imports is going to cause flake8 errors on all the subsequent imports. I'd suggest to just keep the previous technique where the shim is a package and a module of the package can be imported to invoke the behavior. i.e. mv _distutils_hack setuptools/_distutils_shim

[nitzmahone]

This one's easy to solve any number of ways since it's a real import (I was just lazy with the PoC), but I think you're already starting to see the problems that the extra top-level package can cause (eg, d0da014)

[jaraco]

I believe accessing __file__ may cause problems under a zipimport scenario. That's something we'd want to verify before accepting.

[jaraco]

My biggest concern about this change is moving the hack into the setuptools namespace. I would have liked to have this package in some shared namespace, but the fact of the matter is:

• importing setuptools invokes a lot of behavior, including monkeypatching distutils.
• importing setuptools imports pkg_resources, which invokes a lot more behavior and is known to be slow.

It's simply not acceptable to force every environment to import setuptools on startup. Therefore, the shim needs not to import pkg_resources or setuptools in order to install the shim. The import of either should be deferred until distutils is imported.

[nitzmahone]

My biggest concern about this change is moving the hack into the setuptools namespace

It's very explicitly not importing setuptools or pkg_resources- merely direct loading the module code for the shim from the sibling setuptools structure. Try it:

>>> import sys
>>> sys.meta_path
[<_shimmod.DistutilsMetaFinder object at 0x7f053265e790>, <_virtualenv._Finder object at 0x7f05327dcf70>, <class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]
>>> any(m for m in sys.modules.keys() if m in ['setuptools', 'distutils', 'pkg_resources'])
False

It's using the assumption that the setuptools package that's a sibling to the .pth in the filesystem is the only one that it should be allowed to service. I haven't played with the zip side of things, but assuming it works the way I expect, it could probably be located with a find_spec without tripping the package init, though I'm not sure what assumptions could be made about co-location in that case...

[jaraco]

In 9c9c91c, I've applied a similar patch to address a separate concern (race between .pth handling and import invocation). We'll need to resolve the conflict.

It's very explicitly not importing setuptools or pkg_resources- merely direct loading the module code for the shim from the sibling setuptools structure.

Oh, nice. I'd missed that nuance. I like that behavior.

I do regret the added complexity to the implementation.

import shims attempt to remove themselves/others when it's clear they're not needed

I wonder if this is a premature optimization... Can we consider this change separately if it's not needed to fix the reported issue?

[jaraco]

Oh, this is unfortunate. Because the shim might be invoked independently several times, it has to detect different instances of the same name and origin. This complexity is one argument for keeping _distutils_hack as a top-level module in order to rely on import semantics to resolve exactly one instance of the module/finder per origin.

I'd like to explore an approach that doesn't require touching some many different aspects of the implementation.

[nitzmahone]

That was the point though- since the requirements for the shim are certainly a moving target, the only sure way to get the right shim behavior for any given version of setuptools is to let each setuptools load its own locally, then clean up the others when one actually "fires". It's definitely "icky", but given the restrictions of what you can actually do in a .pth file, I can't think of another way that will also accommodate evolving shims.

[jaraco]

Let's try #2962 first and see how that goes.