Unexpected behavior when distutils is invoked before Setuptools

[webknjaz]

TL;DR We are forced to use distutils because setuptools has broken symlink processing and this causes distutils.errors.DistutilsClassError: command class <class '__main__.SDistCommand'> must subclass Command.

It works with setuptools<48 and the changelog doesn't document any breaking behaviors for this version.

Repro:

$ git clone https://github.com/ansible/ansible.git
$ cd ansible
$ pip install -U 'setuptools>=48'
$ python setup.py sdist

(tried under Python 3.8)

Ref: ansible/ansible#70456

[jaraco]

Thanks for the report.

Setuptools 49.1 is out, should address the issue for early adopters while I triage and correct the issue.

[jaraco]

I'm able to replicate the failure:

ansible devel $ pip-run -q setuptools==48 -- setup.py sdist
/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/distutils_patch.py:17: UserWarning: Setuptools is replacing distutils
  warnings.warn("Setuptools is replacing distutils")
Traceback (most recent call last):
  File "setup.py", line 338, in <module>
    main()
  File "setup.py", line 333, in main
    setup(**setup_params)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/__init__.py", line 164, in setup
    return distutils.core.setup(**attrs)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/core.py", line 134, in setup
    ok = dist.parse_command_line()
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/dist.py", line 484, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/dist.py", line 925, in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-run-2ua0qbu2/setuptools/_distutils/dist.py", line 547, in _parse_command_opts
    raise DistutilsClassError(
distutils.errors.DistutilsClassError: command class <class '__main__.SDistCommand'> must subclass Command

Note the warning "Setuptools is replacing distutils".

That was meant to be a hard failure, except that PyPy somehow ends up with distutils unconditionally imported during interpreter startup.

[jaraco]

I'm able to work around the issue by ensuring that setuptools is imported prior to running the command:

$ pip-run -q setuptools==48 -- -c "import setuptools; exec(open('setup.py').read())" sdist
...
hard linking test/units/vars/test_variable_manager.py -> ansible-base-2.11.0.dev0/test/units/vars
creating dist
Creating tar archive
removing 'ansible-base-2.11.0.dev0' (and everything under it)
<string>:180: RuntimeWarning: When setup.py sdist is run from outside of the Makefile, the generated tarball may be incomplete.  Use `make snapshot` to create a tarball from an arbitrary checkout or use `cd packaging/release && make release version=[..]` for official builds.

[jaraco]

Also using PEP 517:

ansible devel $ cat > pyproject.toml
[build-system]
requires = ['setuptools', 'wheel']
build-backend = 'setuptools.build_meta'

ansible devel $ pip-run -q pep517 -- -m pep517.build -s .
running egg_info
writing lib/ansible_base.egg-info/PKG-INFO
writing dependency_links to lib/ansible_base.egg-info/dependency_links.txt
writing requirements to lib/ansible_base.egg-info/requires.txt
writing top-level names to lib/ansible_base.egg-info/top_level.txt
reading manifest file 'lib/ansible_base.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'docs/docsite/rst_warnings'
warning: no previously-included files matching '*' found under directory 'docs/docsite/_build'
warning: no previously-included files matching '*.pyc' found under directory 'docs/docsite/_extensions'
warning: no previously-included files matching '*.pyo' found under directory 'docs/docsite/_extensions'
warning: no files found matching '*.ps1' under directory 'lib/ansible/modules/windows'
warning: no files found matching '*.psm1' under directory 'test/support'
writing manifest file 'lib/ansible_base.egg-info/SOURCES.txt'
running sdist
...
Creating tar archive
removing 'ansible-base-2.11.0.dev0' (and everything under it)
setup.py:180: RuntimeWarning: When setup.py sdist is run from outside of the Makefile, the generated tarball may be incomplete.  Use `make snapshot` to create a tarball from an arbitrary checkout or use `cd packaging/release && make release version=[..]` for official builds.
  warnings.warn('When setup.py sdist is run from outside of the Makefile,'

[jaraco]

It also works with this diff on ansible:

diff --git a/setup.py b/setup.py
index 0398473d5e..7c86075495 100644
--- a/setup.py
+++ b/setup.py
@@ -9,8 +9,6 @@ import sys
 import warnings
 
 from collections import defaultdict
-from distutils.command.build_scripts import build_scripts as BuildScripts
-from distutils.command.sdist import sdist as SDist
 
 try:
     from setuptools import setup, find_packages
@@ -23,6 +21,9 @@ except ImportError:
           " install setuptools).", file=sys.stderr)
     sys.exit(1)
 
+from distutils.command.build_scripts import build_scripts as BuildScripts
+from distutils.command.sdist import sdist as SDist
+
 sys.path.insert(0, os.path.abspath('lib'))
 from ansible.release import __version__, __author__

The underlying issue is that with setuptools 48, you cannot import distutils before setuptools.

My recommendation is to (a) move the imports so distutils imports come after setuptools (quick, painless, compatible), then (b) adopt PEP 517 and use pep517 to build your sdists.

I'll update the changelog to make this change more clear.

[jaraco]

In the referenced commit, I've updated the changelog to clarify the weaknesses identified from this bug report and guide users to the best practices. Can you confirm that this guidance and applying the recommended changes in Ansible addresses the concern?

[webknjaz]

Can you confirm that this guidance and applying the recommended changes in Ansible addresses the concern?

I've submitted the patch that you suggested and it'll probably get us going short-term. But we cannot use pyproject.toml atm, nor we ship wheels. The reason is that we have a bunch of symlinks that need to be preserved and it's done by distutils but not setuptools.

[jaraco]

Great. Thanks for enacting that.

I'm wondering if there's something that Setuptools should do here to help other projects with similar usages from encountering the same issue as we bring the adopted distutils back. For example, the project could:

1. Prior to re-enabling the adoption, warn when 'distutils' appears in sys.modules (except on PyPy, where that's the status quo on startup).
2. When re-enabling the adoption, error with a helpful message when distutils is replaced (again exempting PyPy).

I think I'll enact (1) now and give it some time to percolate over the weekend and maybe longer.

Do please consider filing something with Ansible to investigate a long-term solution for what Ansible will do when distutils is sunset and all that's left is Setuptools (as that's the plan).

[omry]

This seems a bit heavy handed.
I suddenly started to get this warning for code that is well tested and works fine.
running just one additional program (isort) I saw that warning from it too.

Requiring specific import order is a bad practice that makes for fragile code.
Automatic tools like isort work by the common rules and users will have to constantly hack around this odd new requirement.
Please reconsider.

Is it possible to have a more refined detection of the problem you are trying to avoid?

[jaraco]

The problem we're trying to avoid is that Setuptools wishes to ensure that anything the user has imported is imported after Setuptools has ensured that import distutils effectively imports setuptools._distutils. It's a transitional hack until distutils can be deprecated entirely.

Ideally, most users should be able to simply import everything from Setuptools.

I did consider another approach, where Setuptools could add a .pth file that would import setuptools.distutils_patch. Such an approach would always happen earlier and so would not be subject to the race that's happening here, but it would also happen whether or not setuptools was imported (on any invocation of Python in that environment).

I elected for the current approach as it's more surgical.

[omry]

Can you explain what the actual problem is that requires controlling the import order?

specifically in my case I am using distutils.cmd.Command. I noticed that there is a Command in setuptools but it seems different.

[jaraco]

setuptools.Command is a subclass of distutils.cmd.Command, so while it may seem different, it's actually the same with some small changes. Probably you can use setuptools.Command in its place. Maybe give it a try?

[jorisvandenbossche]

We also run into this warning in pandas, where we use distutils.version for version checking in the actual library (so not just in the packaging code).

[omry]

setuptools.Command is a subclass of distutils.cmd.Command, so while it may seem different, it's actually the same with some small changes. Probably you can use setuptools.Command in its place. Maybe give it a try?

That worked, next in line is a distutils.log usage I have :

      self.announce(
                f"Generating parser for Python3: {command}", level=distutils.log.INFO,
            )

and I am not seeing those constants in setuptools. Are they already ported?

[vklohiya]

Hi,

I am also facing this issue with latest version of setup tools v50.0.0. After downgrading the version worked for me.

[webknjaz]

@vklohiya this behavior has been re-introduced. You may temporarily revert back to the old one by setting SETUPTOOLS_USE_DISTUTILS=stdlib env var. But the ultimate solution would be to fix your setup.py.