Resolution is halted by sdists whose metadata extraction fail. #12212

jsirois · 2023-08-08T03:59:32Z

Description

During a resolve (I encountered using pip download), newer versions of Pip halt instead of backtracking when a candidate sdist metadata extraction process fails. Older versions logged these errors and backtracked.

Expected behavior

It seems that treating an sdist whose metadata fails to be extractable as a rejected candidate is more sensible than halting the resolve all together. This allows a project to make a faulty release, but still be resolvable by Pip in many circumstances (consumers have requirements satisfiable by earlier, non-broken releases).

pip version

=22.0

Python version

3.11

OS

WSL Linux (Ubuntu 22.04)

How to Reproduce

Successful backtracks in the face of sdist metadata extraction using Pip 20.3.4:

$ python3.11 -mvenv example.venv
$ example.venv/bin/pip -q install -U pip==20.3.4
$ example.venv/bin/pip -V
pip 20.3.4 from /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip (python 3.11)
$ example.venv/bin/pip download awscli==1.28.1
Collecting awscli==1.28.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/awscli-1.28.1-py3-none-any.whl
Collecting botocore==1.30.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/botocore-1.30.1-py3-none-any.whl
Collecting colorama<0.4.5,>=0.2.5
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/colorama-0.4.4-py2.py3-none-any.whl
Collecting docutils<0.17,>=0.10
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/docutils-0.16-py2.py3-none-any.whl
Collecting s3transfer<0.7.0,>=0.6.0
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/s3transfer-0.6.1-py3-none-any.whl
Collecting rsa<4.8,>=3.1.2
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/rsa-4.7.2-py3-none-any.whl
Collecting PyYAML<5.5,>=3.10
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/bin/python3.11 /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp3871j3iy
       cwd: /tmp/pip-download-egvptpng/pyyaml_bc01200998b544a4828b298a72463864
  Complete output (68 lines):
  /tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
  !!

          ********************************************************************************
          The license_file parameter is deprecated, use license_files instead.

          By 2023-Oct-30, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.

          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************

  !!
    parsed = self.parsers.get(option_name, lambda x: x)(value)
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 314, in run
      self.find_sources()
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
      mm.run()
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
      self.add_defaults()
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
      super().add_defaults()
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<string>", line 201, in get_source_files
    File "/tmp/pip-build-env-fek1ubjb/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz#sha256=607774cbba28732bfa802b54baa7484215f530991055bb562efbed5b2f20a45e (from https://pypi.org/simple/pyyaml/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*). Command errored out with exit status 1: /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/bin/python3.11 /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp3871j3iy Check the logs for full command output.
  Using cached PyYAML-5.4.tar.gz (174 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/bin/python3.11 /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpnik8i6qu
       cwd: /tmp/pip-download-egvptpng/pyyaml_b679807bbe794a39a2311d0b8e3b0f83
  Complete output (68 lines):
  /tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
  !!

          ********************************************************************************
          The license_file parameter is deprecated, use license_files instead.

          By 2023-Oct-30, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.

          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************

  !!
    parsed = self.parsers.get(option_name, lambda x: x)(value)
  running egg_info
  writing lib3/PyYAML.egg-info/PKG-INFO
  writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
  writing top-level names to lib3/PyYAML.egg-info/top_level.txt
  Traceback (most recent call last):
    File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 271, in <module>
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 314, in run
      self.find_sources()
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
      mm.run()
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
      self.add_defaults()
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
      super().add_defaults()
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
      self._add_defaults_ext()
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "<string>", line 201, in get_source_files
    File "/tmp/pip-build-env-4br16vm8/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/b5/fd/15638de2da0a5aa91c095718444624aa565f766fc178249ca6faa372f71a/PyYAML-5.4.tar.gz#sha256=3c49e39ac034fd64fd576d63bb4db53cda89b362768a67f07749d55f128ac18a (from https://pypi.org/simple/pyyaml/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*). Command errored out with exit status 1: /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/bin/python3.11 /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpnik8i6qu Check the logs for full command output.
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/PyYAML-5.3.1.tar.gz
Collecting urllib3<1.27,>=1.25.4
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/urllib3-1.26.16-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/python_dateutil-2.8.2-py2.py3-none-any.whl
Collecting jmespath<2.0.0,>=0.7.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/jmespath-1.0.1-py3-none-any.whl
Collecting six>=1.5
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/six-1.16.0-py2.py3-none-any.whl
Collecting pyasn1>=0.1.3
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/pyasn1-0.5.0-py2.py3-none-any.whl
Successfully downloaded awscli botocore colorama docutils jmespath python-dateutil PyYAML rsa pyasn1 s3transfer six urllib3
WARNING: You are using pip version 20.3.4; however, version 23.2.1 is available.
You should consider upgrading via the '/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/bin/python3.11 -m pip install --upgrade pip' command.

Likewise success up through 21.3.1:

$ example.venv/bin/pip -q install -U "pip<22"
$ example.venv/bin/pip -V
pip 21.3.1 from /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip (python 3.11)
$ example.venv/bin/pip download awscli==1.28.1
...
    File "<string>", line 201, in get_source_files
    File "/tmp/pip-build-env-zfpin_0i/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/b5/fd/15638de2da0a5aa91c095718444624aa565f766fc178249ca6faa372f71a/PyYAML-5.4.tar.gz#sha256=3c49e39ac034fd64fd576d63bb4db53cda89b362768a67f07749d55f128ac18a (from https://pypi.org/simple/pyyaml/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*). Command errored out with exit status 1: /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/bin/python3.11 /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp9z4erva2 Check the logs for full command output.
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/PyYAML-5.3.1.tar.gz
  Preparing metadata (setup.py) ... done
Collecting s3transfer<0.7.0,>=0.6.0
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/s3transfer-0.6.1-py3-none-any.whl
Collecting rsa<4.8,>=3.1.2
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/rsa-4.7.2-py3-none-any.whl
Collecting botocore==1.30.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/botocore-1.30.1-py3-none-any.whl
Collecting docutils<0.17,>=0.10
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/docutils-0.16-py2.py3-none-any.whl
Collecting urllib3<1.27,>=1.25.4
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/urllib3-1.26.16-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/python_dateutil-2.8.2-py2.py3-none-any.whl
Collecting jmespath<2.0.0,>=0.7.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/jmespath-1.0.1-py3-none-any.whl
Collecting pyasn1>=0.1.3
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/pyasn1-0.5.0-py2.py3-none-any.whl
Collecting six>=1.5
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/six-1.16.0-py2.py3-none-any.whl
Successfully downloaded awscli botocore colorama docutils PyYAML rsa s3transfer jmespath pyasn1 python-dateutil urllib3 six
WARNING: You are using pip version 21.3.1; however, version 23.2.1 is available.
You should consider upgrading via the '/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/bin/python3.11 -m pip install --upgrade pip' command.

But failure starting at Pip 22.0:

$ example.venv/bin/pip -q install -U "pip==22"
$ example.venv/bin/pip -V
pip 22.0 from /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip (python 3.11)
$ example.venv/bin/pip download awscli==1.28.1
Collecting awscli==1.28.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/awscli-1.28.1-py3-none-any.whl
Collecting colorama<0.4.5,>=0.2.5
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/colorama-0.4.4-py2.py3-none-any.whl
Collecting botocore==1.30.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/botocore-1.30.1-py3-none-any.whl
Collecting rsa<4.8,>=3.1.2
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/rsa-4.7.2-py3-none-any.whl
Collecting s3transfer<0.7.0,>=0.6.0
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/s3transfer-0.6.1-py3-none-any.whl
Collecting docutils<0.17,>=0.10
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/docutils-0.16-py2.py3-none-any.whl
Collecting PyYAML<5.5,>=3.10
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [68 lines of output]
      /tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!

              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.

              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 314, in run
          self.find_sources()
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
          mm.run()
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-0miimahn/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: You are using pip version 22.0; however, version 23.2.1 is available.
You should consider upgrading via the '/home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/bin/python3.11 -m pip install --upgrade pip' command.

And that style of fail-fast error persists to present day released Pip:

$ example.venv/bin/pip -q install -U pip
$ example.venv/bin/pip -V
pip 23.2.1 from /home/jsirois/dev/pantsbuild/jsirois-pex/example.venv/lib/python3.11/site-packages/pip (python 3.11)
$ example.venv/bin/pip download awscli==1.28.1
...
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-fu25rkcp/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Output

N/A - commands and output are above.

Code of Conduct

I agree to follow the PSF Code of Conduct.

The text was updated successfully, but these errors were encountered:

jsirois · 2023-08-08T04:02:31Z

For historical context, the underlying build issue is well known by the PyYAML team and documented here: yaml/pyyaml#601. This is of course a more general issue, but this was the exemplar case that brought the problem to my attention.

notatallshaw · 2023-08-08T04:06:05Z

I believe this was an intentional change when the build process fails, more details here: #10722

jsirois · 2023-08-08T04:12:49Z

Aha! Ok - thank you, I did not think to poke around features (vs flags). I now know about backtrack-on-build-failures but I don't see that as an available feature in modern Pip. I'll research more ... without an opt-out this seems unfortunate, but I'm sure there were deliberations.

jsirois · 2023-08-08T04:18:54Z

Hrm, that was removed in 22.2 here: 8bebea8

I think the removal of the opt out is really unfortunate. In this case the issue is unresolvable without reverting to an older Pip because the problematic dependency is a (transitive) PEP-518 unbounded Cython dep that --constraint cannot solve (by experiment and as I understand it in general).

I'm guessing this was driven by desire for a simpler code base and maybe not anticipating issues like this (or else a calculation that these sorts of issues were rare enough / or people should upgrade their deps to work past broken releases downstream).

jsirois · 2023-08-08T04:32:28Z

@notatallshaw thanks again for cluing me in to the context. I'm pretty sure this is all working as intended and the opt-out is unlikely to be revived. I'll close this out.

uranusjr · 2023-08-08T04:42:29Z

The change of default behaviour was motivated by user feedback, since backtracking on build failure, while reasonable in theory, causes too much user confusion in practice. The backtrack-on-build-failures flag was provided to offer a transition window for users that happen to rely on the behaviour, and eventually removed because there was no more feedback from users that need the flag.

Note that although using backtrack-on-build-failures in this particular situation would resolve the immediate issue, it is very likely to introduce further problems at some point in the future, if an upgrade somewhere introduces a new build failure and starts causing unwanted backtracking. The flag is therefore only designed to be used as a stopgap measure while you work on a proper fix. In this case, a better fix would be to first build and install pyyaml separately (with --no-build-isolation and a manually populated build environment), so the pacakge is skipped later during the “main” installation phase.

jsirois · 2023-08-08T04:46:04Z

Thanks @uranusjr. This all comes from a hermetic build context where setting up (mutating) a venv just so ahead of the main resolve event is not an option.

jsirois added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Aug 8, 2023

jsirois mentioned this issue Aug 8, 2023

KeyError when locking awscli on Python 3.11 pex-tool/pex#2211

Closed

jsirois closed this as completed Aug 8, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Resolution is halted by sdists whose metadata extraction fail. #12212

Resolution is halted by sdists whose metadata extraction fail. #12212

jsirois commented Aug 8, 2023

jsirois commented Aug 8, 2023

notatallshaw commented Aug 8, 2023 •

edited

Loading

jsirois commented Aug 8, 2023 •

edited

Loading

jsirois commented Aug 8, 2023 •

edited

Loading

jsirois commented Aug 8, 2023

uranusjr commented Aug 8, 2023

jsirois commented Aug 8, 2023

Resolution is halted by sdists whose metadata extraction fail. #12212

Resolution is halted by sdists whose metadata extraction fail. #12212

Comments

jsirois commented Aug 8, 2023

Description

Expected behavior

pip version

Python version

OS

How to Reproduce

Output

Code of Conduct

jsirois commented Aug 8, 2023

notatallshaw commented Aug 8, 2023 • edited Loading

jsirois commented Aug 8, 2023 • edited Loading

jsirois commented Aug 8, 2023 • edited Loading

jsirois commented Aug 8, 2023

uranusjr commented Aug 8, 2023

jsirois commented Aug 8, 2023

notatallshaw commented Aug 8, 2023 •

edited

Loading

jsirois commented Aug 8, 2023 •

edited

Loading

jsirois commented Aug 8, 2023 •

edited

Loading