Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flit's dynamic description is not working with pip 22.2 along with python 3.8, 3.9 #11294

Closed
furiosamg opened this issue Jul 22, 2022 · 19 comments · Fixed by #11298
Closed

Flit's dynamic description is not working with pip 22.2 along with python 3.8, 3.9 #11294

furiosamg opened this issue Jul 22, 2022 · 19 comments · Fixed by #11298
Labels
C: build logic Stuff related to metadata generation / wheel generation type: bug A confirmed bug or unintended behavior
Milestone

Comments

@furiosamg
Copy link

Minimal reproduction: https://github.com/furiosamg/flit-dynamic-pip-22

I think flit's dynamic description generation is broken in pip 22.2.
When trying to install above package with pip 22.2, following error occurred.

$ pip install pip==22.2 && pip install .
Looking in indexes: https://pypi.org/simple, https://internal-pypi.furiosa.dev/simple
Requirement already satisfied: pip==22.2 in /root/miniconda3/envs/env-3.9/lib/python3.9/site-packages (22.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Looking in indexes: https://pypi.org/simple, https://internal-pypi.furiosa.dev/simple
Processing /root/flit-dynamic-pip-22
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [45 lines of output]
      Looking in indexes: https://pypi.org/simple, https://internal-pypi.furiosa.dev/simple, https://internal-pypi.furiosa.dev/simple
      Collecting flit_core<4,>=3.2
        Using cached flit_core-3.7.1-py3-none-any.whl (60 kB)
      Installing collected packages: flit_core
      /root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py:103: _ExperimentalProjectMetadata: Support for project metadata in `pyproject.toml` is still experimental and may be removed (or change) in future releases.
        warnings.warn(msg, _ExperimentalProjectMetadata)
      ERROR: Exception:
      Traceback (most recent call last):
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
          status = run_func(*args)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
          return func(self, options, args)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 461, in run
          installed = install_given_reqs(
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
          requirement.install(
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 752, in install
          scheme = get_scheme(
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/locations/__init__.py", line 246, in get_scheme
          old = _distutils.get_scheme(
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/locations/_distutils.py", line 130, in get_scheme
          scheme = distutils_scheme(dist_name, user, home, root, isolated, prefix)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip/_internal/locations/_distutils.py", line 46, in distutils_scheme
          d.parse_config_files()
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/dist.py", line 854, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 55, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 135, in read_configuration
          return expand_configuration(asdict, root_dir, ignore_option_errors, dist)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 190, in expand_configuration
          return _ConfigExpander(config, root_dir, ignore_option_errors, dist).expand()
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 237, in expand
          self._expand_all_dynamic(dist, package_dir)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 272, in _expand_all_dynamic
          obtained_dynamic = {
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 273, in <dictcomp>
          field: self._obtain(dist, field, package_dir)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 310, in _obtain
          self._ensure_previously_set(dist, field)
        File "/root/miniconda3/envs/env-3.9/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 296, in _ensure_previously_set
          raise OptionError(msg)
      distutils.errors.DistutilsOptionError: No configuration found for dynamic 'description'.
      Some dynamic fields need to be specified via `tool.setuptools.dynamic`
      others must be specified via the equivalent attribute in `setup.py`.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip
@takluyver
Copy link
Member

Your package builds and installs fine for me (see below). 😕

It looks like pip is using some distutils/setuptools code to work out destination paths, and setuptools ends up trying to parse the metadata from pyproject.toml assuming that it's built with setuptools. What version of setuptools do you have? I've just tried updating to the latest and I still don't see the issue.

Do you have any pip config in files or environment variables?

$ pip install .
Defaulting to user installation because normal site-packages is not writeable
Processing /home/takluyver/Code/flit-dynamic-pip-22
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: flitdynamicpip22
  Building wheel for flitdynamicpip22 (pyproject.toml) ... done
  Created wheel for flitdynamicpip22: filename=flitdynamicpip22-0.0.1-py2.py3-none-any.whl size=1101 sha256=0a1efe3e78bd38c63c5ce540cc616d72da889be62233e2e61033f5b67ade985e
  Stored in directory: /home/takluyver/.cache/pip/wheels/df/08/86/74c3e5aa6010da77b6ecae2b31a34efd35815ddc26e6b382b3
Successfully built flitdynamicpip22
Installing collected packages: flitdynamicpip22
Successfully installed flitdynamicpip22-0.0.1

$ pip --version
pip 22.2 from /home/takluyver/.local/lib/python3.10/site-packages/pip (python 3.10)

@furiosamg
Copy link
Author

Thanks for the comment.

When I try to install my package with Python 3.10, the installation succeeds. 🤔 But not with 3.8, 3.9.
Could you please try with Python 3.8 or 3.9?

P.S. I think I haven't set any pip configuration.

@furiosamg furiosamg changed the title Dynamic description is not working with pip 22.2 Dynamic description is not working with pip 22.2 along with python 3.8, 3.9 Jul 22, 2022
@takluyver
Copy link
Member

Thanks, when I make a Python 3.9 environment, I do see the same error.

I still don't think this is a Flit bug - I think the issue is somewhere between pip & setuptools, because it looks like setuptools is trying to get the package metadata as if it was a setuptools package, which is wrong. But I don't know when I'll have time to dig into it to work out exactly what is going wrong.

Other than digging through the traceback to figure out what is going on, you could try using git bisect to identify which change in pip introduced or revealed the problem.

@furiosamg
Copy link
Author

I agree with that flit is innocent 😇
The reason I raised the issue here was because I thought dynamic description generation was a flit's functionality.

Thanks for gentle and quick response.

@pradyunsg
Copy link
Member

Pinging @abravalheri since setuptools' pyproject.toml support seems to be implicated in this.

@abravalheri
Copy link

abravalheri commented Jul 22, 2022

Any chance setuptools got imported before distutils? It seems that pip is using the monkey patched version of Distribution.

@pradyunsg
Copy link
Member

pradyunsg commented Jul 22, 2022

I can confirm why this isn't failing on 3.10 -- on 3.10+, pip's using the sysconfig paths and avoid importing/using distutils entirely. That's why the code never reaches the locations/_distutils.py file, which means it never ends up using that.

Any chance setuptools got imported before distutils?

I don't think that should happen? The only reason that I can think of is if setuptools' distutils hack has some logic to detect pip that's broken in 22.2.


FWIW, this only happens for local installations and not when installing from a zipfile.

❯ python --version
Python 3.9.6

❯ pip --version
pip 22.2 from /Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip (python 3.9)

❯ pip install https://github.com/furiosamg/flit-dynamic-pip-22/archive/refs/heads/main.zip
Collecting https://github.com/furiosamg/flit-dynamic-pip-22/archive/refs/heads/main.zip
  Downloading https://github.com/furiosamg/flit-dynamic-pip-22/archive/refs/heads/main.zip
     - 1.2 kB 8.5 MB/s 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: flitdynamicpip22
  Building wheel for flitdynamicpip22 (pyproject.toml) ... done
  Created wheel for flitdynamicpip22: filename=flitdynamicpip22-0.0.1-py2.py3-none-any.whl size=1101 sha256=178bba12298d9ad6341796ffee219b2348817d481367f0501ed4bd8f436c4489
  Stored in directory: /private/var/folders/y1/j465wvf92vs938kmgqh63bj80000gn/T/pip-ephem-wheel-cache-0h1nj302/wheels/cf/e8/77/17cc0fd5e981a5997f17e4fa054ed3cd7d06eb9522308f4722
Successfully built flitdynamicpip22
Installing collected packages: flitdynamicpip22
Successfully installed flitdynamicpip22-0.0.1

❯ g cl https://github.com/furiosamg/flit-dynamic-pip-22
Cloning into 'flit-dynamic-pip-22'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 12 (delta 2), reused 12 (delta 2), pack-reused 0
Receiving objects: 100% (12/12), done.
Resolving deltas: 100% (2/2), done.

❯ cd flit-dynamic-pip-22

❯ pip install .
Processing /Users/pradyunsg/Developer/github/flit-dynamic-pip-22
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 2
  ╰─> [42 lines of output]
      Collecting flit_core<4,>=3.2
        Using cached flit_core-3.7.1-py3-none-any.whl (60 kB)
      Installing collected packages: flit_core
      ERROR: Exception:
      Traceback (most recent call last):
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
          status = run_func(*args)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
          return func(self, options, args)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 461, in run
          installed = install_given_reqs(
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
          requirement.install(
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/req/req_install.py", line 752, in install
          scheme = get_scheme(
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/locations/__init__.py", line 246, in get_scheme
          old = _distutils.get_scheme(
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/locations/_distutils.py", line 130, in get_scheme
          scheme = distutils_scheme(dist_name, user, home, root, isolated, prefix)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/pip/_internal/locations/_distutils.py", line 46, in distutils_scheme
          d.parse_config_files()
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/dist.py", line 868, in parse_config_files
          pyprojecttoml.apply_configuration(self, filename, ignore_option_errors)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 58, in apply_configuration
          config = read_configuration(filepath, True, ignore_option_errors, dist)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 136, in read_configuration
          return expand_configuration(asdict, root_dir, ignore_option_errors, dist)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 191, in expand_configuration
          return _ConfigExpander(config, root_dir, ignore_option_errors, dist).expand()
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 238, in expand
          self._expand_all_dynamic(dist, package_dir)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 275, in _expand_all_dynamic
          obtained_dynamic = {
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 276, in <dictcomp>
          field: self._obtain(dist, field, package_dir)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 322, in _obtain
          self._ensure_previously_set(dist, field)
        File "/Users/pradyunsg/Developer/flit-bug/.venv/lib/python3.9/site-packages/setuptools/config/pyprojecttoml.py", line 301, in _ensure_previously_set
          raise OptionError(msg)
      distutils.errors.DistutilsOptionError: No configuration found for dynamic 'description'.
      Some dynamic fields need to be specified via `tool.setuptools.dynamic`
      others must be specified via the equivalent attribute in `setup.py`.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

@abravalheri
Copy link

The only reason that I can think of is if setuptools' distutils hack has some logic to detect pip that's broken in 22.2.

I can confirm that the custom import logic don't seem to be working for pip 22.2: in pip/_internal/locations/_distutils.py, distutils is loaded from setuptools/_distutils, which is related to #8761.

Maybe we can avoid this error if PipImportRedirectingFinder is added after DistutilsMetaFinder?

Something like:

- sys.meta_path.insert(0, PipImportRedirectingFinder())
+ def _install():
+    for i, finder in enumerate(sys.meta_path):
+        if "DistutilsMetaFinder" in str(type(finder)):
+            sys.meta_path.insert(i + 1, PipImportRedirectingFinder())
+            return
+
+    sys.meta_path.insert(0, PipImportRedirectingFinder())
+
+
+ _install()

@pradyunsg
Copy link
Member

Ah, well, that's interesting and would fix the immediate bleeding. I think it's reasonable for us to do a sys.meta_path.append(...) instead of the sys.meta_path.insert(0, ...).

We actually changed this to the start on the rationale that no one else should try modifying pip before us (#11257 (comment)) but setuptools use case seems like a reasonable one.

@furiosamg
Copy link
Author

Thank you all for your support and interest. I'd be happy to wait.

@abravalheri
Copy link

abravalheri commented Jul 22, 2022

Thank you very much for the feedback @pradyunsg.

There might be other ways of handling this in the setuptools side, but I think that this is the most straight forward... Without the custom finder intercepting pip, we would need some kind of signalling from pip to avoid replacing distutils (e.g. the approach described by Jason). (I am more than happy to discuss other alternatives if any of you guys has a different idea).

@furiosamg, meanwhile you can try exporting SETUPTOOLS_USE_DISTUTILS=stdlib as an environment variable.

@furiosamg
Copy link
Author

@abravalheri Thanks, I checked that workaround works. Great!

@pradyunsg
Copy link
Member

I'll move this over to pip, since we need to make a change on pip's end.

@pradyunsg pradyunsg transferred this issue from pypa/flit Jul 23, 2022
@sbidoul sbidoul added this to the 22.2.1 milestone Jul 23, 2022
@sbidoul sbidoul added type: bug A confirmed bug or unintended behavior C: build logic Stuff related to metadata generation / wheel generation labels Jul 23, 2022
@ctb
Copy link

ctb commented Jul 24, 2022

thank you very much for posting this, @furiosamg!

we ran into a related problem over in sourmash, sourmash-bio/sourmash#2139, where our ReadTheDocs build was failing with python3.8 and pip 22.2.

I verified that downgrading to pip 22.1.2 fixed the problem, as did upgrading to python3.10 in the conda environment used for building the docs. I'm trying the latter out for the readthedocs fix.

Nothing to be done, just wanted to chime in with another breaking build! Happy to help test or debug things if and as it would be helpful.

(If others are having problems with pip 22.2 and readthedocs build, here is our fix)

@furiosamg furiosamg changed the title Dynamic description is not working with pip 22.2 along with python 3.8, 3.9 Flit's dynamic description is not working with pip 22.2 along with python 3.8, 3.9 Jul 24, 2022
@pradyunsg
Copy link
Member

OK, could someone test and confirm that #11298 fixes the issue for them?

@ctb
Copy link

ctb commented Jul 24, 2022

The following (1) demonstrated the problem and (2) showed that the provided fix worked:

cd $(mktemp -d)
git clone --no-single-branch --depth 50 https://github.com/sourmash-bio/sourmash . 
# adjust environment.yml to point at python=3.8
# vi doc/environment.yml 

mamba env create --name latest --file doc/environment.yml 
conda activate latest
pip show pip
# pip version was 22.2

python -m pip install --upgrade --upgrade-strategy eager --no-cache-dir .[doc]
# FAILED

# now, update to @pradyunsg version of pip
mkdir foo
cd foo
git clone https://github.com/pradyunsg/pip.git .
git switch remove-distutils-shim
pip install .
cd ..

pip show pip
# Version: 22.3.dev0
python -m pip install --upgrade --upgrade-strategy eager --no-cache-dir .[doc]
# SUCCEEDED

@Jorricks
Copy link

OK, could someone test and confirm that #11298 fixes the issue for them?

Just tried this out. Used to give me an error with Flit. Now confirmed it works with your fix @pradyunsg on

% python3 -V
Python 3.8.11

@furiosamg
Copy link
Author

furiosamg commented Jul 25, 2022

My initial minimal reproduction was also fixed.

Python 3.8

(env-3.8) root@mg-shin-npu-0:~/flit-dynamic-pip-22# cd ~/pip
(env-3.8) root@mg-shin-npu-0:~/pip# git switch remove-distutils-shim
Branch 'remove-distutils-shim' set up to track remote branch 'remove-distutils-shim' from 'origin'.
Switched to a new branch 'remove-distutils-shim'
(env-3.8) root@mg-shin-npu-0:~/pip# pip install .
# Installed
(env-3.8) root@mg-shin-npu-0:~# cd ~/flit-dynamic-pip-22/
(env-3.8) root@mg-shin-npu-0:~/flit-dynamic-pip-22# python --version
Python 3.8.13
(env-3.8) root@mg-shin-npu-0:~/flit-dynamic-pip-22# pip --version
pip 22.3.dev0 from /root/.miniconda3/envs/env-3.8/lib/python3.8/site-packages/pip (python 3.8)
(env-3.8) root@mg-shin-npu-0:~/flit-dynamic-pip-22# pip install .
# SUCCEEDED

Python 3.9

(env-3.9) root@mg-shin-npu-0:~/pip# git branch
  main
* remove-distutils-shim
(env-3.9) root@mg-shin-npu-0:~/pip# pip install .
# Installed
(env-3.9) root@mg-shin-npu-0:~/pip# cd ~/flit-dynamic-pip-22/
(env-3.9) root@mg-shin-npu-0:~/flit-dynamic-pip-22# python --version
Python 3.9.12
(env-3.9) root@mg-shin-npu-0:~/flit-dynamic-pip-22# pip --version
pip 22.3.dev0 from /root/.miniconda3/envs/env-3.9/lib/python3.9/site-packages/pip (python 3.9)
(env-3.9) root@mg-shin-npu-0:~/flit-dynamic-pip-22# pip install .
# SUCCEEDED

I also tested Python 3.10 version just in case, and it worked. Thanks!

JonathanWillitts added a commit to effect-trial/effect-edc that referenced this issue Jul 25, 2022
…es did not run successfully."

when installing effect-form-validators in Python 3.9 environment.

See:
 - pypa/pip#11294 for discussion on issue being introduced in pip 22.2
 - https://github.com/pypa/pip/pull/11298/files for PR with possible fix
paavopere added a commit to paavopere/palabras that referenced this issue Jul 25, 2022
stephane-caron added a commit to qpsolvers/qpsolvers that referenced this issue Jul 26, 2022
ska-kialo added a commit to kialo/stubalyzer that referenced this issue Jul 26, 2022
@ofek
Copy link
Contributor

ofek commented Jul 26, 2022

Hatch is affected too pypa/hatch#371

JonathanWillitts added a commit to effect-trial/effect-form-validators that referenced this issue Jul 27, 2022
…dencies did not run successfully."

when pip installing in Python 3.9 environment.

See:
- pypa/pip#11294 for discussion on issue being introduced in pip 22.2
- pypa/pip#11298 for fix, due to be included in pip 22.2.1 release
stephane-caron added a commit to qpsolvers/qpsolvers that referenced this issue Aug 8, 2022
stephane-caron added a commit to stephane-caron/aiorate that referenced this issue Aug 8, 2022
stephane-caron added a commit to stephane-caron/aiorate that referenced this issue Aug 8, 2022
stephane-caron added a commit to stephane-caron/pink that referenced this issue Aug 8, 2022
stephane-caron added a commit to stephane-caron/pink that referenced this issue Aug 8, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: build logic Stuff related to metadata generation / wheel generation type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants