Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

upgrade from 22.2 to 22.2.1. fails with ImportError for name '_distutils' #11319

Closed
1 task done
mrclmh opened this issue Jul 28, 2022 · 13 comments · Fixed by #11324
Closed
1 task done

upgrade from 22.2 to 22.2.1. fails with ImportError for name '_distutils' #11319

mrclmh opened this issue Jul 28, 2022 · 13 comments · Fixed by #11324
Labels
type: bug A confirmed bug or unintended behavior
Milestone

Comments

@mrclmh
Copy link

mrclmh commented Jul 28, 2022

Description

When trying to upgrade from 22.2 to 22.2.1, the install fails because of an ImportError.
The setuptools version is 63.2.0.

Traceback (most recent call last):
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/commands/install.py", line 461, in run
    installed = install_given_reqs(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/req_install.py", line 752, in install
    scheme = get_scheme(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py", line 244, in get_scheme
    from . import _distutils
ImportError: cannot import name '_distutils' from 'pip._internal.locations' (/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py)

Expected behavior

Pip should be upgraded from 22.2 to 22.2.1

pip version

22.2

Python version

3.9.13

OS

macOS 10.15.7

How to Reproduce

With version 22.2 user-installed:

python3 -m pip install --user --upgrade pip

Output

% python3 -m pip install --user --upgrade pip    
Requirement already satisfied: pip in ./Library/Python/3.9/lib/python/site-packages (22.2)
Collecting pip
  Using cached pip-22.2.1-py3-none-any.whl (2.0 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2
    Uninstalling pip-22.2:
      Successfully uninstalled pip-22.2
  Rolling back uninstall of pip
  Moving to /Users/mm/Library/Python/3.9/bin/pip
   from /private/var/folders/v_/3yqdc_t15jjb82bnmh1366ym0000gn/T/pip-uninstall-7x3mha7f/pip
  Moving to /Users/mm/Library/Python/3.9/bin/pip3
   from /private/var/folders/v_/3yqdc_t15jjb82bnmh1366ym0000gn/T/pip-uninstall-7x3mha7f/pip3
  Moving to /Users/mm/Library/Python/3.9/bin/pip3.9
   from /private/var/folders/v_/3yqdc_t15jjb82bnmh1366ym0000gn/T/pip-uninstall-7x3mha7f/pip3.9
  Moving to /Users/mm/Library/Python/3.9/lib/python/site-packages/pip-22.2.dist-info/
   from /Users/mm/Library/Python/3.9/lib/python/site-packages/~ip-22.2.dist-info
  Moving to /Users/mm/Library/Python/3.9/lib/python/site-packages/pip/
   from /Users/mm/Library/Python/3.9/lib/python/site-packages/~ip
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/commands/install.py", line 461, in run
    installed = install_given_reqs(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/req/req_install.py", line 752, in install
    scheme = get_scheme(
  File "/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py", line 244, in get_scheme
    from . import _distutils
ImportError: cannot import name '_distutils' from 'pip._internal.locations' (/Users/mm/Library/Python/3.9/lib/python/site-packages/pip/_internal/locations/__init__.py)

Code of Conduct

@mrclmh mrclmh added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Jul 28, 2022
@mrclmh mrclmh changed the title upgrade from 22.2 to 22.2.1. fails upgrade from 22.2 to 22.2.1. fails with ImportError for name '_distutils' Jul 28, 2022
@uranusjr
Copy link
Member

Does installing anything else (not pip) work?

@mrclmh
Copy link
Author

mrclmh commented Jul 28, 2022

Yes. A test install of numpy, pandas, seaborn works.

@uranusjr
Copy link
Member

Interesting. I wonder if for all our effort to make the distutils import lazy, we actually made it too lazy that it’s not imported in time, before the file is moved out of place.

@zhihaoy
Copy link

zhihaoy commented Jul 29, 2022

Same problem on Windows:

ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\cli\base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\cli\req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\commands\install.py", line 461, in run
    installed = install_given_reqs(
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\req\__init__.py", line 73, in install_given_reqs
    requirement.install(
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\req\req_install.py", line 752, in install
    scheme = get_scheme(
  File "C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\locations\__init__.py", line 244, in get_scheme
    from . import _distutils
ImportError: cannot import name '_distutils' from 'pip._internal.locations' (C:\Users\[...]\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pip\_internal\locations\__init__.py)

@sbidoul
Copy link
Member

sbidoul commented Jul 29, 2022

Can someone provide a reproducer for this ?

@gurnec
Copy link

gurnec commented Jul 29, 2022

@sbidoul:

  1. Spin up an instance of Ubuntu 20.04 on your favorite VPS provider, or see below.

  2. The VPS provider I chose does an immediate update/upgrade, I don't know if this is necessary, but to keep things consistent:

     apt update
     apt upgrade
     reboot
    
  3. Run:

     pip3 install --upgrade --user pip==22.2
     ~/.local/bin/pip3 install --upgrade --user pip
    

Here's a VPS you can test with, just skip straight to step 3 on it:

IP: 144.202.6.36
username: pip

Please give me an SSH pubkey for the account above, or email me for a password.

@sbidoul
Copy link
Member

sbidoul commented Jul 29, 2022

@gurnec thanks, I could reproduce now with your procedure.

@gurnec
Copy link

gurnec commented Jul 29, 2022

@sbidoul You may have already figured this out, but upgrading pip from 22.1.2 to 22.2.1 works fine, it's only 22.2 → 22.2.1 which generates an ImportError.

@sbidoul
Copy link
Member

sbidoul commented Jul 29, 2022

As @uranusjr suspected above, the error is due to _distutils.py being imported lazily after it was removed by the uninstall step.
#11324 should fix this.

@gurnec your reproducer was helpful, thanks a lot. It would be fantastic if you could test the proposed fix.

@sbidoul sbidoul added this to the 22.2.2 milestone Jul 29, 2022
@sbidoul sbidoul removed the S: needs triage Issues/PRs that need to be triaged label Jul 29, 2022
@fossum
Copy link

fossum commented Jul 29, 2022

A reinstall fixed it for me.

python -m pip uninstall -y pip
python -m pip install --user --upgrade pip

@gurnec
Copy link

gurnec commented Jul 29, 2022

@sbidoul It mostly worked for me (22.2.1a is the patched version):

1. 22.2    → 22.2.1  from pypi:   failed
2. 22.2    → 22.2    from pypi:   failed
3. 22.2    → 22.2.1  from source: unexpectedly worked
4. 22.2.1  → 22.2    from pypi:   failed
5. 22.2.1  → 22.2.1  from pypi:   failed
6. 22.2.1  → 22.2.1  from source: unexpectedly worked
7. 22.2.1  → 22.2.1a from source: worked
8. 22.2.1a → 22.2.1  from pypi:   worked
9. 22.2.1a → 22.2    from pypi:   worked
A. 22.2.1a → 22.2.1a from source: worked

In particular, 8 & 9 worked, which I assumes means that this is fixed. 7 & A also worked, but it's not a good test because 3 & 6 didn't fail for me—something about installing from source causes the bug to not manifest for me.

@sbidoul
Copy link
Member

sbidoul commented Jul 30, 2022

Reinstalling from source works because the metadata preparation steps imports _distutils before the uninstall step, so it is in memory when reinstalling.

So a workaround for people affected by this is to reinstall pip from source:

python -m pip install --user --upgrade https://github.com/pypa/pip/archive/refs/tags/22.2.1.zip

@sbidoul
Copy link
Member

sbidoul commented Jul 30, 2022

Thanks for the report, reproducer and tests!

inmantaci pushed a commit to inmanta/inmanta-core that referenced this issue Aug 4, 2022
Bumps [pip](https://github.com/pypa/pip) from 22.2.1 to 22.2.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>22.2.2 (2022-08-03)</h1>
<h2>Bug Fixes</h2>
<ul>
<li>Avoid  <code>AttributeError</code> when removing the setuptools-provided <code>_distutils_hack</code> and it is missing its implementation. (<code>[#11314](pypa/pip#11314) &lt;https://github.com/pypa/pip/issues/11314&gt;</code>_)</li>
<li>Fix import error when reinstalling pip in user site. (<code>[#11319](pypa/pip#11319) &lt;https://github.com/pypa/pip/issues/11319&gt;</code>_)</li>
<li>Show pip deprecation warnings by default. (<code>[#11330](pypa/pip#11330) &lt;https://github.com/pypa/pip/issues/11330&gt;</code>_)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/10902ac52d49ec3f5a03a4ed67c4dee533bf8bc4"><code>10902ac</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/534e4d23386f43114f3deed65b7ebdc821aacb4a"><code>534e4d2</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/96675e90cc9b51d3ab2c4f33f8e5e7f2438dfb91"><code>96675e9</code></a> Do not parse JSON content with HTML parser</li>
<li><a href="https://github.com/pypa/pip/commit/4514d1f47300e6df8ad43b309811c876287f8f03"><code>4514d1f</code></a> Revert &quot;PipDeprecationWarning subclass DeprecationWarning&quot;</li>
<li><a href="https://github.com/pypa/pip/commit/1ac427b647b276acd92def9e6bfa223c0d74d103"><code>1ac427b</code></a> Remove TODO</li>
<li><a href="https://github.com/pypa/pip/commit/450fd26c2d4f97284eccb1e6a6f81e9a460b17af"><code>450fd26</code></a> Import distutils only if needed, but sooner</li>
<li><a href="https://github.com/pypa/pip/commit/0c9eb969ebc833454468e16a5fd8fe90548d4c75"><code>0c9eb96</code></a> Ensure that removing shim in older setuptools does not error</li>
<li>See full diff in <a href="https://github.com/pypa/pip/compare/22.2.1...22.2.2">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.2.1&new-version=22.2.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
inmantaci pushed a commit to inmanta/inmanta-core that referenced this issue Aug 4, 2022
Bumps [pip](https://github.com/pypa/pip) from 22.2.1 to 22.2.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>22.2.2 (2022-08-03)</h1>
<h2>Bug Fixes</h2>
<ul>
<li>Avoid  <code>AttributeError</code> when removing the setuptools-provided <code>_distutils_hack</code> and it is missing its implementation. (<code>[#11314](pypa/pip#11314) &lt;https://github.com/pypa/pip/issues/11314&gt;</code>_)</li>
<li>Fix import error when reinstalling pip in user site. (<code>[#11319](pypa/pip#11319) &lt;https://github.com/pypa/pip/issues/11319&gt;</code>_)</li>
<li>Show pip deprecation warnings by default. (<code>[#11330](pypa/pip#11330) &lt;https://github.com/pypa/pip/issues/11330&gt;</code>_)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/10902ac52d49ec3f5a03a4ed67c4dee533bf8bc4"><code>10902ac</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/534e4d23386f43114f3deed65b7ebdc821aacb4a"><code>534e4d2</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/96675e90cc9b51d3ab2c4f33f8e5e7f2438dfb91"><code>96675e9</code></a> Do not parse JSON content with HTML parser</li>
<li><a href="https://github.com/pypa/pip/commit/4514d1f47300e6df8ad43b309811c876287f8f03"><code>4514d1f</code></a> Revert &quot;PipDeprecationWarning subclass DeprecationWarning&quot;</li>
<li><a href="https://github.com/pypa/pip/commit/1ac427b647b276acd92def9e6bfa223c0d74d103"><code>1ac427b</code></a> Remove TODO</li>
<li><a href="https://github.com/pypa/pip/commit/450fd26c2d4f97284eccb1e6a6f81e9a460b17af"><code>450fd26</code></a> Import distutils only if needed, but sooner</li>
<li><a href="https://github.com/pypa/pip/commit/0c9eb969ebc833454468e16a5fd8fe90548d4c75"><code>0c9eb96</code></a> Ensure that removing shim in older setuptools does not error</li>
<li>See full diff in <a href="https://github.com/pypa/pip/compare/22.2.1...22.2.2">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.2.1&new-version=22.2.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants