pip should not destroy instead of upgrade itself on Windows

[chrullrich]

Environment

• pip version: Any
• Python version: Any
• OS: Windows

Description
pip upgrade pip in a venv on Windows will inevitably end in failure to delete the pip.exe wrapper, which is a running process. The next line of output uselessly suggests using --user. The correct approach is to run python -m pip install -U pip instead, but the user is not told this, and at the time of the error the pip installation is already gone anyway.

There are two ways to recover, the faster and far less discoverable one is python -m ensurepip, the other is to nuke the venv and start over.

Upgrading pip in the same way in a system-wide installation also throws the error, but appears to successfully upgrade itself first. I suspect it upgrades the Python bits of itself first and only then fails to recreate the executable wrapper.

Expected behavior
pip should probably do the same in a venv that it does in the system-wide installation, i.e. upgrade its innards first, then fail to update the executable wrapper (because it is somewhat version-independent). If that is not possible, at least it should not uninstall (most of) itself before failing, but exit early and with a comprehensible error message that points out the correct command to run.

(Better yet, it should re-run itself using the correct command line, but that interferes with returning the correct exit code. Oh, exec(), how we miss you.)

How to Reproduce

py -m venv venv
venv\scripts\activate
pip install -U pip

Output:

Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'c:\\daten\\venv\\scripts\\pip.exe'
Consider using the `--user` option or check the permissions.

[uranusjr]

pip 20.3 and later no longer fail in an unusable state. It still suggests --user (which should be corrected).

[uranusjr]

> py -m venv v

> v\Scripts\python -m pip install -q -U pip

> v\Scripts\pip --version
pip 20.3.3 from \v\lib\site-packages\pip (python 3.9)

> v\Scripts\pip install "pip!=20.3.3"
Collecting pip!=20.3.3
  Using cached pip-20.3.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.3.3
    Uninstalling pip-20.3.3:
      Successfully uninstalled pip-20.3.3
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: '\\AppData\\Local\\Temp\\pip-uninstall-l31vxw52\\pip.exe'
Consider using the `--user` option or check the permissions.

> v\Scripts\pip --version
pip 20.3.1 from \v\lib\site-packages\pip (python 3.9)

[chrullrich]

After some thinking I was able to reproduce this. It only works if the venv is on the same volume as %TEMP%.

It does not work if the venv is:

• on a mounted volume under the same root as %TEMP%
• on a different drive letter

Moving files across volumes does not work on Windows, of course, and procmon shows that whatever method pip uses to move the wrapper instead tries copy-then-delete, and pip does not even notice it is setting itself up to fail.

[dfidler]

pip 20.3 and later no longer fail in an unusable state. It still suggests --user (which should be corrected).

If you aren't running the terminal as Administrator, this still happens... new issue or should I track it under this one?

pip 21.0.1

C:\projects\personal\Python\sample>python -m pip install pip --upgrade
Requirement already satisfied: pip in c:\python39\lib\site-packages (21.0.1)
Collecting pip
  Downloading pip-21.1.3-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 1.6 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.0.1
    Uninstalling pip-21.0.1:
ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'c:\\python39\\scripts\\pip.exe'
Consider using the `--user` option or check the permissions.

C:\projects\personal\Python\sample>where python
C:\Python39\python.exe

C:\projects\personal\Python\sample>python -m pip install pip --upgrade
C:\Python39\python.exe: No module named pip

[chrullrich]

I do not think this is the same bug. This one is about pip install pip, but you are already using python -m pip install pip instead, which should work.

That said, I'm not sure what is going on there. You obviously have write permission on C:\Python39\lib (because pip can damage itself), and C:\Python39\scripts\pip.exe is not involved in the operation. It should be possible to delete it (unless it was running at the time for some other reason). Did you by any chance give the user you are running as write permission on lib, but not on scripts? (In the pip install pip case, the "access denied" error is because the program is running. Here it may just be actual missing permissions instead.)

[baterflyrity]

Hi there. Here is a fix at #10454.

Another quick fix is to run the command in administrator shell/user that have write access to c:\daten\venv\scripts\ as stated in the error message.

[chrullrich]

Hi there. Here is a fix at #10454.

Another quick fix is to run the command in administrator shell/user that have write access to _c:\daten\venv\scripts_ as stated in the error message.

Hello, unfortunately neither your PR nor your suggestion of admin rights will fix this issue. It has nothing to do with file permissions, not least because (as you can clearly see in my first example) I create the venv immediately before attempting to upgrade pip. Obviously I have all the permissions I need on the files in the venv.

The problem here is that the venv is on a different volume from the executing user's %TEMP%, and pip does not check for that. It attempts to move the pip.exe wrapper to %TEMP% so it can create a new one for the updated version, but Windows does not let you move (e.g. os.rename()) a file to a different volume. (The file is also in use, which might break the move as well, but it fails as soon as the filesystem notices that the destination is not on the same volume.)

I suspect that pip uses existing library code, such as shutil.move(), to do the move, and that code falls back to copy-then-delete (which shutil.move() does) because it is not aware that this is a situation where that does not work. Copy-then-delete is fine if your goal is to have the file in the other location (it will be there even if deleting the original fails), but here the point is to not have it in the original directory. The copy is successful, and the delete fails with "access denied" because the file is still in use by the running pip itself.

There are two possible fixes that I can imagine. One is to rename the wrapper in the same directory rather than attempt to move it to %TEMP%; renaming an open file is possible. The other is to check whether %TEMP% is on the same volume, and if it is not, bail out with a useful error message that directs the user to python -m pip. And in either case, do it early enough that the existing pip installation survives the experience.

[baterflyrity]

@chrullrich thanks for clarification of your case. I think i will make further research about access problems.