Can not install editable pip into venv on Windows

[notatallshaw]

Description

It is not possible to install an editable version of pip into a virtual environment on Windows using Python 3.12 and pip 24.0, I have tried this multiple ways on multiple machines now

Expected behavior

It is possible to install an editable version of pip

pip version

24.0

Python version

3.12.3

OS

Windows 10

How to Reproduce

1. Install Python 3.12.3
2. Open terminal powershell in normal user mode
3. cd C:\temp (or similiar)
4. git clone https://github.com/pypa/pip
5. cd pip
6. py -3.12 -m venv .venv
7. .\.venv\Scripts\activate
8. python -m pip install -e .

Output

(.venv) PS C:\temp\pip> python -m pip install -e .
Obtaining file:///C:/temp/pip
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: pip
  Building editable for pip (pyproject.toml) ... done
  Created wheel for pip: filename=pip-24.1.dev0-0.editable-py3-none-any.whl size=9576 sha256=fb94e098d7a3b8a37a51ca7d80e534734358e3ba327d8de1723e8c91d43d2354
  Stored in directory: C:\Users\damia\AppData\Local\Temp\pip-ephem-wheel-cache-xomgbjc4\wheels\ae\13\10\f08f9034efc6531b45d1b33b8e18d1fcd461a050eda59dfdf1
Successfully built pip
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
  Rolling back uninstall of pip
  Moving to c:\temp\pip\.venv\lib\site-packages\pip-24.0.dist-info\
   from C:\temp\pip\.venv\Lib\site-packages\~ip-24.0.dist-info
  Moving to c:\temp\pip\.venv\lib\site-packages\pip\
   from C:\temp\pip\.venv\Lib\site-packages\~ip
  Moving to c:\temp\pip\.venv\scripts\pip.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip.exe
  Moving to c:\temp\pip\.venv\scripts\pip3.12.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip3.12.exe
  Moving to c:\temp\pip\.venv\scripts\pip3.exe
   from C:\Users\damia\AppData\Local\Temp\pip-uninstall-dxzgkapv\pip3.exe
ERROR: Exception:
Traceback (most recent call last):
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\cli\base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\cli\req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\commands\install.py", line 452, in run
                ^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\req\__init__.py", line 72, in install_given_reqs
    requirement.install(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\req\req_install.py", line 856, in install
    install_wheel(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 725, in install_wheel
    _install_wheel(
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 642, in _install_wheel
    generated_console_scripts = maker.make_multiple(scripts_to_generate)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 451, in make_multiple
    filenames.extend(self.make(specification, options))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_internal\operations\install\wheel.py", line 423, in make
    return super().make(specification, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 440, in make
    self._make_script(entry, filenames, options=options)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 339, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 260, in _write_script
    launcher = self._get_launcher('t')
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\temp\pip\.venv\Lib\site-packages\pip\_vendor\distlib\scripts.py", line 419, in _get_launcher
    raise ValueError(msg)
ValueError: Unable to find resource t64.exe in package pip._vendor.distlib

### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).

[notatallshaw]

My workaround is to use uv:

(.venv) PS C:\temp\pip> python -m uv pip install -e .
   Built file:///C:/temp/pip                                                                                                                                    Built 1 editable in 1.86s
Resolved 1 package in 1ms
Installed 1 package in 4ms
 - pip==24.0
 - pip==24.1.dev0
 + pip==24.1.dev0 (from file:///C:/temp/pip)

[notatallshaw]

This might be a duplicate of #11349, but I find this 100% reproducible, it seems like it might be an edge case with pip specifically, and if not for uv I would struggle to test pip on Windows.

[pfmoore]

I can confirm this.

I note the lines

Moving to c:\temp\pip\.venv\lib\site-packages\pip\
   from C:\temp\pip\.venv\Lib\site-packages\~ip

in the "rolling back" section of the output. I wonder if the problem here is that pip renames itself to ~ip as part of the process of uninstalling the old version of pip, but then when it comes to writing the new wrappers, distlib looks for t64.exe relative to __file__, which was calculated before the rename.

I don't know how we'd fix this, to be honest. It seems pretty fundamental that moving the package that you're using while it's running would make it impossible for it to find its own resources. The best I can think of is for distlib to load all of its wrapper executables into memory at import time - or for us to patch distlib to do that, as our use case is almost unique, and I wouldn't be surprised if distlib wasn't willing to do this.

I'll see if I can work up a patch for this.

[pfmoore]

Adding a note here so I don't forget later.

The reproducer is tricky. The version of pip that's being tested is the one that was installed in non-editable mode as part of creating the venv. So when I patch pip, in order to test, I need to:

1. Install the patched version in non-editable mode. Installing in editable mode deosn't work, as editable mode just puts a .pth file in site-packages, and the pip directory never gets renamed.
2. Try to install any version of pip in editable mode. The version of pip you use here is irrelevant, as all that matters is that it triggers the rename of the in-use pip.

Writing a test for this is going to be fun! I suspect I'll be fighting our test infrastructure all the way 🙁

[pfmoore]

OK. I have a test that demonstrates the issue all the way back to Python 3.8. I also have a fix, but it relies on the importlib.resources.files API, which wasn't available in 3.8, so I need to work out how to add 3.8 support (without relying on importlib_resources as that would involve a new vendored dependency just for a weird edge case in a very old Python version...)

I think I know how to fix 3.8, but that will have to wait for tomorrow.