Tests are broken with virtualenv's rewrite (>=20.0.0)

[mgorny]

Environment

• pip version: 20.1.1
• Python version: all (tested 2.7.18, 3.6.10, 3.7.7, 3.8.3)
• OS: Gentoo Linux

Description
After upgrading to virtualenv-20.0.18, all pip tests fail due to missing path_locations. I know you know that but I couldn't find any bug tracking this.

Expected behavior
I would expect pip tests to work with current releases of virtualenv, however crazy that sounds.

How to Reproduce
Change tools/requirements/tests.txt to list plain virtualenv instead of legacy branch. Run tox.

Output

$ sed -i -e '/virtualenv/s:.*:virtualenv:' tools/requirements/tests.txt 
$ tox -e py38 -- -x
GLOB sdist-make: /tmp/pip/setup.py
py38 inst-nodeps: /tmp/pip/.tox/.tmp/package/1/pip-20.2.dev0.zip
^CERROR: got KeyboardInterrupt signal
_____________________________________________________________________ summary _____________________________________________________________________
  py38: commands succeeded
  congratulations :)
^Cmgorny@pomiot /tmp/pip $ ^C
mgorny@pomiot /tmp/pip $ rm -rf .tox/
mgorny@pomiot /tmp/pip $ tox -e py38 -- -x
GLOB sdist-make: /tmp/pip/setup.py
py38 create: /tmp/pip/.tox/py38
py38 installdeps: -r/tmp/pip/tools/requirements/tests.txt
py38 inst: /tmp/pip/.tox/.tmp/package/1/pip-20.2.dev0.zip
py38 installed: apipkg==1.5,appdirs==1.4.4,atomicwrites==1.4.0,attrs==19.3.0,cffi==1.14.0,coverage==5.1,cryptography==2.8,csv23==0.3.1,distlib==0.3.0,execnet==1.7.1,filelock==3.0.12,freezegun==0.3.15,mock==4.0.2,more-itertools==8.3.0,pip @ file:///tmp/pip/.tox/.tmp/package/1/pip-20.2.dev0.zip,pluggy==0.13.1,pretend==1.0.9,py==1.8.1,pycparser==2.20,pytest==3.8.2,pytest-cov==2.8.1,pytest-forked==1.1.3,pytest-rerunfailures==6.0,pytest-timeout==1.3.4,pytest-xdist==1.27.0,python-dateutil==2.8.1,PyYAML==5.3.1,scripttest==1.3,setuptools==42.0.2,six==1.14.0,virtualenv==20.0.20,Werkzeug==0.16.0,wheel==0.33.6
py38 run-test-pre: PYTHONHASHSEED='39579200'
py38 run-test-pre: commands[0] | python -c 'import shutil, sys; shutil.rmtree(sys.argv[1], ignore_errors=True)' /tmp/pip/tests/data/common_wheels
py38 run-test-pre: commands[1] | python /tmp/pip/tools/tox_pip.py wheel -w /tmp/pip/tests/data/common_wheels -r /tmp/pip/tools/requirements/tests-common_wheels.txt
Collecting setuptools>=40.8.0
  Using cached setuptools-46.4.0-py3-none-any.whl (583 kB)
  Saved ./tests/data/common_wheels/setuptools-46.4.0-py3-none-any.whl
Collecting wheel
  Using cached wheel-0.34.2-py2.py3-none-any.whl (26 kB)
  Saved ./tests/data/common_wheels/wheel-0.34.2-py2.py3-none-any.whl
Skipping setuptools, due to already being wheel.
Skipping wheel, due to already being wheel.
py38 run-test: commands[0] | pytest --timeout 300 -x
=============================================================== test session starts ===============================================================
platform linux -- Python 3.8.3, pytest-3.8.2, py-1.8.1, pluggy-0.13.1
rootdir: /tmp/pip, inifile: setup.cfg
plugins: xdist-1.27.0, forked-1.1.3, timeout-1.3.4, rerunfailures-6.0, cov-2.8.1
timeout: 300.0s
timeout method: signal
timeout func_only: False
collected 2006 items                                                                                                                              

tests/functional/test_broken_stdout.py ...                                                                                                  [  0%]
tests/functional/test_cache.py E

===================================================================== ERRORS ======================================================================
________________________________________________________ ERROR at setup of test_cache_dir _________________________________________________________

request = <SubRequest 'virtualenv_template' for <Function 'test_cache_dir'>>
tmpdir_factory = <_pytest.tmpdir.TempdirFactory object at 0x7f9499734910>, pip_src = Path('/tmp/pytest-of-mgorny/pytest-0/pip_src0/pip_src')
setuptools_install = Path('/tmp/pytest-of-mgorny/pytest-0/setuptools0/install'), common_wheels = Path('/tmp/pip/tests/data/common_wheels')

    @pytest.fixture(scope='session')
    def virtualenv_template(request, tmpdir_factory, pip_src,
                            setuptools_install, common_wheels):
    
        if six.PY3 and request.config.getoption('--use-venv'):
            venv_type = 'venv'
        else:
            venv_type = 'virtualenv'
    
        # Create the virtual environment
        tmpdir = Path(str(tmpdir_factory.mktemp('virtualenv')))
>       venv = VirtualEnvironment(
            tmpdir.joinpath("venv_orig"), venv_type=venv_type
        )

tests/conftest.py:303: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/lib/venv.py:31: in __init__
    self._update_paths()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <VirtualEnvironment /tmp/pytest-of-mgorny/pytest-0/virtualenv0/venv_orig>

    def _update_paths(self):
>       home, lib, inc, bin = _virtualenv.path_locations(self.location)
E       AttributeError: module 'virtualenv' has no attribute 'path_locations'

tests/lib/venv.py:35: AttributeError
============================================================= short test summary info =============================================================
ERROR tests/functional/test_cache.py::test_cache_dir
======================================================== 3 passed, 1 error in 6.14 seconds ========================================================
ERROR: InvocationError for command /tmp/pip/.tox/py38/bin/pytest --timeout 300 -x (exited with code 1)
_____________________________________________________________________ summary _____________________________________________________________________
ERROR:   py38: commands failed

[deveshks]

Note that there has been some ongoing efforts in #7698 and #7718 towards moving the test infra to virtualenv-20

[pradyunsg]

Huh, I thought I'd filed a tracking issue for this task, but seems like I hadn't. :)

[pradyunsg]

#8441 should also get a back-link.

[hugovk]

Adding 3.10-dev to the CI (e.g. hugovk@578c9c8) fails because only latest virtualenv 20.4.7 supports Python 3.10 (due to the fix in pypa/virtualenv#2109):

  File "/opt/hostedtoolcache/Python/3.10.0-beta.3/x64/lib/python3.10/site-packages/virtualenv.py", line 1806, in fix_local_scheme
    if sysconfig._get_default_scheme() == "posix_local":
AttributeError: module 'sysconfig' has no attribute '_get_default_scheme'. Did you mean: 'get_default_scheme'?

https://github.com/hugovk/pip/actions/runs/981825354

Python 3.10 beta 3 is out, with 3.10.0 final due on 2021-10-04:

• https://www.python.org/dev/peps/pep-0619/#schedule

[pradyunsg]

Looks like we just got a deadline for finishing this migration. :)

[uranusjr]

I hope we can find enough resource for this to happen. Rewriting the test helpers is a giant time sink and I don’t think we’ll get enough contribution without outside help.

[pradyunsg]

This is now a major annoyance for me, since the legacy virtualenv does not have support for my shiny new Apple Silicon laptop. :/

[pfmoore]

Excellent, that means you're motivated to fix it now 🙂

More seriously, if you do look at this, ping me if you want reviews/help, I took a brief look at it a while ago but didn't have enough time.

[mgorny]

…and now this became even more important since virtualenv-16 doesn't work on Python 3.11 at all:

ERROR: The executable /home/runner/work/pip/pip/.nox/test-3-11/bin/python3.11 is not functioning
ERROR: It thinks sys.prefix is '/opt/hostedtoolcache/Python/3.11.0-beta.1/x64' (should be '/home/runner/work/pip/pip/.nox/test-3-11')
ERROR: virtualenv is not compatible with this system or executable

(I've confirmed with GHA in case it was my install: https://github.com/mgorny/pip/runs/6400925082)

It's also broken with PyPy3.9 but AFAIR pip's tests never worked well with PyPy3.

[pradyunsg]

#11044 (comment) has some more discussion.

[mgorny]

Hmm, it seems that the new releases pass tests with the new virtualenv. Should this be closed now, or is there anything left to do?

[pradyunsg]

#11288 is the relevant PR. We're still using it on < 3.10.

[uranusjr]

It’s quite unlikely we’re ever going to fix <3.10 though, since it needs us to monkey-patch either distutils or virtualenv, which is not really worthwhile IMO (these are just test setup, not actual pip code—not even code to test pip). So I’d say it’s OK to close this anyway.

[mgorny]

Well, maybe I'm doing something wrong but it seems that the test suite passes for me with on py3.9 with the newest virtualenv. That said, 3.9's going out of Gentoo in 3-4 months anyway, so not a big deal for me.

[ichard26]

I'm going to say that this is done. We have better things to focus our resources on than getting the test suite to play nicely with modern virtualenv on Python 3.8 and 3.9.