Error while creating Python2.7 virtualenv

[tzulberti]

Issue

Hi. I am trying to create a virtualenv for python 2.7 but it fails:

$ virtualenv --python=/usr/bin/python2.7 /tmp/foobar
RuntimeError: No virtualenv implementation for PythonInfo(spec=CPython2.7.16.final.0-64, exe=/usr/bin/python2.7, platform=linux2, version='2.7.16 (default, Jul 19 2019, 22:59:28) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=UTF-8-None)

But, if I run: /usr/bin/python2.7 it finishes ok.

On the same computer, I try to create a virtualenv without specifing the python version and it creates everyting:

$ virtualenv /tmp/foobar
created virtual environment CPython3.6.10.final.0-64 in 231ms
  creator CPython3Posix(dest=/tmp/foobar, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/tmp/tmp3yblu868/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

I am using the following virtualenv version:

$ virtualenv --version
virtualenv 20.0.7 from /usr/local/lib/python3.6/site-packages/virtualenv/__init__.py

Output of the virtual environment creation

Make sure to run the creation with -vvv --with-traceback:

$ virtualenv --python=/usr/bin/python2.7 -vvv --with-traceback /tmp/foobar 
56 setup logging to NOTSET [DEBUG report:43]
56 could not create app data folder /home/foobar/.local/share/virtualenv due to FileNotFoundError(2, 'No such file or directory') [INFO app_data:54]
67 created temporary app data folder /tmp/tmp19dm6vc0 [DEBUG app_data:32]
68 find interpreter for spec PythonSpec(path=/usr/bin/python2.7) [INFO builtin:44]
68 Attempting to acquire lock 140619915224400 on /tmp/tmp19dm6vc0/py_info/20.0.7/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
68 Lock 140619915224400 acquired on /tmp/tmp19dm6vc0/py_info/20.0.7/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
68 get interpreter info via cmd: /usr/bin/python2.7 /usr/local/lib/python3.6/site-packages/virtualenv/discovery/py_info.py [DEBUG cached_py_info:105]
98 write PythonInfo to /tmp/tmp19dm6vc0/py_info/20.0.7/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python2.7 [DEBUG cached_py_info:91]
99 Attempting to release lock 140619915224400 on /tmp/tmp19dm6vc0/py_info/20.0.7/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
99 Lock 140619915224400 released on /tmp/tmp19dm6vc0/py_info/20.0.7/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
99 proposed PythonInfo(spec=CPython2.7.16.final.0-64, exe=/usr/bin/python2.7, platform=linux2, version='2.7.16 (default, Jul 19 2019, 22:59:28) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=UTF-8-None) [INFO builtin:50]
99 accepted PythonInfo(spec=CPython2.7.16.final.0-64, exe=/usr/bin/python2.7, platform=linux2, version='2.7.16 (default, Jul 19 2019, 22:59:28) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=UTF-8-None) [DEBUG builtin:52]
101 filesystem is case-sensitive [DEBUG info:28]
106 CPython2Posix missing PathRefToDest(src=/usr/lib/python2.7/os.py) [DEBUG via_global_self_do:44]
Traceback (most recent call last):
  File "/usr/local/bin/virtualenv", line 11, in <module>
    sys.exit(run_with_catch())
  File "/usr/local/lib/python3.6/site-packages/virtualenv/__main__.py", line 50, in run_with_catch
    run(args, options)
  File "/usr/local/lib/python3.6/site-packages/virtualenv/__main__.py", line 20, in run
    session = cli_run(args, options)
  File "/usr/local/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 27, in cli_run
    session = session_via_cli(args, options)
  File "/usr/local/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 35, in session_via_cli
    parser = build_parser(args, options)
  File "/usr/local/lib/python3.6/site-packages/virtualenv/run/__init__.py", line 75, in build_parser
    CreatorSelector(interpreter, parser),
  File "/usr/local/lib/python3.6/site-packages/virtualenv/run/plugin/creators.py", line 17, in __init__
    raise RuntimeError("No virtualenv implementation for {}".format(interpreter))
RuntimeError: No virtualenv implementation for PythonInfo(spec=CPython2.7.16.final.0-64, exe=/usr/bin/python2.7, platform=linux2, version='2.7.16 (default, Jul 19 2019, 22:59:28) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]', encoding_fs_io=UTF-8-None)

But if I check the output of the os module it is at:

$ python2.7
Python 2.7.16 (default, Jul 19 2019, 22:59:28) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.__file__
'/usr/lib64/python2.7/os.pyc'

[gaborbernat]

Interesting, can you reproduce this with a CentOS docker image? Does RHEL packages python differently? @vstinner can you provide some insight into this?

@tzulberti can you also post the content of /tmp/tmp19dm6vc0/py_info/20.0.7/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json?

[tzulberti]

this is the content of the json file:

{
  "st_mtime": 1563577194.0,
  "path": "/usr/bin/python2.7",
  "content": {
    "base_exec_prefix": null,
    "system_stdlib": "/usr/lib/python2.7",
    "has_venv": false,
    "prefix": "/usr",
    "stdout_encoding": null,
    "executable": "/usr/bin/python2.7",
    "implementation": "CPython",
    "exec_prefix": "/usr",
    "platform": "linux2",
    "version": "2.7.16 (default, Jul 19 2019, 22:59:28) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]",
    "sysconfig_paths": {
      "platstdlib": "{platbase}/lib64/python{py_version_short}",
      "platlib": "{platbase}/local/lib64/python{py_version_short}/site-packages",
      "purelib": "{base}/local/lib/python{py_version_short}/site-packages",
      "stdlib": "{base}/lib/python{py_version_short}",
      "scripts": "{base}/local/bin",
      "include": "{base}/local/include/python{py_version_short}",
      "data": "{base}/local"
    },
    "base_prefix": null,
    "_creators": null,
    "file_system_encoding": "UTF-8",
    "version_info": {
      "major": 2,
      "minor": 7,
      "micro": 16,
      "releaselevel": "final",
      "serial": 0
    },
    "sysconfig_vars": {
      "base": "/usr",
      "platbase": "/usr",
      "PYTHONFRAMEWORK": "",
      "py_version_short": "2.7"
    },
    "path": [
      "/usr/local/lib/python3.6/site-packages/virtualenv/discovery",
      "/usr/lib/python27.zip",
      "/usr/lib64/python2.7",
      "/usr/lib64/python2.7/plat-linux2",
      "/usr/lib64/python2.7/lib-tk",
      "/usr/lib64/python2.7/lib-old",
      "/usr/lib64/python2.7/lib-dynload",
      "/usr/local/lib64/python2.7/site-packages",
      "/usr/local/lib/python2.7/site-packages",
      "/usr/local/lib/python2.7/site-packages/StreamPublisher-1.3.2-py2.7.egg",
      "/usr/lib64/python2.7/site-packages",
      "/usr/lib/python2.7/site-packages",
      "/usr/lib64/python2.7/dist-packages",
      "/usr/lib64/python2.7/dist-packages/PIL",
      "/usr/lib/python2.7/dist-packages"
    ],
    "max_size": 9223372036854775807,
    "real_prefix": null,
    "distutils_install": {
      "purelib": "lib/python2.7/dist-packages",
      "headers": "include/python2.7/UNKNOWN",
      "platlib": "lib64/python2.7/dist-packages",
      "data": "",
      "scripts": "bin"
    },
    "architecture": 64,
    "original_executable": "/usr/bin/python2.7",
    "os": "posix",
    "system_executable": "/usr/bin/python2.7"
  }
}

I will check if I can create a Docker file to recreate the issue

[gaborbernat]

Here's the part which does not seem to add up:

"sysconfig_paths": {
      "stdlib": "{base}/lib/python{py_version_short}",
    },
    "sysconfig_vars": {
      "base": "/usr",
      "platbase": "/usr",
      "PYTHONFRAMEWORK": "",
      "py_version_short": "2.7"
    },

So this tells us that the standard library (of which os.py is part of) should live under {base}/lib/python{py_version_short} which translates to /usr/lib/python2.7 not /usr/lib64/python2.7 🤔

[gaborbernat]

I see the bug here, seems platstdlib should also be checked.

[gaborbernat]

@tzulberti can you give #1697 a try?

[gaborbernat]

Hello, a fix for this issue has been released via virtualenv 20.0.9; see https://pypi.org/project/virtualenv/20.0.9/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-9-2020-03-08). Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

[tzulberti]

No, thank you for improving the python environment for all of us. I will test this tomorrow

[tzulberti]

@gaborbernat Thank you. I tested virtualenv 20.0.10 and I could create the virtualenv. Thank you again for all your work

[gaborbernat]

You're welcome 👍thanks for the detailed error report and testing.