Support for Python 3.2 broken

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/323
• Originally reported by: @Ch00k
• Originally created at: 2016-04-11T10:45:39.325

I just came across a problem with tox and Python 3.2 that is described here: travis-ci/travis-ci#5485
I am running tox by the ShiningPanda Jenkins plugin which installs tox like this:

#!

[py32] $ /var/lib/jenkins/shiningpanda/jobs/f517a249/tools/bin/python -c "import pip; pip.main();" install --upgrade tox
Collecting tox
  Using cached tox-2.3.1-py2.py3-none-any.whl
Collecting virtualenv>=1.11.2 (from tox)
  Using cached virtualenv-15.0.1-py2.py3-none-any.whl
Collecting py>=1.4.17 (from tox)
  Using cached py-1.4.31-py2.py3-none-any.whl
Collecting pluggy<0.4.0,>=0.3.0 (from tox)
  Using cached pluggy-0.3.1-py2.py3-none-any.whl

This installs virtualenv version 15.0.1 which comes from the tox's setup.py
Looks like changing that to virtualenv<14.0.0 would fix the problem.

[pytoxbot]

Original comment by dahlia

+1 for envcreate_command option.

[pytoxbot]

Original comment by danilobellini

I think tox might help with that job, and that would be useful to many people. I propose 2 new options for tox.ini that would solve this and many other problems.

A lot of people are abandoning Python 3.2 because of its prefix-less-only unicode strings, but I'd argue the same kind of incompatibility might appear in the future, and tox virtual environments should work with as many Python versions as possible, even if tox itself isn't compatible with them.

By now, tox only works with Python 3.2 if the interpreter that runs tox has the tools (pip/setuptools/wheel/virtualenv) in compatible (old) versions. One can do that by hand:

#!shell

virtualenv -p python3.2 ~/venv32 --no-pip --no-setuptools --no-wheel
source ~/venv32/bin/activate
curl https://bootstrap.pypa.io/3.2/get-pip.py | python3.2
pip install tox 'virtualenv<14'
tox

This alternative way to run tox inside a virtual environment is required to run tests with Python 3.2 without downgrading the related packages, but I think it might be simpler. Can't tox add a way to use that custom get-pip in Python 3.2 virtual environments? Something like (supposing curl is in the white-list externals):

#!ini

[testenv:py32]
envcreate_command = virtualenv -p {basepython} {envdir} --no-pip --no-setuptools --no-wheel
prepare_commands =
  curl https://bootstrap.pypa.io/3.2/get-pip.py | python

That prepare_commands would be called with tox --notest, and happens in the virtual environment. That would be useful in a lot of other contexts like segregating the install step in CIs (which might show "error" for installation errors instead "failure"), or running the "non-pip" install steps just once for a faster way of repeating tests locally (the user would need to call tox -r manually to update the environment, but tox might also watch for updates in that parameter). That would also be a way to install easy_install.

The envcreate_command is based on the install_command option, and that's an idea about controlling the command used for virtual environment generation (the step whereby this issue happens). Anyone who doesn't install packages or wishes to use another install command instead of pip probably doesn't need to have it installed. Still not enough for radically different environments (e.g. using docker containers), but that's perhaps that's a step in that direction.

[pytoxbot]

Original comment by @jayvdb

The current version of tox has a list of dependencies which isnt helpful on Python 3.2.

Surprisingly, the classifiers focus on Python 3 support and don't single out Python 2 support.

classifiers=[
            ...
            'Programming Language :: Python',
            'Programming Language :: Python :: 3'],

Given the decision by virtualenv, IMO the current tox should explicitly drop support for Python 3.2, or include a environment marker extra that specifies virtualenv<14 for Python 3.2 only (setup.py already has a similar version specific dependency, and a very nice has_environment_marker_support() for that), with the existing non-specific virtualenv dependency for other versions of Python.

[pytoxbot]

Original comment by @The-Compiler

I don't think it should be tox' job to handle this.

Virtualenv dropped support for Python 3.2, tox works fine with an older virtualenv, but I think it's not tox' job to ensure an older (3.2-compatible) virtualenv is installed.

Alternatively, why not run tox with a newer python?

[hpk42]

I agree with @The-Compiler that py32 support is not tox core concern. You could do a "tox-withpy32" package on pypi which pulls in the right dependencies before installing tox.

[jayvdb]

I agree with @The-Compiler that py32 support is not tox core concern.
You could do a "tox-withpy32" package on pypi which pulls in the right dependencies before installing tox.

Note that tox-travis has done this, and it isnt easy to do properly due to broken environment marker support in older versions of pip and setuptools (and as a result I appreciate why tox core doesnt want to provide the solution for this).

Anyone wanting to do this should steal the code from tox-travis' setup.py, which is ugly but necessary to be effective.

[jaraco]

Perhaps it would make sense for tox to limit its dependency on virtualenv using environment markers. e.g.:

install_requires=[
    'virtualenv<14; python_version=="3.2"',
],

Only thing is, I'm not sure that's compatible with the existing requirement virtualenv >= 1.11.2.

Ideally, the packaging infrastructure would exclude versions of packages that aren't compatible, but that functionality doesn't yet exist.

If an environment marker solution could be devised, would tox consider adopting it?

[obestwalter]

Hi @jaraco I am not sure about this as I have no experience with working with environment markers, but I'll reopen this issue for now to explore if this would be an option.

It would be great If you could make some tests if and how this might work to have a basis for deciding if this could work.

[jaraco]

In a test project, I tried this:

	install_requires=[
		'virtualenv>=1.11',
		'virtualenv<14; python_version=="3.6"',
	],

Installing with easy_install did the right thing, installed virtualenv 13.1.2 on Python 3.6. However, installing with pip installed virtualenv 15.0.1. But that's not too unexpected. pip is designed to interpret version numbers loosely. And since pip installs the requirements in order, the install of "virtualenv>=1.11" satisfies the second requirement.

Swapping the order of the requirements, it does the right thing for pip, installing virtualenv 13.1.2, but fails when installing with easy_install:

$ env/bin/python setup.py develop    
running develop
running egg_info
writing marked.egg-info/PKG-INFO
writing dependency_links to marked.egg-info/dependency_links.txt
writing namespace_packages to marked.egg-info/namespace_packages.txt
writing requirements to marked.egg-info/requires.txt
writing top-level names to marked.egg-info/top_level.txt
writing manifest file 'marked.egg-info/SOURCES.txt'
running build_ext
Creating /Users/jaraco/Dropbox/code/main/marked/env/lib/python3.6/site-packages/marked.egg-link (link to .)
marked 0.1.dev45+nb1133de.d20161223 is already the active version in easy-install.pth

Installed /Users/jaraco/Dropbox/code/main/marked
Processing dependencies for marked==0.1.dev45+nb1133de.d20161223
Searching for virtualenv>=1.11
Reading https://pypi.python.org/simple/virtualenv/
Downloading https://pypi.python.org/packages/d4/0c/9840c08189e030873387a73b90ada981885010dd9aea134d6de30cd24cb8/virtualenv-15.1.0.tar.gz#md5=44e19f4134906fe2d75124427dc9b716
Best match: virtualenv 15.1.0
Processing virtualenv-15.1.0.tar.gz
Writing /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-b3ijlfl6/virtualenv-15.1.0/setup.cfg
Running virtualenv-15.1.0/setup.py -q bdist_egg --dist-dir /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/easy_install-b3ijlfl6/virtualenv-15.1.0/egg-dist-tmp-blr07fy0
warning: no previously-included files matching '*' found under directory 'docs/_templates'
warning: no previously-included files matching '*' found under directory 'docs/_build'
creating /Users/jaraco/Dropbox/code/main/marked/env/lib/python3.6/site-packages/virtualenv-15.1.0-py3.6.egg
Extracting virtualenv-15.1.0-py3.6.egg to /Users/jaraco/Dropbox/code/main/marked/env/lib/python3.6/site-packages
Adding virtualenv 15.1.0 to easy-install.pth file
Installing virtualenv script to /Users/jaraco/Dropbox/code/main/marked/env/bin

Installed /Users/jaraco/Dropbox/code/main/marked/env/lib/python3.6/site-packages/virtualenv-15.1.0-py3.6.egg
error: virtualenv 15.1.0 is installed but virtualenv<14; python_version == "3.6" is required by {'marked'}

That's because easy_install installs requirements in reverse order (opposite of pip), installing virtualenv 15 to satisfy virtualenv>=1.11 then failing when 15 doesn't satisfy the next requirement processed.

So it seems it's not viable in the current environment to provide two separate requirements for the same package.

[jaraco]

Somewhat more verbose, I've found a technique that seems to work.

	install_requires=[
		'virtualenv>=1.11,<14; python_version=="3.6"',
		'virtualenv>=1.11; python_version!="3.6"',
	],

It works with both easy_install (including setup.py develop) and pip install and isn't sensitive to order (because the environment markers are mutually exclusive and only one will be processed).

Environment markers do require newer versions of the tools (setuptools 0.7+ and pip 5.0+ IIRC). I'll prepare a pull request.

[obestwalter]

I have trouble understanding why you use version 3.6 in your tests now. I thought this was about supporting specifically version 3.2.

[jaraco]

Sorry for the confusion. Yes, I'm just using 3.6 as the test bed as I don't have a 3.2 handy. I'm assuming that the technique translates to any Python version.

[jaraco]

I also realized that the pkgspec; marker syntax isn't supported until a much later version of setuptools, so I'll need to use a slightly more convoluted technique for tox.

[jayvdb]

@jaraco, one approach that works is in my previous comment here #323 (comment) ; i.e. https://github.com/ryanhiebert/tox-travis/blob/5c51e4cd738bcfc9cde80940df932d5d8e88c62a/setup.py

[obestwalter]

Yes, I'm just using 3.6 as the test bed as I don't have a 3.2 handy. I'm assuming that the technique translates to any Python version.

ah - o.k. then - :)

Thanks for trying this. I start to realize why this has been rejected in the past as it involves some really ugly hackery ... let's see how it goes this time.