Using non-existent environment does not fail

[Mr-Pepe]

Issue

Using an environment that does not exist still executes without failure but does nothing useful.

Environment

Provide at least:

• OS: Linux
• pip list of the host Python where tox is installed:

Package       Version
------------- -------
cachetools    5.2.1
chardet       5.1.0
colorama      0.4.6
distlib       0.3.6
filelock      3.9.0
packaging     23.0
pip           22.3.1
platformdirs  2.6.2
pluggy        1.0.0
pyproject_api 1.4.0
setuptools    65.7.0
tomli         2.0.1
tox           4.2.8
virtualenv    20.17.1
wheel         0.38.4

Minimal example

If possible, provide a minimal reproducer for the issue:

tox run -e abcdefg

[jugmac00]

Looks like this is a regression as with tox v3...

❯ tox -e abcdefgh
ERROR: unknown environment 'abcdefgh'

[gaborbernat]

This is expected 😊 and feature with tox 4.

[jugmac00]

I was not specific enough. Above output is from tox version 3.

tox version 4 falsly runs tests for an unknown env.

So...
tox3 -> shows error message that env is unknown
tox4 -> runs tests for the version tox was installed with

This is not intended, right?

[gaborbernat]

It is intended. I refer to it as the on-demand definition of new tox environments. Allows you to test never python versions without defining them in the config.

tox4 -> runs tests for the version tox was installed with

Well, that depends on the config, 😊 you might define basepython in testenv, and in that case will run with that python.

[jugmac00]

While I am happy to allow this for e.g. tox -e py313, but I think this is dangerous for e.g. tox -e lynt (ie with a typo).

[gaborbernat]

There's no easy way to allow just one and not the other. I am mainly considering factors like a-py313-ok. But feel free to put in a PR to try.

[Mr-Pepe]

I bumped into this issue when I removed an environment from the tox.ini but did not remove the call from my pipeline script. I would have expected the pipeline to fail and was really confused when it didn't. Is this behavior documented?

[gaborbernat]

It's undocumented in either direction.

[bmw]

This may not matter unless this issue is worked on, but I'm even seeing some strange behavior here with tox 3.28.0. For this tox.ini file:

[tox]
skipsdist = true

[testenv]
allowlist_externals = echo
commands =
    echo do pytest things

[testenv:coverage]
commands =
    echo do coverage things

I get this:

$ tox -e py39-coverage
ERROR: unknown environment 'py39-coverage'

If I add a value for envlist to the above file, this no longer happens. For

[tox]
envlist = py,coverage
skipsdist = true

[testenv]
allowlist_externals = echo
commands =
    echo do pytest things

[testenv:coverage]
commands =
    echo do coverage things

I get:

$ tox -e py39-coverage
py39-coverage create: /private/var/folders/tr/_jqrlt554c107b5lcxl3td_40000gn/T/tmp.LCUaVCXRI1/.tox/py39-coverage
py39-coverage run-test-pre: PYTHONHASHSEED='1933119295'
py39-coverage run-test: commands[0] | echo do pytest things
do pytest things
________________________________________________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________________________________________________
  py39-coverage: commands succeeded
  congratulations :)

This behavior resulted in the CI config for a project I work on not running the tests we expect for months. I suspect many other people are unknowingly bitten by behavior like this. I think more sanity checking on environment names could provide a lot of value.

[gaborbernat]

This is a trade off between flexibility of on demand envs and too much validation.

[manueljacob]

A workaround could be to add a default testenv that always fails:

[testenv]
commands =
    false

This requires the actual test commands to be in a section like [testenv:{py39,py310,py311}].

I agree with some people earlier in this issue that it would be a better default tradeoff if only py* environments but no other environments were created implicitly.

[bmw]

Expanding on the above a bit, the approach I'm currently using is:

...
whitelist_externals =
    echo
    false
# This and the next few testenvs are a workaround for
# https://github.com/tox-dev/tox/issues/2858.
commands =
    echo "Unrecognized environment name {envname}"
    false

[testenv:py]
commands = do whatever

[testenv:py3{,7,8,9,10,11}]
commands = {[testenv:py]commands}

[testenv:py3.{7,8,9,10,11}]
commands = {[testenv:py]commands}
...

[gaborbernat]

I agree with some people earlier in this issue that it would be a better default tradeoff if only py* environments but no other environments were created implicitly.

PR welcome 👍

[bmw]

I've been looking at this a little bit. I see the code that handled this in 3.28.0, but it's not immediately clear to me where the equivalent code should go with the 4.0 refactoring. Any pointers would be much appreciated.

[jamesbraza]

Hit this today, and was initially surprised that passing an unknown recipe tox run -e complete-typo lead to tox passing.

From reading the above, I see now it provides a back door to testing on newer Python versions.

A few ideas here:

• Provide a bool config option to turn on strict recipe matching
• Issuing a warning if the recipe doesn't match one from tox list and didn't use a recipe's back door

[benhoyt]

For the record, this trips me up all the time. Some projects I work on have a "static" testenv, others call it "static-charm" or "pyright" or similar. When I type tox -e static and it succeeds, giving me some nice green text, I assume it's run the static type checking and succeeded. Sometimes I realize, "boy, that was fast, I wonder if it did anything?" but sometimes I don't.

[tjsmart]

I opened a PR following the suggestion to disallow implicit environment creation outside of the special case for py*. It's my first attempt at contributing to tox 😄, so won't be surprised if I've overlooked something. My team has hit this issue a few times and I'd love to be able to run tox with stricter environment creation and generally don't use the implicit creation of environments.

I'm curious if it would be better to create a configuration option to opt in to this behavior since this change seems like it could break backwards compatibility for some.

Suggestions for improvement are welcomed 👍.

[markferry]

Ah nuts. We were relying on this behaviour. (ALL THE CI THINGS broke today...)

In particular it:

• allowed us to extend a matrix of builds without changing the tox.ini
• allowed devs to just run tox -e test without having to specify the tags

For example:

[tox]
minversion = 3.24
envlist =
    test
    lint
isolated_build = True
skip_missing_interpreters = True

[testenv]
commands =
    test: python -m pytest --junitxml=test-reports/pytest.{envname}.xml --cov-report xml:test-reports/coverage.{envname}.xml {posargs}
    lint-!py27: pre-commit run --all-files
    lint-py27: pre-commit run --all-files --config .pre-commit-config.27.yaml

(pseudo) Jenkinsfile:

  dynamicMatrix(
      [
          platform: ['linux','windows'],
          python: ['27', '38', '310'],    // add python versions here
      ]) {
        ...
        stage(cell) {
            sh label: "test", script: "tox -e test-py${python}-${platform}"
            sh label: "lint", script: "tox -e lint-py${python}-${platform}"
        }
    }

While I understand the argument for rejecting tox -e garbage, this fix now also rejects tox -e test-py311 which I'd consider a legitimate environment (assuming the interpreter can be found).

Now I guess the alternative is we write:

[tox]
minversion = 4.9
envlist =
    {test,lint}-py{27,38,310,311}-{linux,windows}

And retrain those who are used to running tox -e test.

Is there a better way?

[gaborbernat]

While I understand the argument for rejecting tox -e garbage, this fix now also rejects tox -e test-py311 which I'd consider a legitimate environment (assuming the interpreter can be found).

PR welcome to accept this.

[markferry]

I found a workaround, which is to put the py tag first.

And then realised that, humorously, this means that tox -e py311-garbage still runs an (empty) testenv and passes.

@tjsmart

[tjsmart]

A couple thoughts:

1. We could add an option to the configuration (something along the lines of explicit_environment_creation: bool) which would default to False which is essentially the pre-existing tox 4 behavior. When set to True, all tox environment must be explicitly specified in the configuration file (even things like py310 or test-py310 must be explicitly added).
2. There may be opportunity to improve the current "hybrid" solution, but it seems problematic.