Disallow command line environments which are not explicitly specified in the config file #3089

tjsmart · 2023-08-13T21:48:06Z

Checks that all command line provided environments correspond with an environment in the configuration file.

This does not apply to configurations starting with py* or .pkg (I don't know what the latter is perhaps it shouldn't be special cased).

I'd love to update/extend the documentation but could use some help finding a home for this.

Fixes: #2858.

ran the linter to address style issues (tox -e fix)
wrote descriptive pull request text
ensured there are test(s) validating the fix
added news fragment in docs/changelog folder
updated/extended the documentation

…g file Special exception is made for 'py*' which allows implicit creation of environments.

tjsmart · 2023-08-13T22:12:21Z

tests/session/cmd/test_parallel.py

@@ -141,7 +141,7 @@ def test_keyboard_interrupt(tox_project: ToxProjectCreator, demo_pkg_inline: Pat
    )
    cmd = ["-c", str(proj.path / "tox.ini"), "p", "-p", "1", "-e", f"py,py{sys.version_info[0]},dep"]
    process = Popen([sys.executable, "-m", "tox", *cmd], stdout=PIPE, stderr=PIPE, universal_newlines=True)
-    while not marker.exists():
+    while not marker.exists() and (process.poll() is None):


While in a bit of broken state, this test was hanging for me. Suggesting that we use Popen.poll to check if the child process has already terminated (likely with an error if marker does not exist).

gaborbernat

gaborbernat

Generraly the idea looks good, but you need to fix the CI.

gaborbernat

Signed-off-by: Bernát Gábor <bgabor8@bloomberg.net>

atugushev · 2023-08-17T20:41:58Z

@tjsmart FYI, I've reported a bug #3096, which might be related to this PR.

# Description Test of examples starts failing as tox no longer discover tox.ini in a subfolder and/or do not allow non-defined environments tox-dev/tox#3089

hroncok · 2023-08-30T17:51:09Z

tests/session/test_env_select.py

+    assert "provided environments not found in configuration file: ['does_not_exist']" in outcome.out, outcome.out
+
+
+@pytest.mark.parametrize("env_name", ["py", "py310", ".pkg"])


test_allowed_implicit_cli_envs[py310] now fails when Python 3.10 is not installed. Is that expected, or should I open an issue?

PR welcome, likely should be pyx.y where x.y is the current Python interpreter.

…on version See tox-dev#3089 (review)

…on version See tox-dev#3089 (review) test_allowed_implicit_cli_envs[py310] assumed Python 3.10 is installed. It failed without it: ____________________ test_allowed_implicit_cli_envs[py310] _____________________ [gw3] linux -- Python 3.12.0 /usr/bin/python3 env_name = 'py310' tox_project = <function init_fixture.<locals>._init at 0x7febd4e1ca40> @pytest.mark.parametrize("env_name", ["py", "py310", ".pkg"]) def test_allowed_implicit_cli_envs(env_name: str, tox_project: ToxProjectCreator) -> None: proj = tox_project({"tox.ini": ""}) outcome = proj.run("r", "-e", env_name) > outcome.assert_success() env_name = 'py310' outcome = code: -1 cmd: /usr/bin/python3 -m tox r -e py310 cwd: /tmp/pytest-of-mockbuild/pytest-0/popen-gw3/test_allowed_implici...uld not find python interpreter with spec(s): py310 py310: SKIP (0.23 seconds) evaluation failed :( (0.27 seconds) proj = ToxProject(path=/tmp/pytest-of-mockbuild/pytest-0/popen-gw3/test_allowed_implicit_cli_envs1/p) at 140650867026656 tox_project = <function init_fixture.<locals>._init at 0x7febd4e1ca40> tests/session/test_env_select.py:158: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = code: -1 cmd: /usr/bin/python3 -m tox r -e py310 cwd: /tmp/pytest-of-mockbuild/pytest-0/popen-gw3/test_allowed_implici...uld not find python interpreter with spec(s): py310 py310: SKIP (0.23 seconds) evaluation failed :( (0.27 seconds) def assert_success(self) -> None: > assert self.success, repr(self) # noqa: S101 E AssertionError: code: -1 E cmd: /usr/bin/python3 -m tox r -e py310 E cwd: /tmp/pytest-of-mockbuild/pytest-0/popen-gw3/test_allowed_implicit_cli_envs1/p E standard output E py310: skipped because could not find python interpreter with spec(s): py310 E py310: SKIP (0.23 seconds) E evaluation failed :( (0.27 seconds) E E assert False E + where False = code: -1\ncmd: /usr/bin/python3 -m tox r -e py310\ncwd: /tmp/pytest-of-mockbuild/pytest-0/popen-gw3/test_allowed_implicit_cli_envs1/p\nstandard output\npy310: skipped because could not find python interpreter with spec(s): py310\n py310: SKIP (0.23 seconds)\n evaluation failed :( (0.27 seconds)\n.success self = code: -1 cmd: /usr/bin/python3 -m tox r -e py310 cwd: /tmp/pytest-of-mockbuild/pytest-0/popen-gw3/test_allowed_implici...uld not find python interpreter with spec(s): py310 py310: SKIP (0.23 seconds) evaluation failed :( (0.27 seconds) .../usr/lib/python3.12/site-packages/tox/pytest.py:352: AssertionError This way, we use an environment that is available (as long as the tests run on CPython). This could be improved to support PyPy as well, but I decided to take one step at a time.

Related: - tox-dev/tox#3089 - tox-dev/tox@2124458

* tox.ini: Fix broken path for --cov= argument Broken since the move in commit d076fa2 . * tox.yml: Stop using non-existent environment "coverage" for tox >=4.9.0 Related: - tox-dev/tox#3089 - tox-dev/tox@2124458 * readthedocs.yml: Remove discontinued key "python.system_packages" Error was: > Problem in your project's configuration. Invalid configuration option > "python.system_packages". This configuration key has been deprecated > and removed. Refer to https://blog.readthedocs.com/use-system-packages-deprecated/ > to read more about this change and how to upgrade your config file. Build log at https://readthedocs.org/projects/ansi2html/builds/21826996/ . Upstream docs at https://blog.readthedocs.com/drop-support-system-packages/ . Fix may be incomplete and may need a closer look.

Disallow environments which are not explicitly specified in the confi…

e7acbf7

…g file Special exception is made for 'py*' which allows implicit creation of environments.

tjsmart requested a review from gaborbernat as a code owner August 13, 2023 21:48

tjsmart added 2 commits August 13, 2023 15:56

Fix py37 tests

8c2af1d

Fix mypy complaint

7aa17bb

tjsmart commented Aug 13, 2023

View reviewed changes

Add changelog entry

d4ca1c3

psf-chronographer bot added the bot:chronographer:provided label Aug 13, 2023

gaborbernat approved these changes Aug 14, 2023

View reviewed changes

gaborbernat requested changes Aug 14, 2023

View reviewed changes

gaborbernat approved these changes Aug 14, 2023

View reviewed changes

PR Feedback

d6f67a6

Signed-off-by: Bernát Gábor <bgabor8@bloomberg.net>

gaborbernat merged commit 2124458 into tox-dev:main Aug 14, 2023

dcermak mentioned this pull request Aug 16, 2023

Add missing helm, git & nginx environments to tox.ini SUSE/BCI-tests#287

Merged

WilliamJamieson mentioned this pull request Aug 16, 2023

Update coverage job spacetelescope/roman_datamodels#261

Merged

4 tasks

mvdbeek mentioned this pull request Aug 17, 2023

Fix: serialize tool_shed_urls directly from the API galaxyproject/galaxy#16561

Merged

3 tasks

atugushev mentioned this pull request Aug 17, 2023

Getting environments not found error when running a partial set of factors #3096

Closed

BeyondEvil mentioned this pull request Aug 18, 2023

Chore: Fix tox pytest-dev/pytest-html#718

Merged

tjsmart mentioned this pull request Aug 19, 2023

Add doctesting-coverage to envlist pytest-dev/pytest#11321

Merged

Czaki mentioned this pull request Aug 22, 2023

Do not clone repository to subdirectory in examples test napari/napari#6181

Merged

hroncok reviewed Aug 30, 2023

View reviewed changes

hroncok added a commit to hroncok/tox that referenced this pull request Aug 30, 2023

Tests: Don't assume Python 3.10 is always installed, use current Pyth…

fe1bef1

…on version See tox-dev#3089 (review)

hroncok mentioned this pull request Aug 30, 2023

Tests: Don't assume Python 3.10 is always installed, use current Python version #3108

Merged

5 tasks

hartwork added a commit to hartwork/ansi2html that referenced this pull request Sep 5, 2023

tox.yml: Stop using non-existent environment "coverage" for tox >=4.9.0

9c4a718

Related: - tox-dev/tox#3089 - tox-dev/tox@2124458

hartwork added a commit to hartwork/ansi2html that referenced this pull request Sep 5, 2023

tox.yml: Stop using non-existent environment "coverage" for tox >=4.9.0

da06d55

Related: - tox-dev/tox#3089 - tox-dev/tox@2124458

lucabello mentioned this pull request Sep 26, 2023

CI should fail if a tox env doesn't exist canonical/observability#64

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disallow command line environments which are not explicitly specified in the config file #3089

Disallow command line environments which are not explicitly specified in the config file #3089

tjsmart commented Aug 13, 2023

tjsmart Aug 13, 2023

gaborbernat left a comment

gaborbernat left a comment

gaborbernat left a comment

atugushev commented Aug 17, 2023

hroncok Aug 30, 2023

gaborbernat Aug 30, 2023

hroncok Aug 30, 2023

		assert "provided environments not found in configuration file: ['does_not_exist']" in outcome.out, outcome.out


		@pytest.mark.parametrize("env_name", ["py", "py310", ".pkg"])

Disallow command line environments which are not explicitly specified in the config file #3089

Disallow command line environments which are not explicitly specified in the config file #3089

Conversation

tjsmart commented Aug 13, 2023

tjsmart Aug 13, 2023

Choose a reason for hiding this comment

gaborbernat left a comment

Choose a reason for hiding this comment

gaborbernat left a comment

Choose a reason for hiding this comment

gaborbernat left a comment

Choose a reason for hiding this comment

atugushev commented Aug 17, 2023

hroncok Aug 30, 2023

Choose a reason for hiding this comment

gaborbernat Aug 30, 2023

Choose a reason for hiding this comment

hroncok Aug 30, 2023

Choose a reason for hiding this comment