tox 4 does not support factor all conditionals as 3

[dataflake]

We use factors to differentiate test setups like this:

[testenv]
...
commands_pre =
    py27,py35: DO X
    !py27,!py35: DO Y
...

This worked fine until version 4.0.13. Now the py27 and py35 environments match both the first and second condition, both commands_pre are run.

Environment

Provide at least:

• OS: macOS
• Python: 3.8.15
• pip list of the host Python where tox is installed:

$ bin/pip list
Package            Version
------------------ ---------
bleach             5.0.1
build              0.9.0
cachetools         5.2.0
certifi            2022.12.7
chardet            5.1.0
charset-normalizer 2.1.1
check-manifest     0.49
colorama           0.4.6
commonmark         0.9.1
distlib            0.3.6
docutils           0.19
filelock           3.8.2
idna               3.4
importlib-metadata 5.1.0
jaraco.classes     3.2.3
keyring            23.11.0
more-itertools     9.0.0
packaging          22.0
pep517             0.13.0
pip                22.3.1
pkginfo            1.9.2
platformdirs       2.6.0
pluggy             1.0.0
py                 1.11.0
Pygments           2.13.0
pyproject_api      1.2.1
readme-renderer    37.3
requests           2.28.1
requests-toolbelt  0.10.1
rfc3986            2.0.0
rich               12.6.0
setuptools         65.6.3
six                1.16.0
tomli              2.0.1
tox                4.0.13
twine              4.0.2
typing_extensions  4.4.0
urllib3            1.26.13
virtualenv         20.17.1
webencodings       0.5.1
wheel              0.38.4
zc.buildout        3.0.1
zipp               3.11.0

[gaborbernat]

This change is not just 4.0.13; it's any 4.0 version; the reporting claims otherwise, but I get the same behavior with 4.0. I'm surprised this worked in tox 3 because we use the , as an OR operator... So py27,py35 means the environment is py27 or py35, so we match. Then !py27,!py35 means not py27 or not py35, so both lines are valid. What would you expect the second conditional to explode to? Note, in tox 4, you can test this with:

tox c -e py27 -k commands_pre

Here are two more filter expressions that worked with 3.0 and are not parsed with 4.0, taken from our docs here https://tox.wiki/en/legacy/config.html#complex-factor-conditions

[tox]
envlist = py{27,34,36}-django{15,16}-{sqlite,mysql}

[testenv]
deps =
    sqlite-!py34: E2     # (same as the line above)
    !py34-!py36: neither # use if neither py34 nor py36 are in the env name

[gaborbernat]

A PR trying to address this regression would be welcomed.

[dataflake]

The specific expression we used, !py27-!py35, worked with tox 4.0.12 and broke with 4.0.13. I tested this manually today.

Can you suggest an expression hat works with tox 4.0.13 and up that means "environment is NOT py27 AND NOT py35"?

[gaborbernat]

My test shows it doesn't work with 4.0.12 🤔 PR welcome. I don't plan to work on this. This is a free open source project.

[frenzymadness]

I can confirm this bug. Fox example zope.event uses:

commands =
    zope-testrunner --test-path=src {posargs:-vc}
    !py27-!pypy: sphinx-build -b doctest -d {envdir}/.cache/doctrees docs {envdir}/.cache/doctest

With tox 4.0.12:

$ tox -e py311 config -k commands
[testenv:py311]
commands =
  zope-testrunner --test-path=src -vc
  sphinx-build -b doctest -d /home/lbalhar/Software/zope.event/.tox/py311/.cache/doctrees docs /home/lbalhar/Software/zope.event/.tox/py311/.cache/doctest

With tox 4.0.13:

$ tox -e py311 config -k commands
[testenv:py311]
commands =
  zope-testrunner --test-path=src -vc
  '!py27-!pypy:' sphinx-build -b doctest -d /home/lbalhar/Software/zope.event/.tox/py311/.cache/doctrees docs /home/lbalhar/Software/zope.event/.tox/py311/.cache/doctest

git bisect says that this is the change: 864332f

[frenzymadness]

To go a little bit deeper, this function

tox/src/tox/config/loader/ini/factor.py

Lines 70 to 81 in 9972a82

	def expand_env_with_negation(value: str) -> Iterator[str]:
	"""transform '{py,!pi}-{a,b},c' to ['py-a', 'py-b', '!pi-a', '!pi-b', 'c']"""
	for key, group in groupby(re.split(r"((?:{[^}]+})+)|,", value), key=bool):
	if key:
	group_str = "".join(group).strip()
	elements = re.split(r"{([^}]+)}", group_str)
	parts = [[i.strip() for i in elem.split(",")] for elem in elements]
	for variant in product(*parts):
	variant_str = "".join(variant)
	if not re.fullmatch(r"!?[\w._][\w._-]*", variant_str):
	raise ValueError(variant_str)
	yield variant_str

raises ValueError because, from the example above, the !py27-!pypy does not fully match the r"!?[\w._][\w._-]*" regex – it stops on the second exclamation mark after the dash.

[dhomeier]

Confirmed as well, with 4.0.13 and 4.0.14 astropy jobs are failing on
https://github.com/astropy/astropy/actions/runs/3733576067/jobs/6334552302#step:9:47