More fixes to support tox 4

[mkoeppe]

The tox.ini files in pkgs/*/ are broken with tox 4.

Early versions of tox 4.x have additional defects.

• Environment variables containing strings with spaces are not accepted any longer for passenv/pass_env. tox-dev/tox#2699 (comment) - factors no longer work in passenv - fixed in tox 4.0.13

• negated factors in deps, like !pipenv-!norequirements: -rrequirements.txt do not work - tox 4 does not support factor all conditionals as 3 tox-dev/tox#2747 - fixed in tox 4.0.15

• And autoprovisioning for downgrade to tox 3 is not working - tox auto-provisioning not working tox-dev/tox#2634

• In SAGE_ROOT/tox.ini, in tox 4 earlier than 4.2.7, numerical factors are broken because they are misinterpreted as Python versions (https://tox.wiki/en/latest/changelog.html#bugfixes-4-2-7); e.g., 4.0.14 is broken:

  tox -e docker-linuxmint-21.1-standard                                                                                                      git:t/34903/tox_ini___gh_actions__update_linux_platforms*
  docker-linuxmint-21.1-standard: skipped because could not find python interpreter with spec(s): 21.1
    docker-linuxmint-21.1-standard: SKIP (1.98 seconds)
  

Here we update affected tox.ini files so that:

• src/tox.ini and pkgs/*/tox.ini work both with suitable versions of tox 3 and tox 4. We change the tox spkg-configure.m4 to check for such versions.
• SAGE_ROOT/tox.ini work with tox >= 4.2.7 (autoprovisioning if necessary)

Follow up:

• Upgrade tox to 4.x #34840 Upgrade tox to 4.x

Depends on #34839

CC: @fchapoton @dimpase

Component: scripts

Author: Matthias Koeppe

Branch/Commit: a2eb816

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/34849

[mkoeppe]

Branch: u/mkoeppe/more_fixes_to_support_tox_4

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

d302a62	tox.ini, src/tox.ini: Adjust to tox 4, autoprovision tox >= 3.18
f8703ae	.github/workflows/lint.yml: Invoke tox with --
9d76204	fix some rst issues in sage_docbuild and sage_setup
4a27a3d	Merge #34839
95fe89a	build/pkgs/tox/spkg-configure.m4: Check for good tox 3, tox 4 versions separately
cc7196a	src/tox.ini: More allowlist_externals for tox 4
1999bd6	src/tox.ini: Quoting fix for tox 4

[sagetrac-git]

Commit: 1999bd6

[mkoeppe]

Author: Matthias Koeppe

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

071c113

Merge tag '9.8.beta6' into t/34849/more_fixes_to_support_tox_4

[sagetrac-git]

Changed commit from 1999bd6 to 071c113

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

70da140	Merge branch 'u/chapoton/support_tox_4' in 9.8.b6
6b8c666	Merge #34839

[sagetrac-git]

Changed commit from 071c113 to 6b8c666

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

a2eb816

tox.ini: Require tox >= 4.2.7

[sagetrac-git]

Changed commit from 6b8c666 to a2eb816

[dimpase]

comment:17

with tox 4.2.8, I get

$ tox -e docker-linuxmint-21.1-standard
docker-linuxmint-21.1-standard: commands[0]> bash -c 'build/bin/write-dockerfile.sh debian "--has-file=spkg-configure.m4 :standard:" yes yes "_bootstrap      " > /mnt/opt/Sage/sage-dev/.tox/docker-linuxmint-21.1-standard/Dockerfile'
docker-linuxmint-21.1-standard: failed with bash is not allowed, use allowlist_externals to allow it
  docker-linuxmint-21.1-standard: FAIL code 1 (0.44 seconds)
  evaluation failed :( (0.75 seconds)

• not sure what it is.

[dimpase]

comment:18

and with this branch, I get

$ tox -e docker-linuxmint-21.1-standard
docker-linuxmint-21.1-standard: commands[0]> bash -c 'build/bin/write-dockerfile.sh debian "--has-file=spkg-configure.m4 :standard:" yes yes "_bootstrap      " > /mnt/opt/Sage/sage-dev/.tox/docker-linuxmint-21.1-standard/Dockerfile'
docker-linuxmint-21.1-standard: commands[1]> bash -c 'if [ x"" != x ]; then mkdir -p /mnt/opt/Sage/sage-dev/.tox/docker-linuxmint-21.1-standard/.docker && ln -sf $(realpath "") /mnt/opt/Sage/sage-dev/.tox/docker-linuxmint-21.1-standard/.docker/; fi'
docker-linuxmint-21.1-standard: commands[2]> bash -c 'for docker_target in with-targets; do BUILD_IMAGE_STEM=sage-$(echo docker-linuxmint-21.1-standard | sed s/-incremental//); BUILD_IMAGE=$DOCKER_PUSH_REPOSITORY$BUILD_IMAGE_STEM-$docker_target; BUILD_TAG=$(git describe --dirty --always); TAG_ARGS=$(for tag in $BUILD_TAG ; do echo --tag $BUILD_IMAGE:$tag; done); DOCKER_BUILDKIT=0 docker build . -f /mnt/opt/Sage/sage-dev/.tox/docker-linuxmint-21.1-standard/Dockerfile --target $docker_target $TAG_ARGS --build-arg EXTRA_CONFIGURE_ARGS="--enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes  " --build-arg BASE_IMAGE=-amd64:latest --build-arg BOOTSTRAP="./bootstrap" --build-arg TARGETS_PRE="$(if test -n "$TARGETS_PRE"; then echo $TARGETS_PRE; else echo all-sage-local; fi)" --build-arg TARGETS="build" --build-arg TARGETS_OPTIONAL="ptest" ; status=$?; if [ $status != 0 ]; then BUILD_TAG="$BUILD_TAG-failed"; docker commit $(docker ps -l -q) $BUILD_IMAGE:$BUILD_TAG; PUSH_TAGS=$BUILD_IMAGE:$BUILD_TAG; else PUSH_TAGS=$(echo $BUILD_IMAGE:$BUILD_TAG; for tag in ; do echo "$BUILD_IMAGE:$tag"; done); fi; echo $BUILD_IMAGE:$BUILD_TAG >> /mnt/opt/Sage/sage-dev/.tox/docker-linuxmint-21.1-standard/Dockertags; if [ x"" != x ]; then echo Pushing $PUSH_TAGS; for tag in $PUSH_TAGS; do docker push $tag || echo "(ignoring errors)"; done; fi; if [ $status != 0 ]; then exit $status; fi; done'
Sending build context to Docker daemon  749.1MB
Step 1/45 : ARG BASE_IMAGE
Step 2/45 : FROM ${BASE_IMAGE} as with-system-packages
invalid reference format
sha256:1839a6c05ff9a58a9ffe9488ff229a64cc35fca8c23ca670792fb6b6c70baa50
docker-linuxmint-21.1-standard: exit 1 (38.18 seconds) /mnt/opt/Sage/sage-dev> bash -c 'for docker_target in with-targets; do BUILD_IMAGE_STEM=sage-$(echo docker-linuxmint-21.1-standard | sed s/-incremental//); BUILD_IMAGE=$DOCKER_PUSH_REPOSITORY$BUILD_IMAGE_STEM-$docker_target; BUILD_TAG=$(git describe --dirty --always); TAG_ARGS=$(for tag in $BUILD_TAG ; do echo --tag $BUILD_IMAGE:$tag; done); DOCKER_BUILDKIT=0 docker build . -f /mnt/opt/Sage/sage-dev/.tox/docker-linuxmint-21.1-standard/Dockerfile --target $docker_target $TAG_ARGS --build-arg EXTRA_CONFIGURE_ARGS="--enable-experimental-packages --enable-download-from-upstream-url  --with-system-python3=yes  " --build-arg BASE_IMAGE=-amd64:latest --build-arg BOOTSTRAP="./bootstrap" --build-arg TARGETS_PRE="$(if test -n "$TARGETS_PRE"; then echo $TARGETS_PRE; else echo all-sage-local; fi)" --build-arg TARGETS="build" --build-arg TARGETS_OPTIONAL="ptest" ; status=$?; if [ $status != 0 ]; then BUILD_TAG="$BUILD_TAG-failed"; docker commit $(docker ps -l -q) $BUILD_IMAGE:$BUILD_TAG; PUSH_TAGS=$BUILD_IMAGE:$BUILD_TAG; else PUSH_TAGS=$(echo $BUILD_IMAGE:$BUILD_TAG; for tag in ; do echo "$BUILD_IMAGE:$tag"; done); fi; echo $BUILD_IMAGE:$BUILD_TAG >> /mnt/opt/Sage/sage-dev/.tox/docker-linuxmint-21.1-standard/Dockertags; if [ x"" != x ]; then echo Pushing $PUSH_TAGS; for tag in $PUSH_TAGS; do docker push $tag || echo "(ignoring errors)"; done; fi; if [ $status != 0 ]; then exit $status; fi; done' pid=4130
  docker-linuxmint-21.1-standard: FAIL code 1 (38.78=setup[0.06]+cmd[0.53,0.00,38.18] seconds)
  evaluation failed :( (39.09 seconds)

[dimpase]

comment:20

oops, sorry, I just got logged out of docker, that's the reason for the error

[dimpase]

comment:21

seems to work

[dimpase]

Reviewer: Dima Pasechnik

[mkoeppe]

comment:22

Thanks!

[vbraun]

Changed branch from u/mkoeppe/more_fixes_to_support_tox_4 to a2eb816