ci: Make build error on warnings

[matthewfeickert]

Description

Add PYTHONWARNINGS=error,default::DeprecationWarning to the environment to cause Python to treat warnings as errors (but use default for DeprecationWarning as probably internal to build tools) during the package build to ensure that the build configuration conforms to the latest standards. Use the 'PYTHONWARNINGS' environment variable as build runs in a new process and so something like python -Werror -m build . won't be able to pass the flags into the new process, but the shell environment will be picked up. c.f. pypa/build#482

Thanks to @agoose77 for their help pointing out that PYTHONWARNINGS should be used. 👍

Checklist Before Requesting Reviewer

• Tests are passing
• "WIP" removed from the title of the pull request
• Selected an Assignee for the PR to be responsible for the log summary

Before Merging

For the PR Assignees:

• Summarize commit messages into a comprehensive review of the PR

* Add `PYTHONWARNINGS=error,default::DeprecationWarning` to the environment to cause
Python to treat warnings as errors (but use default for DeprecationWarning as probably
internal to build tools) during the package build to ensure that the build configuration
conforms to the latest standards. Use the 'PYTHONWARNINGS' environment variable as `build` runs in a new process and so something like `python -Werror -m build .` won't be able to
pass the flags into the new process, but the shell environment will be picked up.
   - c.f. https://docs.python.org/3/library/warnings.html#describing-warning-filters
   - c.f. https://github.com/pypa/build/issues/482
* Remove `--outdir` option from `build` to use default.

[codecov]

Codecov Report

Merging #1887 (9bfba76) into master (ca77a63) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1887   +/-   ##
=======================================
  Coverage   98.26%   98.26%           
=======================================
  Files          69       69           
  Lines        4439     4439           
  Branches      748      748           
=======================================
  Hits         4362     4362           
  Misses         45       45           
  Partials       32       32           

Flag	Coverage Δ
contrib	25.81% <ø> (ø)
doctest	61.02% <ø> (ø)
unittests-3.10	96.14% <ø> (ø)
unittests-3.7	96.13% <ø> (ø)
unittests-3.8	96.17% <ø> (ø)
unittests-3.9	96.19% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[matthewfeickert]

Hm. CI is hitting a

DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

This is internal stuff that we shouldn't care about, so need to filter the warnings that can be labeled as an Error better.

[matthewfeickert]

Hm. CI is hitting a

DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives

This is internal stuff that we shouldn't care about, so need to filter the warnings that can be labeled as an Error better.

This was happening with pip v22.1.2

Details:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/runpy.py", line [8](https://github.com/scikit-hep/pyhf/runs/6865420250?check_suite_focus=true#step:7:9)6, in _run_code
    exec(code, run_globals)
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/__main__.py", line 2[9](https://github.com/scikit-hep/pyhf/runs/6865420250?check_suite_focus=true#step:7:10), in <module>
    from pip._internal.cli.main import main as _main
  File "/tmp/build-env-rv_juzk7/lib/python3.[10](https://github.com/scikit-hep/pyhf/runs/6865420250?check_suite_focus=true#step:7:11)/site-packages/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/cli/parser.py", line [12](https://github.com/scikit-hep/pyhf/runs/6865420250?check_suite_focus=true#step:7:13), in <module>
    from pip._internal.configuration import Configuration, ConfigurationError
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/configuration.py", line 26, in <module>
    from pip._internal.utils.logging import getLogger
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 39, in <module>
    from pip._internal.locations import get_major_minor_version
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/locations/__init__.py", line [14](https://github.com/scikit-hep/pyhf/runs/6865420250?check_suite_focus=true#step:7:15), in <module>
    from . import _distutils, _sysconfig
  File "/tmp/build-env-rv_juzk7/lib/python3.10/site-packages/pip/_internal/locations/_distutils.py", line 9, in <module>
    from distutils.cmd import Command as DistutilsCommand
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/distutils/__init__.py", line [19](https://github.com/scikit-hep/pyhf/runs/6865420250?check_suite_focus=true#step:7:20), in <module>
    warnings.warn(_DEPRECATION_MESSAGE,
DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
* Creating venv isolated environment...

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/build/__main__.py", line 370, in main
    built = build_call(
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/build/__main__.py", line [23](https://github.com/scikit-hep/pyhf/runs/6865420250?check_suite_focus=true#step:7:24)0, in build_package_via_sdist
    sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/build/__main__.py", line 140, in _build
    return _build_in_isolated_env(builder, outdir, distribution, config_settings)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/build/__main__.py", line 104, in _build_in_isolated_env
    with _IsolatedEnvBuilder() as env:
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/build/env.py", line 108, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/build/env.py", line 288, in _create_isolated_env_venv
    _subprocess([executable, '-m', 'pip', 'uninstall', 'setuptools', '-y'])
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/build/env.py", line 80, in _subprocess
    raise e
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/site-packages/build/env.py", line 77, in _subprocess
    subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/subprocess.py", line 5[24](https://github.com/scikit-hep/pyhf/runs/6865420250?check_suite_focus=true#step:7:25), in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/build-env-rv_juzk7/bin/python', '-m', 'pip', 'uninstall', 'setuptools', '-y']' returned non-zero exit status 1.

ERROR Command '['/tmp/build-env-rv_juzk7/bin/python', '-m', 'pip', 'uninstall', 'setuptools', '-y']' returned non-zero exit status 1.

and now passing with pip v22.2.2. pip v22.2 brought with it

pip’s deprecation warnings now subclass the built-in DeprecationWarning, and can be suppressed by running the Python interpreter with -W ignore::DeprecationWarning.

but for the reasons mentioned in the body of the PR, the interpreter flags are insufficient when working with build. This is probably fine for now, but if we want to ignore DeprecationWarning in general then we should see if it is possible to AND different warning filters.

[matthewfeickert]

Okay, so this wasn't necessarily pip related but here's an example to show things are fine now with PYTHONWARNINGS=error,default::DeprecationWarning

$ cat warning_test.py  # taken and minorly edited from distutils
"""distutils

The main package for the Python Module Distribution Utilities.  Normally
used from a setup script as

   from distutils.core import setup

   setup (...)
"""

import sys
import warnings

__version__ = sys.version[:sys.version.index(' ')]

_DEPRECATION_MESSAGE = ("The distutils package is deprecated and slated for "
                        "removal in Python 3.12. Use setuptools or check "
                        "PEP 632 for potential alternatives")
warnings.warn(_DEPRECATION_MESSAGE,
              DeprecationWarning)
$ PYTHONWARNINGS=error python warning_test.py && echo $?
Traceback (most recent call last):
  File "/home/feickert/tmp/warning_test.py", line 21, in <module>
    warnings.warn(_DEPRECATION_MESSAGE,
DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
$ PYTHONWARNINGS=error,ignore::DeprecationWarning python warning_test.py && echo $?
0
$ PYTHONWARNINGS=error,default::DeprecationWarning python warning_test.py && echo $?
/home/feickert/tmp/warning_test.py:19: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
  warnings.warn(_DEPRECATION_MESSAGE,
0