docs: Add how to run tests in development (#1698)

* Add docs for how to run tests in development

* Update the docs to use nox primarily and not require CIBW_PLATFORM

* Reorganise this file

docs/contributing.md

@@ -12,8 +12,7 @@ If you have an idea for a modification or feature, it's probably best to raise a
 
 Everyone contributing to the cibuildwheel project is expected to follow the [PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md).
 
-Design Goals
-------------
+## Design Goals
 
 - `cibuildwheel` should wrap the complexity of wheel building.
 - The user interface to `cibuildwheel` is the build script (e.g. `.travis.yml`). Feature additions should not increase the complexity of this script.
@@ -33,41 +32,94 @@ We're not responsible for errors in those tools, for fixing errors/crashes there
 
 So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/pypa/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the Linux builds are happening in a different OS via Docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves.
 
-Maintainer notes
-----------------
+## Development
 
-### Nox support
+### Running the tests
 
-Most developer tasks have a nox interface. This allows you to very simply run tasks without worrying about setting up a development environment (as shown below). This is a slower than setting up a development environment and reusing it, but has the (important) benefit of being highly reproducible; an earlier run does not affect a current run, or anything else on your machine.
+When making a change to the codebase, you can run tests locally for quicker feedback than the CI runs on a PR. You can [run them directly](#making-a-venv), but the easiest way to run tests is using [nox](https://nox.thea.codes/).
 
-Install [nox](https://nox.thea.codes); homebrew is recommend on macOS, otherwise, pipx is a great choice - in fact, you can use `pipx run nox` and avoid installing completely.
+You can run all the tests locally by doing:
 
-You can see a list of sessions by typing `nox -l`; here are a few common ones:
+```bash
+nox -s tests
+```
 
-```console
-nox -s lint          # Run the linters (default)
-nox -s tests         # Run the tests   (default)
-nox -s docs -- serve # Build and serve the documentation
-nox -s build         # Make SDist and wheel
+However, because this takes a while, you might prefer to be more specific.
+
+#### Unit tests
+
+To run the project's unit tests, do:
+
+```bash
+nox -s tests -- unit_test
 ```
 
-More advanced users can run the update scripts. `update_pins` should work directly, but `update_constraints` needs all versions of Python installed. If you don't want to do that locally, a fast way to run it to use docker to run nox:
+There are a few custom options to enable different parts of the test suite - check `nox -s tests -- unit_test --help` for details.
 
-```console
-docker run --rm -itv $PWD:/src -w /src quay.io/pypa/manylinux_2_24_x86_64:latest pipx run nox -s update_constraints
+If you're calling this a lot, you might consider using the `-r` or `-R` arguments to nox to make it a bit faster. This calls pytest under the hood, so to target a specific test, use pytest's `-k` option after the `--` above to select a specific test.
+
+#### Integration tests
+
+To run the project's integration tests, do:
+
+```bash
+nox -s tests -- test
 ```
 
-### Local testing
+The integration test suite is big - it can take more than 30 minutes to run the whole thing.
 
-You should run:
+Because it takes such a long time, normally you'd choose specific tests to run locally, and rely on the project's CI for the rest. Use pytest's `-k` option to choose specific tests. You can pass a test name or a filename, it'll run everything that matches.
 
-```console
-python3 -m venv venv
-. venv/bin/activate
+```bash
+nox -s tests -- test -k <test_name_or_filename>
+# e.g.
+nox -s tests -- test -k before_build
+```
+
+A few notes-
+
+- Because they run inside a container, Linux tests can run on all platforms where Docker is installed, so they're convenient for running integration tests locally. Set CIBW_PLATFORM to do this: `CIBW_PLATFORM=linux nox -s tests -- test`.
+
+- Running the macOS integration tests requires _system installs_ of Python from python.org for all the versions that are tested. We won't attempt to install these when running locally, but you can do so manually using the URL in the error message that is printed when the install is not found.
+
+#### Making a venv
+
+More advanced users might prefer to invoke pytest directly-
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
 pip install -e .[dev]
+# run the unit tests
+pytest unit_test
+# run the whole integration test suite
+pytest test
+# run a specific integration test
+pytest test -k test_build_frontend_args
+# run a specific integration test on a different platform
+CIBW_PLATFORM=linux pytest test -k test_build_frontend_args
+```
+
+### Linting, docs
+
+Most developer tasks have a nox interface. This allows you to very simply run tasks without worrying about setting up a development environment (as shown below). This is a slower than setting up a development environment and reusing it, but has the (important) benefit of being highly reproducible; an earlier run does not affect a current run, or anything else on your machine.
+
+You can see a list of sessions by typing `nox -l`; here are a few common ones:
+
+```console
+nox -s lint                    # Run the linters (default)
+nox -s tests [-- PYTEST-ARGS]  # Run the tests   (default)
+nox -s docs -- serve           # Build and serve the documentation
+nox -s build                   # Make SDist and wheel
+```
+
+More advanced users can run the update scripts. `update_pins` should work directly, but `update_constraints` needs all versions of Python installed. If you don't want to do that locally, a fast way to run it to use docker to run nox:
+
+```console
+docker run --rm -itv $PWD:/src -w /src quay.io/pypa/manylinux_2_24_x86_64:latest pipx run nox -s update_constraints
 ```
 
-To prepare a development environment.
+## Maintainer notes
 
 ### Testing sample configs