From 0b2dd18e9d024ba8953cbedd7a459fe5e6eb2e25 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Wed, 13 Dec 2023 19:03:24 +0000 Subject: [PATCH 1/3] Add docs for how to run tests in development --- docs/contributing.md | 68 +++++++++++++++++++++++++++++++------------- 1 file changed, 48 insertions(+), 20 deletions(-) diff --git a/docs/contributing.md b/docs/contributing.md index 097fa6e28..afd3d2fe0 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -12,8 +12,49 @@ If you have an idea for a modification or feature, it's probably best to raise a Everyone contributing to the cibuildwheel project is expected to follow the [PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md). -Design Goals ------------- +## Running the tests + +When making a change to the codebase, you can run tests locally for quicker feedback than the CI runs on a PR. + +#### Setup + +To prepare a development environment, do: + +```console +python3 -m venv .venv +source .venv/bin/activate +pip install -e .[dev] +``` + +Alternatively, you can have `nox` handle the development environment for you. + +#### Unit tests + +To run the project's unit tests, do: + +```console +pytest unit_test +``` + +There are a few custom options to enable different parts of the test suite - check `pytest unit_test --help` for details. + +#### Integration tests + +The integration test suite is big - it can take more than 30 minutes to run the whole thing. So normally you'd choose specific tests to run locally, and rely on the project's CI for the rest. + +To run them locally, you must pass the platform you want to test via the `CIBW_PLATFORM` environment variable. The `-k` pytest option can be used to select the specific tests. For example, you could do: + +```console +CIBW_PLATFORM=linux pytest test -k before_build +``` + +A few notes on this- + +- Because they run inside a container, Linux tests can run on all platforms where Docker is installed, so they're convenient for running integration tests locally. + +- Running the macOS integration tests requires _system installs_ of Python from python.org for all the versions that are tested. We won't attempt to install these when running locally, but you can do so manually using the URL in the error message that is printed when the install is not found. + +## Design Goals - `cibuildwheel` should wrap the complexity of wheel building. - The user interface to `cibuildwheel` is the build script (e.g. `.travis.yml`). Feature additions should not increase the complexity of this script. @@ -33,8 +74,7 @@ We're not responsible for errors in those tools, for fixing errors/crashes there So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/pypa/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the Linux builds are happening in a different OS via Docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves. -Maintainer notes ----------------- +## Maintainer notes ### Nox support @@ -45,10 +85,10 @@ Install [nox](https://nox.thea.codes); homebrew is recommend on macOS, otherwise You can see a list of sessions by typing `nox -l`; here are a few common ones: ```console -nox -s lint # Run the linters (default) -nox -s tests # Run the tests (default) -nox -s docs -- serve # Build and serve the documentation -nox -s build # Make SDist and wheel +nox -s lint # Run the linters (default) +nox -s tests [-- PYTEST-ARGS] # Run the tests (default) +nox -s docs -- serve # Build and serve the documentation +nox -s build # Make SDist and wheel ``` More advanced users can run the update scripts. `update_pins` should work directly, but `update_constraints` needs all versions of Python installed. If you don't want to do that locally, a fast way to run it to use docker to run nox: @@ -57,18 +97,6 @@ More advanced users can run the update scripts. `update_pins` should work direct docker run --rm -itv $PWD:/src -w /src quay.io/pypa/manylinux_2_24_x86_64:latest pipx run nox -s update_constraints ``` -### Local testing - -You should run: - -```console -python3 -m venv venv -. venv/bin/activate -pip install -e .[dev] -``` - -To prepare a development environment. - ### Testing sample configs cibuildwheel's example configs can be tested on a simple project on cibuildwheel's existing CI. These should be run whenever the minimal configs change. From c975724299e0a1831fbcdd52c232e26a43c39eb0 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Sat, 2 Mar 2024 19:40:22 +0000 Subject: [PATCH 2/3] Update the docs to use nox primarily and not require CIBW_PLATFORM --- docs/contributing.md | 62 ++++++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 20 deletions(-) diff --git a/docs/contributing.md b/docs/contributing.md index afd3d2fe0..54e8deba7 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -14,46 +14,68 @@ Everyone contributing to the cibuildwheel project is expected to follow the [PSF ## Running the tests -When making a change to the codebase, you can run tests locally for quicker feedback than the CI runs on a PR. +When making a change to the codebase, you can run tests locally for quicker feedback than the CI runs on a PR. You can [run them directly](#making-a-venv), but the easiest way to run tests is using [nox](https://nox.thea.codes/). -#### Setup +You can run all the tests locally by doing: -To prepare a development environment, do: - -```console -python3 -m venv .venv -source .venv/bin/activate -pip install -e .[dev] +```bash +nox -s tests ``` -Alternatively, you can have `nox` handle the development environment for you. +However, because this takes a while, you might prefer to be more specific. -#### Unit tests +### Unit tests To run the project's unit tests, do: -```console -pytest unit_test +```bash +nox -s tests -- unit_test ``` -There are a few custom options to enable different parts of the test suite - check `pytest unit_test --help` for details. +There are a few custom options to enable different parts of the test suite - check `nox -s tests -- unit_test --help` for details. -#### Integration tests +If you're calling this a lot, you might consider using the `-r` or `-R` arguments to nox to make it a bit faster. This calls pytest under the hood, so to target a specific test, use pytest's `-k` option after the `--` above to select a specific test. -The integration test suite is big - it can take more than 30 minutes to run the whole thing. So normally you'd choose specific tests to run locally, and rely on the project's CI for the rest. +### Integration tests -To run them locally, you must pass the platform you want to test via the `CIBW_PLATFORM` environment variable. The `-k` pytest option can be used to select the specific tests. For example, you could do: +The integration test suite is big - it can take more than 30 minutes to run the whole thing. -```console -CIBW_PLATFORM=linux pytest test -k before_build +```bash +nox -s tests -- test +``` + +Because it takes such a long time, normally you'd choose specific tests to run locally, and rely on the project's CI for the rest. Use pytest's `-k` option to choose specific tests. You can pass a test name or a filename, it'll run everything that matches. + +```bash +nox -s tests -- test -k +# e.g. +nox -s tests -- test -k before_build ``` -A few notes on this- +A few notes- -- Because they run inside a container, Linux tests can run on all platforms where Docker is installed, so they're convenient for running integration tests locally. +- Because they run inside a container, Linux tests can run on all platforms where Docker is installed, so they're convenient for running integration tests locally. Set CIBW_PLATFORM to do this: `CIBW_PLATFORM=linux nox -s tests -- test`. - Running the macOS integration tests requires _system installs_ of Python from python.org for all the versions that are tested. We won't attempt to install these when running locally, but you can do so manually using the URL in the error message that is printed when the install is not found. +### Making a venv + +More advanced users might prefer to invoke pytest directly- + +```bash +python3 -m venv .venv +source .venv/bin/activate +pip install -e .[dev] +# run the unit tests +pytest unit_test +# run the whole integration test suite +pytest test +# run a specific integration test +pytest test -k test_build_frontend_args +# run a specific integration test on a different platform +CIBW_PLATFORM=linux pytest test -k test_build_frontend_args +``` + ## Design Goals - `cibuildwheel` should wrap the complexity of wheel building. From 742f876f1d5b92cbb87fa3b4a32ebd97133416c7 Mon Sep 17 00:00:00 2001 From: Joe Rickerby Date: Sat, 2 Mar 2024 19:55:20 +0000 Subject: [PATCH 3/3] Reorganise this file --- docs/contributing.md | 62 +++++++++++++++++++++++--------------------- 1 file changed, 32 insertions(+), 30 deletions(-) diff --git a/docs/contributing.md b/docs/contributing.md index 54e8deba7..31333f4e4 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -12,7 +12,29 @@ If you have an idea for a modification or feature, it's probably best to raise a Everyone contributing to the cibuildwheel project is expected to follow the [PSF Code of Conduct](https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md). -## Running the tests +## Design Goals + +- `cibuildwheel` should wrap the complexity of wheel building. +- The user interface to `cibuildwheel` is the build script (e.g. `.travis.yml`). Feature additions should not increase the complexity of this script. +- Options should be environment variables (these lend themselves better to YML config files). They should be prefixed with `CIBW_`. +- Options should be generalise to all platforms. If platform-specific options are required, they should be namespaced e.g. `CIBW_TEST_COMMAND_MACOS` + +Other notes: + +- The platforms are very similar, until they're not. I'd rather have straight-forward code than totally DRY code, so let's keep airy platform abstractions to a minimum. +- I might want to break the options into a shared config file one day, so that config is more easily shared. That has motivated some of the design decisions. + +### cibuildwheel's relationship with build errors + +cibuildwheel doesn't really do anything itself - it's always deferring to other tools (pip, wheel, auditwheel, delocate, docker). Without cibuildwheel, the process is really fragmented. Different tools, across different OSs need to be stitched together in just the right way to make it work. + +We're not responsible for errors in those tools, for fixing errors/crashes there. But cibuildwheel's job is providing users with an 'integrated' user experience across those tools. We provide an abstraction. The user says 'build me some wheels', not 'open the docker container, build a wheel with pip, fix up the symbols with auditwheel' etc. However, errors have a habit of breaking abstractions. And this is where users get confused, because the mechanism of cibuildwheel is laid bare, and they must understand a little bit how it works to debug. + +So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/pypa/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the Linux builds are happening in a different OS via Docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves. + +## Development + +### Running the tests When making a change to the codebase, you can run tests locally for quicker feedback than the CI runs on a PR. You can [run them directly](#making-a-venv), but the easiest way to run tests is using [nox](https://nox.thea.codes/). @@ -24,7 +46,7 @@ nox -s tests However, because this takes a while, you might prefer to be more specific. -### Unit tests +#### Unit tests To run the project's unit tests, do: @@ -36,14 +58,16 @@ There are a few custom options to enable different parts of the test suite - che If you're calling this a lot, you might consider using the `-r` or `-R` arguments to nox to make it a bit faster. This calls pytest under the hood, so to target a specific test, use pytest's `-k` option after the `--` above to select a specific test. -### Integration tests +#### Integration tests -The integration test suite is big - it can take more than 30 minutes to run the whole thing. +To run the project's integration tests, do: ```bash nox -s tests -- test ``` +The integration test suite is big - it can take more than 30 minutes to run the whole thing. + Because it takes such a long time, normally you'd choose specific tests to run locally, and rely on the project's CI for the rest. Use pytest's `-k` option to choose specific tests. You can pass a test name or a filename, it'll run everything that matches. ```bash @@ -58,7 +82,7 @@ A few notes- - Running the macOS integration tests requires _system installs_ of Python from python.org for all the versions that are tested. We won't attempt to install these when running locally, but you can do so manually using the URL in the error message that is printed when the install is not found. -### Making a venv +#### Making a venv More advanced users might prefer to invoke pytest directly- @@ -76,34 +100,10 @@ pytest test -k test_build_frontend_args CIBW_PLATFORM=linux pytest test -k test_build_frontend_args ``` -## Design Goals - -- `cibuildwheel` should wrap the complexity of wheel building. -- The user interface to `cibuildwheel` is the build script (e.g. `.travis.yml`). Feature additions should not increase the complexity of this script. -- Options should be environment variables (these lend themselves better to YML config files). They should be prefixed with `CIBW_`. -- Options should be generalise to all platforms. If platform-specific options are required, they should be namespaced e.g. `CIBW_TEST_COMMAND_MACOS` - -Other notes: - -- The platforms are very similar, until they're not. I'd rather have straight-forward code than totally DRY code, so let's keep airy platform abstractions to a minimum. -- I might want to break the options into a shared config file one day, so that config is more easily shared. That has motivated some of the design decisions. - -### cibuildwheel's relationship with build errors - -cibuildwheel doesn't really do anything itself - it's always deferring to other tools (pip, wheel, auditwheel, delocate, docker). Without cibuildwheel, the process is really fragmented. Different tools, across different OSs need to be stitched together in just the right way to make it work. - -We're not responsible for errors in those tools, for fixing errors/crashes there. But cibuildwheel's job is providing users with an 'integrated' user experience across those tools. We provide an abstraction. The user says 'build me some wheels', not 'open the docker container, build a wheel with pip, fix up the symbols with auditwheel' etc. However, errors have a habit of breaking abstractions. And this is where users get confused, because the mechanism of cibuildwheel is laid bare, and they must understand a little bit how it works to debug. - -So, if we can, I'd like to improve the experience on errors as well. In [this](https://github.com/pypa/cibuildwheel/issues/139) case, it takes a bit of knowledge to understand that the Linux builds are happening in a different OS via Docker, that the linked symbols won't match, that auditwheel will fail because of this. A problem with how the tools fit together, instead of the tools themselves. - -## Maintainer notes - -### Nox support +### Linting, docs Most developer tasks have a nox interface. This allows you to very simply run tasks without worrying about setting up a development environment (as shown below). This is a slower than setting up a development environment and reusing it, but has the (important) benefit of being highly reproducible; an earlier run does not affect a current run, or anything else on your machine. -Install [nox](https://nox.thea.codes); homebrew is recommend on macOS, otherwise, pipx is a great choice - in fact, you can use `pipx run nox` and avoid installing completely. - You can see a list of sessions by typing `nox -l`; here are a few common ones: ```console @@ -119,6 +119,8 @@ More advanced users can run the update scripts. `update_pins` should work direct docker run --rm -itv $PWD:/src -w /src quay.io/pypa/manylinux_2_24_x86_64:latest pipx run nox -s update_constraints ``` +## Maintainer notes + ### Testing sample configs cibuildwheel's example configs can be tested on a simple project on cibuildwheel's existing CI. These should be run whenever the minimal configs change.