From 9a6016598a02a4ce7ff588de8156b9ae7ca3c695 Mon Sep 17 00:00:00 2001 From: Wolf Vollprecht Date: Fri, 5 Jan 2024 10:23:52 +0100 Subject: [PATCH] expand the test documentation --- docs/index.md | 16 ++- docs/recipe_file.md | 245 ++++++++++++++++++++++++++++---------------- docs/testing.md | 113 ++++++++++++-------- mkdocs.yml | 12 ++- 4 files changed, 238 insertions(+), 148 deletions(-) diff --git a/docs/index.md b/docs/index.md index 2420ea9f..f950012d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -118,8 +118,8 @@ requirements: run_constrained: - xsimd >=8.0.3,<10 -test: - commands: +tests: + - script: - if: unix or emscripten then: - test -d ${PREFIX}/include/xtensor @@ -181,13 +181,11 @@ requirements: - python 3.10 - typing_extensions >=4.0.0,<5.0.0 -test: - imports: - - rich - commands: - - pip check - requires: - - pip +tests: + - python: + imports: + - rich + pip_check: true about: homepage: https://github.com/Textualize/rich diff --git a/docs/recipe_file.md b/docs/recipe_file.md index 26bf26ec..66bd046f 100644 --- a/docs/recipe_file.md +++ b/docs/recipe_file.md @@ -30,10 +30,11 @@ The reason for a new spec are: - no full Jinja2 support: no conditional or `{% set ...` support, only string interpolation. Variables can be set in the toplevel "context" which is valid YAML -- Jinja string interpolation needs to be preceded by a dollar sign at the beginning of a string, - e.g. `- ${{ version }}` in order for it to be valid YAML -- Selectors use a YAML dictionary style (vs. comments in conda-build). Instead of `- somepkg #[osx]` - we use +- Jinja string interpolation needs to be preceded by a dollar sign at the + beginning of a string, e.g. `- ${{ version }}` in order for it to be valid + YAML +- Selectors use a YAML dictionary style (vs. comments in conda-build). Instead + of `- somepkg #[osx]` we use ```yaml if: osx then: @@ -60,9 +61,10 @@ The recipe spec has the following parts: Spec reference -------------- -The spec is also made available through a JSON Schema (which is used for validation). -The schema (and pydantic source file) can be found in this repository: https://github.com/prefix-dev/recipe-format. -To use with VSCode or other IDEs, start the document with the following line: +The spec is also made available through a JSON Schema (which is used for +validation). The schema (and pydantic source file) can be found in this +repository: https://github.com/prefix-dev/recipe-format. To use with VSCode or +other IDEs, start the document with the following line: ```yaml # yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json @@ -193,9 +195,11 @@ source: lfs: true # defaults to false ``` -Note: `git_rev` may not be available within commit depth range, consider avoiding use of both simultaneously. +Note: `git_rev` may not be available within commit depth range, consider +avoiding use of both simultaneously. -When you want to use git-lfs, you need to set `lfs: true`. This will also pull the lfs files from the repository. +When you want to use git-lfs, you need to set `lfs: true`. This will also pull +the lfs files from the repository. #### Source from a local path @@ -208,8 +212,9 @@ source is copied to the work directory before building. use_gitignore: false # (defaults to true) ``` -By default, all files in the local path that are ignored by git are also ignored by rattler-build. -You can disable this behavior by setting `use_gitignore` to `false`. +By default, all files in the local path that are ignored by git are also ignored +by rattler-build. You can disable this behavior by setting `use_gitignore` to +`false`. #### Patches @@ -227,10 +232,10 @@ Patches may optionally be applied to the source. #### Destination path Within boa's work directory, you may specify a particular folder to place source -into. `rattler-build` will always drop you into the same folder (build folder/work), but -it's up to you whether you want your source extracted into that folder, or -nested deeper. This feature is particularly useful when dealing with multiple -sources, but can apply to recipes with single sources as well. +into. `rattler-build` will always drop you into the same folder (build +folder/work), but it's up to you whether you want your source extracted into +that folder, or nested deeper. This feature is particularly useful when dealing +with multiple sources, but can apply to recipes with single sources as well. ```yaml source: @@ -498,118 +503,180 @@ This is the version bound consistent with CentOS 6. Software built against glibc mamba tell the user that a given package can't be installed if their system glibc version is too old. -Test section ------------- +Tests section +------------- -If this section exists or if there is a `run_test.[py,pl,sh,bat]` file in the -recipe, the package is installed into a test environment after the build is -finished and the tests are run there. +Rattler-build supports 4 different types of tests. The "script" test installs +the package and runs a list of commands. The python test attempts to import a +list of python modules and runs `pip check`. The downstream test runs the tests +of a downstream package that reverse depends on the package being built. - +#### Extra Test Files -### Test requirements +Test files that are copied from the source work directory into the temporary +test directory and are needed during testing (note that the source work +directory is otherwise not available at all during testing). -In addition to the runtime requirements, you can specify requirements needed -during testing. The runtime requirements that you specified in the "run" section -described above are automatically included during testing. +You can also include files that come from the `recipe` folder. They are copied +into the test directory as well. + +At test execution time, the test directory is the current working directory. ```yaml -test: - requires: - - nose +tests: + - script: + - ls + files: + source: + - myfile.txt + - tests/ + - some/directory/pattern*.sh + recipe: + - extra-file.txt ``` -### Test commands +--> -Commands that are run as part of the test. +#### Test requirements + +In addition to the runtime requirements, you can specify requirements needed +during testing. The runtime requirements that you specified in the "run" section +described above are automatically included during testing (because the built +package is installed like regular). + +In the `build` section you can specify additional requirements that are only +needed on the build system for cross-compilation (e.g. emulators or compilers). ```yaml -test: - commands: - - bsdiff4 -h - - bspatch4 -h +tests: + - script: + - echo "hello world" + requirements: + build: + - myemulator + run: + - nose ``` -### Python imports +### Python tests + +For this test type you can list a set of Python modules that need to be +importable. The test will fail if any of the modules cannot be imported. + +The test will also automatically run `pip check` to check for any broken +dependencies. This can be disabled by setting `pip_check: false` in the YAML. -List of Python modules or packages that will be imported in the test -environment. ```yaml -test: - imports: - - bsdiff4 +tests: + - python: + imports: + - bsdiff4 + - bspatch4 + pip_check: true # can be left out because this is the default ``` -This would be equivalent to having a `run_test.py` with the following: +Internally this will write a small Python script that imports the modules: ```python import bsdiff4 +import bspatch4 ``` + ### Check for package-contents -Checks if the built package contains the mentioned items. +Checks if the built package contains the mentioned items. These checks are executed directly at +the end of the build process to make sure that all expected files are present in the package. ```yaml -test: - package-contents: - # checks for the existence of files inside $PREFIX or %PREFIX% - # or, checks that there is at least one file matching the specified `glob` - # pattern inside the prefix - files: - - etc/libmamba/test.txt - - etc/libmamba - - etc/libmamba/*.mamba.txt - - # checks for the existence of `mamba/api/__init__.py` inside of the - # Python site-packages directory (note: also see Python import checks) - site_packages: - - mamba.api - - - # looks in $PREFIX/bin/mamba for unix and %PREFIX%\Library\bin\mamba.exe on Windows - # note: also check the `commands` and execute something like `mamba --help` to make - # sure things work fine - bins: - - mamba - - # searches for `$PREFIX/lib/libmamba.so` or `$PREFIX/lib/libmamba.dylib` on Linux or macOS, - # on Windows for %PREFIX%\Library\lib\mamba.dll & %PREFIX%\Library\bin\mamba.bin - libs: - - mamba - - # searches for `$PREFIX/include/libmamba/mamba.hpp` on unix, and - # on Windows for `%PREFIX%\Library\include\mamba.hpp` - includes: - - libmamba/mamba.hpp +tests: + - package-contents: + # checks for the existence of files inside $PREFIX or %PREFIX% + # or, checks that there is at least one file matching the specified `glob` + # pattern inside the prefix + files: + - etc/libmamba/test.txt + - etc/libmamba + - etc/libmamba/*.mamba.txt + + # checks for the existence of `mamba/api/__init__.py` inside of the + # Python site-packages directory (note: also see Python import checks) + site_packages: + - mamba.api + + + # looks in $PREFIX/bin/mamba for unix and %PREFIX%\Library\bin\mamba.exe on Windows + # note: also check the `commands` and execute something like `mamba --help` to make + # sure things work fine + bin: + - mamba + + # searches for `$PREFIX/lib/libmamba.so` or `$PREFIX/lib/libmamba.dylib` on Linux or macOS, + # on Windows for %PREFIX%\Library\lib\mamba.dll & %PREFIX%\Library\bin\mamba.bin + lib: + - mamba + + # searches for `$PREFIX/include/libmamba/mamba.hpp` on unix, and + # on Windows for `%PREFIX%\Library\include\mamba.hpp` + includes: + - libmamba/mamba.hpp ``` +### Downstream tests + +!!! warning + Downstream tests are not yet implemented in `rattler-build`. + +A downstream test can mention a single package that has a dependency on the package being built. +The test will install the package and run the tests of the downstream package with our current +package as a dependency. + +Sometimes downstream packages do not resolve. In this case, the test is ignored. + +```yaml +tests: + - downstream: numpy +``` Outputs section --------------- diff --git a/docs/testing.md b/docs/testing.md index 9536a2d4..5c82db90 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1,18 +1,23 @@ # Testing packages -When you are developing a package, you should write tests for it. -The tests are automatically executed right after the package build has finished. +When you are developing a package, you should write tests for it. The tests are +automatically executed right after the package build has finished. -The tests from the test section are actually packaged _into_ your package and can -also be executed straight from the existing package. For this, we have the `test` subcommand: +The tests from the test section are actually packaged _into_ your package and +can also be executed straight from the existing package. + +The idea behind adding the tests into the package is that you can execute the +tests independently from building the package. That is also why we are shipping +a `test` subcommand that takes as input an existing package and executes the +tests: ```bash rattler-build test --package-file ./xtensor-0.24.6-h60d57d3_0.tar.bz2 ``` -Running the above command will extract the package and create a clean environment -where the package and dependencies are installed. Then the tests are executed in -this environment. +Running the above command will extract the package and create a clean +environment where the package and dependencies are installed. Then the tests are +executed in this environment. If you inspect the package contents, you would find the test files under `info/test/*`. @@ -22,52 +27,70 @@ If you inspect the package contents, you would find the test files under The test section allows you to specify the following things: ```yaml -test: - # commands to run to test the package. If any of the commands - # returns with an error code, the test is considered failed. - commands: - - echo "Hello world" - - pip check +tests: + - script: + # commands to run to test the package. If any of the commands + # returns with an error code, the test is considered failed. + - echo "Hello world" + - pytest ./tests + + # additional requirements at test time + requirements: + run: + - pytest + + files: + # Extra files to be copied to the test directory from the "work directory" + source: + - tests/ + - test.py + - *.sh + recipe: + - more_tests/*.py # This test section tries to import the Python modules and errors if it can't - imports: - - mypkg - - mypkg.subpkg - - # additional requirements at test time (only in the target platform architecture) - requires: - - pip - - # Extra files to be copied to the test directory from the build dir (can be globs) - files: - - test.py - - "*.sh" - - # Extra files to be copied to the test directory from the source directory (can be globs) - source_files: - - test_files/ + - python: + imports: + - mypkg + - mypkg.subpkg ``` -The files from the `files` and `source_files` sections are copied into the -`info/test/` folder. The `commands` section is turned into a `run_test.sh` -or `run_test.bat` file, depending on the platform. For a `noarch` package, -both are created. The imports section is turned into a `run_test.py` script. +When you are writing a test for your package, additional files are created and +added to your package. These files are placed under the `info/tests/{index}/` +folder per test. -## Internals +For a script test: -When you are writing a test for your package, additional files are created and added to your package. +- All the files are copied straight into the test folder (under + `info/tests/{index}/`) +- The script is turned into a `run_test.sh` or `run_test.bat` file +- The extra requirements are stored as a JSON file called + `test_time_dependencies.json` -The files are: +For a Python import test: -- `run_test.sh` (Unix) -- `run_test.bat` (Windows) -- `run_test.py` (for the Python import tests) +- A JSON file is created that is called `python_test.json` and stores the + imports to be tested and wether to execute `pip check` or not. This file is + placed under `info/tests/{index}/` + +For a downstream test: + +- A JSON file is created that is called `downstream_test.json` and stores the + downstream tests to be executed. This file is placed under + `info/tests/{index}/` -These files are created under the `info/test` directory of the package. -Additionally, any `source_files` or `files` are also moved into this directory. +## Legacy tests -The tests are executed pointing to this directory as the current working directory. +Legacy tests (from conda-build) are still supported for execution. These tests +are stored as files under the `info/test/` folder. + +The files are: + +- `run_test.sh` (Unix) +- `run_test.bat` (Windows) +- `run_test.py` (for the Python import tests) +- `test_time_dependencies.json` (for additional dependencies at test time) -The idea behind adding the tests into the package is that you can execute the tests independent -from building the package. That is also why we are shipping a `test` subcommand that takes -as input an existing package and executes the tests. +Additionally, the `test` folder contains all the files specified in the test +section as `source_files` and `files`. The tests are executed pointing to this +directory as the current working directory. diff --git a/mkdocs.yml b/mkdocs.yml index 54bdf852..09ba44c1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -72,20 +72,22 @@ extra: nav: - Home: index.md - Highlevel overview: highlevel.md + - Recipe file: recipe_file.md + - Selector syntax: selectors.md - Build script: build_script.md - Variants: variants.md - - Package specification: package_spec.md - Compilers and cross compilation: compilers.md + + - CLI Usage: cli_usage.md + - Automatic recipe linting: automatic_linting.md + - Testing packages: testing.md - Reproducible builds: rebuild.md + - Package specification: package_spec.md - Internals: internals.md - - Recipe file: recipe_file.md - - - CLI Usage: cli_usage.md - - Automatic recipe linting: automatic_linting.md plugins: - search