run pytest with or without docs and notebooks in run_pytest.sh

[dorotat-nv]

Description

We've streamlined our testing workflow to improve CI efficiency. By default, documentation tests and notebook validations are skipped to reduce pipeline duration.

Type of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactor
• Documentation update
• Other (please describe): user guideline, updated to CI workflow

CI Pipeline Configuration

Configure CI behavior by checking relevant boxes below. This will automatically apply labels.

• SKIP_CI - Skip all continuous integration tests
• INCLUDE_NOTEBOOKS_TESTS - Execute notebook validation tests in pytest

Note

By default, the notebooks validation tests are skipped unless explicitly enabled.

Usage

When running tests locally, use these flags to control test execution:

./ci/scripts/run_pytest.sh --skip-docs
./ci/scripts/run_pytest.sh --no-nbval
./ci/scripts/run_pytest.sh --no-nbval --skip-docs

Control test pipeline execution by adding these labels to your PR:
RUN_DOCS - Enables documentation testing
RUN_NOTEBOOKS_VALIDATION - Enables notebook validation

Pre-submit Checklist

• I have tested these changes locally
• I have updated the documentation accordingly
• I have added/updated tests as needed
• All existing tests pass successfully

[dorotat-nv]

/build-ci

[dorotat-nv]

/build-ci

[dorotat-nv]

/build-ci

[dorotat-nv]

/build-ci

[dorotat-nv]

/build-ci

[pstjohn]

awesome, LGTM, thanks!

[jstjohn]

Approved, but see my comment about the --nbval-lax argument. Maybe change the option to --skip-notebooks and if that's set then drop that kwarg.

[dorotat-nv]

/build-ci

[dorotat-nv]

/build-ci

[dorotat-nv]

/build-ci

[dorotat-nv]

/build-ci

[jstjohn]

One thought: when you run docs, there is only notebook validation in there. I am a bit worried a user will add a notebook to docs, set RUN_NOTEBOOKS_VALIDATION and not realize they also need to add RUN_DOCS_VALIDATION if their notebook happens to be in the docs folder rather than a sub-module. Basically you currently support:

• No notebook tests (default)
• Only notebooks tests in sub-packages (if you say RUN_NOTEBOOKS_VALIDATION)
• All notebooks tests (if you say both RUN_NOTEBOOKS_VALIDATION and RUN_DOCS_VALIDATION)

Note that only saying RUN_DOCS_VALIDATION would be more or less a no-op since the only tests in there are notebook tests so without RUN_NOTEBOOKS_VALIDATION also it would not do anything other than run the base tests in the sub-modules, which would be pretty confusing behavior given that option name.

My proposal would be to, for now, only have RUN_NOTEBOOKS_VALIDATION and have that option check in docs as well as sub-packages for notebooks to test. Thoughts? Also I don't think the argument would only run notebooks, it would include notebooks. So maybe the least confusing thing would be to say INCLUDE_NOTEBOOKS_VALIDATION and then it's clearer that all tests would run but notebooks would be included.

[dorotat-nv]

/build-ci

[dorotat-nv]

/build-ci