Run Notebook Tests in Parallel GitHub Workflow Runs

[ckunki]

Enable to run Notebook Tests in Parallel GitHub Workflow Runs.

Currently,

• executing the tests for all Jupyter Notebooks inside the AI-Lab take very long.
• Some notebook tests are disabled:
  • disabled_nbtest_cloud.py by prefix of filename
  • nbtest_sagemaker.py by using @pytest.mark.skip(reason="Want to get the results for other tests quickly")

Sub Tasks

• Implement a Nox task enumerating all notebook tests and returning a build matrix in JSON format
• Add support for a CLI parameter enabling to execute only the specified notebook test
• Change the GitHub workflow to
  • call the Nox task
  • build a dynamic build matrix
  • use the CLI parameter accordingly
• Re-enable the disabled notebook tests named above

Enhancing notebook_test_runner.test_notebook()

• By default execute all test
• If CLI parameter is specified then only execute the selected test case

[tkilias]

Suggestion for the parameterized fixture with cli parameter

@fixture(scope="module", params=[
    python_file.name
    for python_file in sorted(TEST_RESOURCE_PATH.glob("nbtest_*.py"))
    if python_file.is_file()
    ])
def notebook_test_file(request):
    if request.options.get("cli-parameter") == request.param:
        return request.param
    else:
        pytest.skip("Test disabled")


def test_notebook(notebook_test_container_with_log, notebook_test_file):
    ...