Split test suite with separate docker-compose.

[dblock]

Description

Split the test suite to allow for custom docker-compose.yml.

This is needed in #464 and #368.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Changes Analysis

Commit SHA: ba6ddd3
Comparing To SHA: cfe357b

API Changes

Summary

NO CHANGES

Report

The full API changes report is available at: https://github.com/opensearch-project/opensearch-api-specification/actions/runs/10287706061/artifacts/1786454233

API Coverage

	Before	After	Δ
Covered (%)	510 (49.95 %)	510 (49.95 %)	0 (0 %)
Uncovered (%)	511 (50.05 %)	511 (50.05 %)	0 (0 %)
Unknown	24	24	0

[github-actions]

Spec Test Coverage Analysis

Total	Tested
534	116 (21.72 %)

[nhtruong]

Love the feature!!!

This approach though, is some sort of hidden logic, and can create a knowledge silo in spite of the guide update, in my opinion. It's not obvious to connect that .github/opensearch-cluster/**/* must share an identical folder structure with tests/**/* or a github workflow will fail. The information on which cluster a story should be tested against is also hidden in the name of the grandparent folder the story is in, instead of being explicitly defined in the story itself. I'd like to propose a different approach:

Move the docker-compose files into tests/clusters and stories into tests/stories:

tests
│
├── clusters
│   ├── default.yml
│   ├── index_management.yml
│   └── ...
│
├── stories
│   ├── _core
│   ├── cat
│   ├── sql
│   ├── index
│   └── ...
│
└── test_story.schema.yaml

Inside each story, we can explicitly state which non-default cluster we want to run it against:

$schema: ../../json_schemas/test_story.schema.yaml
cluster: index_management # when not provided, `default` will be used

description: Test cat/count endpoints.
chapters:
...

[dblock]

Inside each story, we can explicitly state which non-default cluster we want to run it against:

How will we modify the test tool to accommodate that?

I don't think the test tool should be in charge of starting a cluster, but it will somehow need to know what cluster is running and to know to skip certain tests. It also will make it confusing for developers who run npm run test:spec--insecure that some tests in the middle of the set are skipped.

Would it be simpler to co-locate tests and cluster configuration (move .github/opensearch-cluster/docker-compose.yml into tests/default) and just use folders to avoid the confusion problem?

tests
│
|── default
|   ├── docker_compose.yml
│   ├── _core
│   ├── cat
│   ├── sql
│   ├── index
│   └── ...
|── plugin
|   ├── docker_compose.yml
│   ├── ...
└── test_story.schema.yaml

[nhtruong]

Inside each story, we can explicitly state which non-default cluster we want to run it against:

How will we modify the test tool to accommodate that?

We can add --cluster option for the test command (default to default cluster). The framework will only read stories for that cluster and display SKIPPED status for other stories (with a message saying why it's skipped)

Would it be simpler to co-locate tests and cluster configuration (move .github/opensearch-cluster/docker-compose.yml into tests/default) and therefore just use folders?

As in each folder immediately under tests determines a cluster, and within each of that folder, there's gonna be a docker-compose.yml file and subfolders for stories? YES that would address the concern I have too.

[nhtruong]

YES, I actually like your 2nd solution better. Let's do that!

[dblock]

YES, I actually like your 2nd solution better. Let's do that!

Done. Take a look.