devcontainer features test documentation (#219)

* first pass at devcontainer features test docs

* add example project structure

* Apply suggestions from code review

Co-authored-by: JP Ungaretti <19893438+jungaretti@users.noreply.github.com>
Co-authored-by: Brigit Murtaugh <brigit.murtaugh@microsoft.com>
Co-authored-by: Alexander Lanin <alex@lanin.de>

* code review

Co-authored-by: JP Ungaretti <19893438+jungaretti@users.noreply.github.com>
Co-authored-by: Brigit Murtaugh <brigit.murtaugh@microsoft.com>
Co-authored-by: Alexander Lanin <alex@lanin.de>

README.md

@@ -15,9 +15,9 @@ This CLI is in active development. Current status:
 - [x] `devcontainer run-user-commands` - Runs lifecycle commands like `postCreateCommand`
 - [x] `devcontainer read-configuration` - Outputs current configuration for workspace
 - [x] `devcontainer exec` - Executes a command in a container with `userEnvProbe`, `remoteUser`, `remoteEnv`, and other properties applied
+- [x] `devcontainer features <...>` - Tools to assist in authoring and testing [dev container Features](https://containers.dev/implementors/features/)
 - [ ] `devcontainer stop` - Stops containers
 - [ ] `devcontainer down` - Stops and deletes containers
-- [ ] `devcontainer features <...>` - Tools to assist in authoring and testing [dev container 'features'](https://github.com/devcontainers/spec/blob/main/proposals/devcontainer-features.md)
 
 ## Try it out
 
@@ -133,6 +133,10 @@ Learn more on the [dev container spec website](https://devcontainers.github.io/)
 
 You may review other resources part of the specification in the [`devcontainers` GitHub organization](https://github.com/devcontainers).
 
+### Documentation
+
+- Additional information on using the built-in [Features testing command](./docs/features/test.md).
+
 ## Contributing
 
 Check out how to contribute to the CLI in [CONTRIBUTING.md](CONTRIBUTING.md).

docs/features/test.md

@@ -0,0 +1,133 @@
+# Testing Dev Container Features
+
+A built-in testing framework for Features is in active development.  This command helps you iterate on [self-authored Features](https://containers.dev/implementors/features-distribution/).
+
+The `test` command utilizes the CLI's `build` and `exec` commands to test Features in your local source tree.  The command will look at the `target` path for mirrored `src` and `test` directories ([example](https://github.com/devcontainers/features).  Without any additional arguments, the `test` command will auto-generate a test for each Feature (pulling the source code directly from `src/<FEATURE>`), and exec `test/<FEATURE>/test.sh` inside of the running container.  
+
+For the test to pass, the container must (1) build and start successfully, and (2) execute the `test.sh` with a success (zero) exit code.  Note that auto-generated tests will execute the given Feature with default options.
+
+Additonally, 'scenarios' can be defined for each Feature to test more complicated cases.  For example, you can use a scenario to test Feature options or >1 Feature in a container.  You can find more information about [writing scenario tests below](#scenarios).
+
+The source code of the sub-command is [here](../../src/spec-node/featuresCLI/test.ts). An example of the command being used in CI can be [found in the `devcontainers/features-template` repo](https://github.com/devcontainers/feature-template/blob/main/.github/workflows/test.yaml) and the [`devcontainers/features` repo](https://github.com/devcontainers/features).
+
+For more information on the `test` command, run `devcontainer features test --help`.
+
+An example project structure can be found below.
+
+```
+.
+├── README.md
+├── src
+│   ├── dotnet
+│   │   ├── devcontainer-feature.json
+│   │   └── install.sh
+│   ├── oryx
+│   │   ├── devcontainer-feature.json
+│   │   └── install.sh
+|   ├── ...
+│   │   ├── devcontainer-feature.json
+│   │   └── install.sh
+├── test
+│   ├── _global
+│	│	├── scenarios.json
+│   │   └── some_test_scenario.sh
+│   ├── dotnet
+│   │   └── test.sh
+│   ├── oryx
+|   |   ├── scenarios.json
+|   |   ├── install_dotnet_and_oryx.sh
+│   |   └── test.sh
+|   ├── ...
+│   │   └── test.sh
+...
+```
+
+To run all the `dotnet` related tests from a repo structured above, the command would be:
+
+```bash
+devcontainer features test  -f dotnet -b ubuntu
+```
+
+## Scenarios 
+
+Scenarios are an additional mode that augments the auto-generated test (that is asserted with the `test/<FEATURE>/test.sh` script).  
+
+Scenarios are snippets of `devcontainer.json` configuration.  The scenario is a JSON object, where the key is the test name, and the object is a `devcontainer.json`.
+
+> The following example references the [`oryx` Feature](https://github.com/devcontainers/features/tree/2d89dc301ed834d74b07350c7d8567eee78f966d/test/oryx).
+
+The following `scenarios.json` defines a single test scenario named `install_dotnet_and_oryx`.  The scenario will install the `dotnet` and `oryx` Features in the target repo with the provided options. 
+
+
+##### test/oryx/scenarios.json
+```json
+{
+    "install_dotnet_and_oryx": {
+        "image": "ubuntu:focal",
+        "features": {
+            "dotnet": {
+                "version": "6",
+                "installUsingApt": "false"
+            },
+            "oryx": {}
+        }
+    }
+}
+```
+
+The test command will build a container with the config above, and then look for a `.sh` test file with the same name.  The test will pass if the container builds successfully and the `install_dotnet_and_oryx.sh` shell script exits will a successful exit code (0).
+
+##### test/install_dotnet_and_oryx.sh
+```
+#!/bin/bash
+
+set -e
+
+# Import test library for `check` command
+source dev-container-features-test-lib
+
+check "Oryx version" oryx --version
+check "Dotnet is not removed if it is not installed by the Oryx Feature" dotnet --version
+
+# Install platforms with oryx build tool
+check "oryx-install-dotnet-2.1" oryx prep --skip-detection --platforms-and-versions dotnet=2.1.30
+check "dotnet-2-installed-by-oryx" ls /opt/dotnet/ | grep 2.1
+
+....
+....
+
+# Replicates Oryx's behavior for universal image
+mkdir -p /opt/oryx
+echo "vso-focal" >> /opt/oryx/.imagetype
+
+mkdir -p /opt/dotnet/lts
+cp -R /usr/local/dotnet/current/dotnet /opt/dotnet/lts
+cp -R /usr/local/dotnet/current/LICENSE.txt /opt/dotnet/lts
+cp -R /usr/local/dotnet/current/ThirdPartyNotices.txt /opt/dotnet/lts
+
+....
+....
+
+# Report result
+reportResults
+```
+
+The flags `--global-scenarios-only`, `--skip-scenarios`, and `--skip-autogenerated` can be passed to run a subset of tests.
+
+### Global Scenarios
+
+The `test/_global` directory is a special directory that holds scenario tests not tied to a specific Feature. This directory is useful for scenarios that broadly tests several Features in a given repository.
+
+The `--global-scenarios-only` can be passed to only run the global scenarios.
+
+## dev-container-features-test-lib
+
+The `dev-container-features-test-lib` is convenience helper [defined in the CLI](https://github.com/devcontainers/cli/blob/1910ca41015c627b884ddd69ebc52d1e8cdd8cf0/src/spec-node/featuresCLI/utils.ts#L59) that adds several bash functions to organize test asserts. Note that using this libary **is not required**.
+
+#### `check <LABEL> <cmd> [args...]`
+Description: Executes `cmd` and prints success/failed depending on exit code (0 === success) of `cmd`.
+Note: Use quotes to include whitespace in the label or individual arguments for the command.
+Example: `check "python is available" python3 --version`
+
+##### reportResults
+Prints results of check and checkMultiple

src/spec-node/featuresCLI/test.ts

@@ -14,8 +14,8 @@ export function featuresTestOptions(y: Argv) {
 			'global-scenarios-only': { type: 'boolean', default: false, description: 'Run only scenario tests under \'tests/_global\' .  Cannot be combined with \'-f\'.' },
 			'skip-scenarios': { type: 'boolean', default: false, description: 'Skip all \'scenario\' style tests.  Cannot be combined with \'--global--scenarios-only\'.' },
 			'skip-autogenerated': { type: 'boolean', default: false, description: 'Skip all \'autogenerated\' style tests.' },
-			'base-image': { type: 'string', alias: 'i', default: 'ubuntu:focal', description: 'Base Image' },  // TODO: Optionally replace 'scenario' configs with this value?
-			'remote-user': { type: 'string', alias: 'u', default: 'root', describe: 'Remote user', },  // TODO: Optionally replace 'scenario' configs with this value?
+			'base-image': { type: 'string', alias: 'i', default: 'ubuntu:focal', description: 'Base Image. Not used for scenarios.' },  // TODO: Optionally replace 'scenario' configs with this value?
+			'remote-user': { type: 'string', alias: 'u', default: 'root', describe: 'Remote user.  Not used for scenarios.', },  // TODO: Optionally replace 'scenario' configs with this value?
 			'log-level': { choices: ['info' as 'info', 'debug' as 'debug', 'trace' as 'trace'], default: 'info' as 'info', description: 'Log level.' },
 			'quiet': { type: 'boolean', alias: 'q', default: false, description: 'Quiets output' },
 		})