devcontainer features test documentation

README.md

@@ -15,9 +15,9 @@ This CLI is in active development. Current status:
 - [x] `devcontainer run-user-commands` - Runs lifecycle commands like `postCreateCommand`
 - [x] `devcontainer read-configuration` - Outputs current configuration for workspace
 - [x] `devcontainer exec` - Executes a command in a container with `userEnvProbe`, `remoteUser`, `remoteEnv`, and other properties applied
+- [x] `devcontainer features <...>` - Tools to assist in authoring and testing [dev container 'features'](https://containers.dev/implementors/features/)
 - [ ] `devcontainer stop` - Stops containers
 - [ ] `devcontainer down` - Stops and deletes containers
-- [ ] `devcontainer features <...>` - Tools to assist in authoring and testing [dev container 'features'](https://github.com/devcontainers/spec/blob/main/proposals/devcontainer-features.md)
 
 ## Try it out
 
@@ -133,6 +133,10 @@ Learn more on the [dev container spec website](https://devcontainers.github.io/)
 
 You may review other resources part of the specification in the [`devcontainers` GitHub organization](https://github.com/devcontainers).
 
+### Documentation
+
+- Additional information on using the built-in [Features testing command](./docs/features/test.md).
+
 ## Contributing
 
 Check out how to contribute to the CLI in [CONTRIBUTING.md](CONTRIBUTING.md).

docs/features/test.md

@@ -0,0 +1,157 @@
+# Testing Dev Container Features
+
+A built-in testing framework for Features is currently under active development.  This command aims to make it easy to iterate on [self-authored Features](https://containers.dev/implementors/features-distribution/).
+
+This command utilizes the CLI's `build` and `exec` commands to test Features in your local source tree.  The command will look at the `target` path for mirrored `src` and `test` directories ([example](https://github.com/devcontainers/features).  Without any additional arguments, the `test` command will auto-generate a test for each Feature (pulling the source code directly from `src/<FEATURE>`), and exec `test/<FEATURE>/test.sh` inside of the running container.  
+
+For the test to pass, the container must (1) build and start successfully, and (2) execute the `test.sh` with a success (zero) exit code.  Note that auto-generated tests will execute the given Feature
+with default options.
+
+Additonally, 'scenarios' can be defined for each Feature to test more complicated cases (ie: Passing options to tests, testing >1 Feature together, etc.).  More information on [writing scenarios can be found below](#scenarios).
+
+The source code can of the sub-command is [here](../../src/spec-node/featuresCLI/test.ts). An example of the command being used in CI can be [found in the `devcontainers/features-template` repo](https://github.com/devcontainers/feature-template/blob/main/.github/workflows/test.yaml) and the [`devcontainers/features` repo](https://github.com/devcontainers/features).
+
+```
+devcontainer features test <target>
+
+Test features
+
+Positionals:
+  target  Path to folder containing 'src' and 'test' sub-folders.                     [string] [required] [default: "."]
+
+Options:
+      --help                   Show help                                                                       [boolean]
+      --version                Show version number                                                             [boolean]
+  -f, --features               Feature(s) to test as space-separated parameters. Omit to run all tests.  Cannot be
+                               combined with '--global-scenarios-only'.                                          [array]
+      --global-scenarios-only  Run only scenario tests under 'tests/_global' .  Cannot be combined with '-f'.
+                                                                                              [boolean] [default: false]
+      --skip-scenarios         Skip all 'scenario' style tests.  Cannot be combined with '--global--scenarios-only'.
+                                                                                              [boolean] [default: false]
+      --skip-autogenerated     Skip all 'autogenerated' style tests.                          [boolean] [default: false]
+  -i, --base-image             Base Image. Not used for scenarios.                            [string] [default: "ubuntu:focal"]
+  -u, --remote-user            Remote user. Not used for scenarios.                           [string] [default: "root"]
+      --log-level              Log level.                          [choices: "info", "debug", "trace"] [default: "info"]
+  -q, --quiet                  Quiets output                                                  [boolean] [default: false]
+```
+
+An example project structure can be found below.
+
+```
+.
+├── README.md
+├── src
+│   ├── dotnet
+│   │   ├── devcontainer-feature.json
+│   │   └── install.sh
+│   ├── oryx
+│   │   ├── devcontainer-feature.json
+│   │   └── install.sh
+|   ├── ...
+│   │   ├── devcontainer-feature.json
+│   │   └── install.sh
+├── test
+│   ├── _global
+│	│	├── scenarios.json
+│   │   └── some_test_scenario.json
+│   ├── dotnet
+│   │   └── test.sh
+│   ├── oryx
+|   |   ├── scenarios.json
+|   |   ├── install_dotnet_and_oryx.json
+│   |   └── test.sh
+|   ├── ...
+│   │   └── test.sh
+...
+```
+
+## Scenarios 
+
+Scenarios are an additional mode that augments the auto-generated test (that is asserted with the `test/<FEATURE>/test.sh` script).  
+
+Scenarios are snippets of `devcontainer.json` configuration.  The scenario is a JSON object, where the key is the test name, and the object is a `devcontainer.json`.
+
+> I'll use the [`oryx` Feature](https://github.com/devcontainers/features/tree/2d89dc301ed834d74b07350c7d8567eee78f966d/test/oryx) as an example if you'd like a live code sample.
+
+The following `scenarios.json` defines a single test scenario named `install_dotnet_and_oryx`.  The scenario will install the `dotnet` and `oryx` Features in the target repo with the provided options. 
+
+
+##### test/oryx/scenarios.json
+```json
+{
+    "install_dotnet_and_oryx": {
+        "image": "ubuntu:focal",
+        "features": {
+            "dotnet": {
+                "version": "6",
+                "installUsingApt": "false"
+            },
+            "oryx": {}
+        }
+    }
+}
+```
+
+The test command will build a container with the config above, and then look for a `.sh` test file with the same name.  The test will pass if the container builds successfully
+and the `install_dotnet_and_oryx.sh` shell script exits will a successful exit code (0).
+
+##### test/install_dotnet_and_oryx.sh
+```
+#!/bin/bash
+
+set -e
+
+# Optional: Import test library
+source dev-container-features-test-lib
+
+check "Oryx version" oryx --version
+check "Dotnet is not removed if it is not installed by the Oryx Feature" dotnet --version
+
+# Install platforms with oryx build tool
+check "oryx-install-dotnet-2.1" oryx prep --skip-detection --platforms-and-versions dotnet=2.1.30
+check "dotnet-2-installed-by-oryx" ls /opt/dotnet/ | grep 2.1
+
+....
+....
+
+# Replicates Oryx's behavior for universal image
+mkdir -p /opt/oryx
+echo "vso-focal" >> /opt/oryx/.imagetype
+
+mkdir -p /opt/dotnet/lts
+cp -R /usr/local/dotnet/current/dotnet /opt/dotnet/lts
+cp -R /usr/local/dotnet/current/LICENSE.txt /opt/dotnet/lts
+cp -R /usr/local/dotnet/current/ThirdPartyNotices.txt /opt/dotnet/lts
+
+....
+....
+
+# Report result
+reportResults
+```
+
+The flags `--global-scenarios-only`, `--skip-scenarios`, and `--skip-autogenerated` can be passed to run a subset of tests.
+
+
+### Global Scenarios
+
+The `test/_global` directory is a special directory that holds scenario tests not tied to a specific Feature. This directory is useful for scenarios that broadly tests several Features in a given repository.
+
+The `--global-scenarios-only` can be passed to only run the global scenarios.
+
+
+## dev-container-features-test-lib
+
+The `dev-container-features-test-lib` is convenience helper [defined in the CLI](https://github.com/devcontainers/cli/blob/1910ca41015c627b884ddd69ebc52d1e8cdd8cf0/src/spec-node/featuresCLI/utils.ts#L59).  This shell script can be sourced in any shell script test file when running the `devcontainer features test` command, **although it is not required**.
+
+#### `check <LABEL> <cmd>`
+Description: Executes `cmd` and prints success/failed depending on exit code (0 === success) of `cmd`.
+Note: Do not use quotes for cmd as there is no such executable as e.g. "python3 --version".
+Example: `check "python is available" python3 --version`
+
+#### `checkMultiple <LABEL> <MINIMUMPASSED> <cmd1> [cmd2] [cmd3] ...`
+Description: Executes all provided commands succeeds if the number of `cmdX` is greater than `$MINIMUMPASSED`
+Note: Do not use quotes for cmd as there is no such executable as e.g. "python3 --version".
+
+##### reportResults
+Prints results of check and checkMultiple

src/spec-node/featuresCLI/test.ts

@@ -14,8 +14,8 @@ export function featuresTestOptions(y: Argv) {
 			'global-scenarios-only': { type: 'boolean', default: false, description: 'Run only scenario tests under \'tests/_global\' .  Cannot be combined with \'-f\'.' },
 			'skip-scenarios': { type: 'boolean', default: false, description: 'Skip all \'scenario\' style tests.  Cannot be combined with \'--global--scenarios-only\'.' },
 			'skip-autogenerated': { type: 'boolean', default: false, description: 'Skip all \'autogenerated\' style tests.' },
-			'base-image': { type: 'string', alias: 'i', default: 'ubuntu:focal', description: 'Base Image' },  // TODO: Optionally replace 'scenario' configs with this value?
-			'remote-user': { type: 'string', alias: 'u', default: 'root', describe: 'Remote user', },  // TODO: Optionally replace 'scenario' configs with this value?
+			'base-image': { type: 'string', alias: 'i', default: 'ubuntu:focal', description: 'Base Image. Not used for scenarios.' },  // TODO: Optionally replace 'scenario' configs with this value?
+			'remote-user': { type: 'string', alias: 'u', default: 'root', describe: 'Remote user.  Not used for scenarios.', },  // TODO: Optionally replace 'scenario' configs with this value?
 			'log-level': { choices: ['info' as 'info', 'debug' as 'debug', 'trace' as 'trace'], default: 'info' as 'info', description: 'Log level.' },
 			'quiet': { type: 'boolean', alias: 'q', default: false, description: 'Quiets output' },
 		})