Add helm test integration

Fixes #369.

README.md

@@ -31,6 +31,7 @@ performed by the operator.
     or a (local) URL
 * Automated purging on release install failures
 * Automated (optional) rollback on upgrade failures
+* Automated (optional) helm test gating of installs and upgrades.
 * Automated image upgrades [using Flux](https://docs.fluxcd.io/en/stable/references/helm-operator-integration)
 * Automated (configurable) chart dependency updates for Helm charts
   from Git sources on install or upgrade

chart/helm-operator/crds/helmrelease.yaml

@@ -191,6 +191,23 @@ spec:
                 Helm release. The default namespace equals to the namespace of the
                 HelmRelease resource.
               type: string
+            test:
+              description: The test settings for this Helm release.
+              type: object
+              properties:
+                cleanup:
+                  description: Cleanup, when targeting Helm 2, determines whether
+                    to delete test pods between each test run initiated by the Helm
+                    Operator.
+                  type: boolean
+                enable:
+                  description: Enable will mark this Helm release for tests.
+                  type: boolean
+                timeout:
+                  description: Timeout is the time to wait for any individual Kubernetes
+                    operation (like Jobs for hooks) during test.
+                  type: integer
+                  format: int64
             timeout:
               description: Timeout is the time to wait for any individual Kubernetes
                 operation (like Jobs for hooks) during installation and upgrade operations.
@@ -321,13 +338,15 @@ spec:
                     - "False"
                     - Unknown
                   type:
-                    description: Type of the condition, one of ('ChartFetched', 'Released',
-                      'RolledBack').
+                    description: Type of the condition, one of ('ChartFetched', 'Deployed',
+                      'Released', 'RolledBack', 'Tested').
                     type: string
                     enum:
                     - ChartFetched
+                    - Deployed
                     - Released
                     - RolledBack
+                    - Tested
             lastAttemptedRevision:
               description: LastAttemptedRevision is the revision of the latest chart
                 sync, and may be of a failed release.
@@ -339,14 +358,20 @@ spec:
               format: int64
             phase:
               description: Phase the release is in, one of ('ChartFetched', 'ChartFetchFailed',
-                'Installing', 'Upgrading', 'Succeeded', 'RollingBack', 'RolledBack',
+                'Installing', 'Upgrading', 'Deployed', 'DeployFailed', 'Testing',
+                'TestFailed', 'Tested', 'Succeeded', 'RollingBack', 'RolledBack',
                 'RollbackFailed')
               type: string
               enum:
               - ChartFetched
               - ChartFetchFailed
               - Installing
               - Upgrading
+              - Deployed
+              - DeployFailed
+              - Testing
+              - TestFailed
+              - Tested
               - Succeeded
               - Failed
               - RollingBack

deploy/crds.yaml

@@ -192,6 +192,23 @@ spec:
                 Helm release. The default namespace equals to the namespace of the
                 HelmRelease resource.
               type: string
+            test:
+              description: The test settings for this Helm release.
+              type: object
+              properties:
+                cleanup:
+                  description: Cleanup, when targeting Helm 2, determines whether
+                    to delete test pods between each test run initiated by the Helm
+                    Operator.
+                  type: boolean
+                enable:
+                  description: Enable will mark this Helm release for tests.
+                  type: boolean
+                timeout:
+                  description: Timeout is the time to wait for any individual Kubernetes
+                    operation (like Jobs for hooks) during test.
+                  type: integer
+                  format: int64
             timeout:
               description: Timeout is the time to wait for any individual Kubernetes
                 operation (like Jobs for hooks) during installation and upgrade operations.
@@ -322,13 +339,15 @@ spec:
                     - "False"
                     - Unknown
                   type:
-                    description: Type of the condition, one of ('ChartFetched', 'Released',
-                      'RolledBack').
+                    description: Type of the condition, one of ('ChartFetched', 'Deployed',
+                      'Released', 'RolledBack', 'Tested').
                     type: string
                     enum:
                     - ChartFetched
+                    - Deployed
                     - Released
                     - RolledBack
+                    - Tested
             lastAttemptedRevision:
               description: LastAttemptedRevision is the revision of the latest chart
                 sync, and may be of a failed release.
@@ -340,14 +359,20 @@ spec:
               format: int64
             phase:
               description: Phase the release is in, one of ('ChartFetched', 'ChartFetchFailed',
-                'Installing', 'Upgrading', 'Succeeded', 'RollingBack', 'RolledBack',
+                'Installing', 'Upgrading', 'Deployed', 'DeployFailed', 'Testing',
+                'TestFailed', 'Tested', 'Succeeded', 'RollingBack', 'RolledBack',
                 'RollbackFailed')
               type: string
               enum:
               - ChartFetched
               - ChartFetchFailed
               - Installing
               - Upgrading
+              - Deployed
+              - DeployFailed
+              - Testing
+              - TestFailed
+              - Tested
               - Succeeded
               - Failed
               - RollingBack

docs/helmrelease-guide/release-configuration.md

@@ -102,16 +102,16 @@ spec:
 ## Wait for resources to be ready
 
 !!! note
-    When you have many `HelmRelease` resources with the `.wait` flag
-    set, it is likely that you want to increase the amount of `--workers` to
+    When you have many `HelmRelease` resources with the waiting enabled,
+    it is likely that you want to increase the amount of `--workers` to
     ensure other releases are still processed, given the waiting process blocks
     the worker from processing other `HelmRelease` resources.
 
-Normally Helm will mark a release as successfully deployed as soon as the
+By default, Helm will mark a release as successfully deployed as soon as the
 changes have been applied to the cluster. To instruct Helm to wait until
 all resources are in ready state before marking the release as successful
-you can set `.wait`. When set, it will wait for as long as
-[`.timeout`](#configuring-the-timeout):
+you can set `.wait`, or enable [tests](tests.md#enabling-tests) or [rollbacks](rollbacks.md#enabling-rollbacks) which has the same effect.
+When set, it will wait for as long as [`.timeout`](#configuring-the-timeout):
 
 ```yaml
 spec:

docs/helmrelease-guide/rollbacks.md

@@ -13,10 +13,11 @@ rollbacks.
 ## Enabling rollbacks
 
 When rollbacks for a `HelmRelease` are enabled, the Helm Operator will detect
-a faulty upgrade and instruct Helm to perform a rollback, it will not attempt
-a new upgrade unless it detects a change in values and/or the chart, or [retries
-have been enabled](#enabling-retries-of-rolled-back-releases). Changes are
-detected by comparing the failed release to a fresh dry-run release.
+a faulty upgrade, including post-upgrade helm test [if enabled](tests.md#enabling-tests)
+failures, and instruct Helm to perform a rollback, it will not attempt a new
+upgrade unless it detects a change in values and/or the chart, or
+[retries have been enabled](#enabling-retries-of-rolled-back-releases). Changes
+are detected by comparing the failed release to a fresh dry-run release.
 
 Rollbacks can be enabled by setting `.rollback.enable`:
 
@@ -26,6 +27,12 @@ spec:
     enable: true
 ```
 
+## Wait interaction
+
+When rollbacks are enabled, [resource waiting](release-configuration.md#wait-for-resources-to-be-ready)
+defaults to `true` since this is necessary to validate whether the release should
+be rolled back or not.
+
 ## Tweaking the rollback configuration
 
 To get more fine-grained control over how the rollback is performed by Helm,

docs/helmrelease-guide/tests.md

@@ -0,0 +1,70 @@
+# Tests
+
+[Helm tests](https://helm.sh/docs/topics/chart_tests/) are a useful validation
+mechanism for Helm Releases, and thus are supported by the Helm Operator.
+
+## Enabling tests
+
+When tests for a `HelmRelease` are enabled, the Helm Operator will run them
+after any successful installation or upgrade attempt. In the case of a test
+failure, the prior installation or upgrade will be treated as failed, resulting
+in the release being purged or rolled back [if enabled](rollbacks.md#enabling-rollbacks).
+
+Tests can be enabled by setting `.test.enable`:
+
+```yaml
+spec:
+  test:
+    enable: true
+```
+
+## Wait interaction
+
+When tests are enabled, [resource waiting](release-configuration.md#wait-for-resources-to-be-ready)
+defaults to `true` since this is likely needed for test pre-conditions to be satisfied.
+
+## Test timeout
+
+Test timeout can be set via the `.test.timeout` option.
+
+```yaml
+spec:
+  test:
+    enable: true
+    timeout: 600
+```
+
+It is defined as the time to wait for any individual Kubernetes operation during
+tests in seconds. Defaults to `300` when omitted.
+
+## Helm 2 vs 3
+
+In Helm 3, test-specific funcationality was removed in favor of a generic `test`
+hook no different than any other hook. The Helm Operator takes this into account
+as detailed below.
+
+### Test cleanup
+
+Helm 3 removed the `helm test --cleanup` flag in favor of [hook delete policies](https://helm.sh/docs/topics/chart_tests/#notes).
+For `HelmRelease`s targeting Helm 2, test cleanup is enabled by default since
+upgrades are highly likely to cause test pod name conflicts without it.
+This flag currently deletes test pods immediately after they are run, but the only
+guarantee is that tests are cleaned up before running a subsequent test for the
+same `HelmRelease`, as delaying the deletion would allow time to debug failures,
+and thus may be implemented in the future. Test cleanup can be disabled by setting
+`.test.cleanup` to `false`.
+
+```yaml
+spec:
+  helmVersion: v2
+  test:
+    enable: true
+    cleanup: false
+```
+
+### Test parallelism
+
+Helm 2 supported `helm test --parallel --max 10` to run tests in parallel. Helm 3 will
+likely [expand this functionality to all hooks](https://github.com/helm/helm/issues/7763). Once the Helm 3 implementation is available
+this can be integrated into the Helm Operator, and translated into the equivalent
+Helm 2 options for test parallelism as well.

docs/index.md

@@ -27,6 +27,7 @@ performed by the operator.
       or a (local) URL
 * Automated purging on release install failures
 * Automated (optional) rollback on upgrade failures
+* Automated (optional) helm test gating of installs and upgrades.
 * Automated image upgrades [using Flux](https://docs.fluxcd.io/en/stable/references/helm-operator-integration)
 * Automated (configurable) chart dependency updates for Helm charts
   from Git sources on install or upgrade