Create a zarf package publish command (#1336)

## Description ### 💲 `zarf package publish` This PR adds a new `zarf package publish` command. This command will publish a fully built (via `zarf package create`) Zarf package to an OCI compliant registry using the oras library. The command's usage is based off of how Helm's `helm push` command operates and has many of the same restrictions. <https://v3.helm.sh/docs/topics/registries/#the-push-subcommand> ``` $ zarf package publish [PATH_TO_TAR] [REGISTRY_URL] $ zarf package publish zarf-package-strimzi.tar oci://localhost:666/defenseunicorns ``` #### CLI Syntax restrictions: - The OCI URL _must_ start with `oci://` - A basename/version _cannot_ be provided, as it is derived from information present in the built `zarf.yaml` where it _must_ be set. - The command may return an error, permission denied, if the repository does not exist, ie `defenseunicorns/strimzi` as an example on Docker Hub. - A `zarf tools registry login` must be done prior as this will inherit the credentials from docker's cred system (defaultly located at `~/.docker`), or users can create a Docker compatible `config.json` and point its directory: ```sh # example symlinking podman $ mkdir ~/.docker/ $ ln -s $XDG_RUNTIME_DIR/containers/auth.json ~/.docker/config.json $ zarf package publish <...> ``` ### 💲 `zarf package deploy oci://` This PR also adds a new `deploy oci://` feature. This command uses the existing `deploy` system, but does some slightly different behavior from `sget://`. oras is used to pull the package layer by layer to the temp directory. There is no need to unarchive, as the package is already in an uncompressed state. The temp path is then used by the packager to deploy the package as though it is a local package. Usage is denoted in the following example: ``` $ zarf package deploy oci://REGISTRY/NAMESPACE/NAME:VERSION $ zarf package deploy oci://docker.io/defenseunicorns/strimzi:v0.24.0-arm64 $ zarf package deploy oci://localhost:666/strimzi:v0.24.0-arm64 --insecure ``` ### 💲 `zarf package inspect oci://` This PR also adds a new `inspect oci://` feature. ``` $ zarf package inspect oci://docker.io/defenseunicorns/strimzi tags: - v0.23.5-14-arm64 latest: tag: v0.23.5-14-arm64 descriptor: mediaType: application/vnd.oci.image.manifest.v1+json digest: sha256:340f489a105e476f846203e6844b2738f2924fc608522711761a21b176d6b67f size: 41817 ``` ## Related Issue Relates to #1298 Fixes #381 Fixes #823 Blocked by #1331 ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Other (security config, docs update, etc) ## Checklist before merging - [x] Test, docs, adr added or updated as needed - [x] [Contributor Guide Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow) followed --------- Co-authored-by: Wayne Starr <Racer159@users.noreply.github.com> Co-authored-by: Wayne Starr <me@racer159.com>
zarf-dev · Mar 1, 2023 · 061165a · 061165a
1 parent df3f933
commit 061165a
Show file tree

Hide file tree

Showing 27 changed files with 1,086 additions and 30 deletions.
diff --git a/.github/workflows/scan-labels.yml b/.github/workflows/scan-labels.yml
@@ -9,4 +9,4 @@ jobs:
     steps:
       - uses: yogevbd/enforce-label-action@2.2.2
         with:
-          BANNED_LABELS: "needs-docs,needs-tests,needs-adr,needs-git-sign-off"
+          BANNED_LABELS: "needs-docs,needs-tests,needs-adr,needs-git-sign-off,needs-walkthrough"
diff --git a/docs/13-walkthroughs/6-publish-and-deploy.md b/docs/13-walkthroughs/6-publish-and-deploy.md
@@ -0,0 +1,115 @@
+# Publishing + Deploying A Zarf Package Via OCI
+
+## Introduction
+
+In this walkthrough, we are going to run through how to publish a Zarf package to an [OCI](https://github.com/opencontainers/image-spec) compliant registry, allowing end users to pull and deploy packages without needing to build locally, or transfer the package to their environment.
+
+![Walkthrough GIF](../.images/walkthrough-6.gif)
+
+## Prerequisites
+
+For following along locally, please ensure the following prerequisites are met:
+
+1. Zarf binary installed on your $PATH: ([Install Instructions](../3-getting-started.md#installing-zarf))
+2. Access to a [Registry supporting the OCI Distribution Spec](https://oras.land/implementors/#registries-supporting-oci-artifacts), this walkthrough will be using Docker Hub
+3. The following operations:
+
+```bash
+# Setup some variables for the registry we will be using
+$ REGISTRY=docker.io
+$ set +o history
+$ REGISTRY_USERNAME=<username> # <-- replace with your username
+$ REGISTRY_SECRET=<secret> # <-- replace with your password or auth token
+$ set -o history
+
+# Make a new directory to work in
+$ mkdir -p zarf-publish-walkthrough && cd zarf-publish-walkthrough
+
+# For this walkthrough we will use the `helm-oci-chart` example package
+# located here: https://github.com/defenseunicorns/zarf/blob/main/examples/helm-oci-chart/zarf.yaml
+$ cat <<EOF > zarf.yaml 
+kind: ZarfPackageConfig
+metadata:
+  name: helm-oci-chart
+  description: Deploy podinfo using a Helm OCI chart
+  # Note: In order to publish, the package must have a version
+  version: 0.0.1
+
+components:
+  - name: helm-oci-chart
+    required: true
+    charts:
+      - name: podinfo
+        version: 6.3.3
+        namespace: helm-oci-demo
+        url: oci://ghcr.io/stefanprodan/charts/podinfo
+    images:
+      - "ghcr.io/stefanprodan/podinfo:6.3.3"
+EOF
+
+# Authenticate with your registry using Zarf
+$ echo $REGISTRY_SECRET | zarf tools registry login $REGISTRY --username $REGISTRY_USERNAME --password-stdin
+# (Optional) Otherwise, create a Docker compliant auth config file if the Docker CLI is not installed
+$ mkdir -p ~/.docker
+$ AUTH=$(echo -n "$REGISTRY_USERNAME:$REGISTRY_SECRET" | base64)
+# Note: If using Docker Hub, the registry URL is `https://index.docker.io/v1/` for the auth config
+$ cat <<EOF > ~/.docker/config.json
+{
+  "auths": {
+    "$REGISTRY": {
+      "auth": "$AUTH"
+    }
+  }
+}
+EOF
+```
+
+## Publishing A Zarf Package
+
+First create the package locally:
+
+```bash
+# Create the package (interactively)
+$ zarf package create .
+# Make these choices at the prompts:
+# Create this Zarf package? Yes
+# Please provide a value for "Maximum Package Size" 0
+```
+
+Then publish the package to the registry:
+
+```bash
+# Your package tarball may be named differently based on your machine's architecture
+$ zarf package publish zarf-package-helm-oci-chart-arm64.tar.zst oci://$REGISTRY/$USERNAME
+```
+
+:::note
+
+The name and reference of this OCI artifact is derived from the package metadata, e.g.: `helm-oci-chart:0.0.1-arm64`
+
+To modify, edit `zarf.yaml` and re-run `zarf package create .`
+
+:::
+
+## Inspecting a Published Zarf Package
+
+```yaml
+$ zarf package inspect oci://$REGISTRY_URL/$USERNAME/helm-oci-chart
+
+tags:
+- 0.0.1-arm64
+latest:
+  tag: 0.0.1-arm64
+  descriptor:
+    mediaType: application/vnd.oci.artifact.manifest.v1+json
+    digest: sha256:624e97e9b235ebd8eab699238482eebd2f535683755323eb40c819e0efdcd959
+    size: 3338
+```
+
+## Deploying A Zarf Package From The Registry
+
+```bash
+$ zarf package deploy oci://$REGISTRY/$USERNAME/helm-oci-chart:0.0.1-arm64
+# zarf package deploy oci://REGISTRY/NAMESPACE/NAME:VERSION
+# zarf package deploy oci://docker.io/defenseunicorns/strimzi:v0.24.0-arm64
+```
diff --git a/docs/13-walkthroughs/6-publish-and-deploy.tape b/docs/13-walkthroughs/6-publish-and-deploy.tape
@@ -0,0 +1,169 @@
+Output docs/.images/walkthrough-6.gif
+
+Set FontSize 16
+Set Width 1200
+Set Height 800
+Set FontFamily "Hack Nerd Font"
+Set Shell bash
+
+Hide
+Type "cd /tmp" 
+Enter
+Type "alias zarf='~/dev/zarf/build/zarf-mac-apple'"
+Enter
+Ctrl+L
+Show
+
+Type "🦄  Walkthrough: Publish/Deploy a Zarf package to/from Docker Hub"
+Sleep 3s
+Ctrl+C
+Enter
+Type "This walkthrough assumes the following:"
+Sleep 2s
+Ctrl+C
+Type "  - You have a Docker Hub account"
+Sleep 2s
+Ctrl+C
+Type "  - You have access to a Kubernetes cluster"
+Sleep 2s
+Ctrl+C
+Type "  - You have the Zarf binary"
+Sleep 2s
+Ctrl+C
+Type "  - You have run 'zarf init' on your cluster"
+Sleep 2s
+Ctrl+C
+Sleep 10s
+Enter
+
+Type "mkdir -p zarf-publish-walkthrough && cd zarf-publish-walkthrough"
+Sleep 3s
+Enter 2
+
+Type "🦄  Build an example Zarf package"
+Sleep 3s
+Ctrl+C
+Enter
+
+Type "cat <<EOF > zarf.yaml "
+Enter
+Type "kind: ZarfPackageConfig"
+Enter
+Type "metadata:"
+Enter
+Type "  name: helm-oci-chart"
+Enter
+Type "  description: Deploy podinfo using a Helm OCI chart"
+Enter
+Type "  # Note: In order to publish, the package must have a version"
+Enter
+Type "  version: '0.0.1'"
+Enter 2
+Type "components:"
+Enter
+Type "  - name: helm-oci-chart"
+Enter
+Type "    required: true"
+Enter
+Type "    charts:"
+Enter
+Type "      - name: podinfo"
+Enter
+Type "        version: 6.3.3"
+Enter
+Type "        namespace: helm-oci-demo"
+Enter
+Type "        url: oci://ghcr.io/stefanprodan/charts/podinfo"
+Enter
+Type "    images:"
+Enter
+Type '      - "ghcr.io/stefanprodan/podinfo:6.3.3"'
+Enter
+Type "EOF"
+Sleep 500ms
+Enter
+Sleep 3s
+
+Type "zarf package create ."
+Enter
+Sleep 5s
+Type "y"
+Enter
+Sleep 2s
+Enter
+Sleep 7s
+Enter
+
+Type "🦄  Authenticate to Docker Hub"
+Sleep 3s
+Ctrl+C
+Enter
+
+Type "echo $REGISTRY_SECRET | zarf tools registry login --username $REGISTRY_USERNAME --password-stdin"
+Sleep 1s
+Ctrl+C
+Sleep 3s
+
+# temporarily here while package structure is being updated, this is a manually modified package
+Hide
+Type "rm zarf-package-helm-oci-chart-arm64-0.0.1.tar.zst"
+Enter
+Type "cp ~/dev/zarf/examples/helm-oci-chart/zarf-package-helm-oci-chart-arm64.tar.zst ."
+Enter
+Ctrl+L
+Show
+
+Type "🦄  Publish the Zarf package to Docker Hub"
+Sleep 3s
+Ctrl+C
+Enter
+
+Type "zarf package publish zarf-package-helm-oci-chart-arm64.tar.zst oci://docker.io/noxsios"
+Sleep 1s
+Enter
+Sleep 15s
+
+Type "🦄  Inspect the Zarf package"
+Sleep 3s
+Ctrl+C
+Enter
+
+Type "zarf package inspect oci://docker.io/noxsios/helm-oci-chart"
+Enter
+Sleep 1s
+Enter
+Sleep 10s
+
+Type "🦄  Deploy the Zarf package"
+Sleep 3s
+Ctrl+C
+Enter
+
+# currently broken on final step until some file structure changes are made
+Type "zarf package deploy oci://docker.io/noxsios/helm-oci-chart:0.0.1-arm64"
+Sleep 1s
+Enter
+Sleep 2s
+Type "y"
+Enter
+Sleep 30s
+
+Type "🦄  List deployed Zarf packages"
+Sleep 3s
+Ctrl+C
+Enter
+
+Type "zarf package list"
+Sleep 1s
+Enter
+Sleep 10s
+
+Hide
+Type "cd /tmp"
+Enter
+Type "rm -rf /tmp/zarf-publish-walkthrough" 
+Enter
+Type "unalias zarf"
+Enter
+Ctrl+L
+Show
diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package.md
@@ -28,5 +28,7 @@ Zarf package commands for creating, deploying, and inspecting packages
 * [zarf package deploy](zarf_package_deploy.md)	 - Use to deploy a Zarf package from a local file or URL (runs offline)
 * [zarf package inspect](zarf_package_inspect.md)	 - Lists the payload of a Zarf package (runs offline)
 * [zarf package list](zarf_package_list.md)	 - List out all of the packages that have been deployed to the cluster
+* [zarf package publish](zarf_package_publish.md)	 - Publish a Zarf package to a remote registry
+* [zarf package pull](zarf_package_pull.md)	 - Pull a Zarf package from a remote registry and save to the local file system
 * [zarf package remove](zarf_package_remove.md)	 - Use to remove a Zarf package that has been deployed already
 
diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_deploy.md
@@ -15,6 +15,7 @@ zarf package deploy [PACKAGE] [flags]
 
 ```
       --components string    Comma-separated list of components to install.  Adding this flag will skip the init prompts for which components to install
+      --concurrency int      Number of concurrent layer operations to perform when interacting with a remote package.
       --confirm              Confirm package deployment without prompting
   -h, --help                 help for deploy
       --set stringToString   Specify deployment variables to set on the command line (KEY=value) (default [])

diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_publish.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_publish.md
@@ -0,0 +1,38 @@
+# zarf package publish
+<!-- Auto-generated by docs/gen-cli-docs.sh -->
+
+Publish a Zarf package to a remote registry
+
+```
+zarf package publish [PACKAGE] [REPOSITORY] [flags]
+```
+
+## Examples
+
+```
+  zarf package publish my-package.tar oci://my-registry.com/my-namespace
+```
+
+## Options
+
+```
+      --concurrency int   Number of concurrent layer operations to perform when interacting with a remote package. (default 3)
+  -h, --help              help for publish
+```
+
+## Options inherited from parent commands
+
+```
+  -a, --architecture string   Architecture for OCI images
+      --insecure              Allow access to insecure registries and disable other recommended security enforcements. This flag should only be used if you have a specific reason and accept the reduced security posture.
+  -l, --log-level string      Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info")
+      --no-log-file           Disable log file creation
+      --no-progress           Disable fancy UI progress bars, spinners, logos, etc
+      --tmpdir string         Specify the temporary directory to use for intermediate files
+      --zarf-cache string     Specify the location of the Zarf cache directory (default "~/.zarf-cache")
+```
+
+## SEE ALSO
+
+* [zarf package](zarf_package.md)	 - Zarf package commands for creating, deploying, and inspecting packages
+
diff --git a/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_pull.md b/docs/4-user-guide/1-the-zarf-cli/100-cli-commands/zarf_package_pull.md
@@ -0,0 +1,38 @@
+# zarf package pull
+<!-- Auto-generated by docs/gen-cli-docs.sh -->
+
+Pull a Zarf package from a remote registry and save to the local file system
+
+```
+zarf package pull [REFERENCE] [flags]
+```
+
+## Examples
+
+```
+  zarf package pull oci://my-registry.com/my-namespace/my-package:0.0.1-arm64
+```
+
+## Options
+
+```
+      --concurrency int   Number of concurrent layer operations to perform when interacting with a remote package. (default 3)
+  -h, --help              help for pull
+```
+
+## Options inherited from parent commands
+
+```
+  -a, --architecture string   Architecture for OCI images
+      --insecure              Allow access to insecure registries and disable other recommended security enforcements. This flag should only be used if you have a specific reason and accept the reduced security posture.
+  -l, --log-level string      Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info")
+      --no-log-file           Disable log file creation
+      --no-progress           Disable fancy UI progress bars, spinners, logos, etc
+      --tmpdir string         Specify the temporary directory to use for intermediate files
+      --zarf-cache string     Specify the location of the Zarf cache directory (default "~/.zarf-cache")
+```
+
+## SEE ALSO
+
+* [zarf package](zarf_package.md)	 - Zarf package commands for creating, deploying, and inspecting packages
+