Docs for stack registration + deployment wizards (#2814) (#2840)

* Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Adding some context for stack registration * Updated README * Updated README * Updated README * Updated README * Updated README * Updated README * Updated README * various docs changes * typo * first draft before polishing * minor fixes * new screenshots * changing the order * rephrasing * minor fixes * Optimised images with calibre/image-actions * more screenshots and finishing up the page * some links * Optimised images with calibre/image-actions * forgotten breakpoint * updating the readme * register page * minor changes * minor changes * changing the description * Update docs/book/how-to/stack-deployment/deploy-a-cloud-stack.md Co-authored-by: Hamza Tahir <hamza@zenml.io> * added to the warning * adding permission to aws cloudformation * Document AWS permissions * new gif * slight prephrasing * warning regarding the deployed instance * rephrasing --------- Co-authored-by: Baris Can Durak <bariscandurak@hotmail.com> Co-authored-by: Barış Can Durak <36421093+bcdurak@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Stefan Nica <stefan@zenml.io> (cherry picked from commit 9060c52) Co-authored-by: Hamza Tahir <hamza@zenml.io>
zenml-io · Jul 9, 2024 · 3c7ad9e · 3c7ad9e
1 parent 50334c3
commit 3c7ad9e
Show file tree

Hide file tree

Showing 30 changed files with 715 additions and 313 deletions.
diff --git a/README.md b/README.md
@@ -133,9 +133,36 @@ if __name__ == "__main__":
 
 ![Running a ZenML pipeline](/docs/book/.gitbook/assets/readme_basic_pipeline.gif)
 
-### Deploy workloads easily on your production infrastructure
+### Easily provision an MLOps stack or reuse your existing infrastructure
 
-The framework is a gentle entry point for practitioners to build complex ML pipelines with little knowledge required of the underlying infrastructure complexity. ZenML pipelines can be run on AWS, GCP, Azure, Airflow, Kubeflow and even on Kubernetes without having to change any code or know underlying internals.
+The framework is a gentle entry point for practitioners to build complex ML pipelines with little knowledge required of the underlying infrastructure complexity. ZenML pipelines can be run on AWS, GCP, Azure, Airflow, Kubeflow and even on Kubernetes without having to change any code or know underlying internals. 
+
+ZenML provides different features to aid people to get started quickly on a remote setting as well. If you want to deploy a remote stack from scratch on your selected cloud provider, you can use the 1-click deployment feature either through the dashboard:
+
+![Running a ZenML pipeline](/docs/book/.gitbook/assets/one-click-deployment.gif)
+
+Or, through our CLI command:
+
+```bash
+zenml stack deploy --provider aws
+```
+
+Alternatively, if the necessary pieces of infrastructure is already deployed, you can register a cloud stack seamlessly through the stack wizard:
+
+```bash
+zenml stack register <STACK_NAME> --provider aws
+```
+
+Read more about [ZenML stacks](https://docs.zenml.io/user-guide/production-guide/understand-stacks).
+
+### Run workloads easily on your production infrastructure
+
+Once you have your MLOps stack configured, you can easily run workloads on it:
+
+```bash
+zenml stack set <STACK_NAME>
+python run.py
+```
 
 ```python
 from zenml.config import ResourceSettings, DockerSettings
@@ -150,14 +177,8 @@ def training(...):
 	...
 ```
 
-```bash
-zenml stack set k8s  # Set a stack with kubernetes orchestrator
-python run.py
-```
-
 ![Workloads with ZenML](/docs/book/.gitbook/assets/readme_compute.gif)
 
-
 ### Track models, pipeline, and artifacts
 
 Create a complete lineage of who, where, and what data and models are produced.

diff --git a/docs/book/.gitbook/assets/deploy_stack_aws.png b/docs/book/.gitbook/assets/deploy_stack_aws.png
diff --git a/docs/book/.gitbook/assets/deploy_stack_aws_2.png b/docs/book/.gitbook/assets/deploy_stack_aws_2.png
diff --git a/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation.png b/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation.png
diff --git a/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation_intro.png b/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation_intro.png
diff --git a/docs/book/.gitbook/assets/deploy_stack_selection.png b/docs/book/.gitbook/assets/deploy_stack_selection.png
diff --git a/docs/book/.gitbook/assets/one-click-deployment.gif b/docs/book/.gitbook/assets/one-click-deployment.gif
diff --git a/docs/book/.gitbook/assets/register_stack_button.png b/docs/book/.gitbook/assets/register_stack_button.png
diff --git a/docs/book/.gitbook/assets/register_stack_page.png b/docs/book/.gitbook/assets/register_stack_page.png
diff --git a/docs/book/component-guide/artifact-stores/gcp.md b/docs/book/component-guide/artifact-stores/gcp.md
@@ -21,6 +21,13 @@ You should use the GCS Artifact Store when you decide to keep your ZenML artifac
 
 ### How do you deploy it?
 
+{% hint style="info" %}
+Don't want to deploy the artifact store manually? Check out the
+[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md)
+or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md)
+for a shortcut on how to deploy & register this stack component.
+{% endhint %}
+
 The GCS Artifact Store flavor is provided by the GCP ZenML integration, you need to install it on your local machine to be able to register a GCS Artifact Store and add it to your stack:
 
 ```shell

diff --git a/docs/book/component-guide/artifact-stores/s3.md b/docs/book/component-guide/artifact-stores/s3.md
@@ -21,6 +21,13 @@ You should use the S3 Artifact Store when you decide to keep your ZenML artifact
 
 ### How do you deploy it?
 
+{% hint style="info" %}
+Don't want to deploy the artifact store manually? Check out the
+[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md)
+or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md)
+for a shortcut on how to deploy & register this stack component.
+{% endhint %}
+
 The S3 Artifact Store flavor is provided by the S3 ZenML integration, you need to install it on your local machine to be able to register an S3 Artifact Store and add it to your stack:
 
 ```shell

diff --git a/docs/book/component-guide/container-registries/aws.md b/docs/book/component-guide/container-registries/aws.md
@@ -15,6 +15,13 @@ You should use the AWS container registry if:
 
 ### How to deploy it
 
+{% hint style="info" %}
+Don't want to deploy the container registry manually? Check out the
+[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md)
+or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md)
+for a shortcut on how to deploy & register this stack component.
+{% endhint %}
+
 The ECR registry is automatically activated once you create an AWS account. However, you'll need to create a `Repository` in order to push container images to it:
 
 * Go to the [ECR website](https://console.aws.amazon.com/ecr).

diff --git a/docs/book/component-guide/container-registries/gcp.md b/docs/book/component-guide/container-registries/gcp.md
@@ -19,6 +19,13 @@ You should use the GCP container registry if:
 
 ### How to deploy it
 
+{% hint style="info" %}
+Don't want to deploy the container registry manually? Check out the
+[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md)
+or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md)
+for a shortcut on how to deploy & register this stack component.
+{% endhint %}
+
 When using the Google Artifact Registry, you need to:
 
 * enable it [here](https://console.cloud.google.com/marketplace/product/google/artifactregistry.googleapis.com)
@@ -32,7 +39,7 @@ A GCP Container Registry can be deployed directly from the ZenML CLI:
 zenml container-registry deploy gcp_container_registry --flavor=gcp --provider=gcp ...
 ```
 
-You can pass other configurations specific to the stack components as key-value arguments. If you don't provide a name, a random one is generated for you. For more information about how to work use the CLI for this, please refer to the [dedicated documentation section](../../how-to/stack-deployment/deploy-a-stack-component.md).
+You can pass other configurations specific to the stack components as key-value arguments. If you don't provide a name, a random one is generated for you. For more information about how to work use the CLI for this, please refer to the [dedicated documentation section](../../how-to/stack-deployment/deploy-a-stack-using-mlstacks.md).
 
 ## How to find the registry URI
 

diff --git a/docs/book/component-guide/orchestrators/sagemaker.md b/docs/book/component-guide/orchestrators/sagemaker.md
@@ -26,6 +26,13 @@ The ZenML Sagemaker orchestrator works with [Sagemaker Pipelines](https://aws.am
 
 ## How to deploy it
 
+{% hint style="info" %}
+Don't want to deploy the orchestrator manually? Check out the
+[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md)
+or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md)
+for a shortcut on how to deploy & register this stack component.
+{% endhint %}
+
 In order to use a Sagemaker AI orchestrator, you need to first deploy [ZenML to the cloud](../../getting-started/deploying-zenml/README.md). It would be recommended to deploy ZenML in the same region as you plan on using for Sagemaker, but it is not necessary to do so. You must ensure that you are connected to the remote ZenML server before using this stack component.
 
 The only other thing necessary to use the ZenML Sagemaker orchestrator is enabling the relevant permissions for your particular role.

diff --git a/docs/book/component-guide/orchestrators/vertex.md b/docs/book/component-guide/orchestrators/vertex.md
@@ -22,6 +22,13 @@ You should use the Vertex orchestrator if:
 
 ## How to deploy it
 
+{% hint style="info" %}
+Don't want to deploy the orchestrator manually? Check out the
+[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md)
+or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md)
+for a shortcut on how to deploy & register this stack component.
+{% endhint %}
+
 In order to use a Vertex AI orchestrator, you need to first deploy [ZenML to the cloud](../../getting-started/deploying-zenml/README.md). It would be recommended to deploy ZenML in the same Google Cloud project as where the Vertex infrastructure is deployed, but it is not necessary to do so. You must ensure that you are connected to the remote ZenML server before using this stack component.
 
 The only other thing necessary to use the ZenML Vertex orchestrator is enabling Vertex-relevant APIs on the Google Cloud project.

diff --git a/docs/book/how-to/popular-integrations/aws-guide.md b/docs/book/how-to/popular-integrations/aws-guide.md
@@ -6,6 +6,23 @@ description: A simple guide to create an AWS stack to run your ZenML pipelines
 
 This page aims to quickly set up a minimal production stack on AWS. With just a few simple steps, you will set up an IAM role with specifically-scoped permissions that ZenML can use to authenticate with the relevant AWS resources.
 
+{% hint style="info" %}
+**Don't want to learn how to deploy and register stack components manually? 
+Feel free to skip this chapter and read [how to deploy a cloud stack in one click](../../how-to/stack-deployment/deploy-a-cloud-stack.md)
+and learn how to use:**
+
+```shell
+zenml stack deploy --provider aws
+```
+
+**or if you have existing infrastructure, read [how to easily register a cloud stack](../../how-to/stack-deployment/register-a-cloud-stack.md)
+and learn how to use:**
+
+```shell
+zenml stack register <STACK_NAME> --provider aws
+```
+{% endhint %}
+
 ## 1) Set up credentials and local environment
 
 To follow this guide, you need:

diff --git a/docs/book/how-to/popular-integrations/gcp-guide.md b/docs/book/how-to/popular-integrations/gcp-guide.md
@@ -6,6 +6,23 @@ description: A simple guide to quickly set up a minimal stack on GCP.
 
 This page aims to quickly set up a minimal production stack on GCP. With just a few simple steps you will set up a service account with specifically-scoped permissions that ZenML can use to authenticate with the relevant GCP resources.
 
+{% hint style="info" %}
+**Don't want to learn how to deploy and register stack components manually? 
+Feel free to skip this chapter and read [how to deploy a cloud stack in one click](../../how-to/stack-deployment/deploy-a-cloud-stack.md)
+and learn how to use:**
+
+```shell
+zenml stack deploy --provider gcp
+```
+
+**or if you have existing infrastructure, read [how to easily register a cloud stack](../../how-to/stack-deployment/register-a-cloud-stack.md)
+and learn how to use:**
+
+```shell
+zenml stack register <STACK_NAME> --provider gcp
+```
+{% endhint %}
+
 {% hint style="warning" %}
 While this guide focuses on Google Cloud, we are seeking contributors to create a similar guide for other cloud providers. If you are interested, please create a [pull request over on GitHub](https://github.com/zenml-io/zenml/blob/main/CONTRIBUTING.md).
 {% endhint %}

diff --git a/docs/book/how-to/stack-deployment/README.md b/docs/book/how-to/stack-deployment/README.md
@@ -1,16 +1,12 @@
 ---
-description: Deploying your stack components directly from the ZenML CLI
+description: Stacks are the configuration of your infrastructure.
 ---
 
-# ⚒️ Manage stacks
+# Managing stacks & components
 
-The first step in running your pipelines on remote infrastructure is to deploy all the components that you would need, like an MLflow tracking server, a Seldon Core model deployer, and more to your cloud.
+The [stack](../../user-guide/production-guide/understand-stacks.md) is a fundamental component of the ZenML framework. Put simply, a stack represents the configuration of the infrastructure and tooling that defines where and how a pipeline executes.
 
-This can bring plenty of benefits like scalability, reliability, and collaboration. ZenML eases the path to production by providing a seamless way for all tools to interact with others through the use of abstractions. However, one of the most painful parts of this process, from what we see on our Slack and in general, is the deployment of these stack components.
-
-## Deploying and managing MLOps tools is tricky 😭😵‍💫
-
-It is not trivial to set up all the different tools that you might need for your pipeline.
+However, deploying and managing a MLOps stack is tricky 😭😵‍💫. It is not trivial to set up all the different tools that you might need for your pipeline.
 
 * 🌈 Each tool comes with a certain set of requirements. For example, a Kubeflow installation will require you to have a Kubernetes cluster, and so would a Seldon Core deployment.
 * 🤔 Figuring out the defaults for infra parameters is not easy. Even if you have identified the backing infra that you need for a stack component, setting up reasonable defaults for parameters like instance size, CPU, memory, etc., needs a lot of experimentation to figure out.
@@ -21,41 +17,48 @@ It is not trivial to set up all the different tools that you might need for your
 
 All of these points make taking your pipelines to production a more difficult task than it should be. We believe that the expertise in setting up these often-complex stacks shouldn't be a prerequisite to running your ML pipelines.
 
-Thus, to make even this process easier for our users, we have created the `deploy` CLI which allows you to quickly get started with a full-fledged MLOps stack using only a few commands. You can choose to deploy individual stack components through the stack-component CLI or deploy a stack with multiple components together (a tad more manual steps).
-
-## What is `mlstacks`?
-
-[MLStacks](https://mlstacks.zenml.io) is a [Python package](https://pypi.org/project/mlstacks/) that allows you to quickly spin up MLOps infrastructure using Terraform. It is designed to be used with [ZenML](https://zenml.io), but can be used with any MLOps tool or platform.
-
-The ZenML CLI has special subcommands that allow you to deploy individual stack components as well as whole stacks using MLStacks. These stacks will be useful for you if:
-
-* You are at the start of your MLOps journey, and would like to explore different tools.
-* You are looking for guidelines for production-grade deployments.
-
-## Installing the mlstacks extra
-
-To install `mlstacks`, either run `pip install mlstacks` or `pip install "zenml[mlstacks]"` to install it along with ZenML.
-
-MLStacks uses Terraform on the backend to manage infrastructure. You will need to have Terraform installed. Please visit [the Terraform docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform) for installation instructions.
-
-MLStacks also uses Helm to deploy Kubernetes resources. You will need to have Helm installed. Please visit [the Helm docs](https://helm.sh/docs/intro/install/#from-script) for installation instructions.
-
-## Deploying a stack component
-
-The ZenML CLI allows you to deploy individual stack components using the `deploy` subcommand which is implemented for all supported stack components. You can find the list of supported stack components [here](../stack-deployment/README.md).
-
-## Deploying a stack
-
-For deploying a full stack, use the `zenml stack deploy` command. See the [stack deployment](deploy-a-stack-using-mlstacks.md) page for more details of which cloud providers and stack components are supported.
-
-## How does `mlstacks` work?
-
-MLStacks is built around the concept of a stack specification. A stack specification is a YAML file that describes the stack and includes references to component specification files. A component specification is a YAML file that describes a component. (Currently all deployments of components (in various combinations) must be defined within the context of a stack.)
-
-ZenML handles the creation of stack specifications for you when you run one of the `deploy` subcommands using the CLI. A valid specification is generated and used by `mlstacks` to deploy your stack using Terraform. The Terraform definitions and state are stored in your global configuration directory along with any state files generated while deploying your stack.
-
-Your configuration directory could be in a number of different places depending on your operating system, but read more about it in the [Click docs](https://click.palletsprojects.com/en/8.1.x/api/#click.get\_app\_dir) to see which location applies to your situation.
-
-<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:purple;"><strong>Deploy stack components individually</strong></mark></td><td>Individually deploying different stack components.</td><td><a href="deploy-a-stack-component.md">deploy-a-stack-component.md</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy a stack with multiple components using mlstacks</strong></mark></td><td>Deploying an entire stack with ZenML's `mlstacks` package.</td><td><a href="../../how-to/stack-deployment/deploy-a-stack-using-mlstacks.md">../../how-to/stack-deployment/deploy-a-stack-using-mlstacks.md</a></td></tr><tr><td><mark style="color:purple;"><strong>Contribute new components or flavors</strong></mark></td><td>Creating your custom stack component solutions.</td><td><a href="../../../../CONTRIBUTING.md">CONTRIBUTING.md</a></td></tr></tbody></table>
+This docs section consists of information that makes it easier to provision, configure, and extend stacks and components in ZenML.
+
+<table data-view="cards">
+  <thead>
+    <tr>
+      <th></th>
+      <th></th>
+      <th data-hidden data-card-target data-type="content-ref"></th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><mark style="color:purple;"><strong>Deploy a cloud stack</strong></mark></td>
+      <td>Description of deploying a cloud stack.</td>
+      <td><a href="./deploy-a-cloud-stack.md">./deploy-a-cloud-stack.md</a></td>
+    </tr>
+    <tr>
+      <td><mark style="color:purple;"><strong>Register a cloud stack</strong></mark></td>
+      <td>Description of registering a cloud stack.</td>
+      <td><a href="./register-a-cloud-stack.md">./register-a-cloud-stack.md</a></td>
+    </tr>
+    <tr>
+      <td><mark style="color:purple;"><strong>Deploy stack/components using mlstacks</strong></mark></td>
+      <td>Deploying an entire stack with ZenML's `mlstacks` package.</td>
+      <td><a href="./deploy-a-stack-using-mlstacks.md">./deploy-a-stack-using-mlstacks.md</a></td>
+    </tr>
+    <tr>
+      <td><mark style="color:purple;"><strong>Reference secrets in stack configuration</strong></mark></td>
+      <td>Description of referencing secrets in stack configuration.</td>
+      <td><a href="./reference-secrets-in-stack-configuration.md">./reference-secrets-in-stack-configuration.md</a></td>
+    </tr>
+    <tr>
+      <td><mark style="color:purple;"><strong>Implement a custom stack component</strong></mark></td>
+      <td>Creating your custom stack component solutions.</td>
+      <td><a href="./implement-a-custom-stack-component.md">./implement-a-custom-stack-component.md</a></td>
+    </tr>
+    <tr>
+      <td><mark style="color:purple;"><strong>Implement a custom integration</strong></mark></td>
+      <td>Description of implementing a custom integration.</td>
+      <td><a href="./implement-a-custom-integration.md">./implement-a-custom-integration.md</a></td>
+    </tr>
+  </tbody>
+</table>
 
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>