diff --git a/README.md b/README.md index fa209869d2b..f8dde77b535 100644 --- a/README.md +++ b/README.md @@ -133,9 +133,36 @@ if __name__ == "__main__": ![Running a ZenML pipeline](/docs/book/.gitbook/assets/readme_basic_pipeline.gif) -### Deploy workloads easily on your production infrastructure +### Easily provision an MLOps stack or reuse your existing infrastructure -The framework is a gentle entry point for practitioners to build complex ML pipelines with little knowledge required of the underlying infrastructure complexity. ZenML pipelines can be run on AWS, GCP, Azure, Airflow, Kubeflow and even on Kubernetes without having to change any code or know underlying internals. +The framework is a gentle entry point for practitioners to build complex ML pipelines with little knowledge required of the underlying infrastructure complexity. ZenML pipelines can be run on AWS, GCP, Azure, Airflow, Kubeflow and even on Kubernetes without having to change any code or know underlying internals. + +ZenML provides different features to aid people to get started quickly on a remote setting as well. If you want to deploy a remote stack from scratch on your selected cloud provider, you can use the 1-click deployment feature either through the dashboard: + +![Running a ZenML pipeline](/docs/book/.gitbook/assets/one-click-deployment.gif) + +Or, through our CLI command: + +```bash +zenml stack deploy --provider aws +``` + +Alternatively, if the necessary pieces of infrastructure is already deployed, you can register a cloud stack seamlessly through the stack wizard: + +```bash +zenml stack register --provider aws +``` + +Read more about [ZenML stacks](https://docs.zenml.io/user-guide/production-guide/understand-stacks). + +### Run workloads easily on your production infrastructure + +Once you have your MLOps stack configured, you can easily run workloads on it: + +```bash +zenml stack set +python run.py +``` ```python from zenml.config import ResourceSettings, DockerSettings @@ -150,14 +177,8 @@ def training(...): ... ``` -```bash -zenml stack set k8s # Set a stack with kubernetes orchestrator -python run.py -``` - ![Workloads with ZenML](/docs/book/.gitbook/assets/readme_compute.gif) - ### Track models, pipeline, and artifacts Create a complete lineage of who, where, and what data and models are produced. diff --git a/docs/book/.gitbook/assets/deploy_stack_aws.png b/docs/book/.gitbook/assets/deploy_stack_aws.png new file mode 100644 index 00000000000..d0fb71198d3 Binary files /dev/null and b/docs/book/.gitbook/assets/deploy_stack_aws.png differ diff --git a/docs/book/.gitbook/assets/deploy_stack_aws_2.png b/docs/book/.gitbook/assets/deploy_stack_aws_2.png new file mode 100644 index 00000000000..38c704c148e Binary files /dev/null and b/docs/book/.gitbook/assets/deploy_stack_aws_2.png differ diff --git a/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation.png b/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation.png new file mode 100644 index 00000000000..43c5b453175 Binary files /dev/null and b/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation.png differ diff --git a/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation_intro.png b/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation_intro.png new file mode 100644 index 00000000000..0c9a9f3a2a0 Binary files /dev/null and b/docs/book/.gitbook/assets/deploy_stack_aws_cloudformation_intro.png differ diff --git a/docs/book/.gitbook/assets/deploy_stack_selection.png b/docs/book/.gitbook/assets/deploy_stack_selection.png new file mode 100644 index 00000000000..0130587cfcc Binary files /dev/null and b/docs/book/.gitbook/assets/deploy_stack_selection.png differ diff --git a/docs/book/.gitbook/assets/one-click-deployment.gif b/docs/book/.gitbook/assets/one-click-deployment.gif new file mode 100644 index 00000000000..683ead80082 Binary files /dev/null and b/docs/book/.gitbook/assets/one-click-deployment.gif differ diff --git a/docs/book/.gitbook/assets/register_stack_button.png b/docs/book/.gitbook/assets/register_stack_button.png new file mode 100644 index 00000000000..a7a2cfa4371 Binary files /dev/null and b/docs/book/.gitbook/assets/register_stack_button.png differ diff --git a/docs/book/.gitbook/assets/register_stack_page.png b/docs/book/.gitbook/assets/register_stack_page.png new file mode 100644 index 00000000000..dc012385c0e Binary files /dev/null and b/docs/book/.gitbook/assets/register_stack_page.png differ diff --git a/docs/book/component-guide/artifact-stores/gcp.md b/docs/book/component-guide/artifact-stores/gcp.md index c82c367c501..3c279c72253 100644 --- a/docs/book/component-guide/artifact-stores/gcp.md +++ b/docs/book/component-guide/artifact-stores/gcp.md @@ -21,6 +21,13 @@ You should use the GCS Artifact Store when you decide to keep your ZenML artifac ### How do you deploy it? +{% hint style="info" %} +Don't want to deploy the artifact store manually? Check out the +[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md) +for a shortcut on how to deploy & register this stack component. +{% endhint %} + The GCS Artifact Store flavor is provided by the GCP ZenML integration, you need to install it on your local machine to be able to register a GCS Artifact Store and add it to your stack: ```shell diff --git a/docs/book/component-guide/artifact-stores/s3.md b/docs/book/component-guide/artifact-stores/s3.md index 7c84c34218c..c637c89c0b4 100644 --- a/docs/book/component-guide/artifact-stores/s3.md +++ b/docs/book/component-guide/artifact-stores/s3.md @@ -21,6 +21,13 @@ You should use the S3 Artifact Store when you decide to keep your ZenML artifact ### How do you deploy it? +{% hint style="info" %} +Don't want to deploy the artifact store manually? Check out the +[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md) +for a shortcut on how to deploy & register this stack component. +{% endhint %} + The S3 Artifact Store flavor is provided by the S3 ZenML integration, you need to install it on your local machine to be able to register an S3 Artifact Store and add it to your stack: ```shell diff --git a/docs/book/component-guide/container-registries/aws.md b/docs/book/component-guide/container-registries/aws.md index 382fbca5025..8527d7234b3 100644 --- a/docs/book/component-guide/container-registries/aws.md +++ b/docs/book/component-guide/container-registries/aws.md @@ -15,6 +15,13 @@ You should use the AWS container registry if: ### How to deploy it +{% hint style="info" %} +Don't want to deploy the container registry manually? Check out the +[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md) +for a shortcut on how to deploy & register this stack component. +{% endhint %} + The ECR registry is automatically activated once you create an AWS account. However, you'll need to create a `Repository` in order to push container images to it: * Go to the [ECR website](https://console.aws.amazon.com/ecr). diff --git a/docs/book/component-guide/container-registries/gcp.md b/docs/book/component-guide/container-registries/gcp.md index fb4bb7fb39a..1ea39f370b0 100644 --- a/docs/book/component-guide/container-registries/gcp.md +++ b/docs/book/component-guide/container-registries/gcp.md @@ -19,6 +19,13 @@ You should use the GCP container registry if: ### How to deploy it +{% hint style="info" %} +Don't want to deploy the container registry manually? Check out the +[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md) +for a shortcut on how to deploy & register this stack component. +{% endhint %} + When using the Google Artifact Registry, you need to: * enable it [here](https://console.cloud.google.com/marketplace/product/google/artifactregistry.googleapis.com) @@ -32,7 +39,7 @@ A GCP Container Registry can be deployed directly from the ZenML CLI: zenml container-registry deploy gcp_container_registry --flavor=gcp --provider=gcp ... ``` -You can pass other configurations specific to the stack components as key-value arguments. If you don't provide a name, a random one is generated for you. For more information about how to work use the CLI for this, please refer to the [dedicated documentation section](../../how-to/stack-deployment/deploy-a-stack-component.md). +You can pass other configurations specific to the stack components as key-value arguments. If you don't provide a name, a random one is generated for you. For more information about how to work use the CLI for this, please refer to the [dedicated documentation section](../../how-to/stack-deployment/deploy-a-stack-using-mlstacks.md). ## How to find the registry URI diff --git a/docs/book/component-guide/orchestrators/sagemaker.md b/docs/book/component-guide/orchestrators/sagemaker.md index d759f964232..e2931681f07 100644 --- a/docs/book/component-guide/orchestrators/sagemaker.md +++ b/docs/book/component-guide/orchestrators/sagemaker.md @@ -26,6 +26,13 @@ The ZenML Sagemaker orchestrator works with [Sagemaker Pipelines](https://aws.am ## How to deploy it +{% hint style="info" %} +Don't want to deploy the orchestrator manually? Check out the +[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md) +for a shortcut on how to deploy & register this stack component. +{% endhint %} + In order to use a Sagemaker AI orchestrator, you need to first deploy [ZenML to the cloud](../../getting-started/deploying-zenml/README.md). It would be recommended to deploy ZenML in the same region as you plan on using for Sagemaker, but it is not necessary to do so. You must ensure that you are connected to the remote ZenML server before using this stack component. The only other thing necessary to use the ZenML Sagemaker orchestrator is enabling the relevant permissions for your particular role. diff --git a/docs/book/component-guide/orchestrators/vertex.md b/docs/book/component-guide/orchestrators/vertex.md index b5890611bac..be77acb0138 100644 --- a/docs/book/component-guide/orchestrators/vertex.md +++ b/docs/book/component-guide/orchestrators/vertex.md @@ -22,6 +22,13 @@ You should use the Vertex orchestrator if: ## How to deploy it +{% hint style="info" %} +Don't want to deploy the orchestrator manually? Check out the +[easy cloud deployment wizard](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +or the [easy cloud registration wizard](../../how-to/stack-deployment/register-a-cloud-stack.md) +for a shortcut on how to deploy & register this stack component. +{% endhint %} + In order to use a Vertex AI orchestrator, you need to first deploy [ZenML to the cloud](../../getting-started/deploying-zenml/README.md). It would be recommended to deploy ZenML in the same Google Cloud project as where the Vertex infrastructure is deployed, but it is not necessary to do so. You must ensure that you are connected to the remote ZenML server before using this stack component. The only other thing necessary to use the ZenML Vertex orchestrator is enabling Vertex-relevant APIs on the Google Cloud project. diff --git a/docs/book/how-to/popular-integrations/aws-guide.md b/docs/book/how-to/popular-integrations/aws-guide.md index 9396ac57f6e..4d68bdcd685 100644 --- a/docs/book/how-to/popular-integrations/aws-guide.md +++ b/docs/book/how-to/popular-integrations/aws-guide.md @@ -6,6 +6,23 @@ description: A simple guide to create an AWS stack to run your ZenML pipelines This page aims to quickly set up a minimal production stack on AWS. With just a few simple steps, you will set up an IAM role with specifically-scoped permissions that ZenML can use to authenticate with the relevant AWS resources. +{% hint style="info" %} +**Don't want to learn how to deploy and register stack components manually? +Feel free to skip this chapter and read [how to deploy a cloud stack in one click](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +and learn how to use:** + +```shell +zenml stack deploy --provider aws +``` + +**or if you have existing infrastructure, read [how to easily register a cloud stack](../../how-to/stack-deployment/register-a-cloud-stack.md) +and learn how to use:** + +```shell +zenml stack register --provider aws +``` +{% endhint %} + ## 1) Set up credentials and local environment To follow this guide, you need: diff --git a/docs/book/how-to/popular-integrations/gcp-guide.md b/docs/book/how-to/popular-integrations/gcp-guide.md index c92d527b7cd..41c47cc14cc 100644 --- a/docs/book/how-to/popular-integrations/gcp-guide.md +++ b/docs/book/how-to/popular-integrations/gcp-guide.md @@ -6,6 +6,23 @@ description: A simple guide to quickly set up a minimal stack on GCP. This page aims to quickly set up a minimal production stack on GCP. With just a few simple steps you will set up a service account with specifically-scoped permissions that ZenML can use to authenticate with the relevant GCP resources. +{% hint style="info" %} +**Don't want to learn how to deploy and register stack components manually? +Feel free to skip this chapter and read [how to deploy a cloud stack in one click](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +and learn how to use:** + +```shell +zenml stack deploy --provider gcp +``` + +**or if you have existing infrastructure, read [how to easily register a cloud stack](../../how-to/stack-deployment/register-a-cloud-stack.md) +and learn how to use:** + +```shell +zenml stack register --provider gcp +``` +{% endhint %} + {% hint style="warning" %} While this guide focuses on Google Cloud, we are seeking contributors to create a similar guide for other cloud providers. If you are interested, please create a [pull request over on GitHub](https://github.com/zenml-io/zenml/blob/main/CONTRIBUTING.md). {% endhint %} diff --git a/docs/book/how-to/stack-deployment/README.md b/docs/book/how-to/stack-deployment/README.md index 03fe00f97f0..9956f0a686a 100644 --- a/docs/book/how-to/stack-deployment/README.md +++ b/docs/book/how-to/stack-deployment/README.md @@ -1,16 +1,12 @@ --- -description: Deploying your stack components directly from the ZenML CLI +description: Stacks are the configuration of your infrastructure. --- -# โš’๏ธ Manage stacks +# Managing stacks & components -The first step in running your pipelines on remote infrastructure is to deploy all the components that you would need, like an MLflow tracking server, a Seldon Core model deployer, and more to your cloud. +The [stack](../../user-guide/production-guide/understand-stacks.md) is a fundamental component of the ZenML framework. Put simply, a stack represents the configuration of the infrastructure and tooling that defines where and how a pipeline executes. -This can bring plenty of benefits like scalability, reliability, and collaboration. ZenML eases the path to production by providing a seamless way for all tools to interact with others through the use of abstractions. However, one of the most painful parts of this process, from what we see on our Slack and in general, is the deployment of these stack components. - -## Deploying and managing MLOps tools is tricky ๐Ÿ˜ญ๐Ÿ˜ตโ€๐Ÿ’ซ - -It is not trivial to set up all the different tools that you might need for your pipeline. +However, deploying and managing a MLOps stack is tricky ๐Ÿ˜ญ๐Ÿ˜ตโ€๐Ÿ’ซ. It is not trivial to set up all the different tools that you might need for your pipeline. * ๐ŸŒˆ Each tool comes with a certain set of requirements. For example, a Kubeflow installation will require you to have a Kubernetes cluster, and so would a Seldon Core deployment. * ๐Ÿค” Figuring out the defaults for infra parameters is not easy. Even if you have identified the backing infra that you need for a stack component, setting up reasonable defaults for parameters like instance size, CPU, memory, etc., needs a lot of experimentation to figure out. @@ -21,41 +17,48 @@ It is not trivial to set up all the different tools that you might need for your All of these points make taking your pipelines to production a more difficult task than it should be. We believe that the expertise in setting up these often-complex stacks shouldn't be a prerequisite to running your ML pipelines. -Thus, to make even this process easier for our users, we have created the `deploy` CLI which allows you to quickly get started with a full-fledged MLOps stack using only a few commands. You can choose to deploy individual stack components through the stack-component CLI or deploy a stack with multiple components together (a tad more manual steps). - -## What is `mlstacks`? - -[MLStacks](https://mlstacks.zenml.io) is a [Python package](https://pypi.org/project/mlstacks/) that allows you to quickly spin up MLOps infrastructure using Terraform. It is designed to be used with [ZenML](https://zenml.io), but can be used with any MLOps tool or platform. - -The ZenML CLI has special subcommands that allow you to deploy individual stack components as well as whole stacks using MLStacks. These stacks will be useful for you if: - -* You are at the start of your MLOps journey, and would like to explore different tools. -* You are looking for guidelines for production-grade deployments. - -## Installing the mlstacks extra - -To install `mlstacks`, either run `pip install mlstacks` or `pip install "zenml[mlstacks]"` to install it along with ZenML. - -MLStacks uses Terraform on the backend to manage infrastructure. You will need to have Terraform installed. Please visit [the Terraform docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform) for installation instructions. - -MLStacks also uses Helm to deploy Kubernetes resources. You will need to have Helm installed. Please visit [the Helm docs](https://helm.sh/docs/intro/install/#from-script) for installation instructions. - -## Deploying a stack component - -The ZenML CLI allows you to deploy individual stack components using the `deploy` subcommand which is implemented for all supported stack components. You can find the list of supported stack components [here](../stack-deployment/README.md). - -## Deploying a stack - -For deploying a full stack, use the `zenml stack deploy` command. See the [stack deployment](deploy-a-stack-using-mlstacks.md) page for more details of which cloud providers and stack components are supported. - -## How does `mlstacks` work? - -MLStacks is built around the concept of a stack specification. A stack specification is a YAML file that describes the stack and includes references to component specification files. A component specification is a YAML file that describes a component. (Currently all deployments of components (in various combinations) must be defined within the context of a stack.) - -ZenML handles the creation of stack specifications for you when you run one of the `deploy` subcommands using the CLI. A valid specification is generated and used by `mlstacks` to deploy your stack using Terraform. The Terraform definitions and state are stored in your global configuration directory along with any state files generated while deploying your stack. - -Your configuration directory could be in a number of different places depending on your operating system, but read more about it in the [Click docs](https://click.palletsprojects.com/en/8.1.x/api/#click.get\_app\_dir) to see which location applies to your situation. - -
Deploy stack components individuallyIndividually deploying different stack components.deploy-a-stack-component.md
Deploy a stack with multiple components using mlstacksDeploying an entire stack with ZenML's `mlstacks` package.../../how-to/stack-deployment/deploy-a-stack-using-mlstacks.md
Contribute new components or flavorsCreating your custom stack component solutions.CONTRIBUTING.md
+This docs section consists of information that makes it easier to provision, configure, and extend stacks and components in ZenML. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Deploy a cloud stackDescription of deploying a cloud stack../deploy-a-cloud-stack.md
Register a cloud stackDescription of registering a cloud stack../register-a-cloud-stack.md
Deploy stack/components using mlstacksDeploying an entire stack with ZenML's `mlstacks` package../deploy-a-stack-using-mlstacks.md
Reference secrets in stack configurationDescription of referencing secrets in stack configuration../reference-secrets-in-stack-configuration.md
Implement a custom stack componentCreating your custom stack component solutions../implement-a-custom-stack-component.md
Implement a custom integrationDescription of implementing a custom integration../implement-a-custom-integration.md
ZenML Scarf
diff --git a/docs/book/how-to/stack-deployment/deploy-a-cloud-stack.md b/docs/book/how-to/stack-deployment/deploy-a-cloud-stack.md new file mode 100644 index 00000000000..a1deb991567 --- /dev/null +++ b/docs/book/how-to/stack-deployment/deploy-a-cloud-stack.md @@ -0,0 +1,174 @@ +--- +description: Deploy a cloud stack from scratch with a single click +--- + +# Deploy a cloud stack with a single click + +In ZenML, the [stack](../../user-guide/production-guide/understand-stacks.md) +is a fundamental concept that represents the configuration of your +infrastructure. In a normal workflow, creating a stack +requires you to first deploy the necessary pieces of infrastructure and then +define them as stack components in ZenML with proper authentication. + +Especially in a remote setting, this process can be challenging and +time-consuming, and it may create multi-faceted problems. This is why we +implemented a feature, that allows you to **deploy the necessary pieces of +infrastructure on your selected cloud provider and get you started on remote +stack with a single click**. + +{% hint style="info" %} +If you have the required infrastructure pieces already deployed on your cloud, +you can also use [the stack wizard to seamlessly register your stack](register-a-cloud-stack.md). +{% endhint %} + +## How to use the 1-click deployment tool? + +The first thing that you need in order to use this feature is a +deployed instance of ZenML (not a local server via `zenml up`). If you do +not already have it set up for you, feel free to learn how to do so +[here](../../getting-started/deploying-zenml/README.md). + +Once you are connected to your deployed ZenML instance, you can use the +1-click deployment tool either through the dashboard or the CLI: + +{% tabs %} +{% tab title="Dashboard" %} + +In order to create a remote stack over the dashboard go to the stacks page +on the dashboard and click "+ New Stack". + +![The new stacks page](../../.gitbook/assets/register_stack_button.png) + +Since we will be deploying it from scratch, select "New Infrastructure" on the +next page: + +![Options for registering a stack](../../.gitbook/assets/register_stack_page.png) + +{% hint style="warning" %} +Currently, the 1-click deployment only works on AWS. We are working on +bringing support to GCP and Azure as well. Stay in touch for further updates. +{% endhint %} + +![Choosing a cloud provider](../../.gitbook/assets/deploy_stack_selection.png) + +### AWS + +If you choose `aws` as your provider, you will see a page where you will have +to select a region and a name for your new stack: + +![Configuring the new stack](../../.gitbook/assets/deploy_stack_aws.png) + +Once the configuration is finished, you will see a deployment page: + +![Deploying the new stack](../../.gitbook/assets/deploy_stack_aws_2.png) + +During this process, you will be redirected you to a Cloud Formation page +on AWS. + +![Cloudformation page](../../.gitbook/assets/deploy_stack_aws_cloudformation_intro.png) + +You will have to log in to your AWS account, review and confirm the +pre-filled configuration and create the stack. + +![Finalizing the new stack](../../.gitbook/assets/deploy_stack_aws_cloudformation.png) + +{% endtab %} +{% tab title="CLI" %} + +In order to create a remote stack over the CLI you can use the following +command: + +```shell +zenml stack deploy -p aws +``` + +{% hint style="warning" %} +Currently, the 1-click deployment only works on AWS. We are working on +bringing support to GCP and Azure as well. + +In the meanwhile, you can still deploy a GCP or Azure stack manually, meaning +you can deploy the infrastructure yourself and create the necessary ZenML +components yourself using the corresponding integrations. Check [our component +guide](../../component-guide/component-guide.md) for further details. +{% endhint %} + +### AWS + +If you choose `aws` as your provider, this command will redirect you to a +Cloud Formation page on AWS: + +![Cloudformation page](../../.gitbook/assets/deploy_stack_aws_cloudformation_intro.png) + +You will have to log in to your AWS account, have permission to deploy an AWS +CloudFormation script, review and confirm the pre-filled configuration and +create the stack. + +![Finalizing the new stack](../../.gitbook/assets/deploy_stack_aws_cloudformation.png) + +{% endtab %} +{% endtabs %} + +## What will be deployed? + +Here is an overview of the infrastructure that the 1-click deployment will +prepare for you based on your cloud provider: + +{% tabs %} +{% tab title="AWS" %} + +### Resources + +- An S3 bucket that will be used as a ZenML Artifact Store. +- An ECR container registry that will be used as a ZenML Container Registry. +- Permissions to use SageMaker as a ZenML Orchestrator. +- An IAM user and IAM role with the minimum necessary permissions to access +the resources listed above. +- An AWS access key used to give access to ZenML to connect to the above +resources through a ZenML service connector. + +### Permissions + +The configured IAM service account and AWS access key will grant ZenML the +following AWS permissions in your AWS account: + +* S3 Bucket: + * s3:ListBucket + * s3:GetObject + * s3:PutObject + * s3:DeleteObject +* ECR Repository: + * ecr:DescribeRepositories + * ecr:ListRepositories + * ecr:DescribeRegistry + * ecr:BatchGetImage + * ecr:DescribeImages + * ecr:BatchCheckLayerAvailability + * ecr:GetDownloadUrlForLayer + * ecr:InitiateLayerUpload + * ecr:UploadLayerPart + * ecr:CompleteLayerUpload + * ecr:PutImage + * ecr:GetAuthorizationToken +* SageMaker (Client): + * sagemaker:CreatePipeline + * sagemaker:StartPipelineExecution + * sagemaker:DescribePipeline + * sagemaker:DescribePipelineExecution +* SageMaker (Jobs): + * AmazonSageMakerFullAccess + +{% endtab %} +{% tab title="GCP" %} +We are working on bringing the support for the 1-click deployment feature +to GCP! Stay in touch for further updates. +{% endtab %} +{% tab title="Azure" %} +We are working on bringing the support for the 1-click deployment feature +to Azure! Stay in touch for further updates. +{% endtab %} +{% endtabs %} + +There you have it! With a single click, you just deployed a cloud stack +and, you can start running your pipelines on a remote setting. + +
ZenML Scarf
diff --git a/docs/book/how-to/stack-deployment/deploy-a-stack-component.md b/docs/book/how-to/stack-deployment/deploy-a-stack-component.md deleted file mode 100644 index 5b3130245f9..00000000000 --- a/docs/book/how-to/stack-deployment/deploy-a-stack-component.md +++ /dev/null @@ -1,156 +0,0 @@ ---- -description: Individually deploying different stack components. ---- - -# Deploy a stack component - -If you have used ZenML before, you must be familiar with the flow of registering new stack components. It goes something like this: - -``` -zenml artifact-store register my_store --flavor=s3 --path=s3://my_bucket -``` - -Commands like these assume that you already have the stack component deployed. In this case, it would mean that you must already have a bucket called `my_bucket` on AWS S3 to be able to use this component. - -We took inspiration from this design to build something that feels natural to use and is also sufficiently powerful to take care of the deployment of the respective stack components for you. This is where the \ `deploy` CLI comes in! - -The `deploy` command allows you to deploy individual components of your MLOps stack with a single command ๐Ÿš€. You can also customize your components easily by passing in flags (more on that later). - -{% hint style="info" %} -To install `mlstacks`, either run `pip install mlstacks` or `pip install "zenml[mlstacks]"` to install it along with ZenML. - -MLStacks uses Terraform on the backend to manage infrastructure. You will need to have Terraform installed. Please visit [the Terraform docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform) for installation instructions. - -MLStacks also uses Helm to deploy Kubernetes resources. You will need to have Helm installed. Please visit [the Helm docs](https://helm.sh/docs/intro/install/#from-script) for installation instructions. -{% endhint %} - -For example, to deploy an artifact store on a GCP account, you can run: - -{% code overflow="wrap" %} -```bash -# after installing mlstacks -zenml artifact-store deploy -f gcp -p gcp -r us-east1 -x project_id=zenml my_store -``` -{% endcode %} - -The command above takes in the following parameters: - -* **Name**: The name of the stack component. In this case, it is `my_store`. -* **Flavor:** The flavor of the stack component to deploy. Here, we are deploying an artifact store with the `gcp` flavor. -* **Provider:** The provider to deploy this stack component on. Currently, only **GCP, AWS, and K3D** are supported as providers. -* **Region**: The region to deploy the stack component in. -* **Extra Config:** Some components can be customized by the user and these settings are passed as flags to the command. In the example above, we pass the GCP project ID to select what project to deploy the component to. - -Successful execution of this command does the following: - -* It also automatically registers the deployed stack component with your ZenML server, so you don't have to worry about manually configuring components after the deployment! ๐Ÿคฉ - -{% hint style="info" %} -The command currently uses your local credentials for GCP and AWS to provision resources. Integration with your ZenML connectors might be possible soon too! -{% endhint %} - -
- -Want to know what happens in the background? - -The stack component deploy CLI is powered by ZenML's [mlstacks](https://github.com/zenml-io/mlstacks) in the background. This allows you to configure and deploy select stack components. - -Using the values you pass for the cloud, the CLI picks up the right modular recipe to use (one of AWS, GCP, or K3D) and then deploys that recipe with the specific stack component enabled. - -
- -## Destroying a stack component - -Destroying a stack component (i.e. deleting and destroying the underlying -infrastructure) is as easy as deploying one. You can run the following command -to destroy the artifact store we created above: - -```bash -zenml artifact-store destroy -p gcp my_store -``` - -This will destroy the deployed infrastructure and prompt you if you also want to remove and deregister the component from your ZenML server. - -## ๐Ÿจ Available flavors for stack components - -Here's a table of all the flavors that can be deployed through the CLI for every stack component. This is a list that will keep on growing and you can also contribute any flavor or stack component that you feel is missing. Refer to the [Contribution page](../../../../CONTRIBUTING.md) for steps on how to do that :smile: - -
- -How does flavor selection work in the background? - -Whenever you pass in a flavor to any stack-component deploy function, the combination of these two parameters is used to construct a variable name in the following format: - -``` -enable__ -``` - -This variable is then passed as input to the underlying modular recipe. If you check the [`variables.tf`](https://github.com/zenml-io/mlstacks/blob/main/gcp-modular/variables.tf) file for a given recipe, you can find all the supported flavor-stack component combinations there. - -
- -| Component Type | Flavor(s) | -| ------------------ | ------------------------------------ | -| Artifact Store | s3, gcp, minio | -| Container Registry | aws, gcp | -| Experiment Tracker | mlflow | -| Orchestrator | kubernetes, kubeflow, tekton, vertex | -| MLOps Platform | zenml | -| Model Deployer | seldon | -| Step Operator | sagemaker, vertex | - -### โœจ Customizing your stack components - -With simplicity, we didn't want to compromise on the flexibility that this deployment method allows. As such, we have added the option to pass configuration specific to the stack components as key-value arguments to the deploy CLI. Here is an assortment of all possible configurations that can be set. - -
- -How do configuration flags work? - -The flags that you pass to the deploy CLI are passed on as-is to the backing modular recipes as input variables. This means that all the flags need to be defined as variables in the respective recipe. - -For example, if you take a look at the [`variables.tf`](https://github.com/zenml-io/mlstacks/blob/main/gcp-modular/variables.tf) file for a modular recipe, like the `gcp-modular` recipe, you can find variables like `mlflow_bucket` that you could potentially pass in. - -Validation for these flags does not exist yet at the CLI level, so you must be careful in naming them while calling `deploy`. - -All these extra configuration options are passed in with the `-x` option. For example, we already saw this in action above when we passed in the GCP project ID to the artifact store deploy command. - -```bash -zenml artifact-store deploy -f gcp -p gcp -r us-east1 -x project_id=zenml my_store -``` - -Simply pass in as many `-x` flags as you want to customize your stack component. - -
- -**Experiment Trackers** - -You can assign an existing bucket to the MLflow experiment tracker by passing the `-x mlflow_bucket=...` configuration: - -```shell -zenml experiment-tracker deploy mlflow_tracker --flavor=mlflow -p YOUR_DESIRED_PROVIDER -r YOUR_REGION -x mlflow_bucket=gs://my_bucket -``` - -**Artifact Stores** - -For an artifact store, you can pass `bucket_name` as an argument to the command. - -```bash -zenml artifact-store deploy s3_artifact_store --flavor=s3 --provider=aws -r YOUR_REGION -x bucket_name=my_bucket -``` - -**Container Registries** - -For container registries, you can pass the repository name using `repo_name`: - -```bash -zenml container-registry deploy aws_registry --flavor=aws -p aws -r YOUR_REGION -x repo_name=my_repo -``` - -This is only useful for the AWS case since AWS requires a repository to be created before pushing images to it and the deploy command ensures that a repository with the name you provide is created. In case of GCP and other providers, you can choose the repository name at the same time as you are pushing the image via code. This is achieved through setting the `target_repo` attribute of [the `DockerSettings` object](../customize-docker-builds/README.md). - -#### Other configuration - -* In the case of GCP components, it is _required_ that you pass a project ID to the command as extra configuration when you're creating any GCP resource. - -
ZenML Scarf
diff --git a/docs/book/how-to/stack-deployment/deploy-a-stack-using-mlstacks.md b/docs/book/how-to/stack-deployment/deploy-a-stack-using-mlstacks.md index ad8ceaf1345..d7efec3c5bb 100644 --- a/docs/book/how-to/stack-deployment/deploy-a-stack-using-mlstacks.md +++ b/docs/book/how-to/stack-deployment/deploy-a-stack-using-mlstacks.md @@ -1,19 +1,38 @@ --- -description: Deploying an entire stack with mlstacks. +description: Individually deploying different stack components. --- -# Deploy a stack using mlstacks +# โš’๏ธ Deploying stacks and components using `mlstacks` + +The first step in running your pipelines on remote infrastructure is to deploy all the components that you would need, like an [MLflow tracking server](../../component-guide/experiment-trackers/mlflow.md), +[Kubeflow orchestrator](../../component-guide/orchestrators/kubeflow.md), and more to your cloud. + +This can bring plenty of benefits like scalability, reliability, and collaboration. ZenML eases the path to production by providing a seamless way for all tools to interact with others through the use of abstractions. However, one of the most painful parts of this process, from what we see on our Slack and in general, is the deployment of these stack components. [`mlstacks`](https://mlstacks.zenml.io/) is a Python package that allows you to quickly spin up MLOps infrastructure using Terraform. It is designed to be used with [ZenML](https://zenml.io), but can be used with any MLOps tool or platform. You can deploy a modular MLOps stack for AWS, GCP or K3D using mlstacks. Each deployment type is designed to offer a great deal of flexibility in configuring the resources while preserving the ease of application through the use of sensible defaults. +To make even this process easier for our users, we have created the `deploy` command in `zenml`, which allows you to quickly get started with a full-fledged MLOps stack using only a few commands. You can choose to deploy individual stack components through the stack-component CLI or deploy a stack with multiple components together (a tad more manual steps). + Check out [the full documentation for the mlstacks package](https://mlstacks.zenml.io/) for more information. ## When should I deploy something using mlstacks? -To answer this question, here are some pros and cons in comparison to [the stack-component deploy method](deploy-a-stack-component.md) which can help you choose what works best for you! +{% hint style="info" %} +**MLStacks deploys resources using a Kubernetes cluster, which may be expensive and not for ever y user. In order to use stacks which are more basic and cheaper on the cloud, read [how to easily register a cloud stack](../../how-to/stack-deployment/register-a-cloud-stack.md) +if you have existing infrastructure, or read [how to deploy a cloud stack in one click](../../how-to/stack-deployment/deploy-a-cloud-stack.md).** + +Or simply try running one of: + +```shell +zenml stack register --provider aws +zenml stack deploy --provider aws +``` +{% endhint %} + +To answer this question, here are some pros and cons in comparison to the stack-component deploy method which can help you choose what works best for you! {% tabs %} {% tab title="๐Ÿ˜ Pros" %} @@ -31,114 +50,184 @@ To answer this question, here are some pros and cons in comparison to [the stack {% endtab %} {% endtabs %} +The ZenML CLI has special subcommands that allow you to deploy individual stack components as well as whole stacks using MLStacks. These stacks will be useful for you if: + +* You are at the start of your MLOps journey, and would like to explore different tools. +* You are looking for guidelines for production-grade deployments. + +## How does `mlstacks` work? + +MLStacks is built around the concept of a stack specification. A stack specification is a YAML file that describes the stack and includes references to component specification files. A component specification is a YAML file that describes a component. (Currently all deployments of components (in various combinations) must be defined within the context of a stack.) + +ZenML handles the creation of stack specifications for you when you run one of the `deploy` subcommands using the CLI. A valid specification is generated and used by `mlstacks` to deploy your stack using Terraform. The Terraform definitions and state are stored in your global configuration directory along with any state files generated while deploying your stack. + +Your configuration directory could be in a number of different places depending on your operating system, but read more about it in the [Click docs](https://click.palletsprojects.com/en/8.1.x/api/#click.get\_app\_dir) to see which location applies to your situation. + ## Installing the mlstacks extra -To install `mlstacks`, either run `pip install mlstacks` or `pip install -"zenml[mlstacks]"` to install it along with ZenML. +To install `mlstacks`, either run `pip install mlstacks` or `pip install "zenml[mlstacks]"` to install it along with ZenML. -MLStacks uses Terraform on the backend to manage infrastructure. You will need -to have Terraform installed. Please visit [the Terraform -docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform) -for installation instructions. +MLStacks uses Terraform on the backend to manage infrastructure. You will need to have Terraform installed. Please visit [the Terraform docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform) for installation instructions. -MLStacks also uses Helm to deploy Kubernetes resources. You will need to have -Helm installed. Please visit [the Helm -docs](https://helm.sh/docs/intro/install/#from-script) for installation -instructions. +MLStacks also uses Helm to deploy Kubernetes resources. You will need to have Helm installed. Please visit [the Helm docs](https://helm.sh/docs/intro/install/#from-script) for installation instructions. ## Deploying a stack -A simple stack deployment can be done using the following command: +Deploying an end-to-end stack through the ZenML CLI is only possible with the [deployment wizard which does not use `mlstacks`](../../how-to/stack-deployment/deploy-a-cloud-stack.md). However, you can use `mlstacks` directly to deploy various types of stacks and [import them into ZenML](https://mlstacks.zenml.io/reference/zenml). + +```shell +zenml stack import -f +``` + +## Deploying a stack component + +If you have used ZenML before, you must be familiar with the flow of registering new stack components. It goes something like this: + +```shell +zenml artifact-store register my_store --flavor=s3 --path=s3://my_bucket +``` + +Commands like these assume that you already have the stack component deployed. In this case, it would mean that you must already have a bucket called `my_bucket` on AWS S3 to be able to use this component. + +We took inspiration from this design to build something that feels natural to use and is also sufficiently powerful to take care of the deployment of the respective stack components for you. This is where the \ `deploy` CLI comes in! +The `deploy` command allows you to deploy individual components of your MLOps stack with a single command ๐Ÿš€. You can also customize your components easily by passing in flags (more on that later). + +{% hint style="info" %} +To install `mlstacks`, either run `pip install mlstacks` or `pip install "zenml[mlstacks]"` to install it along with ZenML. + +MLStacks uses Terraform on the backend to manage infrastructure. You will need to have Terraform installed. Please visit [the Terraform docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform) for installation instructions. + +MLStacks also uses Helm to deploy Kubernetes resources. You will need to have Helm installed. Please visit [the Helm docs](https://helm.sh/docs/intro/install/#from-script) for installation instructions. +{% endhint %} + +For example, to deploy an artifact store on a GCP account, you can run: + +{% code overflow="wrap" %} ```bash -zenml stack deploy -p aws -a -n basic -r eu-north-1 -x bucket_name=my_bucket -o sagemaker +# after installing mlstacks +zenml artifact-store deploy -f gcp -p gcp -r us-east1 -x project_id=zenml my_store ``` +{% endcode %} + +The command above takes in the following parameters: + +* **Name**: The name of the stack component. In this case, it is `my_store`. +* **Flavor:** The flavor of the stack component to deploy. Here, we are deploying an artifact store with the `gcp` flavor. +* **Provider:** The provider to deploy this stack component on. Currently, only **GCP, AWS, and K3D** are supported as providers. +* **Region**: The region to deploy the stack component in. +* **Extra Config:** Some components can be customized by the user and these settings are passed as flags to the command. In the example above, we pass the GCP project ID to select what project to deploy the component to. -This command deploys a stack on AWS that uses an S3 bucket as an artifact store -and Sagemaker as your orchestrator. The stack will be imported into ZenML once -the deployment is complete and you can start using it right away! +Successful execution of this command does the following: -Supported flavors and component types are as follows: +* It also automatically registers the deployed stack component with your ZenML server, so you don't have to worry about manually configuring components after the deployment! ๐Ÿคฉ -| Component Type | Flavor(s) | -| -------------- | --------- | -| Artifact Store | s3, gcp, minio | -| Container Registry | aws, gcp | -| Experiment Tracker | mlflow | -| Orchestrator | kubernetes, kubeflow, tekton, vertex | -| MLOps Platform | zenml | -| Model Deployer | seldon | -| Step Operator | sagemaker, vertex | +{% hint style="info" %} +The command currently uses your local credentials for GCP and AWS to provision resources. Integration with your ZenML connectors might be possible soon too! +{% endhint %} -MLStacks currently only supports deployments using AWS, GCP, and K3D as providers. +
-Want more details on how this works internally? +Want to know what happens in the background? -The stack recipe CLI interacts with the -[mlstacks](https://github.com/zenml-io/mlstacks) repository to fetch the recipes -and stores them locally in the **Global Config** directory. +The stack component deploy CLI is powered by ZenML's [mlstacks](https://github.com/zenml-io/mlstacks) in the background. This allows you to configure and deploy select stack components. -This is where you could potentially make any changes you want to the recipe files. You can also use native terraform commands like `terraform apply` to deploy components but this would require you to pass the variables manually using the `-var-file` flag to the terraform CLI. +Using the values you pass for the cloud, the CLI picks up the right modular recipe to use (one of AWS, GCP, or K3D) and then deploys that recipe with the specific stack component enabled.
-### CLI Options for `zenml stack deploy` +### Destroying a stack component -Current required options to be passed in to the `zenml stack deploy` subcommand -are: +Destroying a stack component (i.e. deleting and destroying the underlying +infrastructure) is as easy as deploying one. You can run the following command +to destroy the artifact store we created above: -- `-p` or `--provider`: The cloud provider to deploy the stack on. Currently - supported providers are `aws`, `gcp`, and `k3d`. -- `-n` or `--name`: The name of the stack to be deployed. This is used to - identify the stack in ZenML. -- `-r` or `--region`: The region to deploy the stack in. +```bash +zenml artifact-store destroy -p gcp my_store +``` -The remaining options relate to which components you want to deploy. +This will destroy the deployed infrastructure and prompt you if you also want to remove and deregister the component from your ZenML server. -If you want to pass an `mlstacks` stack specification file into the CLI to use -for deployment, you can do so with the `-f` option. Similarly, if you wish to -see more of the Terraform logging, prompts and output, you can pass the `-d` -flag to turn on `debug-mode`. +### ๐Ÿจ Available flavors for stack components -Any extra configuration for specific components (as noted [in the individual -component deployment documentation](deploy-a-stack-component.md)) can be passed -in with the `-x` option. This option can be used multiple times to pass in -multiple configurations. +Here's a table of all the flavors that can be deployed through the CLI for every stack component. This is a list that will keep on growing and you can also contribute any flavor or stack component that you feel is missing. Refer to the [Contribution page](../../../../CONTRIBUTING.md) for steps on how to do that :smile: -### Interactive stack deployment +
-If you would like to be guided through the deployment process, you can use the -`zenml stack deploy` command with the `--interactive` flag. You will still need -to provide the `provider`, `name` and `region` options as described above but -for the rest, you will get prompts in the CLI as to which components you would -like to deploy. For example, using GCP as the provider you might type: +How does flavor selection work in the background? + +Whenever you pass in a flavor to any stack-component deploy function, the combination of these two parameters is used to construct a variable name in the following format: -```bash -zenml stack deploy -p gcp -n my_new_stack -r us-east1 --interactive ``` +enable__ +``` + +This variable is then passed as input to the underlying modular recipe. If you check the [`variables.tf`](https://github.com/zenml-io/mlstacks/blob/main/gcp-modular/variables.tf) file for a given recipe, you can find all the supported flavor-stack component combinations there. + +
+ +| Component Type | Flavor(s) | +| ------------------ | ------------------------------------ | +| Artifact Store | s3, gcp, minio | +| Container Registry | aws, gcp | +| Experiment Tracker | mlflow | +| Orchestrator | kubernetes, kubeflow, tekton, vertex | +| MLOps Platform | zenml | +| Model Deployer | seldon | +| Step Operator | sagemaker, vertex | + +#### โœจ Customizing your stack components + +With simplicity, we didn't want to compromise on the flexibility that this deployment method allows. As such, we have added the option to pass configuration specific to the stack components as key-value arguments to the deploy CLI. Here is an assortment of all possible configurations that can be set. + +
+ +How do configuration flags work? + +The flags that you pass to the deploy CLI are passed on as-is to the backing modular recipes as input variables. This means that all the flags need to be defined as variables in the respective recipe. -## Displaying Terraform outputs for stacks deployed with mlstacks +For example, if you take a look at the [`variables.tf`](https://github.com/zenml-io/mlstacks/blob/main/gcp-modular/variables.tf) file for a modular recipe, like the `gcp-modular` recipe, you can find variables like `mlflow_bucket` that you could potentially pass in. -If you want to view any of the Terraform-generated outputs for a stack deployed -with `mlstacks`, you can do so with the following command: +Validation for these flags does not exist yet at the CLI level, so you must be careful in naming them while calling `deploy`. + +All these extra configuration options are passed in with the `-x` option. For example, we already saw this in action above when we passed in the GCP project ID to the artifact store deploy command. ```bash -zenml stack describe -o +zenml artifact-store deploy -f gcp -p gcp -r us-east1 -x project_id=zenml my_store ``` -This will print any available outputs to the console if you have deployed a -stack with `mlstacks` via ZenML. +Simply pass in as many `-x` flags as you want to customize your stack component. + +
+ +**Experiment Trackers** + +You can assign an existing bucket to the MLflow experiment tracker by passing the `-x mlflow_bucket=...` configuration: + +```shell +zenml experiment-tracker deploy mlflow_tracker --flavor=mlflow -p YOUR_DESIRED_PROVIDER -r YOUR_REGION -x mlflow_bucket=gs://my_bucket +``` -## Deleting / destroying resources +**Artifact Stores** -๐Ÿ—‘๏ธ Once you're done running your pipelines, there's only a single command you need to execute that will take care of cleaning up all the resources that you had created on your cloud. +For an artifact store, you can pass `bucket_name` as an argument to the command. +```bash +zenml artifact-store deploy s3_artifact_store --flavor=s3 --provider=aws -r YOUR_REGION -x bucket_name=my_bucket ``` -zenml stack destroy + +**Container Registries** + +For container registries, you can pass the repository name using `repo_name`: + +```bash +zenml container-registry deploy aws_registry --flavor=aws -p aws -r YOUR_REGION -x repo_name=my_repo ``` -This will offer you the option to delete the underlying stack specifications and -state files as well. You can also choose to delete the stack from your ZenML -server. +This is only useful for the AWS case since AWS requires a repository to be created before pushing images to it and the deploy command ensures that a repository with the name you provide is created. In case of GCP and other providers, you can choose the repository name at the same time as you are pushing the image via code. This is achieved through setting the `target_repo` attribute of [the `DockerSettings` object](../customize-docker-builds/README.md). + +#### Other configuration + +* In the case of GCP components, it is _required_ that you pass a project ID to the command as extra configuration when you're creating any GCP resource.
ZenML Scarf
diff --git a/docs/book/how-to/stack-deployment/register-a-cloud-stack.md b/docs/book/how-to/stack-deployment/register-a-cloud-stack.md new file mode 100644 index 00000000000..a002d6c864d --- /dev/null +++ b/docs/book/how-to/stack-deployment/register-a-cloud-stack.md @@ -0,0 +1,130 @@ +--- +description: Seamlessly register a cloud stack using existing infrastructure +--- + +# Register a cloud stack with existing infrastructure + +In ZenML, the [stack](../../user-guide/production-guide/understand-stacks.md) +is a fundamental concept that represents the configuration of your +infrastructure. In a normal workflow, creating a stack requires you to first +deploy the necessary pieces of infrastructure and then define them as stack +components in ZenML with proper authentication. + +Especially in a remote setting, this process can be challenging and +time-consuming, and it may create multi-faceted problems. This is why we +implemented a feature called the stack wizard, that allows you to **browse +through your existing infrastructure and use it to register a ZenML cloud +stack**. + +{% hint style="info" %} +If you do not have the required infrastructure pieces already deployed +on your cloud, you can also use [the 1-click deployment tool to build your +cloud stack](deploy-a-cloud-stack.md). +{% endhint %} + +# How to use the Stack Wizard? + +At the moment, the stack wizard can only be accessed through our CLI. + +{% tabs %} +{% tab title="CLI" %} + +In order to register a remote stack over the CLI with the stack wizard, +you can use the following command: + +```shell +zenml stack register -p aws +``` + +To register the cloud stack, the first thing that the wizard needs is a service +connector. You can either use an existing connector by providing its ID +`-c ` or the wizard will create one for you. + +Similar to the service connector, you can also use existing stack components. +However, this is only possible if these component are already configured with +the same service connector that you provided through the parameter +described above. + +{% hint style="warning" %} +Currently, the stack wizard only works on AWS. We are working on bringing +support to GCP and Azure as well. Stay in touch for further updates. +{% endhint %} + +### AWS + +If you select `aws` as your cloud provider, and you haven't selected a connector +yet, you will be prompted to select an authentication method for your stack. + +{% code title="Example Command Output" %} +``` + Available authentication methods for aws +โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ +โ”ƒ Choice โ”ƒ Name โ”ƒ Required โ”ƒ +โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ +โ”‚ [0] โ”‚ AWS Secret Key โ”‚ aws_access_key_id (AWS Access Key ID) โ”‚ +โ”‚ โ”‚ โ”‚ aws_secret_access_key (AWS Secret Access Key) โ”‚ +โ”‚ โ”‚ โ”‚ region (AWS Region) โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ [1] โ”‚ AWS STS Token โ”‚ aws_access_key_id (AWS Access Key ID) โ”‚ +โ”‚ โ”‚ โ”‚ aws_secret_access_key (AWS Secret Access Key) โ”‚ +โ”‚ โ”‚ โ”‚ aws_session_token (AWS Session Token) โ”‚ +โ”‚ โ”‚ โ”‚ region (AWS Region) โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ [2] โ”‚ AWS IAM Role โ”‚ aws_access_key_id (AWS Access Key ID) โ”‚ +โ”‚ โ”‚ โ”‚ aws_secret_access_key (AWS Secret Access Key) โ”‚ +โ”‚ โ”‚ โ”‚ region (AWS Region) โ”‚ +โ”‚ โ”‚ โ”‚ role_arn (AWS IAM Role ARN) โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ [3] โ”‚ AWS Session Token โ”‚ aws_access_key_id (AWS Access Key ID) โ”‚ +โ”‚ โ”‚ โ”‚ aws_secret_access_key (AWS Secret Access Key) โ”‚ +โ”‚ โ”‚ โ”‚ region (AWS Region) โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ [4] โ”‚ AWS Federation Token โ”‚ aws_access_key_id (AWS Access Key ID) โ”‚ +โ”‚ โ”‚ โ”‚ aws_secret_access_key (AWS Secret Access Key) โ”‚ +โ”‚ โ”‚ โ”‚ region (AWS Region) โ”‚ +โ”‚ โ”‚ โ”‚ โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` +{% endcode %} + +Based on your selection, you will have to provide the required parameters listed +above. This will allow ZenML to create a Service Connector and +authenticate you to use your cloud resources. + +Next, for each missing component, the available resources will be listed to +you as follows: + +{% code title="Example Command Output for Artifact Stores" %} +``` + Available AWS storages +โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ +โ”ƒ Choice โ”ƒ Storage โ”ƒ +โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ +โ”‚ [0] โ”‚ s3://************** โ”‚ +โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค +โ”‚ [1] โ”‚ s3://************** โ”‚ +โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ +``` +{% endcode %} + +Based on your selection, ZenML will create the stack component and ultimately +register the stack for you. + +{% endtab %} + +{% tab title="Dashboard" %} + +We are working hard to bring this feature to our dashboard as well. Stay in +touch for further updates. + +{% endtab %} +{% endtabs %} + +There you have it! Through the wizard, you just registered a cloud stack +and, you can start running your pipelines on a remote setting. + +
ZenML Scarf
diff --git a/docs/book/how-to/trigger-pipelines/trigger-a-pipeline-from-another.md b/docs/book/how-to/trigger-pipelines/trigger-a-pipeline-from-another.md index aa158a44d7c..513bcebff5c 100644 --- a/docs/book/how-to/trigger-pipelines/trigger-a-pipeline-from-another.md +++ b/docs/book/how-to/trigger-pipelines/trigger-a-pipeline-from-another.md @@ -3,7 +3,7 @@ description: >- Trigger a pipeline from another pipeline. --- -# Trigger a pipeline from another +# Trigger a pipeline from another pipeline {% hint style="info" %} This is a [ZenML Pro](https://zenml.io/pro) only feature. Please [sign up here](https://cloud.zenml.io) get access. @@ -49,7 +49,7 @@ def loads_data_and_triggers_training(): The pipeline that you're triggering (i.e. `training_pipeline` in the above example) has to have been run previously on a remote stack. In other words, the functionality to trigger a pipeline from another only works when a Docker image has previously been built for that pipeline. In most cases this will be because you ran the pipeline already, but in some cases you might have built the image separately. {% endhint %} -Read more about the [PipelineRunConfiguration](https://sdkdocs.zenml.io/latest/core_code_docs/core-config/#zenml.config.pipeline_run_configuration.PipelineRunConfiguration) object in the [SDK Docs](https://sdkdocs.zenml.io/). +Read more about the [PipelineRunConfiguration](https://sdkdocs.zenml.io/latest/core_code_docs/core-config/#zenml.config.pipeline_run_configuration.PipelineRunConfiguration) and [`trigger_pipeline`](https://sdkdocs.zenml.io/0.60.0/core_code_docs/core-client/#zenml.client.Client) function object in the [SDK Docs](https://sdkdocs.zenml.io/). Read more about Unmaterialized Artifacts [here](../handle-data-artifacts/unmaterialized-artifacts.md). diff --git a/docs/book/how-to/trigger-pipelines/trigger-a-pipeline-from-client.md b/docs/book/how-to/trigger-pipelines/trigger-a-pipeline-from-client.md index c015bdd910d..dfbff8f3d05 100644 --- a/docs/book/how-to/trigger-pipelines/trigger-a-pipeline-from-client.md +++ b/docs/book/how-to/trigger-pipelines/trigger-a-pipeline-from-client.md @@ -26,7 +26,7 @@ if __name__ == "__main__": The pipeline that you're triggering (i.e. `training_pipeline` in the above example) has to have been run previously on a remote stack. In other words, the functionality to trigger a pipeline from another only works when a Docker image has previously been built for that pipeline. In most cases this will be because you ran the pipeline already, but in some cases you might have built the image separately. {% endhint %} -Read more about the [PipelineRunConfiguration](https://sdkdocs.zenml.io/latest/core_code_docs/core-config/#zenml.config.pipeline_run_configuration.PipelineRunConfiguration) object in the [SDK Docs](https://sdkdocs.zenml.io/). +Read more about the [PipelineRunConfiguration](https://sdkdocs.zenml.io/latest/core_code_docs/core-config/#zenml.config.pipeline_run_configuration.PipelineRunConfiguration) and [`trigger_pipeline`](https://sdkdocs.zenml.io/0.60.0/core_code_docs/core-client/#zenml.client.Client) in the [SDK Docs](https://sdkdocs.zenml.io/).
Learn how to run a pipeline directly from anotherorchestrators.md
Learn how to run a pipeline from the REST APIorchestrators.md
diff --git a/docs/book/introduction.md b/docs/book/introduction.md index 0c3342c3492..54a2a380311 100644 --- a/docs/book/introduction.md +++ b/docs/book/introduction.md @@ -21,11 +21,11 @@ ZenML enables MLOps infrastructure experts to define, deploy, and manage sophist # Deploy ZenML to any cloud zenml deploy --provider aws - # Deploy MLOps tools and infrastructure to any cloud - zenml orchestrator deploy kfp --flavor kubeflow --provider gcp + # Connect cloud resources with a simple wizard + zenml stack register --provider aws # Deploy entire MLOps stacks at once - zenml stack deploy gcp-vertexai --provider gcp -o kubeflow ... + zenml stack deploy --provider gcp ``` * **Standardization:** With ZenML, you can standardize MLOps infrastructure and tooling across your organization. Simply register your staging and production environments as ZenML stacks and invite your colleagues to run ML workflows on them. @@ -35,9 +35,6 @@ ZenML enables MLOps infrastructure experts to define, deploy, and manage sophist # Register your production environment zenml stack register production --orchestrator kubeflow ... - - # Make it available to your colleagues - zenml stack share production ``` * Registering your environments as ZenML stacks also enables you to browse and explore them in a convenient user interface. Try it out at [https://www.zenml.io/live-demo](https://www.zenml.io/live-demo)! * **No Vendor Lock-In:** Since infrastructure is decoupled from code, ZenML gives you the freedom to switch to a different tooling stack whenever it suits you. By avoiding vendor lock-in, you have the flexibility to transition between cloud providers or services, ensuring that you receive the best performance and pricing available in the market at any time. diff --git a/docs/book/toc.md b/docs/book/toc.md index 5d5eca45e2a..31d1e57c5fd 100644 --- a/docs/book/toc.md +++ b/docs/book/toc.md @@ -84,9 +84,9 @@ * [Access secrets in a step](how-to/build-pipelines/access-secrets-in-a-step.md) * [Fetching pipelines](how-to/build-pipelines/fetching-pipelines.md) * [Get past pipeline/step runs](how-to/build-pipelines/get-past-pipeline-step-runs.md) -* [๐Ÿšจ Build a pipeline](how-to/trigger-pipelines/README.md) +* [๐Ÿšจ Trigger a pipeline](how-to/trigger-pipelines/README.md) * [Trigger a pipeline from Python Client](how-to/trigger-pipelines/trigger-a-pipeline-from-client.md) - * [Trigger a pipeline from another](how-to/trigger-pipelines/trigger-a-pipeline-from-another.md) + * [Trigger a pipeline from another pipeline](how-to/trigger-pipelines/trigger-a-pipeline-from-another.md) * [Trigger a pipeline from REST API](how-to/trigger-pipelines/trigger-a-pipeline-from-rest-api.md) * [๐Ÿ“ƒ Use configuration files](how-to/use-configuration-files/README.md) * [How to configure a pipeline with a YAML](how-to/use-configuration-files/how-to-use-config.md) @@ -105,6 +105,13 @@ * [Use code repositories to automate Docker build reuse](how-to/customize-docker-builds/use-code-repositories-to-speed-up-docker-build-times.md) * [Build the pipeline without running](how-to/customize-docker-builds/build-the-pipeline-without-running.md) * [Define where an image is built](how-to/customize-docker-builds/define-where-an-image-is-built.md) +* [โš’๏ธ Manage stacks & components](how-to/stack-deployment/README.md) + * [Deploy a cloud stack](how-to/stack-deployment/deploy-a-cloud-stack.md) + * [Register a cloud stack](how-to/stack-deployment/register-a-cloud-stack.md) + * [Deploy stack/components using mlstacks](how-to/stack-deployment/deploy-a-stack-using-mlstacks.md) + * [Reference secrets in stack configuration](how-to/stack-deployment/reference-secrets-in-stack-configuration.md) + * [Implement a custom stack component](how-to/stack-deployment/implement-a-custom-stack-component.md) + * [Implement a custom integration](how-to/stack-deployment/implement-a-custom-integration.md) * [๐Ÿšœ Train with GPUs](how-to/training-with-gpus/training-with-gpus.md) * [๐ŸŒฒ Control logging](how-to/control-logging/README.md) * [View logs on the dashboard](how-to/control-logging/view-logs-on-the-dasbhoard.md) @@ -161,12 +168,6 @@ * [GCP Service Connector](how-to/auth-management/gcp-service-connector.md) * [Azure Service Connector](how-to/auth-management/azure-service-connector.md) * [HyperAI Service Connector](how-to/auth-management/hyperai-service-connector.md) -* [โš’๏ธ Manage stacks](how-to/stack-deployment/README.md) - * [Deploy a stack component](how-to/stack-deployment/deploy-a-stack-component.md) - * [Deploy a stack using mlstacks](how-to/stack-deployment/deploy-a-stack-using-mlstacks.md) - * [Reference secrets in stack configuration](how-to/stack-deployment/reference-secrets-in-stack-configuration.md) - * [Implement a custom stack component](how-to/stack-deployment/implement-a-custom-stack-component.md) - * [Implement a custom integration](how-to/stack-deployment/implement-a-custom-integration.md) * [๐Ÿ Configure Python environments](how-to/configure-python-environments/README.md) * [Handling dependencies](how-to/configure-python-environments/handling-dependencies.md) * [Configure the server environment](how-to/configure-python-environments/configure-the-server-environment.md) diff --git a/docs/book/user-guide/production-guide/cloud-orchestration.md b/docs/book/user-guide/production-guide/cloud-orchestration.md index d1e78877d31..2e2604c6341 100644 --- a/docs/book/user-guide/production-guide/cloud-orchestration.md +++ b/docs/book/user-guide/production-guide/cloud-orchestration.md @@ -13,6 +13,23 @@ In order to do this, we need to get familiar with two more stack components: These, along with [remote storage](remote-storage.md), complete a basic cloud stack where our pipeline is entirely running on the cloud. +{% hint style="info" %} +**Don't want to learn how to deploy and register stack components manually? +Feel free to skip this chapter and read [how to deploy a cloud stack in one click](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +and learn how to use:** + +```shell +zenml stack deploy --provider aws +``` + +**or if you have existing infrastructure, read [how to easily register a cloud stack](../../how-to/stack-deployment/register-a-cloud-stack.md) +and learn how to use:** + +```shell +zenml stack register --provider aws +``` +{% endhint %} + ## Starting with a basic cloud stack The easiest cloud orchestrator to start with is the [Skypilot](https://skypilot.readthedocs.io/) orchestrator running on a public cloud. The advantage of Skypilot is that it simply provisions a VM to execute the pipeline on your cloud provider. @@ -31,7 +48,7 @@ To summarize, here is the broad sequence of events that happen when you run a pi 6. As each pipeline runs, it stores artifacts physically in the `artifact store`. Of course, this artifact store needs to be some form of cloud storage. 7. As each pipeline runs, it reports status back to the ZenML server and optionally queries the server for metadata. -## Provisioning and registering a Skypilot orchestrator alongside a container registry +## Provisioning and registering an orchestrator alongside a container registry While there are detailed docs on [how to set up a Skypilot orchestrator](../../component-guide/orchestrators/skypilot-vm.md) and a [container registry](../../component-guide/container-registries/container-registries.md) on each public cloud, we have put the most relevant details here for convenience: @@ -52,8 +69,8 @@ AWS_PROFILE= zenml service-connector register cloud_connector --typ Once the service connector is set up, we can register [a Skypilot orchestrator](../../component-guide/orchestrators/skypilot-vm.md): ```shell -zenml orchestrator register skypilot_orchestrator -f vm_aws -zenml orchestrator connect skypilot_orchestrator --connector cloud_connector +zenml orchestrator register cloud_orchestrator -f vm_aws +zenml orchestrator connect cloud_orchestrator --connector cloud_connector ``` The next step is to register [an AWS container registry](../../component-guide/container-registries/aws.md). Similar to the orchestrator, we will use our connector as we are setting up the container registry: @@ -84,8 +101,8 @@ zenml service-connector register cloud_connector --type gcp --auth-method servic Once the service connector is set up, we can register [a Skypilot orchestrator](../../component-guide/orchestrators/skypilot-vm.md): ```shell -zenml orchestrator register skypilot_orchestrator -f vm_gcp -zenml orchestrator connect skypilot_orchestrator --connect cloud_connector +zenml orchestrator register cloud_orchestrator -f vm_gcp +zenml orchestrator connect cloud_orchestrator --connect cloud_connector ``` The next step is to register [a GCP container registry](../../component-guide/container-registries/gcp.md). Similar to the orchestrator, we will use our connector as we are setting up the container registry: @@ -101,23 +118,29 @@ For more information, you can always check the [dedicated Skypilot orchestrator {% endtab %} {% tab title="Azure" %} -In order to launch a pipeline on Azure with the SkyPilot orchestrator, the first thing that you need to do is to install the Azure and SkyPilot integrations: +As of [v0.60.0](https://github.com/zenml-io/zenml/releases/tag/0.60.0), alongside the switch to `pydantic` v2, due to an incompatibility between the new version `pydantic` and the `azurecli`, the `skypilot[azure]` flavor can not be installed at the same time. Therefore, for Azure users, an alternative is to use the [Kubernetes Orchestrator](../../component-guide/orchestrators/kubernetes.md). You can easily deploy a Kubernetes cluster in your subscription using the [Azure Kubernetes Service](https://azure.microsoft.com/en-us/products/kubernetes-service). + +In order to launch a pipeline on Azure with the Kubernetes orchestrator, the first thing that you need to do is to install the Azure and Kubernetes integrations: ```shell -zenml integration install azure skypilot_azure -y +zenml integration install azure kubernetes -y ``` +You should also ensure you have [kubectl installed](https://kubernetes.io/docs/tasks/tools/). + Before we start registering any components, there is another step that we have to execute. As we [explained in the previous section](remote-storage.md#configuring-permissions-with-your-first-service-connector), components such as orchestrators and container registries often require you to set up the right permissions. In ZenML, this process is simplified with the use of [Service Connectors](../../how-to/auth-management/README.md). For this example, we will need to use the [Service Principal authentication feature of our Azure service connector](../../how-to/auth-management/azure-service-connector.md#azure-service-principal): ```shell zenml service-connector register cloud_connector --type azure --auth-method service-principal --tenant_id= --client_id= --client_secret= ``` -Once the service connector is set up, we can register [a Skypilot orchestrator](../../component-guide/orchestrators/skypilot-vm.md): +Once the service connector is set up, we can register [a Kubernetes orchestrator](../../component-guide/orchestrators/kubernetes.md): ```shell -zenml orchestrator register skypilot_orchestrator -f vm_azure -zenml orchestrator connect skypilot_orchestrator --connect cloud_connector +# Ensure your service connector has access to the AKS cluster: +zenml service-connector list-resources --resource-type kubernetes-cluster -e +zenml orchestrator register cloud_orchestrator --flavor kubernetes +zenml orchestrator connect cloud_orchestrator --connect cloud_connector ``` The next step is to register [an Azure container registry](../../component-guide/container-registries/azure.md). Similar to the orchestrator, we will use our connector as we are setting up the container registry. @@ -129,7 +152,7 @@ zenml container-registry connect cloud_container_registry --connector cloud_conn With the components registered, everything is set up for the next steps. -For more information, you can always check the [dedicated Skypilot orchestrator guide](../../component-guide/orchestrators/skypilot-vm.md). +For more information, you can always check the [dedicated Kubernetes orchestrator guide](../../component-guide/orchestrators/kubernetes.md). {% endtab %} {% endtabs %} @@ -144,7 +167,7 @@ Now that we have our orchestrator and container registry registered, we can [reg {% tabs %} {% tab title="CLI" %} ```shell -zenml stack register minimal_cloud_stack -o skypilot_orchestrator -a cloud_artifact_store -c cloud_container_registry +zenml stack register minimal_cloud_stack -o cloud_orchestrator -a cloud_artifact_store -c cloud_container_registry ``` {% endtab %} {% endtabs %} @@ -165,4 +188,4 @@ You will notice this time your pipeline behaves differently. After it has built Curious to see what other stacks you can create? The [Component Guide](../../component-guide/README.md) has an exhaustive list of various artifact stores, container registries, and orchestrators that are integrated with ZenML. Try playing around with more stack components to see how easy it is to switch between MLOps stacks with ZenML. -
ZenML Scarf
+
ZenML Scarf
\ No newline at end of file diff --git a/docs/book/user-guide/production-guide/configure-pipeline.md b/docs/book/user-guide/production-guide/configure-pipeline.md index 70dff3d2d75..ad411483657 100644 --- a/docs/book/user-guide/production-guide/configure-pipeline.md +++ b/docs/book/user-guide/production-guide/configure-pipeline.md @@ -107,7 +107,7 @@ This is as easy as adding the following section to your local `training_rf.yaml` settings: ... - # Adapt this to vm_azure or vm_gcp accordingly + # Adapt this to vm_gcp accordingly orchestrator.vm_aws: memory: 32 # in GB @@ -119,10 +119,36 @@ steps: cpus: 8 ``` -Here we are configuring the entire pipeline with a certain amount of memory, while for the trainer step we are additionally configuring 8 CPU cores. The `orchestrator.vm_aws` key corresponds to the [`SkypilotBaseOrchestratorSettings`](https://sdkdocs.zenml.io/latest/integration\_code\_docs/integrations-skypilot/#zenml.integrations.skypilot.flavors.skypilot\_orchestrator\_base\_vm\_config.SkypilotBaseOrchestratorSettings) class in the Python SDK. You can adapt it to `vm_gcp` or `vm_azure` depending on which flavor of skypilot you have configured. +Here we are configuring the entire pipeline with a certain amount of memory, while for the trainer step we are additionally configuring 8 CPU cores. The `orchestrator.vm_aws` key corresponds to the [`SkypilotBaseOrchestratorSettings`](https://sdkdocs.zenml.io/latest/integration\_code\_docs/integrations-skypilot/#zenml.integrations.skypilot.flavors.skypilot\_orchestrator\_base\_vm\_config.SkypilotBaseOrchestratorSettings) class in the Python SDK. You can adapt it to `vm_gcp` if you are using the GCP variant of skypilot. + +
+ +Instructions for Microsoft Azure Users + +As discussed [before](cloud-orchestration.md), we are using the [Kubernetes orchestrator](../../component-guide/orchestrators/kubernetes.md) for Azure users. In order to scale compute for the Kubernetes orchestrator, the +YAML file needs to look like this: + +```yaml +# These are the resources for the entire pipeline, i.e., each step +settings: + ... + + resources: + memory: "32GB" + +... +steps: + model_trainer: + settings: + resources: + memory: "8GB" +``` + +
{% hint style="info" %} -Read more about settings in ZenML [here](../../how-to/use-configuration-files/runtime-configuration.md). +Read more about settings in ZenML [here](../../how-to/use-configuration-files/runtime-configuration.md) and +[here](../../how-to/training-with-gpus/training-with-gpus.md) {% endhint %} Now let's run the pipeline again: diff --git a/docs/book/user-guide/production-guide/production-guide.md b/docs/book/user-guide/production-guide/production-guide.md index f287d9a6c50..618d3aafc21 100644 --- a/docs/book/user-guide/production-guide/production-guide.md +++ b/docs/book/user-guide/production-guide/production-guide.md @@ -14,6 +14,7 @@ This guide will focus on shifting gears from running pipelines _locally_ on your * [Understanding stacks](understand-stacks.md) * [Connecting remote storage](remote-storage.md) * [Orchestrating on the cloud](cloud-orchestration.md) +* [Easy stack registration](easy-stack-registration.md) * [Configuring the pipeline to scale compute](configure-pipeline.md) * [Configure a code repository](connect-code-repository.md) diff --git a/docs/book/user-guide/production-guide/remote-storage.md b/docs/book/user-guide/production-guide/remote-storage.md index 75fea33eb0d..8aef02cecff 100644 --- a/docs/book/user-guide/production-guide/remote-storage.md +++ b/docs/book/user-guide/production-guide/remote-storage.md @@ -12,6 +12,23 @@ When using a stack with remote storage, nothing changes except the fact that the

Sequence of events that happen when running a pipeline on a remote artifact store.

+{% hint style="info" %} +**Don't want to learn how to deploy and register stack components manually? +Feel free to skip this chapter and read [how to deploy a cloud stack in one click](../../how-to/stack-deployment/deploy-a-cloud-stack.md) +and learn how to use:** + +```shell +zenml stack deploy --provider aws +``` + +**or if you have existing infrastructure, read [how to easily register a cloud stack](../../how-to/stack-deployment/register-a-cloud-stack.md) +and learn how to use:** + +```shell +zenml stack register --provider aws +``` +{% endhint %} + ## Provisioning and registering a remote artifact store Out of the box, ZenML ships with [many different supported artifact store flavors](../../component-guide/artifact-stores/artifact-stores.md). For convenience, here are some brief instructions on how to quickly get up and running on the major cloud providers: