Add more detailed docs for ZenML Pro (#3065)

* add roles page

* and org and tenants page draft

* add hierarchy diagram

* add images to role page

* update tenants page

* add images to orgs

* update images

* rename file

* Optimised images with calibre/image-actions

* Apply suggestions from code review

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

* update with deployment changes

* Optimised images with calibre/image-actions

* TOC

* TOC

* TOC

* TOC

* TOC

* TOC

* Update deployment links to relative paths

* Update deployment options in README.md

* Update ZenML Pro README with new features

* Update ZenML Pro hierarchy of concepts and TOC links

* Add hierarchy of concepts in ZenML Pro to TOC

* Remove ZenML Pro deployment section from TOC- Reflects removal of ZenML Pro deployment section in TOC

* Update ZenML Pro features and fix README table

* Fix table formatting in ZenML Pro README

* Update organization details in ZenML Pro README

* Update ZenML Pro features in README.md

* Add table linking to pages for each concept in ZenML Pro

* Update hierarchy of concepts in ZenML Pro documentation

* Update organization and tenant creation instructions

* Update roles and permissions in ZenML Pro

* Update roles and teams order in toc.md

* Add steps to create custom roles in roles.md

* Update user description for ZenML Pro instance

* Update roles link and title to "Roles & Permissions".

* Restructure custom role creation in ZenML Pro dashboard

* Update ZenML Pro links to README.md for consistency- Update ZenML Pro links to README.md for consistency

* Add ZenML Pro API documentation to table of contents

* Add initial team management functionality

* Update ZenML Pro deployment scenarios and documentation

* Optimised images with calibre/image-actions

* Update ZenML Pro API authentication information

* Update ZenML Pro API authentication instructions and roles

* Update ZenML Pro early-access features in README.md

* Add share dialog image and update roles.md

* Optimised images with calibre/image-actions

* Update docs/book/getting-started/zenml-pro/README.md

Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>

* Update ZenML Pro README with enhanced features

* Update docs/book/getting-started/zenml-pro/pro-api.md

Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>

* Update docs/book/getting-started/zenml-pro/roles.md

Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>

* Update docs/book/getting-started/zenml-pro/roles.md

Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>

* Update docs/book/getting-started/zenml-pro/roles.md

Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>

* Update docs/book/getting-started/zenml-pro/teams.md

Co-authored-by: Jayesh Sharma <wjayesh@outlook.com>

* Update roles UI with new screenshots and instructions

* Add core concepts diagram to core-concepts.md

* Optimised images with calibre/image-actions

* Update hierarchy of concepts.md with new template section

* Add ZenML Pro deployment architecture images

* Optimised images with calibre/image-actions

* Update ZenML Pro API reference link in table of contents

* Update system architecture title in TOC to emoji

* Update deployment and secret management paths for ZenML Pro

* Add system architectures to getting started section

* Update system architecture title formatting in TOC

* Update deployment architecture image links.- Shorten path for OSS system architecture link

* Add more details on deploying ZenML OSS components

* Update architecture description in system-architectures.md

* Add ZenML Pro components to deployment architecture

* Update image path for ZenML OSS server deployment

* Add detailed ZenML Pro self-hosted deployment architecture

* Update deployment images in assets folder

* Optimised images with calibre/image-actions

* Update architecture diagram images with figure and captions

* Add new deployment images for OSS and Pro

* Remove Zone.Identifier files for deployment images

* Optimised images with calibre/image-actions

* Update ZenML OSS deployment components

* Optimised images with calibre/image-actions

* Update ZenML architecture documentation with deployment info

* Add organizing tenants by business logic and use-case

* Optimised images with calibre/image-actions

* Organize tenants by staging and production + business logic

* Update ZenML Pro hierarchy of concepts to Core Concepts

* Update Pro-only features links in tenants.md

* Update ZenML Pro core concepts link in TOC

* Update system architectures diagram in getting started guide

* Update SaaS deployment architecture diagrams

* Update ZenML Pro description to include SaaS option

* Fix typo in system-architectures.md link

* Optimised images with calibre/image-actions

* Update Pro Metadata Store description in system architectures<commit message>

* New image

* Optimised images with calibre/image-actions

* Add System Architecture to TOC

* Optimised images with calibre/image-actions

* Update ZenML Pro deployment scenarios in documentation

* Delete SVG files and update tenant images to PNG format

* Optimised images with calibre/image-actions

* add scarf

* Apply suggestions from code review

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

* Update ZenML Pro API description in docs

* Update tenant organization best practices text

* Update docs/book/getting-started/zenml-pro/tenants.md

Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>

* Remove Zone.Identifier file attribute from zenml_cloud.png

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Hamza Tahir <hamza@zenml.io>
Co-authored-by: Alex Strick van Linschoten <strickvl@users.noreply.github.com>
Co-authored-by: Hamza Tahir <htahir111@gmail.com>
Co-authored-by: Alex Strick van Linschoten <stricksubscriptions@fastmail.fm>

.gitbook.yaml

@@ -15,3 +15,5 @@ redirects:
   getting-started/deploying-zenml/manage-the-deployed-services/upgrade-the-version-of-the-zenml-server.md: how-to/manage-the-zenml-server/upgrade-zenml-server.md
   getting-started/deploying-zenml/manage-the-deployed-services/troubleshoot-your-deployed-server.md: how-to/manage-the-zenml-server/troubleshoot-your-deployed-server.md
   how-to/stack-deployment/implement-a-custom-integration.md: how-to/contribute-to-zenml/implement-a-custom-integration.md
+
+  getting-started/zenml-pro/system-architectures: getting-started/system-architectures.md

docs/book/getting-started/core-concepts.md

@@ -4,6 +4,8 @@ description: Discovering the core concepts behind ZenML.
 
 # 🪄 Core concepts
 
+![A diagram of core concepts of ZenML OSS](../.gitbook/assets/core_concepts_oss.png)
+
 **ZenML** is an extensible, open-source MLOps framework for creating portable, production-ready **MLOps pipelines**. It's built for data scientists, ML Engineers, and MLOps Developers to collaborate as they develop to production. In order to achieve this goal, ZenML introduces various concepts for different aspects of an ML workflow and we can categorize these concepts under three different threads:
 
 <table data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:purple;"><strong>1. Development</strong></mark></td><td>As a developer, how do I design my machine learning workflows?</td><td></td><td><a href="core-concepts.md#1-development">#1-development</a></td></tr><tr><td><mark style="color:purple;"><strong>2. Execution</strong></mark></td><td>While executing, how do my workflows utilize the large landscape of MLOps tooling/infrastructure?</td><td></td><td><a href="core-concepts.md#2-execution">#2-execution</a></td></tr><tr><td><mark style="color:purple;"><strong>3. Management</strong></mark></td><td>How do I establish and maintain a production-grade and efficient solution?</td><td></td><td><a href="core-concepts.md#3-management">#3-management</a></td></tr></tbody></table>
@@ -133,7 +135,7 @@ To use _stack components_ that are running remotely on a cloud infrastructure, y
 
 #### Server Deployment
 
-In order to benefit from the advantages of using a deployed ZenML server, you can either choose to use the [**ZenML Pro SaaS offering**](zenml-pro/zenml-pro.md) which provides a control plane for you to create managed instances of ZenML servers, or [deploy it in your self-hosted environment](deploying-zenml/README.md).
+In order to benefit from the advantages of using a deployed ZenML server, you can either choose to use the [**ZenML Pro SaaS offering**](zenml-pro/README.md) which provides a control plane for you to create managed instances of ZenML servers, or [deploy it in your self-hosted environment](deploying-zenml/README.md).
 
 #### Metadata Tracking
 

docs/book/getting-started/deploying-zenml/README.md

@@ -4,6 +4,8 @@ description: Why do we need to deploy ZenML?
 
 # 🤔 Deploying ZenML
 
+![ZenML OSS server deployment architecture](../../.gitbook/assets/oss_simple_deployment.png)
+
 Moving your ZenML Server to a production environment offers several benefits over staying local:
 
 1. **Scalability**: Production environments are designed to handle large-scale workloads, allowing your models to process more data and deliver faster results.
@@ -12,19 +14,58 @@ Moving your ZenML Server to a production environment offers several benefits ove
 
 Despite these advantages, transitioning to production can be challenging due to the complexities involved in setting up the needed infrastructure.
 
-### ZenML Server
+## Components
+
+A ZenML deployment consists of multiple infrastructure components:
+
+- [FastAPI server](https://github.com/zenml-io/zenml/tree/main/src/zenml/zen_server) backed with a SQLite or MySQL database
+- [Python Client](https://github.com/zenml-io/zenml/tree/main/src/zenml)
+- An [open-source companion ReactJS](https://github.com/zenml-io/zenml-dashboard) dashboard
+- (Optional) [ZenML Pro API + Database + ZenML Pro dashboard](../system-architectures.md)
+
+You can read more in-depth about the system architecture of ZenML [here](../system-architectures.md).
+This documentation page will focus on the components required to deploy ZenML OSS.
+
+<details>
+
+<summary>Details on the ZenML Python Client</summary>
+
+The ZenML client is a Python package that you can install on your machine. It
+is used to interact with the ZenML server. You can install it using the `pip`
+command as outlined [here](../installation.md).
+
+This Python package gives you [the `zenml` command-line interface](https://sdkdocs.zenml.io/latest/cli/) which
+you can use to interact with the ZenML server for common tasks like managing
+stacks, setting up secrets, and so on. It also gives you the general framework that lets you
+[author and deploy pipelines](../../user-guide/starter-guide/README.md) and so forth.
 
-When you first get started with ZenML, it relies with the following architecture on your machine.
+If you want to have more fine-grained control and access to the metadata that
+ZenML manages, you can use the Python SDK to access the API. This allows you to
+create your own custom automations and scripts and is the most common way teams
+access the metadata stored in the ZenML server. The full documentation for the
+Python SDK can be found [here](https://sdkdocs.zenml.io/latest/). The full HTTP
+[API documentation](../../reference/api-reference.md) can also be found by adding the 
+`/doc` suffix to the URL when accessing your deployed ZenML server.
+
+</details>
+
+### Deployment scenarios
+
+When you first get started with ZenML, you have the following architecture on your machine.
 
 ![ZenML default local configuration](../../.gitbook/assets/Scenario1.png)
 
-The SQLite database that you can see in this diagram is used to store information about pipelines, pipeline runs, stacks, and other configurations. Users can run the `zenml up` command to spin up a local REST server to serve the dashboard. The diagram for this looks as follows:
+The SQLite database that you can see in this diagram is used to store
+information about pipelines, pipeline runs, stacks, and other configurations.
+This default setup allows you to get started and try out the core features but
+you won't be able to use cloud-based components like serverless orchestrators
+and so on.
 
-![ZenML with a local REST Server](../../.gitbook/assets/Scenario2.png)
+Users can run the `zenml up` command to spin up a local ZenML OSS server to serve the
+dashboard. For the local OSS server option, the `zenml up` command implicitly
+connects the client to the server. The diagram for this looks as follows:
 
-{% hint style="info" %}
-For the local REST server option, the `zenml up` command implicitly connects the client to the server.
-{% endhint %}
+![ZenML with a local ZenML OSS Server](../../.gitbook/assets/Scenario2.png)
 
 {% hint style="warning" %}
 Currently the ZenML server supports a legacy and a brand-new version of the dashboard. To use the legacy version simply use the
@@ -33,27 +74,33 @@ following command `zenml up --legacy`
 
 In order to move into production, the ZenML server needs to be deployed somewhere centrally so that the different cloud stack components can read from and write to the server. Additionally, this also allows all your team members to connect to it and share stacks and pipelines.
 
-![Deployed ZenML Server](../../.gitbook/assets/Scenario3.2.png)
+![ZenML centrally deployed for multiple users](../../.gitbook/assets/Scenario3.2.png)
 
-### Deploying a ZenML Server
+You connect to your deployed ZenML server using the `zenml connect` command and
+then you have the full benefits and power of ZenML. You can use all the
+cloud-based components, your metadata will be stored and synchronized across all
+the users of the server and you can leverage features like centralized logs
+storage and pipeline artifact visualization.
+
+## How to deploy ZenML
 
 Deploying the ZenML Server is a crucial step towards transitioning to a production-grade environment for your machine learning projects. By setting up a deployed ZenML Server instance, you gain access to powerful features, allowing you to use stacks with remote components, centrally track progress, collaborate effectively, and achieve reproducible results.
 
 Currently, there are two main options to access a deployed ZenML server:
 
-1. **SaaS:** With [ZenML Pro](../zenml-pro/zenml-pro.md) offering you can utilize a control plane to create ZenML servers, also known as tenants. These tenants are managed and maintained by ZenML's dedicated team, alleviating the burden of server management from your end. Importantly, your data remains securely within your stack, and ZenML's role is primarily to handle tracking of metadata and server maintenance.
-2. **Self-hosted Deployment:** Alternatively, you have the ability to deploy ZenML on your own self-hosted environment. This can be achieved through various methods, including using [Docker](../../component-guide/model-registries/model-registries.md), [Helm](deploy-with-helm.md), or [HuggingFace Spaces](deploy-using-huggingface-spaces.md). We also offer our Pro version for self-hosted deployments, so you can use our full paid feature-set while staying fully in control with an airgapped solution on your infrastructure.
+1. **Managed deployment:** With [ZenML Pro](../zenml-pro/README.md) offering you can utilize a control plane to create ZenML servers, also known as [tenants](../zenml-pro/tenants.md). These tenants are managed and maintained by ZenML's dedicated team, alleviating the burden of server management from your end. Importantly, your data remains securely within your stack, and ZenML's role is primarily to handle tracking of metadata and server maintenance.
+2. **Self-hosted Deployment:** Alternatively, you have the ability to deploy ZenML on your own self-hosted environment. This can be achieved through various methods, including using [Docker](./deploy-with-docker.md), [Helm](./deploy-with-helm.md), or [HuggingFace Spaces](./deploy-using-huggingface-spaces.md). We also offer our Pro version for self-hosted deployments, so you can use our full paid feature-set while staying fully in control with an air-gapped solution on your infrastructure.
 
 {% hint style="warning" %}
 Currently the ZenML server supports a legacy and a brand-new version of the dashboard. To use the legacy version which supports stack registration from the dashboard simply set the following environment variable in the deployment environment: `export ZEN_SERVER_USE_LEGACY_DASHBOARD=True`.
 {% endhint %}
 
 Both options offer distinct advantages, allowing you to choose the deployment approach that best aligns with your organization's needs and infrastructure preferences. Whichever path you select, ZenML facilitates a seamless and efficient way to take advantage of the ZenML Server and enhance your machine learning workflows for production-level success.
 
-## How to deploy ZenML
+### Options for deploying ZenML
 
 Documentation for the various deployment strategies can be found in the following pages below (in our 'how-to' guides):
 
-<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-type="content-ref"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:purple;"><strong>Deploy with Docker</strong></mark></td><td>Deploying ZenML in a Docker container.</td><td></td><td></td><td><a href="deploy-with-docker.md">deploy-with-docker.md</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy with Helm</strong></mark></td><td>Deploying ZenML in a Kubernetes cluster with Helm.</td><td></td><td></td><td><a href="deploy-with-helm.md">deploy-with-helm.md</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy with HuggingFace Spaces</strong></mark></td><td>Deploying ZenML to Hugging Face Spaces.</td><td></td><td></td><td><a href="deploy-using-huggingface-spaces.md">deploy-with-hugging-face-spaces.md</a></td></tr></tbody></table>
+<table data-card-size="large" data-view="cards"><thead><tr><th></th><th></th><th data-hidden></th><th data-hidden data-type="content-ref"></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><mark style="color:purple;"><strong>Deploying ZenML using ZenML Pro</strong></mark></td><td>Deploying ZenML using ZenML Pro.</td><td></td><td></td><td><a href="../zenml-pro/README.md">deploy-with-zenml-cli.md</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy with Docker</strong></mark></td><td>Deploying ZenML in a Docker container.</td><td></td><td></td><td><a href="./deploy-with-docker.md">deploy-with-docker.md</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy with Helm</strong></mark></td><td>Deploying ZenML in a Kubernetes cluster with Helm.</td><td></td><td></td><td><a href="./deploy-with-helm.md">deploy-with-helm.md</a></td></tr><tr><td><mark style="color:purple;"><strong>Deploy with HuggingFace Spaces</strong></mark></td><td>Deploying ZenML to Hugging Face Spaces.</td><td></td><td></td><td><a href="./deploy-using-huggingface-spaces.md">deploy-with-hugging-face-spaces.md</a></td></tr></tbody></table>
 
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>

docs/book/getting-started/deploying-zenml/secret-management.md

@@ -67,6 +67,6 @@ This migration strategy is not necessary if the actual location of the secrets v
 * updating the credentials used to authenticate with the Secrets Store back-end before or after they expire
 * switching to a different authentication method to authenticate with the same Secrets Store back-end (e.g. switching from an IAM account secret key to an IAM role in the AWS Secrets Manager)
 
-If you are a [ZenML Pro](https://zenml.io/pro) user, you can configure your cloud backend based on your [deployment scenario](../zenml-pro/system-architectures.md).
+If you are a [ZenML Pro](https://zenml.io/pro) user, you can configure your cloud backend based on your [deployment scenario](../system-architectures.md).
 
 <figure><img src="https://static.scarf.sh/a.png?x-pxid=f0b4f458-0a54-4fcd-aa95-d5ee424815bc" alt="ZenML Scarf"><figcaption></figcaption></figure>