docs: multi-node airgap documentation

docs/docs-content/clusters/edge/edge-configuration/installer-reference.md

@@ -34,6 +34,7 @@ listed in alphabetical order.
 | `stylus.installationMode`      | Allowed values are `connected` and `airgap`. Default value is `connected`. `connected` means that the Edge host has a connection to Palette; `airgap` means it does not have a connection to Palette.      | `connected` |
 | `stylus.localUI.port`          | Specifies the port that Local UI is exposed on.                                                                                                                                                            | 5080        |
 | `stylus.site`                  | Review Site Parameters for more information.                                                                                                                                                               |             |
+| `stylus.enableMultiNode`       | When set to true, the host is allowed to link with other nodes and form a multi-node cluster. For more information, refer to [Link Hosts](../local-ui//cluster-management/link-hosts.md).                  | `False`     |
 | `stylus.externalRegistries`    | Use this parameter to configure multiple external registries and to apply domain re-mapping rules. Review [External Registry Parameters](#multiple-external-registries) for more information.              | None        |
 | `stylus.registryCredentials`   | Only used when a single external registry in use and no mapping rules are needed. Review [Single External Registry](#single-external-registry) for more information.                                       | None        |
 | `stylus.trace`                 | Enable this parameter to display trace output. Allowed values are `true` or `false`.                                                                                                                       | `False`     |

docs/docs-content/clusters/edge/edgeforge-workflow/palette-canvos/build-content-bundle.md

@@ -55,6 +55,10 @@ Creating a content bundle provides several benefits that may address common use
   version `4.5.3` or earlier to create content bundles. To download the latest version of the Edge CLI, visit the
   [Downloads](../../../../spectro-downloads.md#palette-edge-cli) page.
 
+- Content bundles built with the Palette Edge CLI versions earlier than `4.5.c` cannot be used to provision multi-node
+  clusters. Download the latest version of the Palette Edge CLI from the
+  [Downloads](../../../../spectro-downloads.md#palette-edge-cli) page to build the content bundle instead.
+
 ## Prerequisites
 
 - Linux Machine (Physical or VM) with an AMD64 architecture.

docs/docs-content/clusters/edge/local-ui/cluster-management/cluster-management.md

@@ -7,9 +7,10 @@ sidebar_position: 32
 tags: ["edge"]
 ---
 
-Local UI allows you to upload content bundles to the Edge host. The content in the content bundles can then be used to
-create a single-node cluster consisting of the Edge host itself. You can also import cluster definitions, which is a
-bundle of one or more cluster profiles and their profile variables, from Palette to define your Edge cluster.
+Local UI allows you to upload content bundles to the Edge host. The content in the content bundles can then be used
+during cluster creation. To create a cluster, you import cluster definitions, which is a bundle of one or more cluster
+profiles and their profile variables, from Palette to define your Edge cluster. If you have multiple hosts that are
+linked to each other, you can create multi-node clusters consisting of the linked hosts.
 
 Cluster operations in Local UI are only available to airgapped Edge hosts. You cannot use Local UI to create or manage
 clusters, upload content bundles, or cluster definition on a connected Edge host. You must use Palette to manage the
@@ -27,3 +28,11 @@ Refer to the following resources to learn how to upload content and create a clu
 - [Export Cluster Definition](./export-cluster-definition.md)
 
 - [Create Local Cluster](./create-cluster.md)
+
+- [Link Hosts](./link-hosts.md)
+
+- [Scale a Cluster](./scale-cluster.md)
+
+- [Delete a Cluster](./delete-cluster.md)
+
+- [Share Local UI Authentication](./share-auth.md)

docs/docs-content/clusters/edge/local-ui/cluster-management/create-cluster.md

@@ -1,43 +1,66 @@
 ---
 sidebar_label: "Create Local Cluster"
 title: "Create Local Cluster"
-description: "Instructions for creating a locally manageg cluster in Edge Host Management Console."
+description: "Instructions for creating a locally managed cluster in Edge Host Management Console."
 hide_table_of_contents: false
-sidebar_position: 30
+sidebar_position: 20
 tags: ["edge"]
 ---
 
-You can create and manage a single-node cluster on an airgapped Edge host locally from Local UI. This page guides you
-through how to create a cluster using Local UI.
+You can create and manage single-node as well as multi-node clusters on airgapped hosts locally from Local UI. This page
+teaches you how to create a cluster using Local UI.
 
 :::preview
 
 :::
 
 ## Limitations
 
-- You can only create single-node clusters consisting solely of the Edge host you create the cluster from. You cannot
-  include other Edge hosts in the same local or remote network.
+- For multi-node clusters, all hosts must be deployed in the same deployment mode. For more information, refer to
+  [Deployment Modes](../../../../deployment-modes/deployment-modes.md).
+
+- For hosts that are deployed in agent mode, all hosts must share the same Operating System (OS).
+
+<!-- prettier-ignore -->
+- For multi-node clusters, do not use the
+  <VersionedLink text="Local Path Provisioner Pack" url="/integrations/packs/?pack=csi-local-path-provisioner" />. This
+  is because whenever a node is drained during an upgrade or for any other reason, the volumes will not dynamically move
+  with the local path provisioner.
 
 ## Prerequisites
 
 - Network access to the Edge device’s IP and port where Local UI is exposed. The default port is 5080.
 
+- The `stylus.enableMultiNode` parameter is set to `true` in your user data configuration. For more information, refer
+  to [Prepare User Data](../../edgeforge-workflow/prepare-user-data.md).
+
+- The `stylus.installationMode` parameter is set to `airgap` in your user data configuration for all your hosts.
+
 - Credentials to log into Local UI. Any OS user can be used to log in to Local UI.
 
 - You have uploaded the necessary software artifacts to the Edge host or included the artifacts in the Edge Installer
   ISO during EdgeForge. For more information, refer to [Upload Content Bundle](./upload-content-bundle.md) and
   [Build Content Bundle](../../edgeforge-workflow/palette-canvos/build-content-bundle.md).
 
-- You must ensure that the Edge host has a stable IP address. You have the following options to achieve a stable IP
-  address:
+  :::warning
+
+  Content bundles must be built with a Palette Edge CLI version that is later than `4.5.7`. Visit the
+  [Downloads](../../../../spectro-downloads.md#palette-edge-cli) page to download the appropriate version of the Palette
+  Edge CLI to build the content bundle.
+
+  :::
+
+- You must ensure your hosts have stable IP addresses. You have the following options to achieve a stable IP address:
 
   - Use a static IP address. Contact your network administrator to assign the Edge host a static IP address.
   - Use Dynamic Host Configuration Protocol (DHCP) reservations to reserve an IP address in a DHCP network. Contact your
     network administrator to reserve IP addresses for your Edge hosts in a DHCP network.
   - Enable network overlay on your Edge cluster. Network overlay can only be enabled during cluster creation. For more
     information about network overlay, refer to [Enable Overlay Network](../../networking/vxlan-overlay.md).
 
+- To create multi-node clusters, all nodes to be included in the cluster must be linked. For more information, refer to
+  [Link Hosts](./link-hosts.md).
+
 ## Create Local Cluster
 
 1. Log into Local UI by visiting the 5080 port of your Edge device's IP address or domain name. For more information,
@@ -84,7 +107,12 @@ through how to create a cluster using Local UI.
    more information, refer to [Enable Overlay Network](../../networking/vxlan-overlay.md). If you enable the overlay
    network, you need to specify a CIDR range to be used by the overlay network.
 
-8. In the **Node Config** step, you can specify configurations for the control plane pool of your single-node cluster.
+8. In the **Node Config** step, you can specify configurations for worker pools and control plane pools. To assign a
+   host to a node pool, click **Add Edge Hosts** in the corresponding node pool and select the host to add to the pool.
+   For multi-node clusters, the leader node is a mandatory control plane node and cannot be unassigned. Additionally,
+   you must ensure that you have an odd number of nodes in the control plane. Once a cluster is formed, every node in
+   the control plane will be considered a leader node.
+
    For more information about node pool configurations, refer to [Node pools](../../../cluster-management/node-pool.md).
    After you finish configuration, click **Next**.
 
@@ -93,6 +121,6 @@ through how to create a cluster using Local UI.
 
 ## Validate
 
-1. Log in to Local UI.
+1. Log in to [Local UI](../host-management/access-console.md).
 
 2. Click **Cluster**. Verify that your cluster has entered the running status.

docs/docs-content/clusters/edge/local-ui/cluster-management/delete-cluster.md

@@ -0,0 +1,44 @@
+---
+sidebar_label: "Delete a Cluster"
+title: "Delete a Cluster"
+description: "Learn how to delete a cluster with Local UI."
+hide_table_of_contents: false
+sidebar_position: 90
+tags: ["edge"]
+---
+
+You can delete an active cluster using Local UI. Deleting a cluster will return all nodes in the cluster to **Ready**
+status. Deleting a cluster does not unlink the linked hosts. If you want to use the hosts that were freed from the
+cluster, you must unlink them first. For more information about linking or unlinking hosts, refer to
+[Link Hosts](./link-hosts.md).
+
+When you delete a cluster, the node where you performed the delete action from will be the new leader node of the group.
+For more information about leader nodes, refer to [Link Hosts](link-hosts.md#leader-nodes).
+
+## Prerequisites
+
+- Access to [Local UI](../host-management/access-console.md). Any Operating System user can be used to log in to Local
+  UI.
+
+- An active cluster deployed and managed by Local UI.
+
+## Delete a Cluster
+
+1. Log in to [Local UI](../host-management/access-console.md) on the leader node where your cluster is deployed.
+
+2. From the left **Main Menu**, click **Cluster**.
+
+3. In the upper-right corner of the **Cluster** page, click **Actions**.
+
+4. In the **drop-down Menu** that appears, click **Delete**.
+
+5. In the pop-up window that appears, click **Confirm**. During the deletion of the cluster, Local UI will become
+   unavailable as the nodes reboot after cluster deletion.
+
+## Validate
+
+1. Log in to [Local UI](../host-management/access-console.md) on the leader of the linked hosts.
+
+2. From the left **Main Menu**, click **Cluster**.
+
+3. Confirm that there is no active cluster and that the cluster has been deleted.

docs/docs-content/clusters/edge/local-ui/cluster-management/export-cluster-definition.md

@@ -3,7 +3,7 @@ sidebar_label: "Export Cluster Definition"
 title: "Export Cluster Definition"
 description: "Instructions for exporting cluster definition."
 hide_table_of_contents: false
-sidebar_position: 50
+sidebar_position: 10
 tags: ["edge"]
 ---
 

docs/docs-content/clusters/edge/local-ui/cluster-management/link-hosts.md

@@ -0,0 +1,192 @@
+---
+sidebar_label: "Link Hosts"
+title: "Link Hosts"
+description: "Instructions for linking hosts to prepare for multi-node cluster creation."
+hide_table_of_contents: false
+sidebar_position: 30
+tags: ["edge"]
+---
+
+To create a multi-node cluster with hosts provisioned in the airgap installation mode, the hosts must first be able to
+identify and securely communicate with each other. By default, hosts that are provisioned in the airgap installation
+mode are not aware of each other even if they are on the same network, and they do not have the credentials to
+communicate with each other securely.
+
+![A diagram of the order of operations for linking hosts.](/clusters_edge_localui_cluster-mgmt_link-hosts.webp)
+
+Host linking provides the hosts with the necessary network and security infrastructure to form a cluster together. In a
+group of linked hosts, hosts can broadcast information to all its peers.
+
+## Leader Nodes
+
+To link hosts together, you designate one node as the leader node. The leader node has two obligations:
+
+- Generate tokens with its IP address and One-Time Password (OTP) credentials, which you can use to link other nodes as
+  followers.
+- Sync content bundles uploaded to the node and configuration changes made to the node to the rest of the cluster.
+
+A group of linked hosts that have not formed a cluster only has one leader node, where you can upload content bundles
+and create a cluster.
+
+Once a cluster is formed, every control plane node will be generating the pairing tokens and can act as the leader node.
+When you perform an action on any control plane node, that node will act as the leader and propagate the action to the
+rest of the cluster. For example, if you upload a content bundle to a control plane node, the content bundle will be
+synced from that node to the rest of the cluster, even if the control plane node was not the leader node before a
+cluster is formed. This design removes the need to memorize which node was the leader before forming a cluster.
+
+When you [delete a cluster](../cluster-management/delete-cluster.md), all nodes will return to the **Ready** state, but
+will remain linked. Since the group of hosts no longer have a cluster, they will only have one leader node, which is the
+host where you performed the delete action from.
+
+## Link Hosts
+
+Linked hosts will sync their status and uploaded content, including images for the Palette agent and provider images,
+with each other.
+
+### Limitations
+
+- All hosts must be deployed in the same deployment mode. For more information, refer to
+  [Deployment Modes](../../../../deployment-modes/deployment-modes.md).
+
+- For hosts that are deployed in [agent mode](../../../../deployment-modes/agent-mode/agent-mode.md), all hosts must
+  share the same Operating System (OS).
+
+- You cannot update the IP address of a linked node.
+
+### Prerequisites
+
+- Two or more hosts deployed in the same deployment mode on the same network. For more information, refer to
+  [Appliance Mode Installation](../../site-deployment/stage.md) or
+  [Agent Mode Installation](../../../../deployment-modes/agent-mode/install-agent-host.md).
+
+- The `stylus.enableMultiNode` parameter is set to `true` in your user data configuration for all your hosts. For more
+  information, refer to [Prepare User Data](../../edgeforge-workflow/prepare-user-data.md) and
+  [Installer Reference](../../edge-configuration/installer-reference.md).
+
+- The `stylus.installationMode` parameter is set to `airgap` in your user data configuration for all your hosts.
+
+- No follower host has any current cluster workloads. Refer to [Delete a Cluster](./delete-cluster.md) and
+  [Unlink Hosts](#unlink-hosts) to learn how to delete a cluster and unlink a host to free it up for linking.
+
+### Procedure
+
+1. Decide on a node that you plan to use as the leader of the group. Log in to
+   [Local UI](../host-management/access-console.md) of that node.
+
+2. From the left **Main Menu**, click **Linked Edge Hosts**.
+
+3. Click **Generate token**. This will make the host start generating tokens you will use to link this host with other
+   hosts. The base-64 encoded token contains the IP address of the node, as well as an OTP that will expire in two
+   minutes. Once a token expires, the leader node generates another token automatically.
+
+   If you have already made the
+
+4. Click the **Copy** button to copy the token.
+
+5. Log in to [Local UI](../host-management/access-console.md) on the node that you want to link to the leader node.
+
+6. From the left **Main Menu**, click **Linked Edge Hosts**.
+
+7. Click **Link this device to another**.
+
+8. In the pop-up box that appears, enter the token you copied from the leader node.
+
+9. Click **Confirm**.
+
+10. Repeat this process for every node you want to link to the leader node.
+
+### Validate
+
+1. Log in to [Local UI](../host-management/access-console.md) on the leader node.
+
+2. From the left **Main Menu**, click **Linked Edge Hosts**.
+
+3. Confirm that all nodes you linked together show up in the **Linked Edge Hosts** table.
+
+## Unlink Hosts
+
+You can unlink a host to either link it to another host or to use it for independent workloads. You can only unlink a
+follower host. Once all follower hosts have been unlinked, you can
+[remove its leader status](#remove-leader-node-status) and link it to another node.
+
+You can unlink a follower host on the follower host itself or from the leader node.
+
+### Prerequisites
+
+- Two or more linked hosts.
+
+- [Access to Local UI](../host-management/access-console.md) on the host you want to unlink or on the leader host.
+
+- The host you want to unlink cannot be in-use by a cluster. If you want to remove a node from your cluster and then
+  unlink it, you must first scale down the cluster. Once the host is in **Ready** state instead of **In-Use**, you can
+  proceed to unlink it. For more information, refer to [Scale Down a Cluster](./scale-cluster.md#scale-down-a-cluster).
+
+### Procedure
+
+<Tabs>
+
+<TabItem value="From Leader Node">
+
+1. Log in to Local UI on the leader node.
+
+2. From the left **Main Menu**, click **Linked Edge Hosts**.
+
+3. In the list of linked nodes, identify the node you want to unlink and click the **three-dot button** next to the IP
+   address of the node.
+
+4. Click **Unlink**.
+
+5. Click **Confirm**
+
+</TabItem>
+
+<TabItem value="On Follower Node">
+
+1. Log in to Local UI on the node you want to unlink.
+
+2. From the left **Main Menu**, click **Linked Edge Hosts**.
+
+3. Click **Unlink** in the upper-right corner of the **Linked Edge Hosts** page.
+
+4. Click **Confirm**
+
+</TabItem>
+
+</Tabs>
+
+### Validate
+
+1. Log in to Local UI on the node you unlinked.
+
+2. From the left **Main Menu**, click **Linked Edge Hosts**.
+
+3. Confirm that the node has been unlinked from the leader and is ready to be paired with another node.
+
+## Remove Leader Node Status
+
+Removing the leader node status of a node allows you to link the node to another group of linked nodes as a follower.
+You can only do this when the node is not part of an active cluster.
+
+### Prerequisites
+
+- Access to Local UI on a host that generates pairing tokens.
+
+- The host is not linked with any other host. If your host is still linked with other hosts, you must unlink all its
+  follower nodes.
+
+### Procedure
+
+1. Log in to Local UI.
+
+2. From the left **Main Menu**, click **Linked Edge Hosts**.
+
+3. Click the red **Stop token generation** button to stop the host from generating pairing tokens. This will make it
+   ready to be paired with other nodes again as a follower.
+
+### Validate
+
+1. Log in to Local UI.
+
+2. From the left **Main Menu**, click **Linked Edge Hosts**.
+
+3. Confirm that the host is no longer generating pairing tokens and can be linked to another host as a follower.