From 456094d55dfa4b29f21e5cf9c2d431e1cd47692b Mon Sep 17 00:00:00 2001 From: viniciusdc Date: Tue, 17 Sep 2024 12:54:14 -0300 Subject: [PATCH 1/8] add launch_template usage docs --- .../advanced-provider-configuration.md | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/docs/docs/explanations/advanced-provider-configuration.md b/docs/docs/explanations/advanced-provider-configuration.md index 2126e5c5..38ebc8a2 100644 --- a/docs/docs/explanations/advanced-provider-configuration.md +++ b/docs/docs/explanations/advanced-provider-configuration.md @@ -127,6 +127,47 @@ azure: storage_account_postfix: t65ft6q5 ``` +### Launch Templates (Optional) + +Nebari supports configuring launch templates for your node groups, enabling you to customize settings like the AMI ID and pre-bootstrap commands. This is particularly useful if you need to use a custom AMI or perform specific actions before the node joins the cluster. + +#### Configuring a Launch Template + +To configure a launch template for a node group in your `nebari-config.yaml`, add the `launch_template` section under the desired node group: + +```yaml +amazon_web_services: + region: us-west-2 + kubernetes_version: "1.18" + node_groups: + custom-node-group: + instance: "m5.large" + min_nodes: 1 + max_nodes: 5 + gpu: false # Set to true if using GPU instances + launch_template: + ami_id: ami-0abcdef1234567890 # Replace with your custom AMI ID + pre_bootstrap_command: "echo 'Hello, World!'" # Command to run before the node joins the cluster +``` + +**Parameters:** + +- `ami_id` (Optional): The ID of the custom AMI to use for the nodes in this group. If specified, the `ami_type` is automatically set to `CUSTOM`. +- `pre_bootstrap_command` (Optional): A command or script to execute on the node before it joins the Kubernetes cluster. This can be used for custom setup or configuration tasks. + +:::note +If an `ami_id` is not provided, AWS will use the default Amazon Linux 2 AMI for the +specified instance type. You can find the latest optimized AMI IDs for Amazon EKS in you +cluster region by inspecting its respective SSM parameter. For more information, see +[Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html). +::: + +:::warning **Important:** If you add a `launch_template` to an existing node group that was previously created without one, AWS will treat this as a change requiring the replacement of the entire node group. This action will trigger a reallocation of resources, effectively destroying the current node group and recreating it. This behavior is due to how AWS handles self-managed node groups versus those using launch templates with custom settings. +::: + +:::hint **Recommendation:** To avoid unexpected downtime or data loss, consider creating a new node group with the launch template settings and migrating your workloads accordingly. This approach allows you to implement the new configuration without disrupting your existing resources. +::: + From 65e2e942197703a799ecbd54027456d8da1a8215 Mon Sep 17 00:00:00 2001 From: viniciusdc Date: Tue, 17 Sep 2024 13:01:57 -0300 Subject: [PATCH 2/8] wrong provider block --- .../advanced-provider-configuration.md | 62 +++++++++---------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/docs/docs/explanations/advanced-provider-configuration.md b/docs/docs/explanations/advanced-provider-configuration.md index 38ebc8a2..3a8c6b69 100644 --- a/docs/docs/explanations/advanced-provider-configuration.md +++ b/docs/docs/explanations/advanced-provider-configuration.md @@ -98,35 +98,6 @@ amazon_web_services: permissions_boundary: arn:aws:iam::01234567890:policy/ ``` - - - - -[Microsoft Azure](https://azure.microsoft.com/) has similar settings for Kubernetes version, region, and instance names - using Azure's available values of course. - -Azure also requires a field named `storage_account_postfix` which will have been generated by `nebari init`. This allows nebari to create a Storage Account bucket that should be globally unique. - -```yaml -### Provider configuration ### -azure: - region: Central US - kubernetes_version: 1.19.11 - node_groups: - general: - instance: Standard_D4_v3 - min_nodes: 1 - max_nodes: 1 - user: - instance: Standard_D2_v2 - min_nodes: 0 - max_nodes: 5 - worker: - instance: Standard_D2_v2 - min_nodes: 0 - max_nodes: 5 - storage_account_postfix: t65ft6q5 -``` - ### Launch Templates (Optional) Nebari supports configuring launch templates for your node groups, enabling you to customize settings like the AMI ID and pre-bootstrap commands. This is particularly useful if you need to use a custom AMI or perform specific actions before the node joins the cluster. @@ -162,14 +133,43 @@ cluster region by inspecting its respective SSM parameter. For more information, [Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html). ::: -:::warning **Important:** If you add a `launch_template` to an existing node group that was previously created without one, AWS will treat this as a change requiring the replacement of the entire node group. This action will trigger a reallocation of resources, effectively destroying the current node group and recreating it. This behavior is due to how AWS handles self-managed node groups versus those using launch templates with custom settings. +:::warning If you add a `launch_template` to an existing node group that was previously created without one, AWS will treat this as a change requiring the replacement of the entire node group. This action will trigger a reallocation of resources, effectively destroying the current node group and recreating it. This behavior is due to how AWS handles self-managed node groups versus those using launch templates with custom settings. ::: -:::hint **Recommendation:** To avoid unexpected downtime or data loss, consider creating a new node group with the launch template settings and migrating your workloads accordingly. This approach allows you to implement the new configuration without disrupting your existing resources. +:::tip To avoid unexpected downtime or data loss, consider creating a new node group with the launch template settings and migrating your workloads accordingly. This approach allows you to implement the new configuration without disrupting your existing resources. ::: + + +[Microsoft Azure](https://azure.microsoft.com/) has similar settings for Kubernetes version, region, and instance names - using Azure's available values of course. + +Azure also requires a field named `storage_account_postfix` which will have been generated by `nebari init`. This allows nebari to create a Storage Account bucket that should be globally unique. + +```yaml +### Provider configuration ### +azure: + region: Central US + kubernetes_version: 1.19.11 + node_groups: + general: + instance: Standard_D4_v3 + min_nodes: 1 + max_nodes: 1 + user: + instance: Standard_D2_v2 + min_nodes: 0 + max_nodes: 5 + worker: + instance: Standard_D2_v2 + min_nodes: 0 + max_nodes: 5 + storage_account_postfix: t65ft6q5 +``` + + + DigitalOcean has a restriction with autoscaling in that the minimum nodes allowed (`min_nodes` = 1) is one but is by far the least expensive provider even accounting for `spot/pre-emptible` instances. From 39feed10029bb372d670ce2cf1999023358c8fe4 Mon Sep 17 00:00:00 2001 From: viniciusdc Date: Tue, 17 Sep 2024 13:05:02 -0300 Subject: [PATCH 3/8] Add notes about user_data --- .../explanations/advanced-provider-configuration.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/docs/docs/explanations/advanced-provider-configuration.md b/docs/docs/explanations/advanced-provider-configuration.md index 3a8c6b69..f8595b8f 100644 --- a/docs/docs/explanations/advanced-provider-configuration.md +++ b/docs/docs/explanations/advanced-provider-configuration.md @@ -117,14 +117,20 @@ amazon_web_services: max_nodes: 5 gpu: false # Set to true if using GPU instances launch_template: - ami_id: ami-0abcdef1234567890 # Replace with your custom AMI ID - pre_bootstrap_command: "echo 'Hello, World!'" # Command to run before the node joins the cluster + # Replace with your custom AMI ID + ami_id: ami-0abcdef1234567890 + # Command to run before the node joins the cluster + pre_bootstrap_command: "echo 'Hello, World!'" ``` **Parameters:** - `ami_id` (Optional): The ID of the custom AMI to use for the nodes in this group. If specified, the `ami_type` is automatically set to `CUSTOM`. -- `pre_bootstrap_command` (Optional): A command or script to execute on the node before it joins the Kubernetes cluster. This can be used for custom setup or configuration tasks. +- `pre_bootstrap_command` (Optional): A command or script to execute on the node before + it joins the Kubernetes cluster. This can be used for custom setup or configuration + tasks. The format should be a single string in conformation with the shell syntax. + This command is injected in the `user_data` field of the launch template. For more + information, see [User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html). :::note If an `ami_id` is not provided, AWS will use the default Amazon Linux 2 AMI for the From 24ae90f6dd56a63b837d13c7b5b9fa0ab89551b9 Mon Sep 17 00:00:00 2001 From: viniciusdc Date: Tue, 17 Sep 2024 13:08:41 -0300 Subject: [PATCH 4/8] replace warnings and tips --- .../advanced-provider-configuration.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/docs/explanations/advanced-provider-configuration.md b/docs/docs/explanations/advanced-provider-configuration.md index f8595b8f..150e6c06 100644 --- a/docs/docs/explanations/advanced-provider-configuration.md +++ b/docs/docs/explanations/advanced-provider-configuration.md @@ -102,6 +102,14 @@ amazon_web_services: Nebari supports configuring launch templates for your node groups, enabling you to customize settings like the AMI ID and pre-bootstrap commands. This is particularly useful if you need to use a custom AMI or perform specific actions before the node joins the cluster. +:::warning +If you add a `launch_template` to an existing node group that was previously created without one, AWS will treat this as a change requiring the replacement of the entire node group. This action will trigger a reallocation of resources, effectively destroying the current node group and recreating it. This behavior is due to how AWS handles self-managed node groups versus those using launch templates with custom settings. +::: + +:::tip +To avoid unexpected downtime or data loss, consider creating a new node group with the launch template settings and migrating your workloads accordingly. This approach allows you to implement the new configuration without disrupting your existing resources. +::: + #### Configuring a Launch Template To configure a launch template for a node group in your `nebari-config.yaml`, add the `launch_template` section under the desired node group: @@ -139,12 +147,6 @@ cluster region by inspecting its respective SSM parameter. For more information, [Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html). ::: -:::warning If you add a `launch_template` to an existing node group that was previously created without one, AWS will treat this as a change requiring the replacement of the entire node group. This action will trigger a reallocation of resources, effectively destroying the current node group and recreating it. This behavior is due to how AWS handles self-managed node groups versus those using launch templates with custom settings. -::: - -:::tip To avoid unexpected downtime or data loss, consider creating a new node group with the launch template settings and migrating your workloads accordingly. This approach allows you to implement the new configuration without disrupting your existing resources. -::: - From 7cf7ab3be298625647faebc843f26809fee0a031 Mon Sep 17 00:00:00 2001 From: "Vinicius D. Cerutti" <51954708+viniciusdc@users.noreply.github.com> Date: Wed, 18 Sep 2024 17:18:13 -0300 Subject: [PATCH 5/8] Apply suggestions from code review --- .../advanced-provider-configuration.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/docs/explanations/advanced-provider-configuration.md b/docs/docs/explanations/advanced-provider-configuration.md index 150e6c06..de129eb1 100644 --- a/docs/docs/explanations/advanced-provider-configuration.md +++ b/docs/docs/explanations/advanced-provider-configuration.md @@ -128,23 +128,30 @@ amazon_web_services: # Replace with your custom AMI ID ami_id: ami-0abcdef1234567890 # Command to run before the node joins the cluster - pre_bootstrap_command: "echo 'Hello, World!'" -``` + pre_bootstrap_command: | + #!/bin/bash + # This script is executed before the node is bootstrapped + # You can use this script to install additional packages or configure the node + # For example, to install the `htop` package, you can run: + # sudo apt-get update + # sudo apt-get install -y htop" **Parameters:** -- `ami_id` (Optional): The ID of the custom AMI to use for the nodes in this group. If specified, the `ami_type` is automatically set to `CUSTOM`. +- `ami_id` (Optional): The ID of the custom AMI to use for the nodes in this group; this assumes the AMI provided is an EKS-optimized AMI derivative. If specified, the `ami_type` is automatically set to `CUSTOM`. - `pre_bootstrap_command` (Optional): A command or script to execute on the node before it joins the Kubernetes cluster. This can be used for custom setup or configuration tasks. The format should be a single string in conformation with the shell syntax. This command is injected in the `user_data` field of the launch template. For more information, see [User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html). +> If you're using a `launch_template` with a custom `ami_id`, there's an issue with updating the `scaling`.desired_size` via Nebari configuration (terraform). To scale up, you must recreate the node group or adjust the scaling settings directly in the AWS Console UI (recommended). We are aware of this inconsistency and plan to address it in a future update. + :::note If an `ami_id` is not provided, AWS will use the default Amazon Linux 2 AMI for the -specified instance type. You can find the latest optimized AMI IDs for Amazon EKS in you -cluster region by inspecting its respective SSM parameter. For more information, see -[Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html). +specified instance type. You can find the latest optimized AMI IDs for Amazon EKS in your +cluster region by inspecting its respective SSM parameters. For more information, see +[Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html). ::: From cb05b2def8f52d3977fe9e4f88590e95309bfca6 Mon Sep 17 00:00:00 2001 From: vinicius douglas cerutti Date: Wed, 18 Sep 2024 17:25:12 -0300 Subject: [PATCH 6/8] apply commit fixes --- .../advanced-provider-configuration.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/docs/explanations/advanced-provider-configuration.md b/docs/docs/explanations/advanced-provider-configuration.md index de129eb1..612bde56 100644 --- a/docs/docs/explanations/advanced-provider-configuration.md +++ b/docs/docs/explanations/advanced-provider-configuration.md @@ -114,7 +114,7 @@ To avoid unexpected downtime or data loss, consider creating a new node group wi To configure a launch template for a node group in your `nebari-config.yaml`, add the `launch_template` section under the desired node group: -```yaml +````yaml amazon_web_services: region: us-west-2 kubernetes_version: "1.18" @@ -129,12 +129,12 @@ amazon_web_services: ami_id: ami-0abcdef1234567890 # Command to run before the node joins the cluster pre_bootstrap_command: | - #!/bin/bash - # This script is executed before the node is bootstrapped - # You can use this script to install additional packages or configure the node - # For example, to install the `htop` package, you can run: - # sudo apt-get update - # sudo apt-get install -y htop" + #!/bin/bash + # This script is executed before the node is bootstrapped + # You can use this script to install additional packages or configure the node + # For example, to install the `htop` package, you can run: + # sudo apt-get update + # sudo apt-get install -y htop" **Parameters:** @@ -151,7 +151,7 @@ amazon_web_services: If an `ami_id` is not provided, AWS will use the default Amazon Linux 2 AMI for the specified instance type. You can find the latest optimized AMI IDs for Amazon EKS in your cluster region by inspecting its respective SSM parameters. For more information, see -[Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html). +[Retrieve recommended Amazon Linux AMI IDs](https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html). ::: @@ -181,7 +181,7 @@ azure: min_nodes: 0 max_nodes: 5 storage_account_postfix: t65ft6q5 -``` +```` From 1c17015d2209be7817dd6c28496dbfe1ffbab750 Mon Sep 17 00:00:00 2001 From: vinicius douglas cerutti Date: Wed, 18 Sep 2024 17:34:07 -0300 Subject: [PATCH 7/8] fix code block --- docs/docs/explanations/advanced-provider-configuration.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/docs/explanations/advanced-provider-configuration.md b/docs/docs/explanations/advanced-provider-configuration.md index 612bde56..21a5ba4d 100644 --- a/docs/docs/explanations/advanced-provider-configuration.md +++ b/docs/docs/explanations/advanced-provider-configuration.md @@ -114,7 +114,7 @@ To avoid unexpected downtime or data loss, consider creating a new node group wi To configure a launch template for a node group in your `nebari-config.yaml`, add the `launch_template` section under the desired node group: -````yaml +```yaml amazon_web_services: region: us-west-2 kubernetes_version: "1.18" @@ -135,6 +135,7 @@ amazon_web_services: # For example, to install the `htop` package, you can run: # sudo apt-get update # sudo apt-get install -y htop" +``` **Parameters:** @@ -181,7 +182,7 @@ azure: min_nodes: 0 max_nodes: 5 storage_account_postfix: t65ft6q5 -```` +``` From 30dde5a20e5f358c170d49667eee4fa455639e33 Mon Sep 17 00:00:00 2001 From: vinicius douglas cerutti Date: Wed, 18 Sep 2024 17:38:29 -0300 Subject: [PATCH 8/8] fix typo --- .../advanced-provider-configuration.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/docs/explanations/advanced-provider-configuration.md b/docs/docs/explanations/advanced-provider-configuration.md index 21a5ba4d..fd845d1b 100644 --- a/docs/docs/explanations/advanced-provider-configuration.md +++ b/docs/docs/explanations/advanced-provider-configuration.md @@ -129,12 +129,12 @@ amazon_web_services: ami_id: ami-0abcdef1234567890 # Command to run before the node joins the cluster pre_bootstrap_command: | - #!/bin/bash - # This script is executed before the node is bootstrapped - # You can use this script to install additional packages or configure the node - # For example, to install the `htop` package, you can run: - # sudo apt-get update - # sudo apt-get install -y htop" + #!/bin/bash + # This script is executed before the node is bootstrapped + # You can use this script to install additional packages or configure the node + # For example, to install the `htop` package, you can run: + # sudo apt-get update + # sudo apt-get install -y htop" ``` **Parameters:** @@ -146,7 +146,7 @@ amazon_web_services: This command is injected in the `user_data` field of the launch template. For more information, see [User Data](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html). -> If you're using a `launch_template` with a custom `ami_id`, there's an issue with updating the `scaling`.desired_size` via Nebari configuration (terraform). To scale up, you must recreate the node group or adjust the scaling settings directly in the AWS Console UI (recommended). We are aware of this inconsistency and plan to address it in a future update. +> If you're using a `launch_template` with a custom `ami_id`, there's an issue with updating the `scaling.desired_size` via Nebari configuration (terraform). To scale up, you must recreate the node group or adjust the scaling settings directly in the AWS Console UI (recommended). We are aware of this inconsistency and plan to address it in a future update. :::note If an `ami_id` is not provided, AWS will use the default Amazon Linux 2 AMI for the