Support worker nodes to continue to communicate to local cluster in Outposts even when the Outpost is disconnected from the parent AWS Region #939

chunywan · 2022-06-07T04:12:29Z

Support worker nodes to continue to communicate to local cluster in Outposts even when the Outpost is disconnected from the parent AWS Region.

Issue #, if available:

Description of changes:

How the "local cluster in outpost support" is determined?
- If --enable-local-outpost option is be set with true or false explicitly, it will override the auto-detection of local cluster in the outpost.
- If --enable-local-outpost option is not set and aws eks describe-clsuter is not bypassed [1], it will rely on whether the outpostArn is returned in the response of aws eks describe-clsuter.
How does it support for worker nodes to continue to communicate and connect to the cluster even when the Outpost is disconnected from the AWS Region?
- If "local cluster in outpost support" is turned on, it will
  - append entries to /etc/hosts with the mapping of control plane IP addresses to API server domain name so that the domain name can be resolved to IP addresses locally. This will work for both connected and disconnected states.
  - use aws-iam-authenticator as bootstrap authentication in kubelet bootstrap kubeconfig to download X.509 certificate which will be used in the generated kubelet kubeconfig. With X.509 certificate, worker node’s kubelet will be able to authenticate to API server during disconnected state.

[1] - The DescribeCluster call happens when B64_CLUSTER_CA or APISERVER_ENDPOINT aren't defined. Note that this PR doesn't change this behavior.

Testing:
Add worker nodes to local outpost cluster successfully and the status is Ready:

$ kubectl get nodes
NAME                                       STATUS   ROLES                  AGE     VERSION
ip-10-0-2-124.us-west-2.compute.internal   Ready    control-plane,master   4h      v1.21.6
ip-10-0-2-199.us-west-2.compute.internal   Ready    control-plane,master   4h2m    v1.21.6
ip-10-0-2-247.us-west-2.compute.internal   Ready    control-plane,master   3h57m   v1.21.6
ip-10-0-2-136.us-west-2.compute.internal   Ready    <none>                 64s     v1.21.12-eks-5308cf7
ip-10-0-3-210.us-west-2.compute.internal   Ready    <none>                 82s     v1.21.12-eks-5308cf7

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…utposts even when the Outpost is disconnected from the parent AWS Region

suket22 · 2022-06-09T22:00:43Z

files/bootstrap.sh

@@ -360,7 +367,7 @@ if [[ -z "${B64_CLUSTER_CA}" ]] || [[ -z "${APISERVER_ENDPOINT}" ]]; then
            --region=${AWS_DEFAULT_REGION} \
            --name=${CLUSTER_NAME} \
            --output=text \
-            --query 'cluster.{certificateAuthorityData: certificateAuthority.data, endpoint: endpoint, serviceIpv4Cidr: kubernetesNetworkConfig.serviceIpv4Cidr, serviceIpv6Cidr: kubernetesNetworkConfig.serviceIpv6Cidr, clusterIpFamily: kubernetesNetworkConfig.ipFamily}' > $DESCRIBE_CLUSTER_RESULT || rc=$?
+            --query 'cluster.{certificateAuthorityData: certificateAuthority.data, endpoint: endpoint, serviceIpv4Cidr: kubernetesNetworkConfig.serviceIpv4Cidr, serviceIpv6Cidr: kubernetesNetworkConfig.serviceIpv6Cidr, clusterIpFamily: kubernetesNetworkConfig.ipFamily, outpostArn: outpostConfig.outpostArns[0], id: id}' > $DESCRIBE_CLUSTER_RESULT || rc=$?


Is outpostArn being returned by DescribeCluster today? One issue we've seen in the past is that the version of the AWS CLI that's running on our AMIs isn't regularly updated so we might not pick up a new API parameter without a model change.

Same comment for id. I don't see it here.

What this means is that the value might continue to remain None until we do a subsequent AMI release with an updated CLI. cc: @cartermckinnon

Yep, I discussed this with @chunywan. This will require an updated version of the CLI, which isn't yet available outside of an internal patched version, AFAIK. These changes are for a private beta, so I considered the can kicked.

Yep, this change was tested with current public AWS CLI for backward compatibility. For forward compatibility, it was also tested with AWS CLI preview version (which includes id and outpostArn fields in response)

suket22 · 2022-06-09T22:03:48Z

files/bootstrap.sh

@@ -137,6 +143,7 @@ PAUSE_CONTAINER_VERSION="${PAUSE_CONTAINER_VERSION:-3.1-eksbuild.1}"
 CONTAINER_RUNTIME="${CONTAINER_RUNTIME:-dockerd}"
 IP_FAMILY="${IP_FAMILY:-}"
 SERVICE_IPV6_CIDR="${SERVICE_IPV6_CIDR:-}"
+ENABLE_LOCAL_OUTPOST="${ENABLE_LOCAL_OUTPOST:-}"


I see this in the overview as well - If --enable-local-outpost option is not set with true or false explicitly, bootstrap.sh will auto-detect whether the cluster is running in local outpost environment.

Auto-discovery relies on making an EKS DescribeCluster call, and I don't want to do this for two reasons -

It forces a latency increase on every node launch.

It will be backwards incompatible for anyone running in a fully private VPC, since there's no private endpoints for EKS.

Why not default this to false? Local outposts isn't a common working configuration, so I'd expect this to run false for best backwards compatibility.

@chunywan can you clarify here? I think @suket22 brings up a good point.

This DescribeCluster call only happens when B64_CLUSTER_CA or APISERVER_ENDPOINT aren't defined, but AFAIK these will always be defined for non-custom AMI managed nodegroups (@suket22 please correct me). The only time they wouldn't be defined is for a custom AMI nodegroup, in which case the user could just use the --enable-local-outpost flag, because they're already overriding the bootstrap command. Am I missing something? I admittedly don't have much context on Outpost.

Regardless, this isn't changing when the DescribeCluster call happens, so I don't have any compatibility concerns. I'm just not sure the auto-detection mechanism is really useful.

@suket22 Thanks for the comments. I think I need to refine the description of PR to make it clear.

It forces a latency increase on every node launch.

This PR doesn't change whether the DescribeCluster is called or not.
If it is called and --enable-local-outpost is not defined explicitly, the auto-discovery will be used. And the change is just adding more filters to DescribeCluster call.

It will be backwards incompatible for anyone running in a fully private VPC, since there's no private endpoints for EKS.

As I mentioned above, if DescribeCluster is bypassed for private VPC (e.g., when B64_CLUSTER_CA or API_SERVER_ENDPOINT aren't defined) , this PR change doesn't call DescribeCluster either.

So I think I will modify the description to make it clear.

@suket22 @cartermckinnon
I refined the description of the PR.

suket22 · 2022-06-09T22:21:09Z

files/bootstrap.sh

@@ -360,7 +367,7 @@ if [[ -z "${B64_CLUSTER_CA}" ]] || [[ -z "${APISERVER_ENDPOINT}" ]]; then
            --region=${AWS_DEFAULT_REGION} \
            --name=${CLUSTER_NAME} \
            --output=text \


@cartermckinnon not for this PR, but I checked and our AMIs come with jq, so I wonder if we could get rid of output=text and just rely on jq rather than awk since the line split can be so easy to mess up.

Yeah, this will make the code simple.

suket22 · 2022-06-10T17:56:43Z

files/bootstrap.sh

+    ### kubelet bootstrap kubeconfig uses aws-iam-authenticator with cluster id to authenticate to cluster
+    ###   - if "aws eks describe-cluster" is bypassed, for local outpost, the value of CLUSTER_NAME parameter will be cluster id.
+    ###   - otherwise, the cluster id will use the id returned by "aws eks describe-cluster".
+    CLUSTER_ID="${CLUSTER_ID:-$CLUSTER_NAME}" 
+    sed -i s,CLUSTER_NAME,$CLUSTER_ID,g /var/lib/kubelet/kubeconfig


I don't understand this. So if someone runs /etc/eks/bootstrap.sh $clusterName, then we will attempt to auto discover, and $CLUSTER_ID will be set. We'll wire in the clusterId rather than clusterName in kubeConfig.

But if someone runs /etc/eks/bootstrap.sh --api-server-endpoint-url foo --cluster-ca-data bar --enable-local-outpost true $clusterName then we won't autoDiscover and clusterId won't be used in the kubeConfig at all. Is that correct?

Yes. For the second scenario (support local outpost and bypass DescribeClustger), the cluster ID instead of clusterName should be passed to bootstrap.sh.

suket22 · 2022-06-13T17:40:25Z

files/bootstrap.sh

+	if [[ ! -z "${CLUSTER_ID_IN_DESCRIBE_CLUSTER_RESULT}" ]] && [[ "${CLUSTER_ID_IN_DESCRIBE_CLUSTER_RESULT}" != "None" ]]; then
+	    CLUSTER_ID=${CLUSTER_ID_IN_DESCRIBE_CLUSTER_RESULT}
+	fi


I think the formatting on this might be off.

Thanks. It is fixed by replacing tab with spaces.

Support worker nodes to continue to communicate to local cluster in O…

b01fa0a

…utposts even when the Outpost is disconnected from the parent AWS Region

cartermckinnon self-assigned this Jun 7, 2022

cartermckinnon added the enhancement New feature or request label Jun 7, 2022

chunywan marked this pull request as ready for review June 8, 2022 17:14

cartermckinnon approved these changes Jun 8, 2022

View reviewed changes

suket22 reviewed Jun 9, 2022

View reviewed changes

suket22 reviewed Jun 10, 2022

View reviewed changes

chunywan added 2 commits June 13, 2022 02:36

add --cluster-id option for bootstrap.sh

998a465

replace tab with space

8ca711f

suket22 reviewed Jun 13, 2022

View reviewed changes

replace tab with space

3cd44b9

cartermckinnon merged commit c8fa176 into awslabs:master Jun 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support worker nodes to continue to communicate to local cluster in Outposts even when the Outpost is disconnected from the parent AWS Region #939

Support worker nodes to continue to communicate to local cluster in Outposts even when the Outpost is disconnected from the parent AWS Region #939

chunywan commented Jun 7, 2022 •

edited

Loading

suket22 Jun 9, 2022

cartermckinnon Jun 9, 2022

chunywan Jun 9, 2022

suket22 Jun 9, 2022

cartermckinnon Jun 9, 2022 •

edited

Loading

cartermckinnon Jun 9, 2022

chunywan Jun 9, 2022 •

edited

Loading

chunywan Jun 9, 2022

suket22 Jun 9, 2022

cartermckinnon Jun 9, 2022

chunywan Jun 9, 2022

suket22 Jun 10, 2022

chunywan Jun 10, 2022

suket22 Jun 13, 2022

chunywan Jun 13, 2022

Support worker nodes to continue to communicate to local cluster in Outposts even when the Outpost is disconnected from the parent AWS Region #939

Support worker nodes to continue to communicate to local cluster in Outposts even when the Outpost is disconnected from the parent AWS Region #939

Conversation

chunywan commented Jun 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cartermckinnon Jun 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chunywan Jun 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chunywan commented Jun 7, 2022 •

edited

Loading

cartermckinnon Jun 9, 2022 •

edited

Loading

chunywan Jun 9, 2022 •

edited

Loading