Implementing eksctl apply

[aclevername]

We recently outlined a proposal for the implementation of eksctl apply, this can be found here. This proposal has WIP status so changes may occur, but the general goal of being able to reconcile your cluster config will most likely remain the same. This discussion is about how we should start implementing apply.

Scope of work

eksctl has a quite a large amount of resources it's responsible for managing, as well as a large amount of customization of those resources. Here's an example config that utilises the majority of eksctl resources (note: there are a large amount of customizations not present here):

# An example of ClusterConfig using a large amount of eksctl functionality
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: example-cluster
  region: us-west-2
  version: "1.18"

vpc:
  id: "vpc-0dd338ecf29863c55"
  subnets:
    private:
      eu-north-1a:
        id: "subnet-0b2512f8c6ae9bf30"
      eu-north-1b:
        id: "subnet-08cb9a2ed60394ce3"
      eu-north-1c:
        id: "subnet-00f71956cdec8f1dc"

iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: s3-reader
        namespace: backend-apps
        labels: {aws-usage: "application"}
      attachPolicyARNs:
        - "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
    - metadata:
        name: aws-load-balancer-controller
        namespace: kube-system
      wellKnownPolicies:
        awsLoadBalancerController: true

nodeGroups:
  - name: ng1-spot
    instanceName: custom-node-name
    instancesDistribution:
      maxPrice: 0.017
      instanceTypes: [ "t3.small", "t3.medium" ]
      onDemandBaseCapacity: 0
      onDemandPercentageAboveBaseCapacity: 50
      spotInstancePools: 2
    volumeEncrypted: true

managedNodeGroups:
  - name: managed-ng-public-customized
    instanceType: m5.large
    minSize: 2
    desiredCapacity: 3
    maxSize: 4
    availabilityZones: ["eu-north-1a", "eu-north-1b"]
    volumeSize: 20
    securityGroups:
      attachIDs: ["sg-1", "sg-2"]
    privateNetworking: true
    ssh:
      allow: true
      publicKeyPath: ~/.ssh/ec2_id_rsa.pub
      sourceSecurityGroupIds: ["sg-00241fbb12c607007"]
    labels: {role: worker}
    tags:
      nodegroup-role: worker
    iam:
      withAddonPolicies:
        externalDNS: true
        certManager: true
    preBootstrapCommands:
      # disable hyperthreading
      - "for n in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | cut -s -d, -f2- | tr ',' '\n' | sort -un); do echo 0 > /sys/devices/system/cpu/cpu${n}/online; done"
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh cluster-15

addons:
  - name: vpc-cni
    attachPolicyARNs:
      - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
  - name: another-addon
    serviceAccountRoleARN:  arn:aws:iam::1234567890:role/another-addon-role


fargateProfiles:
  - name: fp-default
    selectors:
      # All workloads in the "default" Kubernetes namespace will be
      # scheduled onto Fargate:
      - namespace: default
      # All workloads in the "kube-system" Kubernetes namespace will be
      # scheduled onto Fargate:
      - namespace: kube-system

secretsEncryption:
  keyARN: "arn:aws:kms:us-west-2:000000000000:key/00000000-0000-0000-0000-000000000000"

identityProviders:
  - name: cognito-user-pool-1
    issuerURL: https://cognito-idp.us-west-2.amazonaws.com/us-west-2_Ur78RxTra
    clientID: 10basodnbu3gs9b1bf9r566btu
    usernameClaim: email
    type: oidc

cloudWatch:
  clusterLogging:
    enableTypes: ["audit", "authenticator", "controllerManager"]

gitops:
  flux:
    gitProvider: github
    owner: dr-who
    repository: my-cluster-gitops
    personal: true
    branch: main
    namespace: "flux-system"
    path: "clusters/cluster-27"

This config shows that there are a large amount of resources that we need to be able to reconcile. There are also some eksctl functionality that cannot be defined in the config file such as:

• --install-vpc-controller when deploying a windows node
• --install-neuron-plugin
• --install-nvidia-plugin

A large amount of the config file also corresponds to immutable resources in AWS, for example the VPC configuration for the cluster cannot be changed without deleting the cluster and recreating, the same goes for a large amount of the nodegroup configuration too. The config file currently doesn’t really have a clear way of distinguishing this and doesn’t document what resources are immutable. We outlined in the proposal that recreation should be the default response to the change of immutable resources, so the implementation of apply needs to clearly show to the user what actions will occur, so that the user is aware of any deletions and recreations that will occur.

Breaking down the work

eksctl apply is a large piece of work that will take a while to implement, so we should try to break it down into achievable features. The proposal outlines adding apply has an experimental flag, so we can release functionality incrementally and document that the behaviour can/will change as we receive new feedback.

Vertical Slicing

We can start to tackle apply by working on adding support for each sub-resource:

• Fargate Profiles
• Addons
• IAMServiceAccounts
• OIDC Providers
• Cloudwatch (currently updated via eksctil utils update-cluster-logging)
• Cluster & nodegroup versions
  • Currently tracked via the top level metadata.Version
• Managed Nodegroups
• Nodegroups

We will most likely find as we tackle each resource that there are a number of desirable config file changes and additions that would improve the experience, we should open a new proposal/discussion for config file changes and document them as we go along. I think the most likely outcome is that a large amount of the apply functionality makes it into v0, with the "completion" of eksctl apply coming alongside a new config file layout in v1