Refactor configuration file handling

[jaypipes]

This is a top-level issue to keep track of a number of sub-tasks needed to refactor the way we handle configuration files for the VPC CNI repo.

Background

The VPC CNI repo keeps a collection of Kubernetes manifests (configuration files) in its config/ directory. There is a subdirectory within that directory for each release. For example, the config/v1.5 directory contains manifests that install the 1.5.x release series of the CNI plugin and IPAM agent.

Helm, for good or bad, is the package management system for Kubernetes. Helm also represents best practices for managing the lifecycle and upgrade of Kubernetes applications.

The EKS Helm Charts repository contains the Helm charts for a number of AWS Kubernetes applications, including the VPC CNI, AppMesh controller, and node termination handler.

The Helm chart for the VPC CNI plugin contains copies of the manifests that are in the VPC CNI source repository along with a set of reasonable production default configuration values.

Problems

It is difficult to keep the Helm chart and the base configuration files in sync without automation, and in fact, the Helm chart serves a different purpose than the configuration files in the VPC CNI source repo: the Helm chart in the eks-charts repository intentionally lags behind the VPC CNI repo in terms of the application binary releases that are associated with Docker images in the chart. The Helm chart is considered "stable" for some definition of that word...

As the number of configuration options for the CNI plugin/agent grows as well as the variety of deployment patterns (custom config, fargate, split/chained plugins, etc) expands, the need to keep a single source of truth increases.

Finally, some users prefer to kubectl apply a set of static manifest files instead of using Helm. We don't want our choice of Helm to negatively impact these users and so need a way of generating static manifests for these users without requiring duplicate configuration files to exist in the source repository.

Proposed solution

We will have a Helm chart in the VPC CNI repo under a charts/ directory. This Helm chart will be tested in CI and gate merges. This Helm chart will install the equivalent of the :latest tagged Docker image for the VPC CNI plugin and agent. The values.yaml file in this Helm chart will have appropriate default values for a production environment. There will be a separate values.ci.yaml file for use in testing environments.

When we tag the VPC CNI source repo with a release tag, automation will create static Kubernetes manifests by running helm template and attach these manifests as release artifacts. These static manifests will not be stored in Git.

We will create a script that will copy the Helm chart from the VPC CNI source repository to the eks-charts source repository under the stable/ directory and set the chart and application version appropriately.

[jaypipes]

@stefanprodan I'm curious if you have any thoughts to add to this issue.

[max-rocket-internet]

We should use the EKS Charts repo for storing the configuration templates for the CNI plugin and use its standard documentation and layout for describing the configuration knobs available for various releases of the CNI.

That's a hard yes from me 😃

[jaypipes]

We should use the EKS Charts repo for storing the configuration templates for the CNI plugin and use its standard documentation and layout for describing the configuration knobs available for various releases of the CNI.

That's a hard yes from me smiley

Heh, I pretty much knew you'd be supportive of this effort, Max! :)

[jaypipes]

We've slightly changed our strategy here after a meeting with a number of AWS source repo maintainers.

We will be replacing the config/ directory and its contents with a single charts/ directory containing a Helm chart that installs the equivalent of :latest Docker image tag with a set of reasonable production default configuration values. We will write a script that runs helm template to generate static manifests and attach those static manifests as artifacts to Github releases of the source repository's release. Finally, we'll be adding a script that automates the process of copying the Helm chart from this repository's charts/ directory into the github.com/aws/eks-charts repository's stable/amazon-vpc-cni-k8s Chart when releases are cut.

I'll update this issue description accordingly.

[TBBle]

In case it's helpful as a reference, kubernetes-sigs/aws-ebs-csi-driver#475 applied the same 'generate static manifests from Helm' change to the aws-ebs-csi-driver repo. Although they commit the static templates rather than generating them on-release, which would be useful for kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/master/aws-k8s-cni.yaml if that's an interesting use-case for anyone.

[jayanthvn]

Merged #1271 to master. Will pick this up in the next CNI release.