Merge pull request #859 from kinvolk/knrt10/update-cluster-autoscaler

Update cluster-autoscaler to v2 version 1.0.2

assets/charts/components/cluster-autoscaler/Chart.yaml

@@ -1,16 +1,20 @@
-apiVersion: v1
-appVersion: 1.14.6
-description: Scales worker nodes within autoscaling groups.
+apiVersion: v2
+appVersion: 1.18.1
+description: Scales Kubernetes worker nodes within autoscaling groups.
 engine: gotpl
 home: https://github.com/kubernetes/autoscaler
 icon: https://github.com/kubernetes/kubernetes/blob/master/logo/logo.png
 maintainers:
-- email: mgoodness@gmail.com
-  name: mgoodness
-- email: e.bailey@sportradar.com
-  name: yurrriq
-name: cluster-autoscaler
+  - email: e.bailey@sportradar.com
+    name: yurrriq
+  - email: mgoodness@gmail.com
+    name: mgoodness
+  - email: guyjtempleton@googlemail.com
+    name: gjtempleton
+  - email: scott.crooks@gmail.com
+    name: sc250024
+name: cluster-autoscaler-chart
 sources:
-- https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
-- https://github.com/spotinst/kubernetes-autoscaler/tree/master/cluster-autoscaler
-version: 6.4.0
+  - https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
+type: application
+version: 1.0.2

assets/charts/components/cluster-autoscaler/README.md.gotmpl

@@ -0,0 +1,288 @@
+{{ template "chart.header" . }}
+
+{{ template "chart.description" . }}
+
+## TL;DR:
+
+```console
+$ helm repo add autoscaler https://kubernetes.github.io/autoscaler
+
+# Method 1 - Using Autodiscovery
+$ helm install my-release autoscaler/cluster-autoscaler-chart \
+--set 'autoDiscovery.clusterName'=<CLUSTER NAME>
+
+# Method 2 - Specifying groups manually
+$ helm install my-release autoscaler/cluster-autoscaler-chart \
+--set "autoscalingGroups[0].name=your-asg-name" \
+--set "autoscalingGroups[0].maxSize=10" \
+--set "autoscalingGroups[0].minSize=1"
+```
+
+## Introduction
+
+This chart bootstraps a cluster-autoscaler deployment on a [Kubernetes](http://kubernetes.io) cluster using the [Helm](https://helm.sh) package manager.
+
+## Prerequisites
+
+- Helm 3+
+- Kubernetes 1.8+
+  - [Older versions](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#releases) may work by overriding the `image`. Cluster autoscaler internally simulates the scheduler and bugs between mismatched versions may be subtle.
+- Azure AKS specific Prerequisites:
+  - Kubernetes 1.10+ with RBAC-enabled.
+
+## Previous Helm Chart
+
+The previous `cluster-autoscaler` Helm chart hosted at [helm/charts](https://github.com/helm/charts) has been moved to this repository in accordance with the [Deprecation timeline](https://github.com/helm/charts#deprecation-timeline). Note that a few things have changed between this version and the old version:
+
+- This repository **only** supports Helm chart installations using Helm 3+ since the `apiVersion` on the charts has been marked as `v2`.
+- Previous versions of the Helm chart have not been migrated, and the version was reset to `1.0.0` at the onset. If you are looking for old versions of the chart, it's best to run `helm pull stable/cluster-autoscaler --version <your-version>` until you are ready to move to this repository's version.
+
+## Installing the Chart
+
+**By default, no deployment is created and nothing will autoscale**.
+
+You must provide some minimal configuration, either to specify instance groups or enable auto-discovery. It is not recommended to do both.
+
+Either:
+
+- Set `autoDiscovery.clusterName` and tag your autoscaling groups appropriately (`--cloud-provider=aws` only) **or**
+- Set at least one ASG as an element in the `autoscalingGroups` array with its three values: `name`, `minSize` and `maxSize`.
+
+To install the chart with the release name `my-release`:
+
+### AWS - Using auto-discovery of tagged instance groups
+
+Auto-discovery finds ASGs tags as below and automatically manages them based on the min and max size specified in the ASG. `cloudProvider=aws` only.
+
+- Tag the ASGs with keys to match `.Values.autoDiscovery.tags`, by default: `k8s.io/cluster-autoscaler/enabled` and `k8s.io/cluster-autoscaler/<YOUR CLUSTER NAME>`
+- Verify the [IAM Permissions](#aws---iam)
+- Set `autoDiscovery.clusterName=<YOUR CLUSTER NAME>`
+- Set `awsRegion=<YOUR AWS REGION>`
+- Set `awsAccessKeyID=<YOUR AWS KEY ID>` and `awsSecretAccessKey=<YOUR AWS SECRET KEY>` if you want to [use AWS credentials directly instead of an instance role](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#using-aws-credentials)
+
+```console
+$ helm install my-release autoscaler/cluster-autoscaler-chart --set autoDiscovery.clusterName=<CLUSTER NAME>
+```
+
+#### Specifying groups manually
+
+Without autodiscovery, specify an array of elements each containing ASG name, min size, max size. The sizes specified here will be applied to the ASG, assuming IAM permissions are correctly configured.
+
+- Verify the [IAM Permissions](#aws---iam)
+- Either provide a yaml file setting `autoscalingGroups` (see values.yaml) or use `--set` e.g.:
+
+```console
+$ helm install my-release autoscaler/cluster-autoscaler-chart \
+--set "autoscalingGroups[0].name=your-asg-name" \
+--set "autoscalingGroups[0].maxSize=10" \
+--set "autoscalingGroups[0].minSize=1"
+```
+
+#### Auto-discovery
+
+For auto-discovery of instances to work, they must be tagged with the keys in `.Values.autoDiscovery.tags`, which by default are
+`k8s.io/cluster-autoscaler/enabled` and `k8s.io/cluster-autoscaler/<ClusterName>`
+
+The value of the tag does not matter, only the key.
+
+An example kops spec excerpt:
+
+```yaml
+apiVersion: kops/v1alpha2
+kind: Cluster
+metadata:
+  name: my.cluster.internal
+spec:
+  additionalPolicies:
+    node: |
+      [
+        {"Effect":"Allow","Action":["autoscaling:DescribeAutoScalingGroups","autoscaling:DescribeAutoScalingInstances","autoscaling:DescribeLaunchConfigurations","autoscaling:DescribeTags","autoscaling:SetDesiredCapacity","autoscaling:TerminateInstanceInAutoScalingGroup"],"Resource":"*"}
+      ]
+      ...
+---
+apiVersion: kops/v1alpha2
+kind: InstanceGroup
+metadata:
+  labels:
+    kops.k8s.io/cluster: my.cluster.internal
+  name: my-instances
+spec:
+  cloudLabels:
+    k8s.io/cluster-autoscaler/enabled: ""
+    k8s.io/cluster-autoscaler/my.cluster.internal: ""
+  image: kops.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2018-01-14
+  machineType: r4.large
+  maxSize: 4
+  minSize: 0
+```
+
+In this example you would need to `--set autoDiscovery.clusterName=my.cluster.internal` when installing.
+
+It is not recommended to try to mix this with setting `autoscalingGroups`
+
+See [autoscaler AWS documentation](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup) for a more discussion of the setup.
+
+### GCE
+
+The following parameters are required:
+
+- `autoDiscovery.clusterName=any-name`
+- `cloud-provider=gce`
+- `autoscalingGroupsnamePrefix[0].name=your-ig-prefix,autoscalingGroupsnamePrefix[0].maxSize=10,autoscalingGroupsnamePrefix[0].minSize=1`
+
+To use Managed Instance Group (MIG) auto-discovery, provide a YAML file setting `autoscalingGroupsnamePrefix` (see values.yaml) or use `--set` when installing the Chart - e.g.
+
+```console
+$ helm install my-release autoscaler/cluster-autoscaler-chart \
+--set "autoscalingGroupsnamePrefix[0].name=your-ig-prefix,autoscalingGroupsnamePrefix[0].maxSize=10,autoscalingGroupsnamePrefi[0].minSize=1" \
+--set autoDiscovery.clusterName=<CLUSTER NAME> \
+--set cloudProvider=gce
+```
+
+Note that `your-ig-prefix` should be a _prefix_ matching one or more MIGs, and _not_ the full name of the MIG. For example, to match multiple instance groups - `k8s-node-group-a-standard`, `k8s-node-group-b-gpu`, you would use a prefix of `k8s-node-group-`.
+
+In the event you want to explicitly specify MIGs instead of using auto-discovery, set members of the `autoscalingGroups` array directly - e.g.
+
+```
+# where 'n' is the index, starting at 0
+-- set autoscalingGroups[n].name=https://content.googleapis.com/compute/v1/projects/$PROJECTID/zones/$ZONENAME/instanceGroupManagers/$FULL-MIG-NAME,autoscalingGroups[n].maxSize=$MAXSIZE,autoscalingGroups[n].minSize=$MINSIZE
+```
+
+### Azure AKS
+
+The following parameters are required:
+
+- `cloudProvider=azure`
+- `autoscalingGroups[0].name=your-agent-pool,autoscalingGroups[0].maxSize=10,autoscalingGroups[0].minSize=1`
+- `azureClientID: "your-service-principal-app-id"`
+- `azureClientSecret: "your-service-principal-client-secret"`
+- `azureSubscriptionID: "your-azure-subscription-id"`
+- `azureTenantID: "your-azure-tenant-id"`
+- `azureClusterName: "your-aks-cluster-name"`
+- `azureResourceGroup: "your-aks-cluster-resource-group-name"`
+- `azureVMType: "AKS"`
+- `azureNodeResourceGroup: "your-aks-cluster-node-resource-group"`
+
+## Uninstalling the Chart
+
+To uninstall `my-release`:
+
+```console
+$ helm uninstall my-release
+```
+
+The command removes all the Kubernetes components associated with the chart and deletes the release.
+
+> **Tip**: List all releases using `helm list` or start clean with `helm uninstall my-release`
+
+## Additional Configuration
+
+### AWS - IAM
+
+The worker running the cluster autoscaler will need access to certain resources and actions:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "autoscaling:DescribeAutoScalingGroups",
+        "autoscaling:DescribeAutoScalingInstances",
+        "autoscaling:DescribeLaunchConfigurations",
+        "autoscaling:DescribeTags",
+        "autoscaling:SetDesiredCapacity",
+        "autoscaling:TerminateInstanceInAutoScalingGroup"
+      ],
+      "Resource": "*"
+    }
+  ]
+}
+```
+
+- `DescribeTags` is required for autodiscovery.
+- `DescribeLaunchConfigurations` is required to scale up an ASG from 0.
+
+If you would like to limit the scope of the Cluster Autoscaler to ***only*** modify ASGs for a particular cluster, use the following policy instead:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "autoscaling:DescribeAutoScalingGroups",
+        "autoscaling:DescribeAutoScalingInstances",
+        "autoscaling:DescribeLaunchConfigurations",
+        "autoscaling:DescribeTags",
+        "ec2:DescribeLaunchTemplateVersions"
+      ],
+      "Resource": "*"
+    },
+    {
+      "Effect": "Allow",
+      "Action": [
+        "autoscaling:SetDesiredCapacity",
+        "autoscaling:TerminateInstanceInAutoScalingGroup",
+        "autoscaling:UpdateAutoScalingGroup"
+      ],
+      "Resource": [
+        "arn:aws:autoscaling:<aws-region>:<account-id>:autoScalingGroup:<some-random-id>:autoScalingGroupName/node-group-1",
+        "arn:aws:autoscaling:<aws-region>:<account-id>:autoScalingGroup:<some-random-id>:autoScalingGroupName/node-group-2",
+        "arn:aws:autoscaling:<aws-region>:<account-id>:autoScalingGroup:<some-random-id>:autoScalingGroupName/node-group-3"
+      ],
+      "Condition": {
+        "StringEquals": {
+          "autoscaling:ResourceTag/k8s.io/cluster-autoscaler/enabled": "true",
+          "autoscaling:ResourceTag/kubernetes.io/cluster/<cluster-name>": "owned"
+        }
+      }
+    }
+  ]
+}
+```
+
+Make sure to replace the variables `<aws-region>`, `<cluster-name>`, `<account-id>`, and the ARNs of the ASGs where applicable.
+
+### AWS - IAM Roles for Service Accounts (IRSA)
+
+For Kubernetes clusters that use Amazon EKS, the service account can be configured with an IAM role using [IAM Roles for Service Accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) to avoid needing to grant access to the worker nodes for AWS resources.
+
+In order to accomplish this, you will first need to create a new IAM role with the above mentions policies.  Take care in [configuring the trust relationship](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html#iam-role-configuration) to restrict access just to the service account used by cluster autoscaler.
+
+Once you have the IAM role configured, you would then need to `--set rbac.serviceAccountAnnotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/MyRoleName` when installing.
+
+## Troubleshooting
+
+The chart will succeed even if the container arguments are incorrect. A few minutes after starting
+`kubectl logs -l "app=aws-cluster-autoscaler" --tail=50` should loop through something like
+
+```
+polling_autoscaler.go:111] Poll finished
+static_autoscaler.go:97] Starting main loop
+utils.go:435] No pod using affinity / antiaffinity found in cluster, disabling affinity predicate for this loop
+static_autoscaler.go:230] Filtering out schedulables
+```
+
+If not, find a pod that the deployment created and `describe` it, paying close attention to the arguments under `Command`. e.g.:
+
+```
+Containers:
+  cluster-autoscaler:
+    Command:
+      ./cluster-autoscaler
+      --cloud-provider=aws
+# if specifying ASGs manually
+      --nodes=1:10:your-scaling-group-name
+# if using autodiscovery
+      --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/<ClusterName>
+      --v=4
+```
+
+### PodSecurityPolicy
+
+Though enough for the majority of installations, the default PodSecurityPolicy _could_ be too restrictive depending on the specifics of your release. Please make sure to check that the template fits with any customizations made or disable it by setting `rbac.pspEnabled` to `false`.
+
+{{ template "chart.valuesSection" . }}

assets/charts/components/cluster-autoscaler/templates/NOTES.txt

@@ -2,7 +2,7 @@
 
 To verify that cluster-autoscaler has started, run:
 
-  kubectl --namespace={{ .Release.Namespace }} get pods -l "app={{ template "cluster-autoscaler.name" . }},release={{ .Release.Name }}"
+  kubectl --namespace={{ .Release.Namespace }} get pods -l "app.kubernetes.io/name={{ template "cluster-autoscaler.name" . }},app.kubernetes.io/instance={{ .Release.Name }}"
 
 {{- else -}}
 

assets/charts/components/cluster-autoscaler/templates/_helpers.tpl

@@ -71,3 +71,14 @@ Return the appropriate apiVersion for podsecuritypolicy.
 {{- print "policy/v1beta1" -}}
 {{- end -}}
 {{- end -}}
+
+{{/*
+Return the service account name used by the pod.
+*/}}
+{{- define "cluster-autoscaler.serviceAccountName" -}}
+{{- if .Values.rbac.serviceAccount.create -}}
+    {{ default (include "cluster-autoscaler.fullname" .) .Values.rbac.serviceAccount.name }}
+{{- else -}}
+    {{ default "default" .Values.rbac.serviceAccount.name }}
+{{- end -}}
+{{- end -}}