Support for tainting kubernetes nodes

[partcyborg]

Description

Our Kubernetes upgrade path involves the following steps:

1. Upgrade control plane
2. Spin up new worker nodes on new version
3. Taint existing worker nodes
4. Wait for all workloads to migrate
5. Remove nodes on previous version

We want to automate this workflow using Terraform. There is support for each of these steps except for the tainting of existing worker nodes.

To accomplish this, we need two things:

• New data source: kubernetes_nodes. This lets us find all nodes in CSP autoscaling group with a specific label.

• New resource: kubernetes_node_taint. This lets us apply a taint to all nodes returned by kubernetes_nodes data source.

Potential Terraform Configuration

data "kubernetes_nodes" "example" {
  metadata {
    labels = {
      "node-role.kubernetes.io/group" = "group1"
    }
  }
}

resource "kubernetes_node_taint" "example" {
  for_each = toset(data.kubernetes_nodes.example.nodes)

  taint {
    key = "node-role.kubernetes.io/retired"
    value = "true"
    effect = "NoSchedule"
  }
}

References

The kubernetes_nodes data source and kubernetes_node_taint resource have been implemented and are ready for review: #1921

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[partcyborg]

Any chance someone could take a look? This is already implemented, I just need a review for the pull request.

Thanks!