[Breaking Change] TLS Bootstrapping

[surajssd]

Add flag enable_tls_bootstrap = false to disable this update for old clusters.

This PR enables TLS bootstrap for Lokomotive.

NOTE: Look at the PR per commit to look at the changes.

Fixes #177

---

• In kube-apiserver config:

  • Add Node authoriser.
  • Enable NodeRestriction admission plugin.
  • Enable bootstrap token auth.

• In kube-controller-manager:

  • Enable tokencleaner controller:
  • Set cluster signing duration flag.

• Remove the ClusterRoleBinding system-nodes which assigned group system:nodes permission to role system:node.

• This PR removes all the certs generated statically for kubelet. These certs are no longer needed with bootstrap kubelet.

• Add terraform plugin random to bootkube which will generate the random strings for bootstrap token.

• Add bootstrap-secrets chart

  • Add chart that will install bootstrap secrets in kube-system namespace corresponding to a node.

  • Add variable bootstrap_tokens to bootkube so that platforms can pass a list of token-id and token-secret corresponding to each node in the cluster.

  • Add ClusterRoleBindings so as to enable kubelet bootstrapping mechanism.

• In kubelet:

  • Provide the bootrap kubeconfig via flag --bootstrap-kubeconfig.

  • Provide a path to store downloaded kubeconfig via existing --kubeconfig flag. This place has to writable.

  • Added flag to rotate ceritificates.

  • Change the path of kubeconfig to delete node.

• For packet: Generate bootstrap tokens per node

  • Create bootstrap kubeconfig, with random token id and secret, per node.

  • Add bootstrap kubeconfig template.

  • Output CA cert, API server endpoint and bootstrap-secrets chart values in controller module.

  • Pass CA cert and API server endpoint to worker pools, so that they can construct bootstrap kubeconfig. To enable this data flow add variables ca_cert and apiserver.

  • Pass tokens from worker pools to controller pool, so that it combined with controller pool tokens and can be passed to bootstrap-secrets chart in bootkube. To enable this data flow add variable worker_bootstrap_tokens in controller module and output worker_bootstrap_tokens in worker module.

  • Don't pass statically generated kubeconfig to the workers.

• cluster-apply: Apply bootstrap-secrets helm chart without node bootup check

  • Bootstrap secrets should be available so that nodes join the cluster. So this chart should be installed without node readiness checks.

• For aws: Generate bootstrap tokens per node

  • Create bootstrap kubeconfig, with random token id and secret, per node.

  • Add bootstrap kubeconfig template.

  • Output CA cert, API server endpoint and bootstrap-secrets chart values in controller module.

  • Pass CA cert and API server endpoint to worker pools, so that they can construct bootstrap kubeconfig. To enable this data flow add variables ca_cert and apiserver.

  • Pass tokens from worker pools to controller pool, so that it combined with controller pool tokens and can be passed to bootstrap-secrets chart in bootkube. To enable this data flow add variable worker_bootstrap_tokens in controller module and output worker_bootstrap_tokens in worker module.

  • Don't pass statically generated kubeconfig to the workers.

  • Pass kubeconfig to controller nodes via ignition config.

• For baremetal: Generate bootstrap tokens per node

  • Create bootstrap kubeconfig, with random token id and secret, per node.

  • Add bootstrap kubeconfig template.

---

Replace a token

This helped in in regenerating the token and replacing the node:

export poolname=pool-1
export workernumber=2
export clustername=suraj-lk-cluster

cd assets/terraform
kubectl drain $clustername-$poolname-worker-$workernumber
kubectl delete node $clustername-$poolname-worker-$workernumber
terraform taint module.worker-$poolname.random_string.bootstrap-token-id[$workernumber]
terraform taint module.worker-$poolname.random_string.bootstrap-token-secret[$workernumber]
terraform taint module.worker-$poolname.packet_device.nodes[$workernumber]

[invidian]

Right now we use same token for each node. Which is not a big deal of a problem if you think about it. The real problem is creating a new token and distributing it across nodes. How do we achieve it later, if the token is leaked accidentally?

What I would do is to create a token per worker group and then collect them in bootkube module and create them from there.

Also, in case the token leaks, one can taint the random resources in Terraform and tokens will be recreated. And so should be worker groups then.

[invidian]

Nice work @surajssd! LGTM.

Would be good if one more person could have a look before we merge.

[surajssd]

@invidian

Also, in case the token leaks, one can taint the random resources in Terraform and tokens will be recreated. And so should be worker groups then.

Can this be triggered from lokoctl? Or can we think about a feature of doing it like that for lokoctl?

I would have ideally wanted a per node token. And a mechanism to ship bootstrap token without node recreation.

[invidian]

Can this be triggered from lokoctl? Or can we think about a feature of doing it like that for lokoctl?

Yes, I can imagine we could implement it as a feature in lokoctl. To be able to manage the tokens somehow.

I would have ideally wanted a per node token.

I don't think that's possible. We use Ignition configuration per worker group to utilize cloud node groups capabilities.

And a mechanism to ship bootstrap token without node recreation.

I think it's OK, if we don't allow updating it on the nodes, as after the node registers, it should use obtained kubeconfig anyway, not the bootstrap token. I wonder for how long the certificates are issued. Because if it's like 90 days, then node would have to be down for this period to require re-using bootstrap token. I'm not even sure if that's implemented.

[surajssd]

@iaguis I deployed a cluster with current master 461abbf and then tried to upgrade it with current PR and I see following changes. These don't seem particularly harmful:

Details

$ lokoctl cluster apply -v
...
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
  - destroy
 <= read (data resources)

Terraform will perform the following actions:

  # module.packet-my-cluster.data.ct_config.controller-ignitions[0] will be read during apply
  # (config refers to values not yet known)
 <= data "ct_config" "controller-ignitions"  {
      + content  = (known after apply)
      + id       = (known after apply)
      + platform = "packet"
      + rendered = (known after apply)
      + snippets = []
    }

  # module.packet-my-cluster.data.ct_config.controller-install-ignitions[0] will be read during apply
  # (config refers to values not yet known)
 <= data "ct_config" "controller-install-ignitions"  {
      + content  = (known after apply)
      + id       = (known after apply)
      + rendered = (known after apply)
    }

  # module.packet-my-cluster.data.template_file.bootstrap-kubeconfig[0] will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "bootstrap-kubeconfig"  {
      + id       = (known after apply)
      + rendered = (known after apply)
      + template = <<~EOT
            apiVersion: v1
            kind: Config
            clusters:
            - name: local
              cluster:
                server: ${server}
                certificate-authority-data: ${ca_cert}
            users:
            - name: kubelet
              user:
                token: ${token_id}.${token_secret}
            contexts:
            - context:
                cluster: local
                user: kubelet
        EOT
      + vars     = {
          + "ca_cert"      = "[REDACTED]"
          + "server"       = "https://my-cluster-private.example.com:6443"
          + "token_id"     = (known after apply)
          + "token_secret" = (known after apply)
        }
    }

  # module.packet-my-cluster.packet_device.controllers[0] will be updated in-place
  ~ resource "packet_device" "controllers" {
        access_private_ipv4              = "10.88.81.1"
        access_public_ipv4               = "1.2.3.4"
        access_public_ipv6               = "2604:1380:1000:cd00::1"
        always_pxe                       = false
        billing_cycle                    = "hourly"
        created                          = "2020-06-19T10:13:11Z"
        deployed_facility                = "sjc1"
        facilities                       = [
            "sjc1",
        ]
        force_detach_volumes             = false
        hostname                         = "my-cluster-controller-0"
        id                               = "02369f5a-f2fb-4630-8278-2c58b2e52182"
        locked                           = false
        network                          = [
            {
                address = "1.2.3.4"
                cidr    = 31
                family  = 4
                gateway = "1.2.3.5"
                public  = true
            },
            {
                address = "2604:1380:1000:cd00::1"
                cidr    = 127
                family  = 6
                gateway = "2604:1380:1000:cd00::"
                public  = true
            },
            {
                address = "10.88.81.1"
                cidr    = 31
                family  = 4
                gateway = "10.88.81.0"
                public  = false
            },
        ]
        network_type                     = "layer3"
        operating_system                 = "flatcar_stable"
        plan                             = "t1.small.x86"
        ports                            = [
            {
                bonded = true
                id     = "f0f53195-8600-4a5f-9a23-eb713e75d1fb"
                mac    = ""
                name   = "bond0"
                type   = "NetworkBondPort"
            },
            {
                bonded = true
                id     = "78ed0f56-35cc-4f5c-8d76-e06f07946b14"
                mac    = "0c:c4:7a:b5:86:ea"
                name   = "eth0"
                type   = "NetworkPort"
            },
            {
                bonded = true
                id     = "2a75c5c7-f65f-43a3-93d8-7408f2e928e2"
                mac    = "0c:c4:7a:b5:86:eb"
                name   = "eth1"
                type   = "NetworkPort"
            },
        ]
        project_id                       = "436c24d4-770d-46c7-a610-8c32d7e8fe22"
        public_ipv4_subnet_size          = 31
        ssh_key_ids                      = [
            "[REDACTED]",
        ]
        state                            = "active"
      ~ tags                             = [
          - "lokoctl-version:v0.1.0-489-g461abbf9",
          + "lokoctl-version:v0.1.0-500-gd36430b7",
        ]
        updated                          = "2020-06-19T10:20:18Z"
      ~ user_data                        = (sensitive value)
        wait_for_reservation_deprovision = false
    }

  # module.packet-my-cluster.random_string.bootstrap-token-id[0] will be created
  + resource "random_string" "bootstrap-token-id" {
      + id          = (known after apply)
      + length      = 6
      + lower       = true
      + min_lower   = 0
      + min_numeric = 0
      + min_special = 0
      + min_upper   = 0
      + number      = true
      + result      = (known after apply)
      + special     = false
      + upper       = false
    }

  # module.packet-my-cluster.random_string.bootstrap-token-secret[0] will be created
  + resource "random_string" "bootstrap-token-secret" {
      + id          = (known after apply)
      + length      = 16
      + lower       = true
      + min_lower   = 0
      + min_numeric = 0
      + min_special = 0
      + min_upper   = 0
      + number      = true
      + result      = (known after apply)
      + special     = false
      + upper       = false
    }

  # module.packet-my-cluster.template_dir.calico_host_protection will be created
  + resource "template_dir" "calico_host_protection" {
      + destination_dir = "../cluster-assets/charts/kube-system/calico-host-protection"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/packet/flatcar-linux/kubernetes/calico-host-protection"
    }

  # module.worker-pool-1.data.ct_config.ignitions[0] will be read during apply
  # (config refers to values not yet known)
 <= data "ct_config" "ignitions"  {
      + content  = (known after apply)
      + id       = (known after apply)
      + platform = "packet"
      + rendered = (known after apply)
      + snippets = []
    }

  # module.worker-pool-1.data.ct_config.install-ignitions[0] will be read during apply
  # (config refers to values not yet known)
 <= data "ct_config" "install-ignitions"  {
      + content  = (known after apply)
      + id       = (known after apply)
      + rendered = (known after apply)
    }

  # module.worker-pool-1.data.template_file.bootstrap-kubeconfig[0] will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "bootstrap-kubeconfig"  {
      + id       = (known after apply)
      + rendered = (known after apply)
      + template = <<~EOT
            apiVersion: v1
            kind: Config
            clusters:
            - name: local
              cluster:
                server: ${server}
                certificate-authority-data: ${ca_cert}
            users:
            - name: kubelet
              user:
                token: ${token_id}.${token_secret}
            contexts:
            - context:
                cluster: local
                user: kubelet
        EOT
      + vars     = {
          + "ca_cert"      = "[REDACTED]"
          + "server"       = "https://my-cluster-private.example.com:6443"
          + "token_id"     = (known after apply)
          + "token_secret" = (known after apply)
        }
    }

  # module.worker-pool-1.packet_device.nodes[0] will be updated in-place
  ~ resource "packet_device" "nodes" {
        access_private_ipv4              = "10.88.81.3"
        access_public_ipv4               = "2.3.4.5"
        access_public_ipv6               = "2604:1380:1000:cd00::3"
        always_pxe                       = false
        billing_cycle                    = "hourly"
        created                          = "2020-06-19T10:13:10Z"
        deployed_facility                = "sjc1"
        facilities                       = [
            "sjc1",
        ]
        force_detach_volumes             = false
        hostname                         = "my-cluster-pool-1-worker-0"
        id                               = "0abff247-04a3-4e25-ad96-ad0d0b7c28f7"
        locked                           = false
        network                          = [
            {
                address = "2.3.4.5"
                cidr    = 31
                family  = 4
                gateway = "2.3.4.6"
                public  = true
            },
            {
                address = "2604:1380:1000:cd00::3"
                cidr    = 127
                family  = 6
                gateway = "2604:1380:1000:cd00::2"
                public  = true
            },
            {
                address = "10.88.81.3"
                cidr    = 31
                family  = 4
                gateway = "10.88.81.2"
                public  = false
            },
        ]
        network_type                     = "layer3"
        operating_system                 = "flatcar_stable"
        plan                             = "c2.medium.x86"
        ports                            = [
            {
                bonded = true
                id     = "bc3a866b-c6d9-4477-987d-5d5c2ed20885"
                mac    = ""
                name   = "bond0"
                type   = "NetworkBondPort"
            },
            {
                bonded = true
                id     = "8e5e8c91-456d-4b5a-b08d-f43e321a3b23"
                mac    = "50:6b:4b:b7:16:b6"
                name   = "eth0"
                type   = "NetworkPort"
            },
            {
                bonded = true
                id     = "89d209db-6f1b-4501-bb87-de4a787dd864"
                mac    = "50:6b:4b:b7:16:b7"
                name   = "eth1"
                type   = "NetworkPort"
            },
        ]
        project_id                       = "436c24d4-770d-46c7-a610-8c32d7e8fe22"
        public_ipv4_subnet_size          = 31
        ssh_key_ids                      = [
            "[REDACTED]",
        ]
        state                            = "active"
      ~ tags                             = [
          - "lokoctl-version:v0.1.0-489-g461abbf9",
          + "lokoctl-version:v0.1.0-500-gd36430b7",
        ]
        updated                          = "2020-06-19T10:23:04Z"
      ~ user_data                        = (sensitive value)
        wait_for_reservation_deprovision = false
    }

  # module.worker-pool-1.random_string.bootstrap-token-id[0] will be created
  + resource "random_string" "bootstrap-token-id" {
      + id          = (known after apply)
      + length      = 6
      + lower       = true
      + min_lower   = 0
      + min_numeric = 0
      + min_special = 0
      + min_upper   = 0
      + number      = true
      + result      = (known after apply)
      + special     = false
      + upper       = false
    }

  # module.worker-pool-1.random_string.bootstrap-token-secret[0] will be created
  + resource "random_string" "bootstrap-token-secret" {
      + id          = (known after apply)
      + length      = 16
      + lower       = true
      + min_lower   = 0
      + min_numeric = 0
      + min_special = 0
      + min_upper   = 0
      + number      = true
      + result      = (known after apply)
      + special     = false
      + upper       = false
    }

  # module.packet-my-cluster.module.bootkube.local_file.bootstrap-secrets will be created
  + resource "local_file" "bootstrap-secrets" {
      + content              = (known after apply)
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "../cluster-assets/charts/kube-system/bootstrap-secrets.yaml"
      + id                   = (known after apply)
    }

  # module.packet-my-cluster.module.bootkube.local_file.kubeconfig-kubelet will be destroyed
  - resource "local_file" "kubeconfig-kubelet" {
      - content              = <<~EOT
            apiVersion: v1
            kind: Config
            clusters:
            - name: local
              cluster:
                server: https://my-cluster-private.example.com:6443
                certificate-authority-data: [REDACTED]
            users:
            - name: kubelet
              user:
                client-certificate-data: [REDACTED]
                client-key-data: [REDACTED]
            contexts:
            - context:
                cluster: local
                user: kubelet
        EOT -> null
      - directory_permission = "0777" -> null
      - file_permission      = "0777" -> null
      - filename             = "../cluster-assets/auth/kubeconfig-kubelet" -> null
      - id                   = "87b108749e4613132f302090c2250614e582e0a1" -> null
    }

  # module.packet-my-cluster.module.bootkube.local_file.kubelet-crt will be destroyed
  - resource "local_file" "kubelet-crt" {
      - content              = <<~EOT
            -----BEGIN CERTIFICATE-----
            [REDACTED]
            -----END CERTIFICATE-----
        EOT -> null
      - directory_permission = "0777" -> null
      - file_permission      = "0777" -> null
      - filename             = "../cluster-assets/tls/kubelet.crt" -> null
      - id                   = "b3ca8228871443057e4795d62a671cb2325593bb" -> null
    }

  # module.packet-my-cluster.module.bootkube.local_file.kubelet-key will be destroyed
  - resource "local_file" "kubelet-key" {
      - content              = <<~EOT
            -----BEGIN RSA PRIVATE KEY-----
            [REDACTED]
            -----END RSA PRIVATE KEY-----
        EOT -> null
      - directory_permission = "0777" -> null
      - file_permission      = "0777" -> null
      - filename             = "../cluster-assets/tls/kubelet.key" -> null
      - id                   = "43c482d7068e1b5d1776874e540a3a5b209e3fac" -> null
    }

  # module.packet-my-cluster.module.bootkube.template_dir.bootstrap-manifests will be created
  + resource "template_dir" "bootstrap-manifests" {
      + destination_dir = "../cluster-assets/bootstrap-manifests"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/bootstrap-manifests"
      + vars            = {
          + "cloud_provider"                = ""
          + "etcd_servers"                  = "https://my-cluster-etcd0.dev.lokomotive-k8s.net:2379"
          + "kube_apiserver_image"          = "k8s.gcr.io/kube-apiserver:v1.18.3"
          + "kube_controller_manager_image" = "k8s.gcr.io/kube-controller-manager:v1.18.3"
          + "kube_scheduler_image"          = "k8s.gcr.io/kube-scheduler:v1.18.3"
          + "pod_cidr"                      = "10.2.0.0/16"
          + "service_cidr"                  = "10.3.0.0/16"
          + "trusted_certs_dir"             = "/usr/share/ca-certificates"
        }
    }

  # module.packet-my-cluster.module.bootkube.template_dir.bootstrap-secrets will be created
  + resource "template_dir" "bootstrap-secrets" {
      + destination_dir = "../cluster-assets/charts/kube-system/bootstrap-secrets"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/charts/bootstrap-secrets"
    }

  # module.packet-my-cluster.module.bootkube.template_dir.calico will be created
  + resource "template_dir" "calico" {
      + destination_dir = "../cluster-assets/charts/kube-system/calico"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/charts/calico"
    }

  # module.packet-my-cluster.module.bootkube.template_dir.kube-apiserver will be created
  + resource "template_dir" "kube-apiserver" {
      + destination_dir = "../cluster-assets/charts/kube-system/kube-apiserver"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/charts/kube-apiserver"
    }

  # module.packet-my-cluster.module.bootkube.template_dir.kubernetes will be created
  + resource "template_dir" "kubernetes" {
      + destination_dir = "../cluster-assets/charts/kube-system/kubernetes"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/charts/kubernetes"
    }

  # module.packet-my-cluster.module.bootkube.template_dir.pod-checkpointer will be created
  + resource "template_dir" "pod-checkpointer" {
      + destination_dir = "../cluster-assets/charts/kube-system/pod-checkpointer"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/charts/pod-checkpointer"
    }

  # module.packet-my-cluster.module.bootkube.tls_cert_request.kubelet will be destroyed
  - resource "tls_cert_request" "kubelet" {
      - cert_request_pem = <<~EOT
            -----BEGIN CERTIFICATE REQUEST-----
            [REDACTED]
            -----END CERTIFICATE REQUEST-----
        EOT -> null
      - id               = "bc1e3d773d9faf14264da127ac8e723cad82e606" -> null
      - key_algorithm    = "RSA" -> null
      - private_key_pem  = (sensitive value)

      - subject {
          - common_name    = "kubelet" -> null
          - organization   = "system:nodes" -> null
          - street_address = [] -> null
        }
    }

  # module.packet-my-cluster.module.bootkube.tls_locally_signed_cert.kubelet will be destroyed
  - resource "tls_locally_signed_cert" "kubelet" {
      - allowed_uses          = [
          - "key_encipherment",
          - "digital_signature",
          - "server_auth",
          - "client_auth",
        ] -> null
      - ca_cert_pem           = "05bb582154ce7258551c8eba1bfc0a71d2fd10d1" -> null
      - ca_key_algorithm      = "RSA" -> null
      - ca_private_key_pem    = (sensitive value)
      - cert_pem              = <<~EOT
            -----BEGIN CERTIFICATE-----
            [REDACTED]
            -----END CERTIFICATE-----
        EOT -> null
      - cert_request_pem      = "b6b17a5743ed0549bd29178536a8ee853d43574d" -> null
      - early_renewal_hours   = 0 -> null
      - id                    = "189863159367432705978437838633353101998" -> null
      - ready_for_renewal     = false -> null
      - validity_end_time     = "2021-06-19T15:43:07.92944791+05:30" -> null
      - validity_period_hours = 8760 -> null
      - validity_start_time   = "2020-06-19T15:43:07.92944791+05:30" -> null
    }

  # module.packet-my-cluster.module.bootkube.tls_private_key.kubelet will be destroyed
  - resource "tls_private_key" "kubelet" {
      - algorithm                  = "RSA" -> null
      - ecdsa_curve                = "P224" -> null
      - id                         = "ce5fc96d008d61bc2c0db0c72bc9665e86b28ba9" -> null
      - private_key_pem            = (sensitive value)
      - public_key_fingerprint_md5 = "82:c6:8f:ec:b6:39:df:22:7d:dc:5a:02:52:be:e8:0d" -> null
      - public_key_openssh         = <<~EOT
            [REDACTED]
        EOT -> null
      - public_key_pem             = <<~EOT
            -----BEGIN PUBLIC KEY-----
            [REDACTED]
            -----END PUBLIC KEY-----
        EOT -> null
      - rsa_bits                   = 2048 -> null
    }

Plan: 12 to add, 2 to change, 6 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Do you want to proceed with cluster apply? [type "yes" to continue]: hellno
INFO[0085] Cluster apply cancelled                       args="[]" command="lokoctl cluster apply"

I went forward with the changes and it failed with following error:

...
  image: kinvolk/pod-checkpointer:d1c58443fe7d7d33aa5bf7d80d65d299be6e5847

Your configurations are stored in ./assets
Controlplane component 'bootstrap-secrets' is missing, reinstalling...Failed!
FATA[0097] Installing controlplane component failed: unable to build kubernetes objects from release manifest: error validating "": error validating data: [unknown object type nil" in Secret.stringData.token-id, unknown object type "nil" in Secret.stringData.token-secret]  action=controlplane-upgrade args="[]" command="lokoctl cluster apply" component=bootstrap-secrets

[invidian]

I installed cluster with old version and tried to upgrade with this version after adding a variable enable_tls_bootstrap = false and it seems that it is trying to recreate controller nodes. I am investigating that now.

@surajssd did you resolve this issue? If so, how? (GitHub does not show diff between the reviews :/)

[surajssd]

@invidian yes resolved it by adding if else in helm chart config and in terraform config.

[invidian]

LGTM

[knrt10]

Just a NIT, rest looks great to me. Also, can you also please update the docs. Thanks

[surajssd]

@knrt10 added docs.

[knrt10]

Thanks for the effort. Looks great.

[surajssd]

Thanks for the continuous feedback @invidian