Rotate kubelet server certificate. #6453

vrlo · 2020-07-25T08:28:35Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
Adds support for kubelet server certificate rotation.

Which issue(s) this PR fixes:
Fixes #6449

Special notes for your reviewer:
New variable kubelet_rotate_server_certificates, when true, sets kubelet config parameter serverTLSBootstrap to true and kubeadm's apiserver parameter kubelet-certificate-authority to the default ca location.

Does this PR introduce a user-facing change?:

Added variable `kubelet_rotate_server_certificates` which enables kubelet server certificate rotation.
If enabled, requires either manual approval of generated certificate requests, or a custom approving controller.

k8s-ci-robot · 2020-07-25T08:28:43Z

Hi @vrlo. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

floryut · 2020-07-28T14:03:38Z

roles/kubernetes/master/templates/kubeadm-config.v1beta2.yaml.j2

@@ -195,7 +195,10 @@ apiServer:
 {% endif %}


Weird that the break line above is mandatory not to break the build

yes, otherwise event-ttl line ends up at the end of the previous line

floryut · 2020-07-28T14:04:31Z

Should this be enabled in one CI test job ?

vrlo · 2020-07-28T14:09:35Z

Should this be enabled in one CI test job ?

That would be wise. Which test would you suggest cloning/editing?

floryut · 2020-07-28T14:12:10Z

Should this be enabled in one CI test job ?

That would be wise. Which test would you suggest cloning/editing?

I would say packet_centos7-flannel-containerd-addons-ha.yml even if it already has a lot of things, but that's kind of the point of this test :)

floryut · 2020-07-28T15:07:15Z

/ok-to-test

floryut · 2020-07-30T08:19:51Z

Nice @vrlo :)
/lgtm

hafe · 2020-08-04T12:16:47Z

Why not have it as default enabled?

hafe · 2020-08-04T12:40:52Z

I can answer myself, because it requires csr approval.

mattymo · 2020-08-04T15:23:31Z

tests/testcases/030_check-network.yml

+        fail_msg: kubelet_rotate_server_certificates is {{ kubelet_rotate_server_certificates }} but no csr's found
+
+    - name: Approve certificates
+      command: "{{ bin_dir }}/kubectl certificate approve {{ get_csr.stdout_lines | join(' ') }}"


I don't really like how this is implemented. What I was expecting in this PR was the following:
1 - needed vars
2 - logic to enable cert renewal in kubelet config
3 - renewal logic

What I see here is kubectl get csr/kubectl certificate approve in the tests directory, not part of the actual Kubespray core Ansible playbook. Why can't we put this in kubernetes/kubeadm role directly and run it on the first controller?

From a security perspective, we should filter csrs for those that belong to nodes we know about from inventory, rather than blindly approving all.

Thanks for the review.

ad 1 - there's var kubelet_rotate_server_certificates that is documented
ad 2 - that's kubelet config parameter ServerTLSBootstrap
ad 3 - both bootstrap and renewal are handled by kubelet when the parameter in 2 is set

I would recommend not putting approval logic in kubespray. There is a reason why serving certificates aren't automatically approved by CSR approving controller like client certificates are. Cluster admin is supposed to manually approve them (or implement a custom approving controller). In any case, approvals are not in scope of this PR. That is why the functionality is disabled by default and the docs warn about the caveat.

Regarding blind approvals in the test, that part takes the role of the cluster admin manually approving certificates and was added just to pass the test which failed otherwise (kubectl exec couldn't connect to the kubelet which was missing the serving certificate).

mattymo · 2020-09-03T14:22:59Z

/approve
/lgtm

k8s-ci-robot · 2020-09-03T14:23:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mattymo, vrlo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mattymo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* 'master' of https://github.com/kubernetes-sigs/kubespray: Add protectKernelDefaults option (default true) to kubelet config file (kubernetes-sigs#6611) Rotate kubelet server certificate. (kubernetes-sigs#6453) Add snapshot-controller for CSI drivers and snapshot CRDs, add a default volumesnapshotclass when running cinder CSI (kubernetes-sigs#6537) Calico: update crds to v1 and cr (kubernetes-sigs#6360) Fix E306 in roles/network_plugin (kubernetes-sigs#6516)

* Rotate kubelet server certificate. * CI test kubelet server cert rotation * Approve kubelet serving certificates in tests.

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 25, 2020

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 25, 2020

k8s-ci-robot requested review from bozzo and floryut July 25, 2020 08:28

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jul 25, 2020

vrlo force-pushed the kubelet-servertlsbootstrap branch from 6d55de1 to 910d9d5 Compare July 27, 2020 13:35

vrlo marked this pull request as draft July 28, 2020 07:21

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 28, 2020

vrlo force-pushed the kubelet-servertlsbootstrap branch 3 times, most recently from 9813a93 to b69b881 Compare July 28, 2020 09:34

Rotate kubelet server certificate.

714e8c2

vrlo force-pushed the kubelet-servertlsbootstrap branch from b69b881 to 714e8c2 Compare July 28, 2020 13:26

vrlo marked this pull request as ready for review July 28, 2020 13:27

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 28, 2020

floryut reviewed Jul 28, 2020

View reviewed changes

vrlo marked this pull request as draft July 28, 2020 14:13

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 28, 2020

CI test kubelet server cert rotation

db932bf

vrlo marked this pull request as ready for review July 28, 2020 15:04

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 28, 2020

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 28, 2020

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 29, 2020

Approve kubelet serving certificates in tests.

320afc4

vrlo force-pushed the kubelet-servertlsbootstrap branch from cf71721 to 320afc4 Compare July 29, 2020 12:49

k8s-ci-robot assigned floryut Jul 30, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 30, 2020

mattymo reviewed Aug 4, 2020

View reviewed changes

k8s-ci-robot assigned mattymo Sep 3, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 3, 2020

k8s-ci-robot merged commit c1ba8e1 into kubernetes-sigs:master Sep 3, 2020

floryut mentioned this pull request Sep 3, 2020

Release Proposal v2.14 #6535

Closed

LuckySB pushed a commit to southbridgeio/kubespray that referenced this pull request Jan 15, 2021

Rotate kubelet server certificate. (kubernetes-sigs#6453)

0eb5c53

* Rotate kubelet server certificate. * CI test kubelet server cert rotation * Approve kubelet serving certificates in tests.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rotate kubelet server certificate. #6453

Rotate kubelet server certificate. #6453

vrlo commented Jul 25, 2020

k8s-ci-robot commented Jul 25, 2020

floryut Jul 28, 2020 •

edited

Loading

vrlo Jul 28, 2020

floryut commented Jul 28, 2020

vrlo commented Jul 28, 2020

floryut commented Jul 28, 2020

floryut commented Jul 28, 2020

floryut commented Jul 30, 2020

hafe commented Aug 4, 2020

hafe commented Aug 4, 2020

mattymo Aug 4, 2020

vrlo Aug 6, 2020

mattymo commented Sep 3, 2020

k8s-ci-robot commented Sep 3, 2020

Rotate kubelet server certificate. #6453

Rotate kubelet server certificate. #6453

Conversation

vrlo commented Jul 25, 2020

k8s-ci-robot commented Jul 25, 2020

floryut Jul 28, 2020 • edited Loading

Choose a reason for hiding this comment

vrlo Jul 28, 2020

Choose a reason for hiding this comment

floryut commented Jul 28, 2020

vrlo commented Jul 28, 2020

floryut commented Jul 28, 2020

floryut commented Jul 28, 2020

floryut commented Jul 30, 2020

hafe commented Aug 4, 2020

hafe commented Aug 4, 2020

mattymo Aug 4, 2020

Choose a reason for hiding this comment

vrlo Aug 6, 2020

Choose a reason for hiding this comment

mattymo commented Sep 3, 2020

k8s-ci-robot commented Sep 3, 2020

floryut Jul 28, 2020 •

edited

Loading