Introduce HA shoot control planes in alpha state #5741

shreyas-s-rao · 2022-04-05T06:44:43Z

How to categorize this PR?

/area control-plane
/area high-availability
/kind enhancement

What this PR does / why we need it:
This PR introduces high availability control planes for shoots in an MVP or alpha state.

New feature gate HAControlPlanes introduced for gardenlet and gardener-scheduler to enable support for the feature on landscapes/seeds. Disabled by default.
Introduce annotation alpha.control-plane.shoot.gardener.cloud/high-availability with supported values single-zone and multi-zone, to enable HA control plane for individual shoot on a HA-supported seed
Currently etcd-main, etcd-events and kube-apiserver are spread across topologies based on this feature
Shoots that request multi-zonal control planes can now be scheduled onto multi-zonal seeds (matched based on shoot annotation alpha.control-plane.shoot.gardener.cloud/high-availability='multi-zone' and seed label seed.gardener.cloud/multi-zonal)
If a shoot requests a HA control plane without the explicit need for zonal spread, then the shoot will be scheduled to a non-multi-zonal seed, and the control plane will be spread at the node level

Which issue(s) this PR fixes:
Fixes partially #2817

Special notes for your reviewer:

Release note:

Introduce feature gate `HAControlPlanes` in alpha state for gardenlet and gardener-scheduler. :warning: This comes with a change to the certs used, which will cause a restart of the etcds.

gardener-prow · 2022-04-05T06:44:45Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

rfranzke · 2022-04-05T06:46:01Z

/assign

pkg/features/features.go

rfranzke

Unit tests are missing

pkg/operation/botanist/component/etcd/etcd.go

pkg/operation/botanist/etcd.go

rfranzke · 2022-04-25T14:25:21Z

Can you please introduce a dedicated CA (with respective client certificates) for the peer communication (do not reuse the ca-etcd CA)?

shreyas-s-rao · 2022-05-05T06:53:42Z

@rfranzke Thanks for your review comments. I have now rebased the PR and made the suggested changes. PTAL

docs/deployment/feature_gates.md

pkg/features/features.go

pkg/operation/botanist/component/etcd/etcd.go

pkg/scheduler/controller/shoot/reconciler.go

timuthy · 2022-06-09T14:21:34Z

The e2e test fails due to #6081.
Apart from that the PR is ready for another round of review. I tested typical scenarios like reconciliation with an existing cluster, creating/hibernating/deleting a single node etcd shoot as well as the same for a multi-node etcd shoot ✅
As discussed, CA rotation doesn't work yet. I'm working on it and will file a separate PR.

timebertt · 2022-06-10T06:20:35Z

I think, the test actually failed because of #6092.
However, I cannot be sure as the shoot state logs are missing in the test (fixed by #6088).
Let's give it another shot, now that #6089 is also in.
/test pull-gardener-e2e-kind

unmarshall · 2022-06-10T08:29:19Z

For multi zonal control plane we have changed the scheduling constraints to either use topologyKey=kubernetes.io/hostname or topologyKey=topology.kubernetes.io/zone for Kube API Server and etcd based on the value of the annotation with possible values as single-zone and multi-zone.

There is another control plane component garden-resource-manager which today always runs with replicas=3 and with podAntiAffinity scheduling constraint which is hard coded to using topologyKey=kubernetes.io/hostname (See here)

Would it make sense to also have GRM be deployed across zones if the the value of annotation: alpha.control-plane.shoot.gardener.cloud/high-availability is multi-zone?

timuthy · 2022-06-10T09:03:41Z

This makes sense @unmarshall. As discussed, let's include this and other improvements in new PRs because:
a) The improvements are not absolutely necessary for this PR.
b) The PR is opened for quite a long time and it'd be good to have it merged before the next Gardener version is released.
c) There were already several rounds of reviews and comments, so I'd propose to not yet open other points.

docs/deployment/feature_gates.md

pkg/apis/core/v1beta1/constants/types_constants.go

pkg/features/features.go

pkg/operation/botanist/component/kubeapiserver/deployment.go

pkg/scheduler/controller/shoot/reconciler.go

pkg/operation/botanist/component/etcd/etcd.go

timuthy · 2022-06-10T16:13:05Z

Thanks for you input @rfranzke. I hope that I could address all of your comments. PTAL.

rfranzke

Thanks @timuthy, let's get this in now :)

/lgtm
/approve

gardener-prow · 2022-06-13T05:23:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rfranzke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rfranzke]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rfranzke · 2022-06-13T06:04:58Z

/unhold

* Introduce feature gate `HAControlPlanes` for gardenlet Signed-off-by: Shreyas Rao <shreyas.sriganesh.rao@sap.com> * Add support for multi-node, multi-zonal etcds * Add multi-AZ kube-apiservers * Schedule multi-AZ shoots only to supported seeds. Introduce feature gate `HAControlPlanes` for Gardener Scheduler. Signed-off-by: Shreyas Rao <shreyas.sriganesh.rao@sap.com> * Adapt tests Signed-off-by: Shreyas Rao <shreyas.sriganesh.rao@sap.com> * Fix etcd labels * Address review comments * add scheduler tests * add HAControlPlanes option validation * update feature gate release version * address review comments Co-authored-by: Tim Usner <tim.usner@sap.com>

gardener-prow bot requested review from BeckerMax and ialidzhikov April 5, 2022 06:45

gardener-robot-ci-2 added the reviewed/ok-to-test label Apr 5, 2022

gardener-prow bot assigned rfranzke Apr 5, 2022

gardener-robot-ci-1 added needs/ok-to-test and removed reviewed/ok-to-test labels Apr 5, 2022

rfranzke reviewed Apr 5, 2022

View reviewed changes

pkg/features/features.go Show resolved Hide resolved

rfranzke reviewed Apr 5, 2022

View reviewed changes

This was referenced Apr 8, 2022

Add SchedulingConstraints to Etcd spec gardener/etcd-druid#329

Merged

[ci:component:github.com/gardener/etcd-druid:v0.8.2->v0.9.0] #5789

Closed

rfranzke changed the title ~~[WIP] Introduce HA shoot control planes in alpha state~~ [WIP] Upgrade to etcd-druid@v0.9 and introduce HA shoot control planes in alpha state Apr 12, 2022

rfranzke mentioned this pull request Apr 25, 2022

☂️ Solution For Automated Shoot CA/Credentials Rotation #3292

Closed

gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 2, 2022

shreyas-s-rao force-pushed the feature/ha-control-plane-mvp branch from 3e86336 to 514b2ad Compare May 5, 2022 06:52

gardener-prow bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 5, 2022

rfranzke self-requested a review May 5, 2022 06:56

rfranzke requested changes May 6, 2022

View reviewed changes

gardener-prow bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 6, 2022

timuthy added 5 commits June 9, 2022 11:32

Fix etcd labels

0b66ffe

Address review comments

7b141e4

add scheduler tests

15fbd4a

add HAControlPlanes option validation

8758406

update feature gate release version

1112b9c

timuthy force-pushed the feature/ha-control-plane-mvp branch from 2c83aff to 1112b9c Compare June 9, 2022 09:32

gardener-prow bot added cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. and removed cla: no Indicates the PR's author has not signed the cla-assistant.io CLA. labels Jun 9, 2022

rfranzke self-requested a review June 10, 2022 09:23

rfranzke reviewed Jun 10, 2022

View reviewed changes

gardener-prow bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jun 10, 2022

address review comments

9a39dd8

timuthy force-pushed the feature/ha-control-plane-mvp branch from 84b8fc1 to 9a39dd8 Compare June 10, 2022 16:28

rfranzke reviewed Jun 13, 2022

View reviewed changes

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jun 13, 2022

gardener-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 13, 2022

gardener-prow bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 13, 2022

gardener-prow bot merged commit f38fd6d into gardener:master Jun 13, 2022

timuthy deleted the feature/ha-control-plane-mvp branch June 13, 2022 09:01

timuthy mentioned this pull request Jun 13, 2022

[HA] Prefer non-zonal seeds for non-zonal shoots #6102

Closed

ialidzhikov mentioned this pull request Jul 27, 2022

[GEP-20] HA Shoot Control Planes #6287

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce HA shoot control planes in alpha state #5741

Introduce HA shoot control planes in alpha state #5741

shreyas-s-rao commented Apr 5, 2022 •

edited

Loading

gardener-prow bot commented Apr 5, 2022

rfranzke commented Apr 5, 2022

rfranzke left a comment

rfranzke commented Apr 25, 2022

shreyas-s-rao commented May 5, 2022

timuthy commented Jun 9, 2022

timebertt commented Jun 10, 2022

unmarshall commented Jun 10, 2022

timuthy commented Jun 10, 2022

timuthy commented Jun 10, 2022

rfranzke left a comment

gardener-prow bot commented Jun 13, 2022

rfranzke commented Jun 13, 2022

Introduce HA shoot control planes in alpha state #5741

Introduce HA shoot control planes in alpha state #5741

Conversation

shreyas-s-rao commented Apr 5, 2022 • edited Loading

gardener-prow bot commented Apr 5, 2022

rfranzke commented Apr 5, 2022

rfranzke left a comment

Choose a reason for hiding this comment

rfranzke commented Apr 25, 2022

shreyas-s-rao commented May 5, 2022

timuthy commented Jun 9, 2022

timebertt commented Jun 10, 2022

unmarshall commented Jun 10, 2022

timuthy commented Jun 10, 2022

timuthy commented Jun 10, 2022

rfranzke left a comment

Choose a reason for hiding this comment

gardener-prow bot commented Jun 13, 2022

rfranzke commented Jun 13, 2022

shreyas-s-rao commented Apr 5, 2022 •

edited

Loading