[operator] Extend `operatorv1alpha1.VirtualCluster` with API for future `kube-apiserver` deployment #7693

rfranzke · 2023-03-22T16:18:17Z

How to categorize this PR?

/area usability
/kind enhancement

What this PR does / why we need it:
This PR extends the Garden API with configuration options for the virtual cluster for the

Kubernetes version that should be used.
configuration of the kube-apiserver.
DNS name the API server is exposed to.
Service CIDR the API server should use.

gardener-operator still does not deploy kube-apiserver, though this will be supported by the upcoming releases.

The API reuses the vast majority of the existing types as well as validation/defaulting code already present for Shoots. For the fields of gardencorev1beta1.KubeAPIServerConfig that are now exposed via the Garden, validation and defaulting happens via the webhook instead of via CRD validation/defaulting to prevent duplicating the logic. As a result, a new mutating webhook has been introduced which performs the defaulting.

For the fields that are not part of gardencorev1beta1.KubeAPIServerConfig but only of operatorv1alpha1.KubeAPIServerConfig, the validation/defaulting is done as part of the CRD validation/defaulting (just like it was done before for other fields).

Which issue(s) this PR fixes:
Part of #7016

Special notes for your reviewer:
/cc @ScheererJ @timuthy @oliver-goetz

Release note:

The `Garden` API was extended with the new `.spec.virtualCluster.{dns,kubernetes,networking}` sections. For now, they only allow configuring the necessary information for the deployment of `kube-apiserver`. Since the API server is not deployed yet, any configuration does not have any effect. Still, you must make sure to already specify at least `.spec.virtualCluster.kubernetes.version`, `.spec.virtualCluster.dns.domain`, and `.spec.virtualCluster.networking.services`. In the upcoming releases, `gardener-operator` will also take over the management of the `kube-apiserver` deployment whilst taking the configuration into account.

The `extensions/pkg/webhook/certificates.AddCertificateManagementToManager` function does now take a list of source webhook configs instead of a single webhook config only.

timuthy · 2023-03-23T07:50:34Z

/assign

ScheererJ

Thanks. Looks good overall.
I have just a few questions.

pkg/apis/operator/v1alpha1/types.go

pkg/utils/validation/admissionplugins/admissionplugins_test.go

pkg/apis/operator/v1alpha1/types.go

pkg/apis/core/v1beta1/defaults.go

rfranzke · 2023-03-24T07:30:57Z

@ScheererJ Thanks for your review, I've answered comments and addressed the feedback with 6cb480a.

I've also added two more commits which expose the DNS and networking section with configuration fields required for the future deployment of the API server, cc @timuthy (I forgot about them initially).

ScheererJ

Thanks for addressing my questions.
I have a few ones about the added DNS/networking parts.

pkg/apis/operator/v1alpha1/types.go

test/integration/gardenlet/seed/seed/seed_test.go

rfranzke · 2023-03-24T16:24:19Z

OK, next round @ScheererJ :) I've addressed your feedback in the last commit. There are two more new commits:

One for validating the Kubernetes version against the versions supported by Gardener
One for fixing a bug in the newly introduced defaulting webhook (found together with @timuthy)

ScheererJ

Thanks a lot. Looks good to me.

ScheererJ · 2023-03-27T12:15:12Z

/lgtm

gardener-prow · 2023-03-27T12:16:16Z

LGTM label has been added.

Git tree hash: 31f17d15b625b1ef4a543ec257d1af8f2052e973

timuthy

Very nice 🚀

pkg/utils/validation/kubernetesversion/version.go

This will be called from the operator in the future

`.import-restrictions` file copied from https://github.com/gardener/gardener/blob/master/pkg/apis/core/validation/.import-restrictions

This commit also enhances the webhook controllers to reconcile multiple webhook configs in the source cluster (now that `gardener-operator` no longer only serves a validating webhook but also a defaulting webhook).

Without this commit, the certificate management reconciler can race with the code that updates the webhook configurations. Since both is started at the same time, it might happen that the reconciler is faster with updating the CA bundle in all webhooks (ref https://github.com/gardener/gardener/blob/5abed1bfedd6f2188e5dcec9af161d1b5c286236/extensions/pkg/webhook/certificates/reconciler.go#L169-L174), and afterwards the code removes the CA bundle again.

gardener-prow · 2023-03-28T05:40:36Z

@rfranzke: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gardener-apidiff	`17020e2`	link	false	`/test pull-gardener-apidiff`

Full PR test history. Your PR dashboard. Command help for this repository.
Please help us cut down on flakes by linking this test failure to an open flake report or filing a new flake report if you can't find an existing one. Also see our testing guideline for how to avoid and hunt flakes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

timuthy

/lgtm
/approve

gardener-prow · 2023-03-28T06:58:19Z

LGTM label has been added.

Git tree hash: aaa4e80c79cd2f23a2659e7e43cdf3d954530864

gardener-prow · 2023-03-28T06:58:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: timuthy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [timuthy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…re `kube-apiserver` deployment (gardener#7693) * Copy example API from `Shoot` example YAML file * Add kubeconfig secret reference for admission plugins * Add configuration for TLS SNI * Add configuration for resources to be stored in `etcd-events` * Add configuration options for {audit,auth{n,z}} webhooks * Export common validation and defaulting code This will be called from the operator in the future * Validation for new fields `.import-restrictions` file copied from https://github.com/gardener/gardener/blob/master/pkg/apis/core/validation/.import-restrictions * Defaulting for new fields This commit also enhances the webhook controllers to reconcile multiple webhook configs in the source cluster (now that `gardener-operator` no longer only serves a validating webhook but also a defaulting webhook). * Run `make generate` * Fix race causing CA bundles to be removed from webhook configs Without this commit, the certificate management reconciler can race with the code that updates the webhook configurations. Since both is started at the same time, it might happen that the reconciler is faster with updating the CA bundle in all webhooks (ref https://github.com/gardener/gardener/blob/5abed1bfedd6f2188e5dcec9af161d1b5c286236/extensions/pkg/webhook/certificates/reconciler.go#L169-L174), and afterwards the code removes the CA bundle again. * Address PR review feedback * Add DNS and networking fields to `Garden` * [make generate] * Do not overwrite inlined config in defaulting webhook * Validate Kubernetes version against supported versions * Address PR review feedback * Address PR review feedback

gardener-prow bot requested review from oliver-goetz, ScheererJ and timuthy March 22, 2023 16:18

gardener-prow bot added area/usability Usability related kind/enhancement Enhancement, improvement, extension labels Mar 22, 2023

rfranzke mentioned this pull request Mar 22, 2023

☂️ Improve Gardener Operator #7016

Closed

65 tasks

gardener-prow bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. labels Mar 22, 2023

rfranzke changed the title ~~[operator] Extend operatorv1alpha1.VirtualCluster with API for kube-apiserver configuration~~ [operator] Extend operatorv1alpha1.VirtualCluster with API for Kubernetes version and kube-apiserver configuration Mar 22, 2023

gardener-prow bot assigned timuthy Mar 23, 2023

ScheererJ reviewed Mar 23, 2023

View reviewed changes

pkg/apis/operator/v1alpha1/types.go Show resolved Hide resolved

pkg/utils/validation/admissionplugins/admissionplugins_test.go Show resolved Hide resolved

pkg/apis/operator/v1alpha1/types.go Show resolved Hide resolved

pkg/apis/core/v1beta1/defaults.go Show resolved Hide resolved

rfranzke mentioned this pull request Mar 24, 2023

Support AdmissionPlugin to reference other files #2879

Closed

rfranzke force-pushed the enh/operator-kapi-api branch from de66f0a to 779f76b Compare March 24, 2023 07:27

rfranzke requested a review from ScheererJ March 24, 2023 07:27

rfranzke changed the title ~~[operator] Extend operatorv1alpha1.VirtualCluster with API for Kubernetes version and kube-apiserver configuration~~ [operator] Extend operatorv1alpha1.VirtualCluster with API for future kube-apiserver deployment Mar 24, 2023

rfranzke force-pushed the enh/operator-kapi-api branch from 779f76b to 98c17f5 Compare March 24, 2023 07:37

ScheererJ reviewed Mar 24, 2023

View reviewed changes

pkg/apis/operator/v1alpha1/types.go Show resolved Hide resolved

test/integration/gardenlet/seed/seed/seed_test.go Show resolved Hide resolved

test/integration/gardenlet/seed/seed/seed_test.go Show resolved Hide resolved

rfranzke force-pushed the enh/operator-kapi-api branch from 98c17f5 to 1186d37 Compare March 24, 2023 16:23

rfranzke requested a review from ScheererJ March 24, 2023 16:24

ScheererJ reviewed Mar 27, 2023

View reviewed changes

gardener-prow bot assigned ScheererJ Mar 27, 2023

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Mar 27, 2023

rfranzke mentioned this pull request Mar 27, 2023

[operator] Manage Virtual Garden kube-apiserver Deployment #7730

Merged

timuthy reviewed Mar 27, 2023

View reviewed changes

pkg/utils/validation/kubernetesversion/version.go Outdated Show resolved Hide resolved

Copy example API from Shoot example YAML file

c0f8453

rfranzke added 16 commits March 28, 2023 07:36

Add kubeconfig secret reference for admission plugins

91728b8

Add configuration for TLS SNI

0c16d4b

Add configuration for resources to be stored in etcd-events

0d98aa8

Add configuration options for {audit,auth{n,z}} webhooks

24b02ce

Export common validation and defaulting code

7f0b331

This will be called from the operator in the future

Validation for new fields

09ac59e

`.import-restrictions` file copied from https://github.com/gardener/gardener/blob/master/pkg/apis/core/validation/.import-restrictions

Defaulting for new fields

c4078c4

This commit also enhances the webhook controllers to reconcile multiple webhook configs in the source cluster (now that `gardener-operator` no longer only serves a validating webhook but also a defaulting webhook).

Run make generate

6ff29fc

Address PR review feedback

8f3efae

Add DNS and networking fields to Garden

73f3dd0

[make generate]

cee4f72

Do not overwrite inlined config in defaulting webhook

2fedcbf

Validate Kubernetes version against supported versions

60d01d4

Address PR review feedback

f4dc21f

Address PR review feedback

17020e2

rfranzke force-pushed the enh/operator-kapi-api branch from 1186d37 to 17020e2 Compare March 28, 2023 05:37

gardener-prow bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 28, 2023

gardener-prow bot requested review from ScheererJ and timuthy March 28, 2023 05:37

timuthy reviewed Mar 28, 2023

View reviewed changes

gardener-prow bot added the lgtm Indicates that a PR is ready to be merged. label Mar 28, 2023

gardener-prow bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 28, 2023

gardener-prow bot merged commit e488729 into gardener:master Mar 28, 2023

rfranzke deleted the enh/operator-kapi-api branch March 28, 2023 06:59

acumino mentioned this pull request Jun 20, 2023

Support AdmissionPlugin to reference other files #8110

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[operator] Extend `operatorv1alpha1.VirtualCluster` with API for future `kube-apiserver` deployment #7693

[operator] Extend `operatorv1alpha1.VirtualCluster` with API for future `kube-apiserver` deployment #7693

rfranzke commented Mar 22, 2023 •

edited

Loading

timuthy commented Mar 23, 2023

ScheererJ left a comment

rfranzke commented Mar 24, 2023 •

edited

Loading

ScheererJ left a comment

rfranzke commented Mar 24, 2023

ScheererJ left a comment

ScheererJ commented Mar 27, 2023

gardener-prow bot commented Mar 27, 2023

timuthy left a comment

gardener-prow bot commented Mar 28, 2023

timuthy left a comment

gardener-prow bot commented Mar 28, 2023

gardener-prow bot commented Mar 28, 2023

[operator] Extend operatorv1alpha1.VirtualCluster with API for future kube-apiserver deployment #7693

[operator] Extend operatorv1alpha1.VirtualCluster with API for future kube-apiserver deployment #7693

Conversation

rfranzke commented Mar 22, 2023 • edited Loading

timuthy commented Mar 23, 2023

ScheererJ left a comment

Choose a reason for hiding this comment

rfranzke commented Mar 24, 2023 • edited Loading

ScheererJ left a comment

Choose a reason for hiding this comment

rfranzke commented Mar 24, 2023

ScheererJ left a comment

Choose a reason for hiding this comment

ScheererJ commented Mar 27, 2023

gardener-prow bot commented Mar 27, 2023

timuthy left a comment

Choose a reason for hiding this comment

gardener-prow bot commented Mar 28, 2023

timuthy left a comment

Choose a reason for hiding this comment

gardener-prow bot commented Mar 28, 2023

gardener-prow bot commented Mar 28, 2023

[operator] Extend `operatorv1alpha1.VirtualCluster` with API for future `kube-apiserver` deployment #7693

[operator] Extend `operatorv1alpha1.VirtualCluster` with API for future `kube-apiserver` deployment #7693

rfranzke commented Mar 22, 2023 •

edited

Loading

rfranzke commented Mar 24, 2023 •

edited

Loading