diff --git a/hack/verify-toc.sh b/hack/verify-toc.sh
index e4f84abc54f8..0d06e18690d7 100755
--- a/hack/verify-toc.sh
+++ b/hack/verify-toc.sh
@@ -47,6 +47,7 @@ echo "Checking table of contents are up to date..."
find keps -name '*.md' \
| grep -Fxvf hack/.notableofcontents \
| xargs mdtoc --inplace --max-depth=5 --dryrun || (
- echo "Table of content not up to date. If this failed silently and you are on mac, try 'brew install grep'"
+ echo "Table of content not up to date. Did you run 'make update-toc' ?"
+ echo "If this failed silently and you are on mac, try 'brew install grep'"
exit 1
)
diff --git a/keps/NNNN-kep-template/README.md b/keps/NNNN-kep-template/README.md
index 2d6e477767b7..18d6c1461139 100644
--- a/keps/NNNN-kep-template/README.md
+++ b/keps/NNNN-kep-template/README.md
@@ -720,6 +720,18 @@ This through this both in small and large cases, again with respect to the
[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
-->
+###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?
+
+
+
### Troubleshooting
+
+### Version Skew Strategy
+
+This feature affects only the kube-apiserver, so there is no issue
+with version skew with other components.
## Production Readiness Review Questionnaire
### Feature Enablement and Rollback
-* **How can this feature be enabled / disabled in a live cluster?** To enable
- priority and fairness, all of the following must be enabled:
- - [x] Feature gate
- - Feature gate name: APIPriorityAndFairness
- - Components depending on the feature gate:
- - kube-apiserver
- - [x] Command-line flags
- - `--enable-priority-and-fairness`, and
- - `--runtime-config=flowcontrol.apiserver.k8s.io/v1alpha1=true`
+###### How can this feature be enabled / disabled in a live cluster?
+
+- [x] Feature gate (also fill in values in `kep.yaml`)
+ - Feature gate name: APIPriorityAndFairness
+ - Components depending on the feature gate:
+ - kube-apiserver
+- [x] Other
+ - Describe the mechanism: command-line flag
+ `enable-priority-and-fairness`, which defaults to `true`.
+ - Will enabling / disabling the feature require downtime of the
+ control plane? Changing a command-line flag requires restarting
+ the kube-apiserver.
+ - Will enabling / disabling the feature require downtime or
+ reprovisioning of a node? (Do not assume `Dynamic Kubelet Config`
+ feature is enabled). No.
-* **Does enabling the feature change any default behavior?** Yes, requests that
- weren't rejected before could get rejected while requests that were rejected
- previously may be allowed. Performance of kube-apiserver under heavy load
- will likely be different too.
+###### Does enabling the feature change any default behavior?
-* **Can the feature be disabled once it has been enabled (i.e. can we roll back
- the enablement)?** Yes.
+Yes, requests that weren't rejected before could get rejected while
+requests that were rejected previously may be allowed. Performance of
+kube-apiserver under heavy load will likely be different too.
-* **What happens if we reenable the feature if it was previously rolled back?**
- The feature will be restored.
+###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
-* **Are there any tests for feature enablement/disablement?** No. Manual tests
- will be run before switching feature gate to beta.
+Yes.
+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+The feature will be restored.
+
+###### Are there any tests for feature enablement/disablement?
+
+No. Manual tests were run before switching feature gate to beta.
### Rollout, Upgrade and Rollback Planning
-* **How can a rollout fail? Can it impact already running workloads?** A
- misconfiguration could cause apiserver requests to be rejected, which could
- have widespread impact such as: (1) rejecting controller requests, thereby
- bringing a lot of things to a halt, (2) dropping node heartbeats, which may
- result in overloading other nodes, (3) rejecting kube-proxy requests to
- apiserver, thereby breaking existing workloads, (4) dropping leader election
- requests, resulting in HA failure, or any combination of the above.
-
-* **What specific metrics should inform a rollback?** An abnormal spike in the
- `apiserver_flowcontrol_rejected_requests_total` metric should potentially be
- viewed as a sign that kube-apiserver is rejecting requests, potentially
- incorrectly. The `apiserver_flowcontrol_request_queue_length_after_enqueue`
- metric getting too close to the configured queue length could be a sign of
- insufficient queue size (or a system overload), which can be precursor to
- rejected requests.
-
-* **Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?**
- No. Manual tests will be run before switching feature gate to beta.
-
-* **Is the rollout accompanied by any deprecations and/or removals of features, APIs,
- fields of API types, flags, etc.?** Yes, `--max-requests-inflights` will be
- deprecated in favor of APF.
+###### How can a rollout or rollback fail? Can it impact already running workloads?
-### Monitoring Requirements
+A misconfiguration could cause apiserver requests to be rejected,
+which could have widespread impact such as: (1) rejecting controller
+requests, thereby bringing a lot of things to a halt, (2) dropping
+node heartbeats, which may result in overloading other nodes, (3)
+rejecting kube-proxy requests to apiserver, thereby breaking existing
+workloads, (4) dropping leader election requests, resulting in HA
+failure, or any combination of the above.
-* **How can an operator determine if the feature is in use by workloads?**
- If the `apiserver_flowcontrol_dispatched_requests_total` metric is non-zero,
- this feature is in use. Note that this isn't a workload feature, but a
- control plane one.
+###### What specific metrics should inform a rollback?
-* **What are the SLIs (Service Level Indicators) an operator can use to determine
-the health of the service?**
- - [x] Metrics
- - Metric name: `apiserver_flowcontrol_request_queue_length_after_enqueue`
- - Components exposing the metric: kube-apiserver
+See [the remarks on service health](#what-are-the-slis-service-level-indicators-an-operator-can-use-to-determine-the-health-of-the-service).
-* **What are the reasonable SLOs (Service Level Objectives) for the above SLIs?**
- No SLOs are proposed for the above SLI.
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+
+Manual tests were run during promotion to Beta in 1.20.
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+
+The rollout replaces the behavior of the max-in-flight filter with the
+more sophisticated behavior of this feature.
+
+### Monitoring Requirements
-* **Are there any missing metrics that would be useful to have to improve observability
-of this feature?** No.
+###### How can an operator determine if the feature is in use by workloads?
+
+If the `apiserver_flowcontrol_dispatched_requests_total` metric is
+non-zero, this feature is in use. Note that this isn't a workload
+feature, but a control plane one.
+
+###### How can someone using this feature know that it is working for their instance?
+
+The classification functionality can be reviewed by looking at the
+`apf_pl` and `apf_fs` attributes that appear in the httplog lines from
+the apiserver when `-v=3` or more verbose. The classification is also
+reported in each HTTP response via the
+`X-Kubernetes-PF-PriorityLevel-UID` and
+`X-Kubernetes-PF-FlowSchema-UID` headers, respectively.
+
+The nominal division of concurrency can be checked by looking at the
+`apiserver_flowcontrol_nominal_limit_seats` metric of the limited
+priority levels to see whether they are in the proportions given by
+the `NominalConcurrencyShares` fields in the corresponding
+`PriorityLevelConfiguration` objects, and sum to the server's
+concurrency limit (the sum of the `--max-requests-inflight` and
+`--max-mutating-requests-inflight` command line parameters) allowing
+for rounding error.
+
+The dynamic adjustment of concurrency limits can be checked for
+validity by checking whether the
+`apiserver_flowcontrol_current_limit_seats` metrics sum to the
+server's concurrency limit allowing for rounding error. The
+responsiveness to load can be checked by comparing those dynamic
+concurrency limits to various indications of load and keeping in mind
+the bounds shown by the `apiserver_flowcontrol_lower_limit_seats` and
+`apiserver_flowcontrol_upper_limit_seats` metrics. Indications of
+load include the `apiserver_flowcontrol_request_wait_duration_seconds`
+histogram vector metric and the
+`apiserver_flowcontrol_demand_seats_smoothed` gauge vector metric.
+
+The dispatching can be checked by checking whether there are
+significant queuing delays only for flow schemeas that go to priority
+levels whose seats are highly utilized. The histogram vector metric
+`apiserver_flowcontrol_request_wait_duration_seconds` shows the
+former, and tThe histogram vector metric
+`apiserver_flowcontrol_priority_level_seat_utilization` shows the
+latter.
+
+###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
+
+None have been identified.
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+Since this feature is about protecting the server from overload,
+trouble in this feature and actual overload can show similar symptoms.
+Following are some thoughts about how to tell them apart.
+
+- high latency / long queue lengths + low request volume / low CPU
+ usage = probably APF misconfiguration or bug
+
+- high latency / long queue lengths + high request volume / high CPU
+ usage = probably APF is saving your cluster from an outage
+
+The following metrics give relevant clues.
+
+- `apiserver_flowcontrol_request_wait_duration_seconds` shows queuing
+ delay in APF.
+- `apiserver_flowcontrol_request_execution_seconds` shows execution
+ durations in APF; that summed with
+ `apiserver_flowcontrol_request_wait_duration_seconds` shows total
+ handling time covered by APF.
+- The `apiserver_request_duration_seconds_count` histogram shows total
+ request delay as measured in a handler tha encloses APF, grouped by
+ request characteristics.
+- The rate of change in
+ `apiserver_flowcontrol_request_execution_seconds_count` shows the
+ rates of request handling, grouped by flow schema.
+- The rate of change in `apiserver_request_total` shows the rates at
+ which requests are being handled, grouped by request
+ characteristics.
+- the `apiserver_flowcontrol_current_inqueue_requests` gauge vector
+ samples queue lengths for each flow schema.
+- the `apiserver_flowcontrol_priority_level_seat_utilization`
+ histogram vector summarizes per-nanosecond samples of the fraction
+ each priority level's seats that are occupied (these should not all
+ be low if the server is heavily loaded).
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+
+No.
### Dependencies
-* **Does this feature depend on any specific services running in the cluster?**
- No.
+###### Does this feature depend on any specific services running in the cluster?
+
+No
### Scalability
-* **Will enabling / using this feature result in any new API calls?** Yes.
- Self-requests for new API objects will be introduced. In addition, the
- request execution order may change, which could occasionally increase the
- number of retries.
+###### Will enabling / using this feature result in any new API calls?
-* **Will enabling / using this feature result in introducing new API types?**
- Yes, a new flowcontrol API group, configuration types, and status types are
- introduced. See `k8s.io/api/flowcontrol/v1alpha1/types.go` for a full list.
+Yes. Self-requests for new API objects will be introduced. In
+addition, the request execution order may change, which could
+occasionally increase the number of retries.
-* **Will enabling / using this feature result in any new calls to the cloud
- provider?** No.
+###### Will enabling / using this feature result in introducing new API types?
-* **Will enabling / using this feature result in increasing size or count of
- the existing API objects?** No.
+Yes, a new flowcontrol API group, configuration types, and status
+types are introduced. See `k8s.io/api/flowcontrol/v1alpha1/types.go`
+for a full list.
-* **Will enabling / using this feature result in increasing time taken by any
- operations covered by [existing SLIs/SLOs]?** Yes, a non-negligible latency
- is added to API calls to kube-apiserver. While [preliminary tests](https://github.com/tkashem/graceful/blob/master/priority-fairness/filter-latency/readme.md)
- shows that the API server latency is still well within the existing SLOs,
- more thorough testing needs to be performed.
+###### Will enabling / using this feature result in any new calls to the cloud provider?
-* **Will enabling / using this feature result in non-negligible increase of
- resource usage (CPU, RAM, disk, IO, ...) in any components?** The proposed
- flowcontrol logic in request handling in kube-apiserver will increase the CPU
- and memory overheads involved in serving each request. Note that the resource
- usage will be configurable and may require the operator to fine-tune some
- parameters.
+No.
-### Troubleshooting
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
-* **How does this feature react if the API server and/or etcd is unavailable?**
- The feature is itself within the API server. Etcd being unavailable would
- likely cause kube-apiserver to fail at processing incoming requests.
+No.
-* **What are other known failure modes?** A misconfiguration could reject
- requests incorrectly. See the rollout and monitoring sections for details on
- which metrics to watch to detect such failures (see the `kep.yaml` file for
- the full list of metrics). The following kube-apiserver log messages could
- also indicate potential issues:
- - "Unable to list PriorityLevelConfiguration objects"
- - "Unable to list FlowSchema objects"
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
-* **What steps should be taken if SLOs are not being met to determine the
- problem?** No SLOs are proposed.
+Yes, a non-negligible latency is added to API calls to
+kube-apiserver. While [preliminary
+tests](https://github.com/tkashem/graceful/blob/master/priority-fairness/filter-latency/readme.md)
+shows that the API server latency is still well within the existing
+SLOs, more thorough testing needs to be performed.
-[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
-[existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos
+###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
-## Implementation History
+The proposed flowcontrol logic in request handling in kube-apiserver
+will cause a small increase the CPU and memory overheads involved in
+serving each request when there is no significant queuing. When there
+_is_ significant queuing, it is because this feature is doing its
+intended function of smoothing out usage of many resources by limiting
+the number of requests being actively handled at once. Holding
+requests in queues involves an added memory cost. Note that the
+resource usage will be configurable and may require the operator to
+fine-tune some parameters.
+
+###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?
+
+This feature does not directly consume any resources on worker nodes.
+It can have indirect effects, through the changes in how the
+kube-apiserver serves requests.
+### Troubleshooting
+
+###### How does this feature react if the API server and/or etcd is unavailable?
+
+This feature is entirely in the apiserver, so will be unavailable when
+the apiserver is unavailable.
+
+This feature does not depend directly on the storage and will continue
+to perform its function while the storage is unavailable, except for
+the parts of the function that involve maintaining its configuration
+objects and reacting to changes in them. While the storage is
+unavailable this feature continues to use the last configuration that
+it read. Unavailability of storage will naturally have severe effects
+on request executions and this feature will react to the relevant
+consequences.
+
+###### What are other known failure modes?
+
+ - Some client makes requests with big responses that the client does
+ not finish reading in the first minute, constantly occupying (hand
+ size) X (max width = 10) seats --- which nearly starves other
+ clients when the priority level's concurrency limit is not much
+ greater than that.
+ - Detection: How can it be detected via metrics? In this situation
+ the requests from the bad client will timeout during execution
+ at a rate of (hand size)/minute, and other requests in the same
+ priority level will almost all timeout while waiting in a queue.
+ - Mitigations: What can be done to stop the bleeding, especially
+ for already running user workloads? Identify the bad client and
+ create a special flow schema and priority level to isolate that
+ client.
+ - Diagnostics: What are the useful log messages and their required
+ logging levels that could help debug the issue? In the
+ kube-apiserver's log at `-v=3` or more verbose the httplog
+ entries will indicate timeouts.
+ - Testing: Are there any tests for failure mode? If not, describe
+ why. We only recently realized this failure mode and we hope to
+ make a change that will prevent it or automatically mitigate it.
+ See https://github.com/kubernetes/kubernetes/issues/115409 .
+
+ - A misconfiguration could reject
+ requests incorrectly.
+ - Detection: See the rollout and monitoring sections for details
+ on which metrics to watch to detect such failures (see the
+ `kep.yaml` file for the full list of metrics). The following
+ kube-apiserver log messages could also indicate potential
+ issues:
+ - "Unable to list PriorityLevelConfiguration objects"
+ - "Unable to list FlowSchema objects"
+
+###### What steps should be taken if SLOs are not being met to determine the problem?
+
+No SLOs are proposed.
-- v1.19: `Alpha` release
+## Implementation History
+
+- v1.18: `Alpha` release
- v1.20: graduated to `Beta`
- v1.22: initial support for width concept and watch initialization
- v1.23: introduce `v1beta2` API
- no changes compared to `v1beta1`
- `v1beta1` remain as storage version
- v1.24: storage version changed to `v1beta2`
+- v1.26: introduce concurrency borrowing between priority levels
## Drawbacks
@@ -2995,6 +3184,5 @@ designs not chosen.
## Infrastructure Needed
-The end-to-end test suite should exercise the functionality introduced
-by this KEP. This may require creating a special client to submit an
-overload of low-priority work.
+Nothing beyond the usual CI.
+
diff --git a/keps/sig-api-machinery/1040-priority-and-fairness/kep.yaml b/keps/sig-api-machinery/1040-priority-and-fairness/kep.yaml
index befebd21bb18..55f16b8c3932 100644
--- a/keps/sig-api-machinery/1040-priority-and-fairness/kep.yaml
+++ b/keps/sig-api-machinery/1040-priority-and-fairness/kep.yaml
@@ -10,6 +10,7 @@ participating-sigs:
- wg-multitenancy
- sig-scheduling
status: implementable
+creation-date: 2019-02-28
reviewers:
- "@deads2k"
- "@lavalamp"
@@ -19,21 +20,20 @@ reviewers:
approvers:
- "@deads2k"
- "@lavalamp"
-creation-date: 2019-02-28
# The target maturity stage in the current dev cycle for this KEP.
-stage: beta
+stage: stable
# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
-latest-milestone: "v1.23"
+latest-milestone: "v1.27"
# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
alpha: "v1.18"
beta: "v1.20"
- stable: "v1.23"
+ stable: "v1.27"
# The following PRR answers are required at alpha release.
# List the feature gate name and the components for which it must be enabled.
@@ -47,9 +47,33 @@ disable-supported: true
metrics:
- apiserver_flowcontrol_rejected_requests_total
- apiserver_flowcontrol_dispatched_requests_total
+- apiserver_flowcontrol_priority_level_seat_utilization
+- apiserver_flowcontrol_priority_level_request_utilization
+- apiserver_flowcontrol_read_vs_write_current_requests
+- apiserver_flowcontrol_current_r
+- apiserver_flowcontrol_dispatch_r
+- apiserver_flowcontrol_latest_s
+- apiserver_flowcontrol_next_s_bounds
+- apiserver_flowcontrol_next_discounted_s_bounds
- apiserver_flowcontrol_current_inqueue_requests
- apiserver_flowcontrol_request_queue_length_after_enqueue
- apiserver_flowcontrol_request_concurrency_limit
- apiserver_flowcontrol_current_executing_requests
+- apiserver_flowcontrol_request_concurrency_in_use
- apiserver_flowcontrol_request_wait_duration_seconds
- apiserver_flowcontrol_request_execution_seconds
+- apiserver_flowcontrol_watch_count_samples
+- apiserver_flowcontrol_epoch_advance_total
+- apiserver_flowcontrol_work_estimated_seats
+- apiserver_flowcontrol_request_dispatch_no_accommodation_total
+- apiserver_flowcontrol_nominal_limit_seats
+- apiserver_flowcontrol_lower_limit_seats
+- apiserver_flowcontrol_upper_limit_seats
+- apiserver_flowcontrol_demand_seats
+- apiserver_flowcontrol_demand_seats_high_watermark
+- apiserver_flowcontrol_demand_seats_average
+- apiserver_flowcontrol_demand_seats_stdev
+- apiserver_flowcontrol_demand_seats_smoothed
+- apiserver_flowcontrol_target_seats
+- apiserver_flowcontrol_seat_fair_frac
+- apiserver_flowcontrol_current_limit_seats
diff --git a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md
index 5c21990c662e..5d2fc7449907 100644
--- a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md
+++ b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/README.md
@@ -105,6 +105,8 @@ tags, and then generate with `hack/update-toc.sh`.
- [Graduation Criteria](#graduation-criteria)
- [Alpha](#alpha)
- [Beta](#beta)
+ - [GA](#ga)
+ - [Version Skew Strategy](#version-skew-strategy)
- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
- [Feature Enablement and Rollback](#feature-enablement-and-rollback)
- [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
@@ -313,11 +315,12 @@ client-side validation, albeit one that is error-prone and not officially
supported).
Long-term, we want to favor using out-of-tree solutions for client-side
-validation, though this idea is still in its infancy.
+validation. The [kubeconform](https://github.com/yannh/kubeconform) project is an example of an out-of-tree solution that does this.
-The [kubeval](https://www.kubeval.com/) project is an example of an out-of-tree solution that does this, and
-we will look into expanding its support of open API to v3, and investigate
-whether it makes sense as a permanent solution to client-side validation.
+SIG API Machinery's effort will go to producing clear and complete (and
+improving over time) API spec documents insteading of making client side
+validation. We strongly recommend that third part integrators make use of such
+documents.
##### Aligning json and yaml errors
@@ -615,6 +618,11 @@ It tests the cross product of all valid permutations along the dimensions of:
With field validation on by default in beta, we will modify
[test/e2e/kubectl/kubectl.go](https://github.com/kubernetes/kubernetes/blob/master/test/e2e/kubectl/kubectl.go) to ensure that kubectl defaults to using server side field validation and detects unknown/duplicate fields as expected.
+[GA]
+We will introduce field validation specific e2e/conformance tests to submit
+requests directly against the API server for both built-in and custom resources
+to test that duplicate and unknown fields are appropriately detected.
+
### Graduation Criteria
### Version Skew Strategy
-If applicable, how will the component handle version skew with other
-components? What are the guarantees? Make sure this is in the test plan.
+Following the graduation of server-side field validation to GA, there will be a
+period of time when a newer client (which expects server-side field validation locked in GA) will send
+requests to an older server that could have server-side field validation
+disabled.
-Consider the following in developing a version skew strategy for this
-enhancement:
-- Does this enhancement involve coordinating behavior in the control plane and
- in the kubelet? How does an n-2 kubelet without this feature available behave
- when this feature is used?
-- Will any other components on the node change? For example, changes to CSI,
- CRI or CNI may require updating that component before the kubelet.
--->
+Until version skew makes it incompatible for a client to hit a server
+with field validation disabled, kubectl must continue to check if the feature is
+enabled server-side. As long as the feature can be disabled server-side, kubectl
+must continue to have client-side validation. When the feature can't be disabled
+server-side, kubectl should remove client-side validation entirely.
+
+Given that kubectl [supports](https://kubernetes.io/releases/version-skew-policy/#kubectl) one minor version skew (older or newer) of kube-apipserver,
+this would mean that removing client-side validation in 1.28 would be
+1-version-skew-compatible with GA in 1.27 (when server-side validation can no
+longer be disabled).
+
+See section on [Out-of-Tree
+Alternatives to Client Side
+Validation](#out-of-tree-alternatives-to-client-side-validation) for
+alternatives to client-side validation
## Production Readiness Review Questionnaire
@@ -749,6 +761,8 @@ Pick one of these and delete the rest.
- [x] Feature gate (also fill in values in `kep.yaml`)
- Feature gate name: ServerSideFieldValidation
- Components depending on the feature gate: kube-apiserver
+ - Note: feature gate locked to true upon graduation to GA (1.27) and removed
+ two releases later (1.29)
- [x] Other
- Describe the mechanism: query parameter
- Will enabling / disabling the feature require downtime of the control
@@ -851,8 +865,15 @@ Recall that end users cannot usually observe component logs or access metrics.
###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
-Users should expect to see an increase in request latency and memory usage
-(~20-25% increase) for requests that desire server side schema validation.
+We have benchmark tests for field validation
+[here](https://github.com/kubernetes/kubernetes/blob/master/test/integration/apiserver/field_validation_test.go#L2936-L2996) and an analysis of those
+benchmarks summarized
+[here](https://github.com/kubernetes/kubernetes/pull/107848#issuecomment-1032216698).
+
+Based on the above, users should should expect to see negligible latency increases (with the
+exception of SMP requests that are ~5% slower), and memory that is negligible
+for JSON (and no change for protobuf), but around 10-20% more memory usage for
+YAML data.
Cluster operators can also use cpu usage monitoring to determine whether
increased usage of server-side schema validation dramatically increases CPU usage after feature enablement.
diff --git a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/kep.yaml b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/kep.yaml
index 3e69f60e4bba..26528b7ff4cd 100644
--- a/keps/sig-api-machinery/2885-server-side-unknown-field-validation/kep.yaml
+++ b/keps/sig-api-machinery/2885-server-side-unknown-field-validation/kep.yaml
@@ -18,21 +18,22 @@ see-also:
replaces:
# The target maturity stage in the current dev cycle for this KEP.
-stage: beta
+stage: stable
# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
-latest-milestone: "v1.25"
+latest-milestone: "v1.27"
# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
alpha: "v1.23"
beta: "v1.25"
- stable: "v1.26"
+ stable: "v1.27"
# The following PRR answers are required at alpha release
# List the feature gate name and the components for which it must be enabled
+# Note: feature-gate removed upon graduation to GA in 1.27
feature-gates:
- name: ServerSideFieldValidation
components:
diff --git a/keps/sig-api-machinery/2896-openapi-v3/README.md b/keps/sig-api-machinery/2896-openapi-v3/README.md
index 43b2a0a8cc62..48e7b257950d 100644
--- a/keps/sig-api-machinery/2896-openapi-v3/README.md
+++ b/keps/sig-api-machinery/2896-openapi-v3/README.md
@@ -63,6 +63,7 @@ Architecture for cross-cutting KEPs). -->
- [Proposal](#proposal)
- [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional)
- [Future Work](#future-work)
+ - [OpenAPI V2 plan](#openapi-v2-plan)
- [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
- [Paths](#paths)
@@ -73,10 +74,12 @@ Architecture for cross-cutting KEPs). -->
- [OpenAPI](#openapi)
- [Version Skew](#version-skew)
- [OpenAPI V3 Proto](#openapi-v3-proto)
+ - [Clients](#clients)
- [Test Plan](#test-plan)
- [Graduation Criteria](#graduation-criteria)
- [Alpha](#alpha)
- [Beta](#beta)
+ - [GA](#ga)
- [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
- [Version Skew Strategy](#version-skew-strategy)
- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
@@ -209,6 +212,17 @@ endpoints.
After OpenAPI v3 is implemented, we can start working on updating clients to
consume OpenAPI v3. A separate KEP will be published for that.
+#### OpenAPI V2 plan
+
+As OpenAPI V3 becomes more and more available and easily accessible,
+clients should move to OpenAPI V3 in favor of OpenAPI V2. This should
+result in a steady decline in the number of requests sent to the
+OpenAPI V2 endpoint. In the short term, we will make OpenAPI V2 make
+computations more lazily, deferring the aggregation and marshaling
+process to be on demand rather than at a set interval. This is not
+directly part of OpenAPI V3's GA plan, but is something we will be
+monitoring closely once OpenAPI V3 is GA.
+
### Risks and Mitigations
Aggregation for OpenAPI v2 already consumes a non-negligible amount of
@@ -277,6 +291,12 @@ updates. If there is a race and the client passes in an outdated etag
value, the server will send a 301 to redirect the client to the URL
with the latest etag value.
+To ensure that the client cache does not grow in an unbounded manner,
+the client-go disk cache will be replaced with a cmopactable disk
+cache which performs occasional cleanup of old cached files. Refer to
+[PR](https://github.com/kubernetes/kubernetes/pull/109701) for more
+details.
+
### Controllers
#### OpenAPI Builder
@@ -383,6 +403,16 @@ This solves the immediate protobuf issue but adds complexity to the OpenAPI sche
The final alternative is to upgrade to OpenAPI 3.1 where the new JSON Schema version it is based off of supports fields alongside a `$ref`. However, OpenAPI does not follow semvar and 3.1 is a major upgrade over 3.0 and introduces various backwards incompatible changes. Furthermore, library support is currently lacking (gnostic) and doesn't fully support OpenAPI 3.1. One important backwards incompatible change is the removal of the nullable field and replacing it by changing the type field from a single string to an array of strings.
+### Clients
+
+Updating all OpenAPI V2 clients to use OpenAPI V3 is outside the scope
+of this KEP and part of future work. However, some key clients will be
+updated to use OpenAPI V3 as part of GA. One example is [OpenAPI V3
+for kubectl
+explain](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/3515-kubectl-explain-openapiv3/README.md).
+Server Side Apply will also be updated to directly use the OpenAPI V3
+schema rather than OpenAPI V2. Finally, the query param verifier in
+kubectl will be updated to use OpenAPI V3.
### Test Plan
@@ -407,6 +437,8 @@ For alpha, unit tests will be included to ensure that v3 schemas are properly
generated and published. A validator will also be used to ensure that the spec
generated is valid OpenAPI v3.
+A fuzzer and performance benchmark tests will also be included.
+
### Graduation Criteria
#### Alpha
@@ -420,13 +452,20 @@ generated is valid OpenAPI v3.
- Incorrect OpenAPI polymorphic types (IntOrString, Quantity) are updated to use `anyOf` in OpenAPI V3
- Definition names of native resources are updated to omit their package paths
- Parameters are reused as components
-- `kubectl explain` to support using the OpenAPI v3 Schema
- Aggregated API servers are queried for their v2 endpoint and converted to
publish v3 if they do not directly publish v3
- Heuristics are used for the OpenAPI v2 to v3 conversion to maximize
correctness of published spec
- Aggregation for OpenAPI v3 will serve as a proxy to downstream OpenAPI paths
+#### GA
+
+- OpenAPI V3 uses the optimized JSON marshaler for increased performance. See [link](https://github.com/kubernetes/kube-openapi/pull/319) for the benefits with OpenAPI V2.
+- [Updated OpenAPI V3 Client Interface](https://docs.google.com/document/d/1HmqJH-yyK8WyU8V1y5z-RyMLq9-Xe8JfTTuaNkm9GZ0)
+- Direct conversion from OpenAPI V3 to SMD schema for Server Side Apply
+- HTTP Disk Cache is replaced with Compactable Disk Cache
+- Conformance tests
+
### Upgrade / Downgrade Strategy
-In order to lower memory consumption while getting a list of data and make it more predictable, we propose to use consistent streaming from the watch-cache instead of paging from etcd.
+In order to lower memory consumption while getting a list of data and make it more predictable, we propose to use streaming from the watch-cache instead of paging from etcd.
Initially, the proposed changes will be applied to informers as they are usually the heaviest users of LIST requests (see [Appendix](#appendix) section for more details on how informers operate today).
The primary idea is to use standard WATCH request mechanics for getting a stream of individual objects, but to use it for LISTs.
This would allow us to keep memory allocations constant.
@@ -266,17 +271,17 @@ plus a few additional allocations, that will be explained later in this document
The rough idea/plan is as follows:
- step 1: change the informers to establish a WATCH request with a new query parameter instead of a LIST request.
-- step 2: upon receiving the request from an informer, contact etcd to get the latest RV. It will be used to make sure the watch cache has seen objects up to the received RV. This step is necessary and ensures we will serve consistent data, even from the cache.
-- step 2a: send all objects currently stored in memory for the given resource.
+- step 2: upon receiving the request from an informer, compute the RV at which the result should be returned (possibly contacting etcd if consistent read was requested). It will be used to make sure the watch cache has seen objects up to the received RV. This step is necessary and ensures we will meet the consistency requirements of the request.
+- step 2a: send all objects currently stored in memory for the given resource type.
- step 2b: propagate any updates that might have happened meanwhile until the watch cache catches up to the latest RV received in step 2.
- step 2c: send a bookmark event to the informer with the given RV.
- step 3: listen for further events using the request from step 1.
Note: the proposed watch-list semantics (without bookmark event and without the consistency guarantee) kube-apiserver follows already in RV="0" watches.
The mode is not used in informers today but is supported by every kube-apiserver for legacy, compatibility reasons.
-A watch started with RV="0" may return stale. It is possible for the watch to start at a much older resource version that the client has previously observed, particularly in high availability configurations, due to partitions or stale caches
+A watch started with RV="0" may return stale data. It is possible for the watch to start at a much older resource version that the client has previously observed, particularly in high availability configurations, due to partitions or stale caches.
-Note 2: informers need consistent lists to avoid time-travel when switching to another HA instance of kube-apiserver with outdated/lagging watch cache.
+Note 2: informers need consistent lists to avoid time-travel when initializing after restart to avoid time travel in case of switching to another HA instance of kube-apiserver with outdated/lagging watch cache.
See the following [issue](https://github.com/kubernetes/kubernetes/issues/59848) for more details.
@@ -310,7 +315,7 @@ required) or even code snippets. If there's any ambiguity about HOW your
proposal will be implemented, this is the place to discuss them.
-->
-### Required changes for a WATCH request with the RV="" and the ResourceVersionMatch=MostRecent
+### Required changes for a WATCH request with the SendInitialEvents=true
The following sequence diagram depicts steps that are needed to complete the proposed feature.
A high-level overview of each was provided in a table that follows immediately the diagram.
@@ -328,11 +333,11 @@ Whereas further down in this section we provided a detailed description of each
2.
- The watch cache contacts etcd for the most up-to-date ResourceVersion.
+ If needed, the watch cache contacts etcd for the most up-to-date ResourceVersion.
2a.
- The watch cache starts streaming initial data. The data it already has in memory.
+ The watch cache starts streaming initial data it already has in memory.
2b.
@@ -352,14 +357,14 @@ Whereas further down in this section we provided a detailed description of each
-Step 1: On initialization the reflector gets a snapshot of data from the server by passing RV=”” (= unset value) and setting resourceVersionMatch=MostRecent (= ensure freshness).
+Step 1: On initialization the reflector gets a snapshot of data from the server by passing RV=”” (= unset value) to ensure freshness and setting resourceVersionMatch=NotOlderThan and sendInitialEvents=true.
We do that only during the initial ListAndWatch call.
Each event (ADD, UPDATE, DELETE) except the BOOKMARK event received from the server is collected.
-Passing resourceVersionMatch=MostRecent tells the cacher it has to guarantee that the cache is at least up to date as a LIST executed at the same time.
+Passing resourceVersion="" tells the cacher it has to guarantee that the cache is at least up to date as a LIST executed at the same time.
Note: This ensures that returned data is consistent, served from etcd via a quorum read and prevents "going back in time".
-Note 2: Unfortunately as of today, the watch cache is vulnerable to stale reads, see https://github.com/kubernetes/kubernetes/issues/59848 for more details.
+Note 2: Watch cache currently doesn't have the feature of supporting resourceVersion="" and thus is vulnerable to stale reads, see https://github.com/kubernetes/kubernetes/issues/59848 for more details.
Step 2: Right after receiving a request from the reflector, the cacher gets the current resourceVersion (aka bookmarkAfterResourceVersion) directly from the etcd.
It is used to make sure the cacher is up to date (has seen data stored in etcd) and to let the reflector know it has seen all initial data.
@@ -447,19 +452,52 @@ It replaces its internal store with the collected items (syncWith) and reuses th
#### API changes
-Extend the optional `ResourceVersionMatch` query parameter of `ListOptions` with the following enumeration value:
+Extend the `ListOptions` struct with the following field:
```
-const (
- // ResourceVersionMatchMostRecent matches data at the most recent ResourceVersion.
- // The returned data is consistent, that is, served from etcd via a quorum read.
- // For watch calls, it begins with synthetic "Added" events of all resources up to the most recent ResourceVersion.
- // It ends with a synthetic "Bookmark" event containing the most recent ResourceVersion.
- // For list calls, it has the same semantics as leaving ResourceVersion and ResourceVersionMatch unset.
- ResourceVersionMatchMostRecent ResourceVersionMatch = "MostRecent"
-)
+type ListOptions struct {
+ ...
+
+ // SendInitialEvents, when set together with Watch option,
+ // begin the watch stream with synthetic init events to build the
+ // whole state of all resources followed by a synthetic "Bookmark"
+ // event containing a ResourceVersion after which the server
+ // continues streaming events.
+ //
+ // When SendInitialEvents option is set, we require ResourceVersionMatch
+ // option to also be set. The semantic of the watch request is as following:
+ // - ResourceVersionMatch = NotOlderThan
+ // It starts with sending initial events for all objects (at some resource
+ // version), potentially followed by an event stream until the state
+ // becomes synced to a resource version as fresh as the one provided by
+ // the ResourceVersion option. At this point, a synthetic bookmark event
+ // is send and watch stream is continued to be send.
+ // If RV is unset, this is interpreted as "consistent read" and the
+ // bookmark event is send when the state is synced at least to the moment
+ // when request started being processed.
+ // - ResourceVersionMatch = Exact
+ // Unsupported error is returned.
+ // - ResourceVersionMatch unset (or set to any other value)
+ // BadRequest error is returned.
+ //
+ // Defaults to true if ResourceVersion="" or ResourceVersion="0" (for backward
+ // compatibility reasons) and to false otherwise.
+ SendInitialEvents bool
+}
```
+The watch bookmark marking the end of initial events stream will have a dedicated
+annotation:
+```
+"k8s.io/initial-events-end": "true"
+```
+(the exact name is subject to change during API review). It will allow clients to
+precisely figure out when the initial stream of events is finished.
+
+It's worth noting that explicitly setting SendInitialEvents to false with ResourceVersion="0"
+will result in not sending initial events, which makes the option works exactly the same
+across every potential resource version passed as a parameter.
+
#### Important optimisations
1. Avoid DeepCopying of initial data/`), they will only need
+to send a request to the aggregated endpoint to obtain all resources
+that the cluster supports.
+
###### Will enabling / using this feature result in introducing new API types?
+Yes, but these API types are not persisted.
+
###### Will enabling / using this feature result in any new calls to the cloud provider?
+No.
+
###### Will enabling / using this feature result in increasing size or count of the existing API objects?
- Estimated amount of new objects: (e.g., new Object X for every
existing Pod) -->
+No.
+
###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+No.
+
###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+No.
+
### Troubleshooting
###### What steps should be taken if SLOs are not being met to determine the problem?
+The feature can be rolled back by setting the AggregatedDiscoveryEndpoint feature flag to false.
+
## Implementation History
## Proposal
@@ -118,6 +114,10 @@ This problem can be mitigated with a fresh build of kube-controller-manager with
updated golang version, but it heavily relies on the go community to keep the
time zone database up-to-date.
+- Outdated time zone database on the system where kube-controller-manager is running
+
+Cluster administrator should always ensure their systems are up-to-date.
+
- Malicious user can create multiple CronJobs with different time zone which can
actually trigger Jobs at the exact same time
@@ -175,9 +175,9 @@ to implement this enhancement.
##### Unit tests
-- `k8s.io/kubernetes/pkg/apis/batch/validation`: `2022-06-09` - `94.4%`
-- `k8s.io/kubernetes/pkg/controller/cronjob`: `2022-06-09` - `50.8%`
-- `k8s.io/kubernetes/pkg/registry/batch/cronjob`: `2022-06-09` - `61.8%`
+- `k8s.io/kubernetes/pkg/apis/batch/validation`: `2023-01-18` - `96.0%`
+- `k8s.io/kubernetes/pkg/controller/cronjob`: `2023-01-18` - `51.7%`
+- `k8s.io/kubernetes/pkg/registry/batch/cronjob`: `2023-01-18` - `56.3%`
##### Integration tests
@@ -185,7 +185,7 @@ None.
##### e2e tests
-None.
+- [CronJob should support timezone](https://github.com/kubernetes/kubernetes/blob/db4c64b1a987675fff7a234e8633f4fd01c69530/test/e2e/apps/cronjob.go#L301): https://storage.googleapis.com/k8s-triage/index.html?sig=apps&test=should%20support%20timezone
### Graduation Criteria
@@ -203,38 +203,7 @@ None.
#### GA
-TBD
-
-
+- Add a new condition when a cronjob controller encounters an invalid time zone.
### Upgrade / Downgrade Strategy
@@ -429,6 +398,7 @@ lies.
- *2022-01-14* - Initial KEP draft
- *2022-06-09* - Updated KEP for beta promotion.
+- *2023-01-18* - Updated KEP for stable promotion.
## Drawbacks
diff --git a/keps/sig-apps/3140-TimeZone-support-in-CronJob/kep.yaml b/keps/sig-apps/3140-TimeZone-support-in-CronJob/kep.yaml
index 9db242e20fdf..90d29535f9f8 100644
--- a/keps/sig-apps/3140-TimeZone-support-in-CronJob/kep.yaml
+++ b/keps/sig-apps/3140-TimeZone-support-in-CronJob/kep.yaml
@@ -18,12 +18,12 @@ see-also:
replaces:
# The target maturity stage in the current dev cycle for this KEP.
-stage: beta
+stage: stable
# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
-latest-milestone: "v1.25"
+latest-milestone: "v1.27"
# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
diff --git a/keps/sig-apps/3329-retriable-and-non-retriable-failures/README.md b/keps/sig-apps/3329-retriable-and-non-retriable-failures/README.md
index 719d620a2b46..87a40386d145 100644
--- a/keps/sig-apps/3329-retriable-and-non-retriable-failures/README.md
+++ b/keps/sig-apps/3329-retriable-and-non-retriable-failures/README.md
@@ -230,7 +230,7 @@ thousands of nodes requires usage of pod restart policies in order
to account for infrastructure failures.
Currently, kubernetes Job API offers a way to account for infrastructure
-failures by setting `.backoffLimit > 0`. However, this mechanism intructs the
+failures by setting `.backoffLimit > 0`. However, this mechanism instructs the
job controller to restart all failed pods - regardless of the root cause
of the failures. Thus, in some scenarios this leads to unnecessary
restarts of many pods, resulting in a waste of time and computational
@@ -354,7 +354,7 @@ As a machine learning researcher, I run jobs comprising thousands
of long-running pods on a cluster comprising thousands of nodes. The jobs often
run at night or over weekend without any human monitoring. In order to account
for random infrastructure failures we define `.backoffLimit: 6` for the job.
-However, a signifficant portion of the failures happen due to bugs in code.
+However, a significant portion of the failures happen due to bugs in code.
Moreover, the failures may happen late during the program execution time. In
such case, restarting such a pod results in wasting a lot of computational time.
@@ -729,7 +729,7 @@ in different messages for pods.
- Reproduction: Run kube-controller-manager with disabled taint-manager (with the
flag `--enable-taint-manager=false`). Then, run a job with a long-running pod and
disconnect the node
-- Comments: handled by node lifcycle controller in: `controller/nodelifecycle/node_lifecycle_controller.go`.
+- Comments: handled by node lifecycle controller in: `controller/nodelifecycle/node_lifecycle_controller.go`.
However, the pod phase remains `Running`.
- Pod status:
- status: Unknown
@@ -762,7 +762,7 @@ In Alpha, there is no support for Pod conditions for failures or disruptions ini
For Beta we introduce handling of Pod failures initiated by Kubelet by adding
the pod disruption condition (introduced in Alpha) in case of disruptions
-initiated by kubetlet (see [Design details](#design-details)).
+initiated by Kubelet (see [Design details](#design-details)).
Kubelet can also evict a pod in some scenarios which are not covered with
adding a pod failure condition:
@@ -863,7 +863,7 @@ dies) between appending a pod condition and deleting the pod.
In particular, scheduler can possibly decide to preempt
a different pod the next time (or none). This would leave a pod with a
condition that it was preempted, when it actually wasn't. This in turn
-could lead to inproper handling of the pod by the job controller.
+could lead to improper handling of the pod by the job controller.
As a solution we implement a worker, part of the disruption
controller, which clears the pod condition added if `DeletionTimestamp` is
@@ -1218,7 +1218,7 @@ the pod failure does not match any of the specified rules, then default
handling of failed pods applies.
If we limit this feature to use `onExitCodes` only when `restartPolicy=Never`
-(see: [limitting this feature](#limitting-this-feature)), then the rules using
+(see: [limiting this feature](#limitting-this-feature)), then the rules using
`onExitCodes` are evaluated only against the exit codes in the `state` field
(under `terminated.exitCode`) of `pod.status.containerStatuses` and
`pod.status.initContainerStatuses`. We may also need to check for the exit codes
@@ -1279,9 +1279,9 @@ the following scenarios will be covered with unit tests:
- handling of a pod failure, in accordance with the specified `spec.podFailurePolicy`,
when the failure is associated with
- a failed container with non-zero exit code,
- - a dedicated Pod condition indicating termmination originated by a kubernetes component
+ - a dedicated Pod condition indicating termination originated by a kubernetes component
- adding of the `DisruptionTarget` by Kubelet in case of:
- - eviciton due to graceful node shutdown
+ - eviction due to graceful node shutdown
- eviction due to node pressure
-The following scenario will be covered with e2e tests:
-- early job termination when a container fails with a non-retriable exit code
+The following scenario are covered with e2e tests:
+- [sig-apps#gce](https://testgrid.k8s.io/sig-apps#gce):
+ - Job Using a pod failure policy to not count some failures towards the backoffLimit Ignore DisruptionTarget condition
+ - Job Using a pod failure policy to not count some failures towards the backoffLimit Ignore exit code 137
+ - Job should allow to use the pod failure policy on exit code to fail the job early
+ - Job should allow to use the pod failure policy to not count the failure towards the backoffLimit
+- [sig-scheduling#gce-serial](https://testgrid.k8s.io/sig-scheduling#gce-serial):
+ - SchedulerPreemption [Serial] validates pod disruption condition is added to the preempted pod
+- [sig-release-master-informing#gce-cos-master-serial](https://testgrid.k8s.io/sig-release-master-informing#gce-cos-master-serial):
+ - NoExecuteTaintManager Single Pod [Serial] pods evicted from tainted nodes have pod disruption condition
+
+The following scenarios are covered with node e2e tests
+([sig-node-presubmits#pr-kubelet-gce-e2e-pod-disruption-conditions](https://testgrid.k8s.io/sig-node-presubmits#pr-kubelet-gce-e2e-pod-disruption-conditions) and
+[sig-node-presubmits#pr-node-kubelet-serial-containerd](https://testgrid.k8s.io/sig-node-presubmits#pr-node-kubelet-serial-containerd)):
+ - GracefulNodeShutdown [Serial] [NodeFeature:GracefulNodeShutdown] [NodeFeature:GracefulNodeShutdownBasedOnPodPriority] graceful node shutdown when PodDisruptionConditions are enabled [NodeFeature:PodDisruptionConditions] should add the DisruptionTarget pod failure condition to the evicted pods
+ - PriorityPidEvictionOrdering [Slow] [Serial] [Disruptive][NodeFeature:Eviction] when we run containers that should cause PIDPressure; PodDisruptionConditions enabled [NodeFeature:PodDisruptionConditions] should eventually evict all of the correct pods
More e2e test scenarios might be considered during implementation if practical.
@@ -1439,7 +1453,7 @@ N/A
An upgrade to a version which supports this feature should not require any
additional configuration changes. In order to use this feature after an upgrade
users will need to configure their Jobs by specifying `spec.podFailurePolicy`. The
-only noticeable difference in behaviour, without specifying `spec.podFailurePolicy`,
+only noticeable difference in behavior, without specifying `spec.podFailurePolicy`,
is that Pods terminated by kubernetes components will have an additional
condition appended to `status.conditions`.
@@ -1654,7 +1668,7 @@ Manual test performed to simulate the upgrade->downgrade->upgrade scenario:
- Scenario 2:
- Create a job with a long running containers and `backoffLimit=0`.
- Verify that the job continues after the node in uncordoned
-1. Disable the feature gates. Verify that the above scenarios result in default behaviour:
+1. Disable the feature gates. Verify that the above scenarios result in default behavior:
- In scenario 1: the job restarts pods failed with exit code `42`
- In scenario 2: the job is failed due to exceeding the `backoffLimit` as the failed pod failed during the draining
1. Re-enable the feature gates
@@ -1953,7 +1967,7 @@ technics apply):
is an increase of the Job controller processing time.
- Inspect the Job controller's `job_pods_finished_total` metric for the
to check if the numbers of pod failures handled by specific actions (counted
- by the `failure_policy_action` label) agree with the expetations.
+ by the `failure_policy_action` label) agree with the expectations.
For example, if a user configures job failure policy with `Ignore` action for
the `DisruptionTarget` condition, then a node drain is expected to increase
the metric for `failure_policy_action=Ignore`.
@@ -1963,7 +1977,7 @@ technics apply):
- 2022-06-23: Initial KEP merged
- 2022-07-12: Preparatory PR "Refactor gc_controller to do not use the deletePod stub" merged
-- 2022-07-14: Preparatory PR "efactor taint_manager to do not use getPod and getNode stubs" merged
+- 2022-07-14: Preparatory PR "Refactor taint_manager to do not use getPod and getNode stubs" merged
- 2022-07-20: Preparatory PR "Add integration test for podgc" merged
- 2022-07-28: KEP updates merged
- 2022-08-01: Additional KEP updates merged
@@ -1972,7 +1986,7 @@ technics apply):
- 2022-08-04: PR "Support handling of pod failures with respect to the configured rules" merged
- 2022-09-09: Bugfix PR for test "Fix the TestRoundTripTypes by adding default to the fuzzer" merged
- 2022-09-26: Prepared PR for KEP Beta update. Summary of the changes:
- - propsal to extend kubelet to add the following pod conditions when evicting a pod (see [Design details](#design-details)):
+ - proposal to extend kubelet to add the following pod conditions when evicting a pod (see [Design details](#design-details)):
- DisruptionTarget for evictions due graceful node shutdown, admission errors, node pressure or Pod admission errors
- ResourceExhausted for evictions due to OOM killer and exceeding Pod's ephemeral-storage limits
- extended the review of pod eviction scenarios by kubelet-initiated pod evictions:
diff --git a/keps/sig-apps/3715-elastic-indexed-job/README.md b/keps/sig-apps/3715-elastic-indexed-job/README.md
new file mode 100644
index 000000000000..cc51802e97c2
--- /dev/null
+++ b/keps/sig-apps/3715-elastic-indexed-job/README.md
@@ -0,0 +1,954 @@
+
+# KEP-3715: Elastic Indexed Job
+
+
+
+
+
+
+- [Release Signoff Checklist](#release-signoff-checklist)
+- [Summary](#summary)
+- [Motivation](#motivation)
+ - [Goals](#goals)
+ - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+ - [User Stories (Optional)](#user-stories-optional)
+ - [Story 1](#story-1)
+ - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+ - [Test Plan](#test-plan)
+ - [Prerequisite testing updates](#prerequisite-testing-updates)
+ - [Unit tests](#unit-tests)
+ - [Integration tests](#integration-tests)
+ - [e2e tests](#e2e-tests)
+ - [Graduation Criteria](#graduation-criteria)
+ - [Beta](#beta)
+ - [GA](#ga)
+ - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+ - [Version Skew Strategy](#version-skew-strategy)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+ - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+ - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+ - [Monitoring Requirements](#monitoring-requirements)
+ - [Dependencies](#dependencies)
+ - [Scalability](#scalability)
+ - [Troubleshooting](#troubleshooting)
+- [Implementation History](#implementation-history)
+- [Alternatives](#alternatives)
+ - [Use StatefulSets](#use-statefulsets)
+ - [Make spec.completions fully mutable](#make-speccompletions-fully-mutable)
+ - [Allow spec.completions=nil for Indexed mode](#allow-speccompletionsnil-for-indexed-mode)
+
+
+## Release Signoff Checklist
+
+
+
+Items marked with (R) are required *prior to targeting to a milestone / release*.
+
+- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [ ] (R) KEP approvers have approved the KEP status as `implementable`
+- [x] (R) Design details are appropriately documented
+- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+ - [ ] e2e Tests for all Beta API Operations (endpoints)
+ - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+ - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+- [x] (R) Graduation criteria is in place
+ - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+- [ ] (R) Production readiness review completed
+- [ ] (R) Production readiness review approved
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://git.k8s.io/enhancements
+[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
+[kubernetes/website]: https://git.k8s.io/website
+
+## Summary
+
+Currently `spec.completions` is an immutable field in Job for both `Indexed` and `NonIndexed` modes.
+This KEP proposes to allow mutating `spec.completions` for `Indexed` job iff `spec.completions` equals
+to `spec.parallelism` before and after the update, meaning that `spec.completions` is mutable only
+in tandem with `spec.parallelism`.
+
+## Motivation
+
+There are cases where a batch workload requires an autoscaled indexed job with stable
+DNS pod names, for example MPI Horovord, Ray, PyTorch etc. This is currently not possible
+because `spec.completions`, which controls the range of indexes in `Indexed` job, is immutable.
+While such workloads could be modeled as a StatefulSet, the Job API is a better fit because it
+offers batch-specific features such as allowing indexes to run to completion, and [pod and container failure handling](https://kubernetes.io/docs/concepts/workloads/controllers/job/#handling-pod-and-container-failures).
+
+In the above mentioned cases, there is no distinction between the concept of "how many completed
+indexes do I need" (i.e., `spec.completions`) and the concept of "how many pods do I want running
+at a time" (i.e., `spec.parallelism`). All indexes are expected to start at the same time, but they
+can finish at different times.
+
+The above behavior can be modeled using an Indexed job with
+- `spec.completions` equal to `spec.parallelism`
+- mutable `spec.completions` and `spec.parallelsim` as long as they are mutated together.
+
+
+
+### Goals
+
+- Allow mutating `spec.completions` for `Indexed` job iff the updated value is equal to `spec.parallelism`.
+
+
+
+### Non-Goals
+
+- Change the existing behavior of `NonIndexed` mode.
+- Change the existing behavior of `Indexed` mode for jobs that never mutate
+ `.spec.completions` and `.spec.parallelism` together.
+
+
+
+## Proposal
+
+The proposal is to allow mutating `spec.completions` for `Indexed` Job when equal to and updated
+in tandem with `spec.parallelsim`.
+
+Success and failure semantics are not changed for jobs that never mutate `spec.completions`;
+however, for ones that do, the semantics are changed in the following manner:
+
+Failures will always count. If `spec.completions` was scaled down, then previous pod failures
+outside the new range will still count against the job's backoffLimit. Morever, the field `status.Failed` will not be
+decremented if previously failed pods are now outside the range. However, `status.Succeeded` will be updated to reflect
+the number of successful indexes within the range as defined below.
+
+In the case where an index successfully completes, and then the job is scaled down rendering the index
+out of range, and then the job is scaled up again including back the index in the range, then the index
+will be restarted.
+
+
+
+
+### User Stories (Optional)
+
+
+
+#### Story 1
+
+I have a batch workload that requires two sets of pods: a manager and a group of workers that can scale up or down.
+The manager communicates with the workers to manage their load; examples of such workloads include distributed HPC
+and districuted ML training. Examples of frameworks include Horovod with MPI, Spark and Ray.
+
+This workload can be modeled as two `Indexed` Jobs, one for the manager and one for the workers. A
+headless service is created to set up stable hostnames for the worker pods. The workers job is
+scaled up/down by updating `spec.completions` and `spec.parallelism` in tandem.
+
+The success semantics of these workloads are tied to the manager, not the workers. However, in
+such workloads, when the job is finished the workers are either deleted directly (effectively scaled down to 0)
+or they potentially exit on their own after communicating their final status to the manager who decides the final status
+of the whole workload.
+
+
+
+
+### Risks and Mitigations
+
+There is a concern that changing completion semantics on the fly can be subject to
+race conditions with declaring the job finished. However, after a closer look, this
+seems to not be an issue because the job controller uses Update when setting the
+finished status on the job object, which is subject to object modified errors if the spec
+changes at the same time. Moreover, `spec.completions` will be immutable once a job has a
+finished condition.
+
+
+
+## Design Details
+
+The controller already counts the number of completed indexes within the range defined by `spec.completions` and
+uses that to evaluate if the job completed successfully and to set the `status.Succeeded` count. This is not impacted
+if `spec.completions` changes over time. Morever, on scale down, the controller also automatically removes any indexes
+outside the range.
+
+As mentioned above, pod failures may have happened to indexes that are outside the range after `spec.completions`
+is updated. Those failures will still count against the backoffLimit of the job and the field `status.Failed` will
+not be decremented.
+
+The only needed change is to modify Job update validation in the kube-apiserver to allow updating an `Indexed` Job's
+`spec.completions` if `spec.completions=spec.parallelism` in both the old and new instance of the Job object.
+
+
+
+### Test Plan
+
+
+
+[x] I/we understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+##### Prerequisite testing updates
+
+
+
+##### Unit tests
+
+
+
+
+
+- `k8s.io/kubernetes/pkg/controller/job`: `1/10/2023` - `90%`
+- `k8s.io/kubernetes/pkg/apis/batch/validation`: `1/10/2023` - `96%`
+
+##### Integration tests
+
+For an `Indexed` Job test that:
+- `spec.completions` is only mutable if equal to `spec.parallelism`
+- When scaled down, higher indexes are removed first.
+- When scaled down to zero, the job is marked as finished successfully.
+- When scaled up, the range is extended and new higher indexes are created.
+- Success, failure and completion semantics are as discussed above.
+
+
+
+
+
+##### e2e tests
+
+Coverage is achieved via integration tests. But we will add one e2e test:
+- Job is allowed to scale down, then up, then successfully finish when all pods in the new range exit with success
+
+
+
+
+
+### Graduation Criteria
+
+Since the change is only relaxing validation that doesn't require multiple releases and
+requires no changes to the job controller, we will start the feature in beta and enabled
+by default since we will not get a lot of value from an alpha release. Just like we
+successfully did for [mutable scheduling directives](https://github.com/kubernetes/enhancements/issues/2926).
+
+Note that if tests reveals a required change that invalidates the above understanding, then we will
+revert and start in Alpha.
+
+#### Beta
+- Validation logic in place to allow mutating `spec.completions` in tandem with `.spec.parallelism`.
+
+#### GA
+- Fix any potentially reported bugs.
+
+
+
+
+### Upgrade / Downgrade Strategy
+
+N/A, when the feature is enabled, mutating completions will be allowed, when disabled (via
+a downgrade or the feature flag) mutating completions will be rejected.
+
+
+
+### Version Skew Strategy
+
+N/A. This feature doesn't impact nodes.
+
+
+
+## Production Readiness Review Questionnaire
+
+
+
+### Feature Enablement and Rollback
+
+The feature can be safely rolled back.
+
+
+
+###### How can this feature be enabled / disabled in a live cluster?
+
+
+
+- [x] Feature gate (also fill in values in `kep.yaml`)
+ - Feature gate name: ElasticIndexedJob
+ - Components depending on the feature gate: kube-apiserver
+- [ ] Other
+ - Describe the mechanism:
+ - Will enabling / disabling the feature require downtime of the control
+ plane?
+ - Will enabling / disabling the feature require downtime or reprovisioning
+ of a node? (Do not assume `Dynamic Kubelet Config` feature is enabled).
+
+###### Does enabling the feature change any default behavior?
+
+No.
+
+
+
+
+###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
+Yes. If disabled, kube-apiserver will reject mutating requests to `spec.completions` for `Indexed` jobs.
+For Jobs that were previously mutated, then the only implication is that future mutating requests will be
+rejected.
+
+
+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+The api-server will accept mutation requests to `spec.completions` for `Indexed` job.
+
+###### Are there any tests for feature enablement/disablement?
+
+N/A
+
+
+
+### Rollout, Upgrade and Rollback Planning
+
+
+
+###### How can a rollout or rollback fail? Can it impact already running workloads?
+
+The change is opt-in, it doesn't impact already running workloads.
+
+
+
+###### What specific metrics should inform a rollback?
+
+N/A
+
+
+
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+
+It will be tested manually prior to beta launch.
+
+
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+
+No.
+
+
+
+### Monitoring Requirements
+
+
+
+###### How can an operator determine if the feature is in use by workloads?
+
+Attempt to mutate `spec.completions` for `Indexed` job.
+
+
+
+###### How can someone using this feature know that it is working for their instance?
+
+
+
+
+- [ ] Events
+ - Event Reason:
+- [ ] API .status
+ - Condition name:
+ - Other field:
+- [x] Other (treat as last resort)
+ - Details: `spec.completions` for `Indexed` jobs are successfully mutated according to the semantics described above.
+
+###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
+
+N/A
+
+
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+
+
+- [X] Metrics
+ - Metric name: apiserver_request_total[resource=job, group=batch, verb=UPDATE, code=400]
+ - [Optional] Aggregation method:
+ - Components exposing the metric:
+- [ ] Other (treat as last resort)
+ - Details:
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+
+
+
+### Dependencies
+
+
+
+###### Does this feature depend on any specific services running in the cluster?
+
+No.
+
+
+
+### Scalability
+
+
+
+###### Will enabling / using this feature result in any new API calls?
+
+Not on of itself.
+
+
+
+###### Will enabling / using this feature result in introducing new API types?
+
+No.
+
+
+
+###### Will enabling / using this feature result in any new calls to the cloud provider?
+
+No.
+
+
+
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+
+No.
+
+
+
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+
+No.
+
+
+
+###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+
+No.
+
+
+
+### Troubleshooting
+
+
+
+###### How does this feature react if the API server and/or etcd is unavailable?
+
+Create requests will be rejected.
+
+###### What are other known failure modes?
+
+In a multi-master setup, when the cluster has skewed apiservers, some create requests
+may get accepted and some may get rejected.
+
+Detection: failed update requests; metric that an operator can monitor is apiserver_request_total[resource=job, group=batch, verb=UPDATE, code=403]
+Diagnostics: apiserver logs indicating rejected/accepted job update requests
+Testing: no testing, this is a transient state until all instances are on the same k8s version.
+
+
+
+###### What steps should be taken if SLOs are not being met to determine the problem?
+
+N/A.
+
+## Implementation History
+
+- 2023-01-10: Proposed KEP.
+
+
+
+
+
+## Alternatives
+
+### Use StatefulSets
+Jobs have different success and retry semantics compared to StatefulSet that makes it better suited
+for batch workloads.
+
+
+### Make spec.completions fully mutable
+Making `spec.completions` fully mutable has no clear use case, and it is in theory a larger scope change.
+Scaling a job is not related to how many pods I want completed, it is more about how many pods I want active,
+and so tying it to parallelism makes sense.
+
+### Allow spec.completions=nil for Indexed mode
+`spec.completions=nil` is a special case that currently allowed only for `NonIndexed` mode.
+The success and completions semantics where designed to address the [work queue pattern](https://kubernetes.io/docs/tasks/job/fine-parallel-processing-work-queue/);
+specifically, a job is marked as successful if at least one pod exits successfully, and all
+remaining pods complete and the number of failures do not exceed `spec.backoffLimit`.
+Currently `spec.completions=nil` is not allowed for `Indexed` mode, and so an alternative is to allow it and use `spec.parallelism` to
+define the range of indexes. However, this has the following drawbacks:
+- the success semantics are not compatible with the use cases we are targeting
+- falling back to `spec.parallelism` to define the range of indexes changes how job completion criteria works for `Indexed` job.
+- Relaxing validation to allow for a nil value may not be a safe change to API clients that expected completions to not be nil.
+
+While we could change the success criteria to work better for our use case, and we could do the change under a new `spec.completionMode`
+to ensure that the change is safe to clients, making `spec.completions` mutable seems like the clearer API.
+
+
+
+
+
diff --git a/keps/sig-apps/3715-elastic-indexed-job/kep.yaml b/keps/sig-apps/3715-elastic-indexed-job/kep.yaml
new file mode 100644
index 000000000000..b7fe39bcb058
--- /dev/null
+++ b/keps/sig-apps/3715-elastic-indexed-job/kep.yaml
@@ -0,0 +1,38 @@
+title: Elastic Indexed Job
+kep-number: 3715
+authors:
+ - "@ahg-g"
+owning-sig: sig-apps
+status: implementable
+creation-date: 2023-01-10
+reviewers:
+ - "@alculquicondor"
+ - "@liggitt"
+approvers:
+ - "@soltysh"
+
+see-also:
+ - "/keps/sig-apps/2214-indexed-job"
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: beta
+
+# The most recent milestone for which work toward delivery of this KEP has been
+# done. This can be the current (upcoming) milestone, if it is being actively
+# worked on.
+latest-milestone: "v1.27"
+
+# The milestone at which this feature was, or is targeted to be, at each stage.
+milestone:
+ beta: "1.27"
+ stable: "v1.28"
+
+feature-gates:
+ - name: ElasticIndexedJob
+ components:
+ - kube-apiserver
+disable-supported: true
+
+# The following PRR answers are required at beta release
+metrics:
+
diff --git a/keps/sig-auth/3257-trust-anchor-sets/README.md b/keps/sig-auth/3257-trust-anchor-sets/README.md
index 0913484d6273..0b8611daebb0 100644
--- a/keps/sig-auth/3257-trust-anchor-sets/README.md
+++ b/keps/sig-auth/3257-trust-anchor-sets/README.md
@@ -319,11 +319,11 @@ that happens.
### Risks and Mitigations
-Scalability: In the limit, ClusterTrustBundle objects will be used by
-every pod in the cluster, which will require watches from all Kubelets. When
-they are updated, workloads will need to receive the updates fairly quickly
-(within 5 minutes across the whole cluster), to accommodate emergency rotation
-of trust anchors for a private CA.
+Scalability: In the limit, ClusterTrustBundle objects will be used by every pod
+in the cluster, which will require one ClusterTrustBundle watch from each
+Kubelet in the cluster. When they are updated, workloads will need to receive
+the updates fairly quickly (within 5 minutes across the whole cluster), to
+accommodate emergency rotation of trust anchors for a private CA.
Security: Should individual trust anchor set entries designate an OSCP endpoint
to check for certificate revociation? Or should we require the URL to be
@@ -853,13 +853,8 @@ and creating new ones, as well as about cluster-level services (e.g. DNS):
Yes.
-A pod that uses a pemTrustAnchors projected volume will result in an additional
-watch on the named ClusterTrustBundle object orginating from Kubelet. This
-watch will be low-throughput.
-
-Similar to the existing kubelet watches on secrets and configmaps, special care
-will need to be taken to ensure that kube-apiserver can efficiently choose which
-single-ClusterTrustBundle watches to update for a given etcd update.
+Kubelet will open a watch on ClusterTrustBundle objects. This watch will be
+low-throughput.
###### Will enabling / using this feature result in introducing new API types?
@@ -894,12 +889,11 @@ arbitrary startup latency for my pod and cause an SLO breach.)
Use of the ClusterTrustBundle objects by themselves should have negligible
resource impact.
-Use of the pemTrustAnchors projected volume type will result in an additional
-watch on kube-apiserver for each unique (Node, ClusterTrustBundle) tuple in the
-cluster. This is similar to existing kubelet support for projecting configmaps
-and secrets into a pod. Just like for secrets and configmaps, we will need to
-make sure that kube-apiserver has the indexes it needs to efficiently map etcd
-watch events to Kubernetes watch channels.
+When the feature gate is enabled, Kubelet will open a watch on all
+ClusterTrustBundle objects in the cluster. We expect there to be a low number
+of ClusterTrustBundle objects that does not scale with the number of nodes or
+workloads in the cluster, although individual ClusterTrustBundle objects could
+be large.
### Troubleshooting
diff --git a/keps/sig-auth/3257-trust-anchor-sets/kep.yaml b/keps/sig-auth/3257-trust-anchor-sets/kep.yaml
index 9d24d2261bfb..e4fb03991332 100644
--- a/keps/sig-auth/3257-trust-anchor-sets/kep.yaml
+++ b/keps/sig-auth/3257-trust-anchor-sets/kep.yaml
@@ -8,9 +8,11 @@ participating-sigs:
status: implementable
creation-date: 2022-02-16
reviewers:
- - TBD
+ - liggitt
+ - enj
approvers:
- - TBD
+ - liggitt
+ - enj
##### WARNING !!! ######
# prr-approvers has been moved to its own location
@@ -25,11 +27,11 @@ stage: alpha
# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
-latest-milestone: "v1.26"
+latest-milestone: "v1.27"
# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
- alpha: "v1.26"
+ alpha: "v1.27"
beta: ""
stable: ""
diff --git a/keps/sig-auth/3325-self-subject-attributes-review-api/README.md b/keps/sig-auth/3325-self-subject-attributes-review-api/README.md
index da707967e3c4..3e3892e12a8c 100644
--- a/keps/sig-auth/3325-self-subject-attributes-review-api/README.md
+++ b/keps/sig-auth/3325-self-subject-attributes-review-api/README.md
@@ -243,16 +243,33 @@ We expect no non-infra related flakes in the last month as a GA graduation crite
#### Alpha
+- `SelfSubjectReview` endpoint is introduced in `authentication.k8s.io/v1alpha1` API
- Feature implemented behind a feature flag
- Initial unit and integration tests completed and enabled
+- Corresponding kubectl command implemented: `kubectl alpha auth whoami`
#### Beta
- Gather feedback from users
+- `SelfSubjectReview` is promoted to `authentication.k8s.io/v1beta1` API (Beta APIs are not enabled by default, [see](https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/3136-beta-apis-off-by-default/README.md)).
+- Promote feature gate to Beta and make it enabled by default
+- Unit tests coverage improved
+- `kubectl alpha auth whoami` command uses `authentication.k8s.io/v1beta1` API, falls back to `authentication.k8s.io/v1alpha1` API
+- Fix [documentation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#self-subject-review):
+ - Change API version
+ - Rewrite conditions to enable the feature
#### GA
-- Corresponding kubectl command implemented
+- `SelfSubjectReview` is promoted to `authentication.k8s.io/v1` API and enable by default
+- Promote feature gate to Stable
+- `kubectl alpha auth whoami` replaced with `kubectl auth whoami`
+- `kubectl auth whoami` command prefers `authentication.k8s.io/v1` API over `authentication.k8s.io/v1beta1` and `authentication.k8s.io/v1alpha1`
+- More integration and e2e tests cases
+- Fix [documentation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#self-subject-review):
+ - Change API version
+ - Rewrite conditions to enable the feature
+ - Change kubectl command
NOTE: Should not be a part of [conformance tests](https://git.k8s.io/community/contributors/devel/sig-architecture/conformance-tests.md).
The fact that a user possesses a token does not necessarily imply the power to know to whom that token belongs.
@@ -263,22 +280,9 @@ The fact that a user possesses a token does not necessarily imply the power to k
###### How can this feature be enabled / disabled in a live cluster?
-
-
-- Feature gate
+- [X] Feature gate (also fill in values in `kep.yaml`)
- Feature gate name: `APISelfSubjectReview`
- - Components depending on the feature gate:
- - kube-apiserver
-
-```go
-FeatureSpec{
- Default: false,
- LockToDefault: false,
- PreRelease: featuregate.Alpha,
-}
-```
+ - Components depending on the feature gate: `kube-apiserver`
###### Does enabling the feature change any default behavior?
diff --git a/keps/sig-auth/3325-self-subject-attributes-review-api/kep.yaml b/keps/sig-auth/3325-self-subject-attributes-review-api/kep.yaml
index 5a40f31de61b..303aff5f4e54 100755
--- a/keps/sig-auth/3325-self-subject-attributes-review-api/kep.yaml
+++ b/keps/sig-auth/3325-self-subject-attributes-review-api/kep.yaml
@@ -10,12 +10,14 @@ reviewers:
- "@enj"
- "@deads2k"
- "@mikedanese"
+- "@liggitt"
approvers:
-- TBD
+- "@deads2k"
+- "@liggitt"
creation-date: "2022-05-30"
status: implementable
-stage: alpha
-latest-milestone: "v1.26"
+stage: beta
+latest-milestone: "v1.27"
milestone:
alpha: "v1.26"
beta: "v1.27"
diff --git a/keps/sig-auth/3766-referencegrant/README.md b/keps/sig-auth/3766-referencegrant/README.md
new file mode 100644
index 000000000000..587751c04060
--- /dev/null
+++ b/keps/sig-auth/3766-referencegrant/README.md
@@ -0,0 +1,941 @@
+# KEP-3766: Move ReferenceGrant to SIG Auth API Group
+
+
+- [Release Signoff Checklist](#release-signoff-checklist)
+- [Summary](#summary)
+- [Motivation](#motivation)
+ - [Goals](#goals)
+ - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+ - [Risks and Mitigations](#risks-and-mitigations)
+ - [No Default Implementation](#no-default-implementation)
+ - [Potential for Variations Among Implementations](#potential-for-variations-among-implementations)
+ - [Cross-Namespace References may Weaken Namespace Boundaries](#cross-namespace-references-may-weaken-namespace-boundaries)
+- [Design Details](#design-details)
+ - [General Notes](#general-notes)
+ - [ReferenceGrant is half of a handshake](#-is-half-of-a-handshake)
+ - [ReferenceGrant authors must have sufficient access](#referencegrant-authors-must-have-sufficient-access)
+ - [Resource vs Kind](#-vs-)
+ - [Revocation behavior](#revocation-behavior)
+ - [Example Usage](#example-usage)
+ - [Gateway API Gateway Referencing Secret](#gateway-api-gateway-referencing-secret)
+ - [Gateway API HTTPRoute Referencing Service](#gateway-api-httproute-referencing-service)
+ - [PersistentVolumeClaim using cross namespace data source](#persistentvolumeclaim-using-cross-namespace-data-source)
+ - [API Spec](#api-spec)
+ - [Outstanding questions and clarifications](#outstanding-questions-and-clarifications)
+ - [Test Plan](#test-plan)
+ - [Prerequisite testing updates](#prerequisite-testing-updates)
+ - [Unit tests](#unit-tests)
+ - [Integration tests](#integration-tests)
+ - [e2e tests](#e2e-tests)
+ - [Graduation Criteria](#graduation-criteria)
+ - [Beta](#beta)
+ - [GA](#ga)
+ - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+ - [Version Skew Strategy](#version-skew-strategy)
+- [Open Questions](#open-questions)
+ - [Do We Need Verbs?](#do-we-need-verbs)
+ - [Do We Need Any or Wildcard Selectors?](#do-we-need-any-or-wildcard-selectors)
+ - [Do We Need Label Selectors?](#do-we-need-label-selectors)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+ - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+ - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+ - [Monitoring Requirements](#monitoring-requirements)
+ - [Dependencies](#dependencies)
+ - [Scalability](#scalability)
+ - [Troubleshooting](#troubleshooting)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+
+
+## Release Signoff Checklist
+
+Items marked with (R) are required *prior to targeting to a milestone / release*.
+
+- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [ ] (R) KEP approvers have approved the KEP status as `implementable`
+- [ ] (R) Design details are appropriately documented
+- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+ - [ ] e2e Tests for all Beta API Operations (endpoints)
+ - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+ - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+- [ ] (R) Graduation criteria is in place
+ - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+- [ ] (R) Production readiness review completed
+- [ ] (R) Production readiness review approved
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://git.k8s.io/enhancements
+[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
+[kubernetes/website]: https://git.k8s.io/website
+
+## Summary
+
+[ReferenceGrant](https://gateway-api.sigs.k8s.io/api-types/referencegrant/) was
+developed by the [Gateway API subproject](https://gateway-api.sigs.k8s.io/) to
+enable certain object references to cross namespaces. More recently, it has also
+been [used by
+sig-storage](https://kubernetes.io/blog/2023/01/02/cross-namespace-data-sources-alpha/)
+to enable cross-namespace data sources.
+
+This KEP proposes moving ReferenceGrant from its current
+`gateway.networking.k8s.io` API group into the `authorization.k8s.io` API
+group.
+
+## Motivation
+
+Any project that wants to enable cross-namespace references currently has to choose
+between introducing a dependency on Gateway API's ReferenceGrant or creating a
+new API that would be partially redundant (leading to confusion for users).
+
+Recent interest between SIGs has made it clear that ReferenceGrant is wanted for use
+cases other than Gateway API. We would like to move ReferenceGrant to a neutral home
+(ideally, under SIG Auth) in order to make it the canonical API for managing references
+across namespaces.
+
+### Goals
+
+* Move ReferenceGrant to an API Group that SIG Auth manages
+* Clearly define how ReferenceGrant should be used, including both current use
+ cases and guidance for future use cases
+* Implement a library to ensure that ReferenceGrant is implemented consistently
+ by all controllers
+
+### Non-Goals
+
+* Develop an authorizer that will automatically implement ReferenceGrant for all
+ use cases. (It would be impossible to represent concepts like "all namespaces"
+ or label selectors that have become important for this KEP).
+
+## Proposal
+
+Move the existing ReferenceGrant resource into a new
+`authorization.k8s.io` API group, defined within the Kubernetes code base
+as part of the 1.27 release.
+
+We will take this opportunity to clarify and update the API after SIG Auth
+feedback. This resource will start with v1alpha1 as the API version.
+
+
+### Risks and Mitigations
+
+#### No Default Implementation
+Similar to the Ingress and Gateway APIs, this API will be dependent on
+implementations by controllers that are not included by default in Kubernetes.
+This could lead to confusion for users. We'll need to rely heavily on
+documentation for this feature, tracking all uses of official Kubernetes APIs
+that support ReferenceGrant in a central place.
+
+#### Potential for Variations Among Implementations
+Because this relies on each individual controller to implement the logic,
+it is possible that implementations may become inconsistent. To avoid that,
+we'll provide a standard library for implementing ReferenceGrant. We'll
+also strongly recommend that every API that relies on ReferenceGrant
+includes robust conformance tests covering this functionality. Existing
+Gateway API conformance tests can serve as a model for this.
+
+#### Cross-Namespace References may Weaken Namespace Boundaries
+Although we believe that the handshake required for cross-namespace references
+with ReferenceGrant ensures these references will be safe, it does potentially
+weaken existing namespace boundaries. We believe ReferenceGrant will have a net
+benefit on the ecosystem as it will allow workloads, secrets, and configuration
+to be deployed in separate namespaces that more clearly match up with desired
+authorization.
+
+## Design Details
+
+### General Notes
+
+#### `ReferenceGrant` is half of a handshake
+
+When thinking about ReferenceGrant, it is important to remember that it does not
+do anything by itself. It *Grants* the *possibility* of making a *Reference*
+across namespaces. It's intended that _another object_ (that is, the From object)
+complete the handshake by creating a reference to the referent object (the To
+object).
+
+#### ReferenceGrant authors must have sufficient access
+
+Anyone creating or updating a ReferenceGrant MUST have read access to the resources
+they are providing access to. If that authorization check fails, the update or
+create action will also fail.
+
+<<[UNRESOLVED Do we need checks beyond read access? ]>>
+Previous Discussion: https://github.com/kubernetes/enhancements/pull/3767#discussion_r1084528657
+
+Comment that started that thread:
+
+does anything ensure the user creating the ReferenceGrant has permissions (read?
+write?) on the object they are granting access to? Translating the existing
+ReferenceGrant into an authz check means translating from Kind to Resource
+
+since this is extending "half of a handshake", it seems important to ensure the
+actor extending the handshake actually has permission to do so
+
+<<[/UNRESOLVED]>>
+
+#### `Resource` vs `Kind`
+
+When creating a metaresource (that is, a resource that targets other resources)
+like ReferenceGrant, it's important to consider if the metaresource uses the more
+common `Kind` or the more correct `Resource`.
+
+In the original Gateway API implementation, we chose to use `Kind` rather than
+`Resource`, mainly to improve the user experience. That is, it's easier users
+to take the value from the `kind` field at the top of the YAML they are already
+using, and put it straight into these fields, rather than needing to do a
+kind-resource lookup for every user's interaction with the API. @robscott even
+ended up making https://github.com/kubernetes/community/pull/5973 to clarify
+the API conventions.
+
+However, in discussion on this KEP, it's clear that the more generic nature of
+_this_ API requires the additional specificity that `Resource` provides.
+
+The Gateway API ReferenceGrant looked like this:
+```yaml
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: ReferenceGrant
+metadata:
+ name: allow-gateways
+ namespace: bar
+spec:
+ from:
+ # Note that in Gateway API, Group is currently defaulted
+ # to this, which means you to explicitly set the group to
+ # the empty string for Core resources. We should definitely
+ # change this.
+ - group: "gateway.networking.kubernetes.io"
+ kind: Gateway
+ namespace: foo
+ to:
+ - group: ""
+ kind: Secret
+```
+
+The new version will look like this instead:
+```yaml
+apiVersion: authorization.k8s.io/v1alpha1
+kind: ReferenceGrant
+metadata:
+ name: allow-gateways
+ namespace: bar
+spec:
+ from:
+ # Assuming that we leave the default for Group to the empty
+ # string, so that Core objects don't need additional config.
+ - group: "gateway.networking.kubernetes.io"
+ resource: gateways
+ namespace: foo
+ to:
+ - group: ""
+ resource: secrets
+
+```
+
+The new version communicates the scope more clearly because `group`+`resource`
+is unambiguous and corresponds to exactly one set of objects on the API Server.
+
+This change also leaves room for an enhancement. Whether we have an in-tree or
+CRD implementation, we can rely on the exact matching that the plural resource
+name gives us, and [warn](https://kubernetes.io/blog/2020/09/03/warnings/) if
+either side of the grant is for an API that's not served by this cluster.
+
+#### Revocation behavior
+
+Unfortunately, there's no way to be specific about what happens when a
+ReferenceGrant is deleted in every possible case - the revocation behavior is
+dependent on what access is being granted (and revoked). With that said, we
+expect the following guidelines to be rules to apply to ALL implementations of
+the API:
+
+* Deletion of a ReferenceGrant means the granted access is revoked.
+* ReferenceGrant controllers must remove any configuration generated by the
+ granted access as soon as possible (eventual consistence permitting).
+* Some actions that have already been enabled by the ReferenceGrant (such as
+ forwarding requests or persisting data) cannot be undone, but no future
+ actions should be allowed.
+
+The examples below include information about what happens when the ReferenceGrant
+is removed as data points.
+
+### Example Usage
+
+#### Gateway API Gateway Referencing Secret
+
+In this example (from the Gateway API docs), we have a Gateway in the
+`gateway-api-example-ns1` namespace, referencing a Secret in the
+`gateway-api-example-ns2` namespace. The following ReferenceGrant allows this:
+
+```yaml
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: Gateway
+metadata:
+ name: cross-namespace-tls-gateway
+ namespace: gateway-api-example-ns1
+spec:
+ gatewayClassName: acme-lb
+ listeners:
+ - name: https
+ protocol: HTTPS
+ port: 443
+ hostname: "*.example.com"
+ tls:
+ # There's a Kind/Resource mismatch here, which sucks, but it is not
+ # easily fixable, since Gateway is already a beta, close to GA
+ # object.
+ certificateRefs:
+ - kind: Secret
+ group: ""
+ name: wildcard-example-com-cert
+ namespace: gateway-api-example-ns2
+---
+apiVersion: authorization.k8s.io/v1alpha1
+kind: ReferenceGrant
+metadata:
+ name: allow-ns1-gateways-to-ref-secrets
+ namespace: gateway-api-example-ns2
+spec:
+ from:
+ - group: gateway.networking.k8s.io
+ resource: gateways
+ namespace: gateway-api-example-ns1
+ to:
+ - group: ""
+ resource: secrets
+```
+
+For Gateway TLS references, if this ReferenceGrant is deleted (revoking,
+the grant), then the Listener will become invalid, and the configuration
+will be removed as soon as possible (eventual consistency permitting).
+
+#### Gateway API HTTPRoute Referencing Service
+
+In this example, a HTTPRoute in the `baz` namespace is directing traffic
+to a Service backend in the `quux` namespace.
+
+```yaml
+apiVersion: gateway.networking.k8s.io/v1beta1
+kind: HTTPRoute
+metadata:
+ name: quuxapp
+ namespace: baz
+spec:
+ parentRefs:
+ - name: example-gateway
+ sectionName: https
+ hostnames:
+ - quux.example.com
+ rules:
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /
+ # BackendRefs are Services by default.
+ backendRefs:
+ - name: quuxapp
+ namespace: quux
+ port: 80
+---
+apiVersion: authorization.k8s.io/v1alpha1
+kind: ReferenceGrant
+metadata:
+ name: allow-baz-httproutes
+ namespace: quux
+spec:
+ from:
+ - group: gateway.networking.k8s.io
+ resource: httproutes
+ namespace: baz
+ to:
+ - group: ""
+ resource: services
+```
+
+For HTTPRoute objects referencing a backend in another namespace, if the
+ReferenceGrant is deleted, the backend will become invalid (since the target
+can't be found). If there was more than one backend, then the valid parts of the
+HTTPRoute's config would persist in the data plane.
+
+But in this case, the cross-namespace reference is the _only_ backend, so the
+removal of the ReferenceGrant will also result in the removal of the HTTPRoute's
+config from the data plane.
+
+#### PersistentVolumeClaim using cross namespace data source
+
+This example is taken from https://kubernetes.io/blog/2023/01/02/cross-namespace-data-sources-alpha/
+and updated to use the proposed new spec.
+
+It allows the PersistentVolumeClaim in the `dev` namespace to use a volume
+snapshot from the `prod` namespace as its data source.
+
+```yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: example-pvc
+ namespace: dev
+spec:
+ storageClassName: example
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: 1Gi
+ dataSourceRef:
+ apiGroup: snapshot.storage.k8s.io
+ kind: VolumeSnapshot
+ name: new-snapshot-demo
+ namespace: prod
+ volumeMode: Filesystem
+---
+apiVersion: authorization.k8s.io/v1alpha1
+kind: ReferenceGrant
+metadata:
+ name: allow-prod-pvc
+ namespace: prod
+spec:
+ from:
+ - resource: persistentvolumeclaims
+ namespace: dev
+ to:
+ - group: snapshot.storage.k8s.io
+ resource: volumesnapshots
+ name: new-snapshot-demo
+```
+
+When a ReferenceGrant is deleted, any existing volumes created from the
+cross-namespace datasource will still persist, but new volumes will be
+rejected".
+
+### API Spec
+
+```golang
+// ReferenceGrant identifies kinds of resources in other namespaces that are
+// trusted to reference the specified kinds of resources in the same namespace
+// as the policy.
+//
+// Each ReferenceGrant can be used to represent a unique trust relationship.
+// Additional ReferenceGrants can be used to add to the set of trusted
+// sources of inbound references for the namespace they are defined within.
+//
+// ReferenceGrant is a form of runtime verification allowing users to assert
+// which cross-namespace object references are permitted. Implementations that
+// support ReferenceGrant MUST NOT permit cross-namespace references which have
+// no grant, and MUST respond to the removal of a grant by revoking the access
+// that the grant allowed.
+//
+// Implementation of ReferenceGrant is eventually consistent, dependent on
+// watch events being received from the Kubernetes API. Although some processing
+// delay is inevitable, any updates that could result in revocation of access MUST
+// be considered high priority and handled as quickly as possible.
+//
+// Implementations of ReferenceGrant MUST treat all of the following scenarios
+// as equivalent:
+//
+// * A reference to a Namespace that doesn't exist
+// * A reference to a Namespace that exists and a Resource that doesn't exist
+// * A reference to Namespace and Resource that exists but a ReferenceGrant
+// allowing the reference does not exist
+//
+// If any of the above occur, a generic error message such as "RefNotPermitted"
+// should be communicated, likely via status on the referring resource.
+//
+// Support: Core
+type ReferenceGrant struct {
+ metav1.TypeMeta `json:",inline"`
+ metav1.ObjectMeta `json:"metadata,omitempty"`
+
+ // Spec defines the desired state of ReferenceGrant.
+ Spec ReferenceGrantSpec `json:"spec,omitempty"`
+
+ // Note that `Status` sub-resource has been excluded at the
+ // moment as it was difficult to work out the design.
+ // A `Status` sub-resource may be added in the future.
+}
+
+// +kubebuilder:object:root=true
+// ReferenceGrantList contains a list of ReferenceGrant.
+type ReferenceGrantList struct {
+ metav1.TypeMeta `json:",inline"`
+ metav1.ListMeta `json:"metadata,omitempty"`
+ Items []ReferenceGrant `json:"items"`
+}
+
+// ReferenceGrantSpec identifies a cross namespace relationship that is trusted
+// for Gateway API.
+type ReferenceGrantSpec struct {
+ // From describes the trusted namespaces and kinds that can reference the
+ // resources described in "To". Each entry in this list MUST be considered
+ // to be an additional place that references can be valid from, or to put
+ // this another way, entries MUST be combined using OR.
+ //
+ // +kubebuilder:validation:MinItems=1
+ // +kubebuilder:validation:MaxItems=16
+ From []ReferenceGrantFrom `json:"from"`
+
+ // To describes the resources in this namespace that may be referenced by
+ // the resources described in "From". Each entry in this list MUST be
+ // considered to be an additional set of objects that references can be
+ // valid to, or to put this another way, entries MUST be combined using OR.
+ //
+ // +kubebuilder:validation:MinItems=1
+ // +kubebuilder:validation:MaxItems=16
+ To []ReferenceGrantTo `json:"to"`
+}
+
+```
+<<[UNRESOLVED Are "To" and "From" the right field names? ]>>
+Previous Discussion: https://github.com/kubernetes/enhancements/pull/3767#discussion_r1084671720
+
+Comments from @thockin:
+
+I would not argue for subject/object - that's confusing, and I love to take
+analogies too far.
+
+NetPol uses To/From but only for actual communications, and that still has been
+critiqued as perhaps "too cute".
+
+Subject/From isn't too bad, but not as symmetric. Subject/Referrer is correct
+but decidedly uncute. Subject/Origin?
+
+I hold opinions from an API review POV, but I'd like sig-auth to own the
+decision :)
+
+For reference, there was an [earlier
+discussion](https://groups.google.com/g/kubernetes-api-reviewers/c/ldmrXXQC4G4)
+on the kubernetes-api-reviewers mailing list that's also relevant to this.
+<<[/UNRESOLVED]>>
+
+```go
+
+// ReferenceGrantFrom describes trusted namespaces and kinds.
+type ReferenceGrantFrom struct {
+ // Group is the group of the referent.
+ // When empty, the Kubernetes core API group is inferred.
+ Group Group `json:"group"`
+
+ // Resource is the resource of the referent.
+ Resource string `json:"resource"`
+
+ // Namespace is the namespace of the referent.
+ Namespace string `json:"namespace"`
+}
+
+// ReferenceGrantTo describes what Kinds are allowed as targets of the
+// references.
+type ReferenceGrantTo struct {
+ // Group is the group of the referent.
+ // When empty, the Kubernetes core API group is inferred.
+ Group string `json:"group"`
+
+ // Resource is the resource of the referent.
+ Resource string `json:"resource"`
+
+ // Name is the name of the referent. When unspecified, this policy
+ // refers to all resources of the specified Group and Kind in the local
+ // namespace.
+ //
+ // +optional
+ Name *string `json:"name,omitempty"`
+}
+```
+
+### Outstanding questions and clarifications
+
+This section lists some of the outstanding questions people have had about
+ReferenceGrant, and other items that we'll be clarifying in the godoc and other
+documentation as part of the transition to the new API group, along with any
+other changes we need to make that aren't already reflected in this document.
+
+Also note that we don't consider any of these blockers for the general _idea_ of
+moving ReferenceGrant to the new API group, just notes to save discussion time.
+
+* Clarify that an implementation is required to reconcile ReferenceGrant for
+ specific `To` Kinds.
+* Corollary for future work, define how controllers interact. Is it a problem if
+ multiple controllers consume the same ReferenceGrant?
+* Status design is still pending, but it's currently expected that controllers
+ will indicate status on the _referring_ resources, not on ReferenceGrant itself.
+<<[UNRESOLVED do we need status? ]>>
+Original Thread: https://github.com/kubernetes/enhancements/pull/3767#discussion_r1084670421
+
+We want to be able to represent the following, in descending order of
+importance:
+
+1. Communicate that the ReferenceGrant is actively being used
+2. Communicate which controllers are using this ReferenceGrant
+3. Communicate how many times it's been used with sufficient granularity that I
+ can see the effects of my changes (if I remove this reference, am I removing
+ a dependency on this ReferenceGrant?)
+
+We could introduce a status structure that allowed each implementing controller
+to write 1 entry:
+
+```yaml
+status:
+ referencesAllowed:
+ - controllerName: gateway.example.com
+ numReferences: 1
+```
+
+@thockin responded with:
+
+If we think that the cardinality of controllers is low, we can put it in status.
+The downsides are:
+
+1. Could frequently require retries because of optimistic concurrency failures
+ (I'm trying to increment my count, but so is everyone else)
+1. If we're wrong about cardinality, there's not an easy way out
+1. Lots of writes to a resource that will be watched fairly often (every
+ controller which needs refs will watch all refgrants)
+1. We need .status.
+1. If we instead put that into a ReferenceGrantUse resource (just a tuple of
+ controller-name and count), then we only have optimistic concurrency
+ problems with ourselves, we have ~infinite cardinality, nobody will be
+ watching them, and RefGrant doesn't need .status.
+
+Downsides:
+
+1. It's another new resource
+1. It's a new pattern, untested in other places.
+
+<<[/UNRESOLVED]>>
+* Clarify that the expected operating model for implementations expects them to
+ have broad, read access to both ReferenceGrant and the specific `To` Kinds they
+ support, and then self-limit to only _use_ the relevant ones.
+* Decide whether to formally add `*` as a special value for Namespace, to mean
+ "all namespaces".
+
+### Test Plan
+
+[x] I/we understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+##### Prerequisite testing updates
+
+This is a net new resource to Kubernetes so it will not require any changes or
+additions to existing tests.
+
+##### Unit tests
+
+Unit tests will be used to cover:
+
+1. ReferenceGrant validation
+2. ReferenceGrant implementation library
+
+Test Cases of the ReferenceGrant implementation library will cover the
+following:
+
+* A reference to a Namespace that doesn't exist
+* A reference to a Namespace that exists and a Resource that doesn't exist
+* A reference to a Namespace and Resource that exists but a ReferenceGrant
+ allowing the reference does not exist
+* Multiple entries in both from and to entries within a ReferenceGrant
+* A ReferenceGrant that allows references to kinds of resources that do not
+ exist
+* Multiple ReferenceGrants with partially overlapping grants
+* Revocation of a ReferenceGrant with partially overlapping grants
+* A ReferenceGrant that does not specify `to.name`
+* A ReferenceGrant that includes overlapping grants for the same namespace both
+ with and without the resource name specified
+* A reference that has not been allowed by any ReferenceGrants
+* A ReferenceGrant that is ineffective due to the wrong `from.namespace` value
+* A ReferenceGrant that is ineffective due to the wrong `from.group` value
+* A ReferenceGrant that is ineffective due to the wrong `from.resource` value
+* A ReferenceGrant that is ineffective due to the wrong `to.group` value
+* A ReferenceGrant that is ineffective due to the wrong `to.resource` value
+* A ReferenceGrant that is ineffective due to the wrong `to.name` value
+* A ReferenceGrant that is ineffective due to being in the wrong namespace
+
+More details will be added as the details of the implementation library are
+clarified.
+
+##### Integration tests
+
+Before this graduates to beta, we will provide a reference implementation with a
+sample CRD that will be used to provide integration tests.
+
+##### e2e tests
+
+We will strongly encourage every API that uses ReferenceGrant to define
+conformance tests for their use of ReferenceGrant.
+
+### Graduation Criteria
+
+#### Beta
+
+[ ] Reference implementation with integration tests.
+[ ] Almost all of the fields and behavior have conformance test coverage in at least one project (for example Gateway API).
+
+#### GA
+
+[ ] Conformance tests that exercise all ReferenceGrant API calls (not the actual implementation of the API).
+[ ] Multiple implementations of this API passing all relevant conformance tests.
+
+### Upgrade / Downgrade Strategy
+
+N/A
+
+### Version Skew Strategy
+
+Version skew is a bit different here. Although we will provide a shared library
+for implementing this API, this will only be used by third-party controllers,
+not built-in components.
+
+There will be some implementations that support both the API defined by Gateway
+API and this API. Since these resources are entirely additive and can be
+duplicative, we can copy Gateway API resources to the new API group and delete
+the old Gateway API resources as part of a seamless migration. We expect that
+many implementations will provide this recommendation to users, and we may even
+provide tooling to simplify this process.
+
+<<[UNRESOLVED open questions that don't clearly fit elsewhere ]>>
+## Open Questions
+
+This KEP was merged in a provisional state with a number of open questions
+remaining. This section highlights the questions we have not resolved yet.
+
+### Do We Need Verbs?
+
+Previous Discussion: https://github.com/kubernetes/enhancements/pull/3767#discussion_r1084509958
+
+We could add a `verbs` field to enable users to specify the kind of referential
+access they want to grant. For example, we could define "read", "route", and
+"backup" as well-known verbs to start. We could also allow implementations to
+support additional domain-prefixed verbs.
+
+This would enable more precise grants, and potentially more open ended fields
+elsewhere in the resource, see the next item for more.
+
+### Do We Need Any or Wildcard Selectors?
+
+Previous Discussions:
+* https://github.com/kubernetes/enhancements/pull/3767#discussion_r1086020464
+* https://github.com/kubernetes/enhancements/pull/3767#discussion_r1086012665
+
+We already allow "Name" to be optional in `To`, effectively resulting in
+wildcard behavior. Should we expand that to allow any of the following?
+
+1. References to any group or resource
+1. References from any group or resource
+1. References from any namespace
+
+Historically, there has been pushback to allowing any group or resource
+because it could enable unknown or unintended kinds of access. If we
+introduced the concepts of "verbs" described above, this would become
+a moot point.
+
+### Do We Need Label Selectors?
+
+Previous Discussions:
+* https://github.com/kubernetes/enhancements/pull/3767#discussion_r1084492070
+* https://github.com/kubernetes/enhancements/pull/3767#discussion_r1084674648
+
+As a natural next step, instead of simply allowing all, we could use
+label selectors to enable:
+
+1. Access to resources with specific labels
+1. Access from namespaces with specific labels
+
+In both cases, this would significantly increase implementation
+complexity.
+
+As a potential middleground, we could explore a solution that left
+room for namespace selectors without actually including them. For example:
+
+```go
+type ReferenceGrantFrom struct {
+ //...
+ Peer ReferenceGrantPeer
+}
+
+// Exatcly one field should be set. If none are found, clients must
+// assume that an unknown value was specified.
+type ReferenceGrantPeer {
+ Namespace *string
+ // Future: Namespace selector
+}
+```
+
+This assumes that group+resource supersedes namespaces - is that true? Or do we
+really want:
+
+```go
+type ReferenceGrant struct {
+ // ...
+ From ReferenceGrantPeer
+}
+
+type ReferenceGrantPeer struct {
+ // ...
+ Namespace *string
+ NamespaceResource *NamespaceResource
+}
+
+type NamespaceResource struct {
+ Namespace string
+ Group string
+ Resource string
+}
+```
+<<[/UNRESOLVED]>>
+
+
+## Production Readiness Review Questionnaire
+
+### Feature Enablement and Rollback
+
+###### How can this feature be enabled / disabled in a live cluster?
+
+- [x] Other
+ - Describe the mechanism: Enable alpha ReferenceGrant API
+ - Will enabling / disabling the feature require downtime of the control
+ plane?
+ No
+ - Will enabling / disabling the feature require downtime or reprovisioning
+ of a node? (Do not assume `Dynamic Kubelet Config` feature is enabled).
+ No
+
+###### Does enabling the feature change any default behavior?
+
+No
+
+###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
+
+Yes
+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+The API would become accessible again, implementing controllers may need to be
+restarted to pick up the presence of this API.
+
+###### Are there any tests for feature enablement/disablement?
+
+No
+
+### Rollout, Upgrade and Rollback Planning
+
+###### How can a rollout or rollback fail? Can it impact already running workloads?
+
+API enablement may not work, but that would not be unique to this API.
+
+###### What specific metrics should inform a rollback?
+
+N/A, this is just an API
+
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+
+N/A, this is just an API
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+
+No
+
+### Monitoring Requirements
+
+N/A, this is just an API
+
+###### How can an operator determine if the feature is in use by workloads?
+```
+kubectl get referencegrants --all-namespaces
+```
+
+###### How can someone using this feature know that it is working for their instance?
+
+This will be dependent on the API that ReferenceGrant is used with. In Gateway API,
+each resource has clear status conditions that reflect the validity of a cross-namespace
+reference.
+
+###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
+
+Changes to ReferenceGrants are processed by the shared library within 10 seconds 99% over a quarter.
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+Changes to ReferenceGrants are processed by the shared library within 10 seconds.
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+
+No
+
+### Dependencies
+
+N/A, this is just an API
+
+###### Does this feature depend on any specific services running in the cluster?
+
+- API Server
+
+### Scalability
+
+###### Will enabling / using this feature result in any new API calls?
+
+Yes, users may install controllers that watch for changes to ReferenceGrants.
+Users may also create ReferenceGrants to enable cross-namespace references.
+
+###### Will enabling / using this feature result in introducing new API types?
+
+API Type: ReferenceGrant
+Supported Number of Objects per Cluster: No limit
+Supported Number of Objects per Namespace: No limit
+
+###### Will enabling / using this feature result in any new calls to the cloud provider?
+
+No
+
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+
+No
+
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+
+No
+
+###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+
+No
+
+### Troubleshooting
+
+###### How does this feature react if the API server and/or etcd is unavailable?
+
+The API would not be accessible. We would likely recommend that controllers revoke
+cross-namespace references if they could not find ReferenceGrants that allow them
+so this could result in a disruption for anything that relied on cross-namespace
+references.
+
+###### What are other known failure modes?
+
+N/A, this is just an API
+
+###### What steps should be taken if SLOs are not being met to determine the problem?
+
+N/A, this is just an API
+
+## Implementation History
+
+* July 2021: [ReferencePolicy is proposed in Gateway API](https://github.com/kubernetes-sigs/gateway-api/pull/711)
+* August 2021: [ReferencePolicy is added to Gateway API](https://github.com/kubernetes-sigs/gateway-api/pulls?page=2&q=is%3Apr+is%3Aclosed+ReferencePolicy)
+* June 2022: [ReferencePolicy is renamed to ReferenceGrant](https://github.com/kubernetes-sigs/gateway-api/pull/1179)
+* December 2022: [SIG-Storage uses ReferenceGrant for cross-namespace data storage sources](https://kubernetes.io/blog/2023/01/02/cross-namespace-data-sources-alpha/)
+* December 2022: [ReferenceGrant graduates to beta in Gateway API v0.6.0](https://github.com/kubernetes-sigs/gateway-api/releases/tag/v0.6.0)
+
+## Drawbacks
+
+N/A
+
+## Alternatives
+
+1. ReferenceGrant could remain as a CRD
+This would probably be fine, we just don't really have a good place for it to
+live. This could also complicate installation of Gateway API and other APIs that
+depended on this.
+
+2. Every API that wanted to support cross-namespace references could maintain their own version of ReferenceGrant
+This would be a confusing mess, we should avoid this at all costs.
+
diff --git a/keps/sig-auth/3766-referencegrant/kep.yaml b/keps/sig-auth/3766-referencegrant/kep.yaml
new file mode 100644
index 000000000000..6f8f282e1982
--- /dev/null
+++ b/keps/sig-auth/3766-referencegrant/kep.yaml
@@ -0,0 +1,29 @@
+title: Move ReferenceGrant to sig-auth API Group
+kep-number: 3766
+authors:
+ - "@robscott"
+ - "@youngnick"
+owning-sig: sig-auth
+participating-sigs:
+ - sig-network
+ - sig-storage
+status: provisional
+creation-date: 2023-01-20
+reviewers:
+ - "@enj"
+ - "@deads2k"
+ - "@liggitt"
+ - "@msau42"
+ - "@thockin"
+approvers:
+ - "@enj"
+ - "@liggitt"
+
+stage: alpha
+
+latest-milestone: "v1.27"
+
+milestone:
+ alpha: "v1.27"
+
+disable-supported: true
diff --git a/keps/sig-cli/2227-kubectl-default-container/README.md b/keps/sig-cli/2227-kubectl-default-container/README.md
index 8fd2b7e22652..0055da193fb3 100644
--- a/keps/sig-cli/2227-kubectl-default-container/README.md
+++ b/keps/sig-cli/2227-kubectl-default-container/README.md
@@ -16,6 +16,11 @@
- [Risks and Mitigations](#risks-and-mitigations)
- [Design Details](#design-details)
- [Test Plan](#test-plan)
+ - [Testing Strategy](#testing-strategy)
+ - [Prerequisite testing updates](#prerequisite-testing-updates)
+ - [Unit tests](#unit-tests)
+ - [Integration tests](#integration-tests)
+ - [e2e tests](#e2e-tests)
- [Graduation Criteria](#graduation-criteria)
- [Alpha -> Beta Graduation](#alpha---beta-graduation)
- [Beta -> GA Graduation](#beta---ga-graduation)
@@ -179,13 +184,37 @@ metadata:
```
### Test Plan
+
+#### Testing Strategy
+
Add a unit test for each command, testing the behavior with the annotation, without the annotation, and with the --container flag
+##### Prerequisite testing updates
+
+##### Unit tests
+
+The main unit test is in package under `vendor/k8s.io/kubectl/pkg/`.
+
+- vendor/k8s.io/kubectl/pkg/cmd/exec:2023-01-13 - 68.9%
+- vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:2023-01-13 - 38.1%
+- vendor/k8s.io/kubectl/pkg/polymorphichelpers/logsforobject.go:2023-01-13 - 79%
+
+See details in .
+
+##### Integration tests
+
+N/A
+
+##### e2e tests
+
+[Kubectl logs default container logs](https://github.com/kubernetes/kubernetes/blob/02f893b6e28549a1008d4fc65c40990b87c07830/test/e2e/kubectl/logs.go#L163):
+
### Graduation Criteria
#### Alpha -> Beta Graduation
As this is an opt-in feature, no gate is expected.
+
- At least 2 release cycles pass to gather feedback and bug reports during
- Documentations, add it to [well-known annotations docs](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/)
- Add a warning deprecation message when using the annotation `kubectl.kubernetes.io/default-logs-container`
@@ -310,11 +339,26 @@ resource usage (CPU, RAM, disk, IO, ...) in any components?**
## Implementation History
-2021-03-10: Alpha implementation completion in 1.21
-- https://github.com/kubernetes/kubernetes/pull/97099
-- https://github.com/kubernetes/kubernetes/pull/99615
-- https://github.com/kubernetes/kubernetes/pull/99581
-- https://github.com/kubernetes/kubernetes/pull/99833
+2021-03-10(v1.21) Alpha: implementation completion
+
+-
+-
+-
+-
+
+2021-10-28(v1.24) Beta: make 'kubectl logs' default to the first container when default container cannot be
+determined or found by annotations
+
+-
+
+2022-04-05(v1.25): remove deprecated message of `kubectl.kubernetes.io/default-logs-container`
+
+-
+
+2023-01-13(v1.27): add an e2e test case
+
+-
+
## Drawbacks
diff --git a/keps/sig-cli/2227-kubectl-default-container/kep.yaml b/keps/sig-cli/2227-kubectl-default-container/kep.yaml
index 36cc39d41ec6..ec491dfb578d 100644
--- a/keps/sig-cli/2227-kubectl-default-container/kep.yaml
+++ b/keps/sig-cli/2227-kubectl-default-container/kep.yaml
@@ -6,7 +6,7 @@ owning-sig: sig-cli
participating-sigs:
status: implementable
creation-date: 2020-12-16
-last-updated: 2021-03-10
+last-updated: 2023-01-20
reviewers:
- "@dougsland"
- "@eddiezane"
@@ -19,18 +19,18 @@ approvers:
see-also:
# The target maturity stage in the current dev cycle for this KEP.
-stage: alpha
+stage: stable
# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
-latest-milestone: "v1.21"
+latest-milestone: "v1.27"
# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
alpha: "v1.21"
- beta: "v1.23"
- stable: "v1.25"
+ beta: "v1.24"
+ stable: "v1.27"
# The following PRR answers are required at alpha release
# List the feature gate name and the components for which it must be enabled
diff --git a/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components/kep.yaml b/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components/kep.yaml
index 34ba8626c52f..d08c499973d0 100644
--- a/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components/kep.yaml
+++ b/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components/kep.yaml
@@ -5,7 +5,7 @@ authors:
owning-sig: sig-instrumentation
participating-sigs:
- sig-arch
-status: implementable
+status: implemented
creation-date: 2021-07-30
reviewers:
- pohly
@@ -16,12 +16,12 @@ approvers:
see-also:
- "/keps/sig-instrumentation/1602-structured-logging"
replaces: []
-stage: beta
-latest-milestone: "v1.24"
+stage: stable
+latest-milestone: "v1.26"
milestone:
alpha: "v1.23"
beta: "v1.24"
- stable: "v1.25"
+ stable: "v1.26"
feature-gates: []
disable-supported: true
diff --git a/keps/sig-instrumentation/3466-kubernetes-component-health-slis/README.md b/keps/sig-instrumentation/3466-kubernetes-component-health-slis/README.md
index ced1109680a6..b7e6db5153f2 100644
--- a/keps/sig-instrumentation/3466-kubernetes-component-health-slis/README.md
+++ b/keps/sig-instrumentation/3466-kubernetes-component-health-slis/README.md
@@ -128,20 +128,20 @@ checklist items _must_ be updated for the enhancement to be released.
Items marked with (R) are required *prior to targeting to a milestone / release*.
-- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
-- [ ] (R) KEP approvers have approved the KEP status as `implementable`
-- [ ] (R) Design details are appropriately documented
-- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
- - [ ] e2e Tests for all Beta API Operations (endpoints)
- - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
- - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+- [X] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [X] (R) KEP approvers have approved the KEP status as `implementable`
+- [X] (R) Design details are appropriately documented
+- [X] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+ - [X] e2e Tests for all Beta API Operations (endpoints)
+ - [X] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+ - [X] (R) Minimum Two Week Window for GA e2e tests to prove flake free
- [ ] (R) Graduation criteria is in place
- [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
-- [ ] (R) Production readiness review completed
+- [X] (R) Production readiness review completed
- [ ] (R) Production readiness review approved
-- [ ] "Implementation History" section is up-to-date for milestone
-- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
-- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+- [X] "Implementation History" section is up-to-date for milestone
+- [X] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [X] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
-
+
- [x] API .status
- Condition name: The Condition name will not be standardized in `alpha` however
implementations are given the `status` field to report what they see fit.
@@ -1175,8 +1355,8 @@ Describe the metrics themselves and the reasons why they weren't added (e.g., co
implementation difficulties, etc.).
-->
-A metric describing the time it takes for the implementation to program the rules
-defined in an AdminNetworkPolicy could be useful. However, some implementations
+A metric describing the time it takes for the implementation to program the rules
+defined in an AdminNetworkPolicy could be useful. However, some implementations
may struggle to report such a metric.
### Dependencies
@@ -1242,10 +1422,10 @@ Describe them, providing:
- Supported number of objects per namespace (for namespace-scoped objects)
-->
-- API Type: AdminNetworkPolicy
-- Supported number of objects per cluster: The total number of AdminNetworkPolicies
-will not be limited. However, it is important to remember that the only users
-creating ANPs will be Cluster-Admins, of which there should only be a handful. This
+- API Type: AdminNetworkPolicy
+- Supported number of objects per cluster: The total number of AdminNetworkPolicies
+will not be limited. However, it is important to remember that the only users
+creating ANPs will be Cluster-Admins, of which there should only be a handful. This
will help limit the total number ANPs being deployed at any given time.
###### Will enabling / using this feature result in any new calls to the cloud provider?
@@ -1256,7 +1436,7 @@ Describe them, providing:
- Estimated increase:
-->
-This depends on the implementation, specifically based on the API used to program
+This depends on the implementation, specifically based on the API used to program
the AdminNetworkPolicy rules into the data-plane.
###### Will enabling / using this feature result in increasing size or count of the existing API objects?
@@ -1295,7 +1475,7 @@ This through this both in small and large cases, again with respect to the
[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md
-->
-Not in any in-tree components, resource efficiency will need to be monitored by the
+Not in any in-tree components, resource efficiency will need to be monitored by the
implementation.
### Troubleshooting
diff --git a/keps/sig-network/2681-pod-host-ip/kep.yaml b/keps/sig-network/2681-pod-host-ip/kep.yaml
index e08ac84cafc7..9de5d23944fb 100644
--- a/keps/sig-network/2681-pod-host-ip/kep.yaml
+++ b/keps/sig-network/2681-pod-host-ip/kep.yaml
@@ -25,13 +25,13 @@ stage: alpha
# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
-latest-milestone: "v1.24"
+latest-milestone: "v1.27"
# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
- alpha: "v1.24"
- beta: "v1.25"
- stable: "v1.27"
+ alpha: "v1.27"
+ beta: "v1.28"
+ stable: "v1.30"
# The following PRR answers are required at alpha release
# List the feature gate name and the components for which it must be enabled
diff --git a/keps/sig-network/3458-remove-transient-node-predicates-from-service-controller/README.md b/keps/sig-network/3458-remove-transient-node-predicates-from-service-controller/README.md
new file mode 100644
index 000000000000..529cc8c90fbb
--- /dev/null
+++ b/keps/sig-network/3458-remove-transient-node-predicates-from-service-controller/README.md
@@ -0,0 +1,381 @@
+# KEP-3458: Remove transient node predicates from KCCM's service controller
+
+
+- [Release Signoff Checklist](#release-signoff-checklist)
+- [Summary](#summary)
+- [Motivation](#motivation)
+ - [Goals](#goals)
+ - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+ - [Risks and Mitigations](#risks-and-mitigations)
+ - [Risk](#risk)
+ - [Mitigations](#mitigations)
+- [Design Details](#design-details)
+ - [Test Plan](#test-plan)
+ - [Prerequisite testing updates](#prerequisite-testing-updates)
+ - [Unit tests](#unit-tests)
+ - [Integration tests](#integration-tests)
+ - [e2e tests](#e2e-tests)
+ - [Graduation Criteria](#graduation-criteria)
+ - [Beta](#beta)
+ - [GA](#ga)
+ - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+ - [Version Skew Strategy](#version-skew-strategy)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+ - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+ - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+ - [Monitoring Requirements](#monitoring-requirements)
+ - [Dependencies](#dependencies)
+ - [Scalability](#scalability)
+ - [Troubleshooting](#troubleshooting)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+
+
+## Release Signoff Checklist
+
+Items marked with (R) are required *prior to targeting to a milestone / release*.
+
+- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [ ] (R) KEP approvers have approved the KEP status as `implementable`
+- [ ] (R) Design details are appropriately documented
+- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+ - [ ] e2e Tests for all Beta API Operations (endpoints)
+ - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+ - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+- [ ] (R) Graduation criteria is in place
+ - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+- [ ] (R) Production readiness review completed
+- [ ] (R) Production readiness review approved
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://git.k8s.io/enhancements
+[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
+[kubernetes/website]: https://git.k8s.io/website
+
+## Summary
+
+The service controller in the Kubernetes cloud controller manager (KCCM)
+currently adds/removes Nodes from the load balancers' node set in the following
+cases:
+
+a) When a node gets the taint `ToBeDeletedByClusterAutoscaler` added/removed
+b) When a node goes `Ready` / `NotReady`
+
+b) however only applies to services with `externalTrafficPolicy: Cluster`. In
+both cases: removing the Node in question from the load balancers' node set will
+cause all connections on that node to get terminated instantly. This can be
+considered a bug / sub-optimal behavior for nodes which are experiencing
+transient readiness state or for terminating nodes, since connections are not
+allowed to drain in those cases, even though the load balancer might support
+that. Moreover: on large clusters with a lot nodes and entropy, re-syncing load
+balancers like this can lead to rate-limiting by the cloud provider due to an
+excessive amount of update calls.
+
+As to enable connection draining, reduce cloud provider API calls and simplify
+the KCCMs sync loop: this KEP proposes that the service controller stops
+synchronizing the load balancer node set in these cases. Seeing as how this has
+always been the case, a new feature gate `StableLoadBalancerNodeSet` will
+be introduced, which will be used to enable the more optimal behavior.
+
+## Motivation
+
+Abruptly terminating connections in the cases defined by a) and b) above can be
+seen as buggy behavior and should be improved. By enabling connection draining,
+applications are allowed profit from graceful shutdown / termination, for what
+concerns cluster ingress connectivity. Users of Kubernetes will also see a
+reduction in the amount of cloud API calls, for what concerns calls stemming
+from syncing load balancers with the Kubernetes cluster state.
+
+### Goals
+
+- Stop re-configuring the load balancers' node set for cases a) and b) above
+
+### Non-Goals
+
+- Stop re-configuring the load balancers' node set for fully deleted /
+ newly added cluster nodes, or for nodes which get annotated with
+ `node.kubernetes.io/exclude-from-external-load-balancers`.
+- Enable load balancer connection draining while Node is draining. This requires
+ health check changes.
+
+## Proposal
+
+### Risks and Mitigations
+
+#### Risk
+
+1. Cloud providers which do not allow VM deletion when the VM is referenced by
+ other constructs, will block the cluster auto-scaler (CA) from deleting the VM
+ upon downscale. This will result in reduced downscale performance by the CA,
+ or completely block VM deletion from happening - this is because the service
+ controller will never proceed to de-reference the VM from the load balancer
+ node set until the Node is fully deleted in the API server, which will never
+ occur until the VM is deleted. The three major cloud providers (GCP/AWS/Azure)
+ do however support this, and it is not expected that other providers don't.
+2. Cloud providers which do not configure their load balancer health checks to
+ target the service proxy's healthz, alternatively: constructs which validate
+ the endpoint's reachability across the data plane; risk experiencing
+ regressions as a consequence of the removal of b). This would happen if a node
+ is faced with a terminal error which does impact the Node's network
+ connectivity. Doing this is considered incorrect, and therefor not expected to
+ be the case.
+3. By removing b) above we are delaying the removal of the Node from the load
+ balancers' node set until the Node is completely deleted in the API server.
+ This might have an impact on CA downscaling. The reason for this is: the CA
+ deletes the VM and expects the node controller in the KCCM to notice this and
+ delete the Node in Kubernetes, as a consequence. If the node controller takes
+ a while to sync that and other Node related events trigger load balancer
+ reconciliation while this is happening, then the service controller will error
+ until the cluster reaches steady-state (because it's trying to sync Nodes for
+ which the VM is non-existent). A mitigation to this is presented in
+ [Mitigations](#mitigations)
+
+#### Mitigations
+
+- Cloud providers/workloads which do not support the behavior mentioned in
+ [Risk](#risk), have the possibility to set the feature flag to false, thus
+ default back to the current mechanism.
+- For point 3. we could implement the following change in the service
+ controller; ensure it only enqueues Node UPDATE on changes to
+ `.metadata.labels["node.kubernetes.io/exclude-from-external-load-balancers"]`.
+ When processing the sync: list only Node following the existing predicates
+ defined for `externalTrafficPolicy: Cluster/Local` services (both currently
+ exclude Nodes with the taint `ToBeDeletedByClusterAutoscaler`). This will
+ ensure that whatever Nodes are included in the load balancer set, always have
+ a corresponding VM. Doing this is however reverting on the goal of the KEP.
+
+## Design Details
+
+- Implement the change in the service controller and ensure it does not add /
+ remove nodes from the load balancers' node set for cases a) and b) mentioned
+ in (Summary)[#Summary]
+- Add the feature gate: `StableLoadBalancerNodeSet`, set it to "on" by
+ default and promote it directly to Beta.
+
+### Test Plan
+
+#### Prerequisite testing updates
+
+#### Unit tests
+
+The service controller in the KCCM currently has a set of tests validating
+expected syncs caused by Node predicates, these will need to be updated.
+
+#### Integration tests
+
+Kubernetes is mostly tested via unit tests and e2e, not integration, and this is
+not expected to change.
+
+#### e2e tests
+
+Kubernetes in general needs to extended its load balancing test suite with
+disruption tests, this might be the right effort we need to get that ball
+rolling. Testing would include:
+
+- validation that an application running on a deleting VM benefits from graceful
+ termination of its TCP connection.
+- validation that Node readiness state changes do not result in load balancer
+ re-syncs.
+
+### Graduation Criteria
+
+#### Beta
+
+This is addressing a sub-optimal solution currently existing in Kubernetes, so
+the feature gate will be moved to Beta and "on" by default from the start.
+
+The feature flag should be kept available until we get sufficient evidence of
+people not being affected by anything mentioned in (Risks)[#Risks] or other.
+
+#### GA
+
+Given the lack of reported issues in Beta: the feature gate will be locked-in in
+GA.
+
+Tentative timeline for this is in v1.29. Services of `type: LoadBalancer` are
+sufficiently common on any given Kubernetes cluster, that any cloud provider
+susceptible to the (Risks)[#Risks] will very likely report issues in Beta.
+
+### Upgrade / Downgrade Strategy
+
+Any upgrade to a version enabling the feature, succeeded by a downgrade to a
+version disabling it, is not expected to be impacted in any way. On upgrade: the
+service controller will add all existing cluster nodes (bar excluded ones) to
+the load balancer set. On downgrade: any nodes `NotReady` / tainted will get
+reconciled by the service controller corresponding to the downgraded control
+plane version and get removed from the load balancer set - as they should.
+
+### Version Skew Strategy
+
+This change is contained to only the control plane and is therefor not
+impacted by any version skew.
+
+## Production Readiness Review Questionnaire
+
+### Feature Enablement and Rollback
+
+###### How can this feature be enabled / disabled in a live cluster?
+
+- [X] Feature gate (also fill in values in `kep.yaml`)
+ - Feature gate name: `StableLoadBalancerNodeSet`
+ - Components depending on the feature gate: Kubernetes cloud controller manager
+
+###### Does enabling the feature change any default behavior?
+
+Yes, Kubernetes Nodes will remain in the load balancers' node set until fully
+deleted in the API server, as opposed to the current behavior: which adds /
+removes the nodes from the set when the Node experience transient state changes.
+Cloud providers which do not support deleting VMs which are still referenced by
+load balancers, will be unable to do so upon downscaling by the cluster
+auto-scaler when it attempts to delete the VM.
+
+###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
+
+Yes, by resetting the feature gate back.
+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+Behavior will be restored back immediately.
+
+###### Are there any tests for feature enablement/disablement?
+
+N/A
+
+### Rollout, Upgrade and Rollback Planning
+
+###### How can a rollout or rollback fail? Can it impact already running workloads?
+
+Rollout and rollback are not expected to fail.
+
+###### What specific metrics should inform a rollback?
+
+Performance degradation by the CA when downscaling / flat out inability to delete VMs.
+
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+
+N/A
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+
+No.
+
+### Monitoring Requirements
+
+The only mechanism currently implemented, is: events for syncing load balancers
+in the KCCM. The events are triggered any time a service is synced or Node
+change triggers a re-sync of all services. This will not change and can be used
+to monitor the implemented change. The implementation will result in less load
+balancer re-syncs.
+
+A new metric `load_balancer_sync_count` will be added for explicitly monitoring
+the amount of load balancer related syncs performed by the service controller.
+This will include load balancer syncs caused by Service and Node changes.
+
+A new metric `nodesync_error_rate` will be added for explicitly monitoring the
+amount of errors produced by syncing Node related events for load balancers. The
+goal is have an indicator of if the service controller is impacted by point 3.
+mentioned in (Risk)[#Risk], and at which frequency.
+
+###### How can an operator determine if the feature is in use by workloads?
+
+Analyze events stemming from the API server, correlating node state changes
+(readiness or addition / removal of the taint: `ToBeDeletedByClusterAutoscaler`)
+to load balancer re-syncs. The events should show a clear reduction in re-syncs
+post the implementation and rollout of the change.
+
+###### How can someone using this feature know that it is working for their instance?
+
+Yes, when a node transition from `Ready` <-> `NotReady` the load balancers are
+not re-synced and the load balancers' node set will remain unchanged for
+`externalTrafficPolicy: Cluster` services. On downscaling by the CA, the node
+will remain the the load balancers' set for a longer period of time, just until
+the Node object in Kubernetes is fully deleted.
+
+###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
+
+Total amount of load balancer re-syncs should be reduced, leading to less cloud
+provider API calls. Also, and more subtle: connections will get a chance to
+gracefully terminate when the CA downscales cluster nodes. For services of type
+`externalTrafficPolicy: Cluster` "traversing" connections through a "nexthop"
+node might not be impacted by that Node's readiness state anymore.
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+
+
+- [X] Metrics
+ - Events: The KCCM triggers events when syncing load balancers. The amount of
+ these events should be reduced.
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+
+N/A
+
+### Dependencies
+
+###### Does this feature depend on any specific services running in the cluster?
+
+It depends on:
+
+- having a `type: LoadBalancer` service with `externalTrafficPolicy: Cluster`
+
+### Scalability
+
+###### Will enabling / using this feature result in any new API calls?
+
+No
+
+###### Will enabling / using this feature result in introducing new API types?
+
+No
+
+###### Will enabling / using this feature result in any new calls to the cloud provider?
+
+No
+
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+
+No
+
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+
+No
+
+###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+
+No
+
+### Troubleshooting
+
+###### How does this feature react if the API server and/or etcd is unavailable?
+
+Not any different than today.
+
+###### What are other known failure modes?
+
+None
+
+###### What steps should be taken if SLOs are not being met to determine the problem?
+
+Validate that services of `type: LoadBalancer` exists on the cluster and that
+Nodes are experiencing a transitioning readiness state, alternatively that the
+CA downscales and deletes VMs.
+
+## Implementation History
+
+- 2023-02-01: Initial proposal
+
+## Drawbacks
+
+## Alternatives
diff --git a/keps/sig-network/3458-remove-transient-node-predicates-from-service-controller/kep.yaml b/keps/sig-network/3458-remove-transient-node-predicates-from-service-controller/kep.yaml
new file mode 100755
index 000000000000..eea8ff138019
--- /dev/null
+++ b/keps/sig-network/3458-remove-transient-node-predicates-from-service-controller/kep.yaml
@@ -0,0 +1,24 @@
+prnumber: "3460"
+name: 3458-remove-transient-node-predicates-from-service-controller
+title: Remove transient node predicates from KCCM's service controller
+kep-number: "3458"
+authors: ['@alexanderConstantinescu']
+owning-sig: sig-network
+participating-sigs: [sig-network]
+reviewers:
+ - "@thockin"
+approvers:
+ - "@thockin"
+creation-date: "2022-08-07"
+last-updated: v1.27
+status: implementeable
+stage: stable
+latest-milestone: "v1.27"
+milestone:
+ beta: "v1.27"
+ stable: "v1.29"
+feature-gates:
+ - name: StableLoadBalancerNodeSet
+ components:
+ - kube-controller-manager
+disable-supported: true
diff --git a/keps/sig-network/3668-reserved-service-nodeport-range/README.md b/keps/sig-network/3668-reserved-service-nodeport-range/README.md
new file mode 100644
index 000000000000..4b0a626b5ade
--- /dev/null
+++ b/keps/sig-network/3668-reserved-service-nodeport-range/README.md
@@ -0,0 +1,468 @@
+# KEP-3668: Reserve Nodeport Ranges For Dynamic And Static Port Allocation
+
+
+- [Release Signoff Checklist](#release-signoff-checklist)
+- [Summary](#summary)
+- [Motivation](#motivation)
+ - [Goals](#goals)
+ - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+ - [Story 1](#story-1)
+ - [Story 2](#story-2)
+ - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+ - [Current Services NodePort allocation model](#current-services-nodeport-allocation-model)
+ - [Proposed Services NodePorts allocation model](#proposed-services-nodeports-allocation-model)
+ - [Test Plan](#test-plan)
+ - [Prerequisite testing updates](#prerequisite-testing-updates)
+ - [Unit tests](#unit-tests)
+ - [Integration tests](#integration-tests)
+ - [e2e tests](#e2e-tests)
+ - [Graduation Criteria](#graduation-criteria)
+ - [Alpha](#alpha)
+ - [Beta](#beta)
+ - [GA](#ga)
+ - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+ - [Version Skew Strategy](#version-skew-strategy)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+ - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+ - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+ - [Monitoring Requirements](#monitoring-requirements)
+ - [Dependencies](#dependencies)
+ - [Scalability](#scalability)
+ - [Troubleshooting](#troubleshooting)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+
+
+## Release Signoff Checklist
+
+
+
+Items marked with (R) are required *prior to targeting to a milestone / release*.
+
+- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [ ] (R) KEP approvers have approved the KEP status as `implementable`
+- [ ] (R) Design details are appropriately documented
+- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+ - [ ] e2e Tests for all Beta API Operations (endpoints)
+ - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+ - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+- [ ] (R) Graduation criteria is in place
+ - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+- [ ] (R) Production readiness review completed
+- [ ] (R) Production readiness review approved
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://git.k8s.io/enhancements
+[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
+[kubernetes/website]: https://git.k8s.io/website
+
+## Summary
+
+Nodeport Service can expose a Service outside the cluster, and allow external applications access to an set of Pods. NodePort has several ports that are widespread in the cluster and allow to load-balance traffic from the external. And the port number can be assigned:
+
+- dynamically, the cluster will pick one within the configured service node port range.
+- statically, the user will set one port within the configured service node port range.
+
+Currently, there is no possibility, before creating a NodePort Service with a static port, to know if the port has been chosen already by any other NodePort using dynamic allocation.
+
+The NodePort Service port range can be logically subdivided to avoid the risk of conflict between NodePort that use static and dynamic port allocation. The idea is mostly the same with [Reserve Service IP Ranges (KEP-3070)](https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/3070-reserved-service-ip-range#release-signoff-checklist) did.
+
+## Motivation
+
+There are many situations that users need to rely on well-known predefined NodePort ports.
+
+The usage scenario is similar in nature to the pre-assigned Cluster IP addresses described in KEP 3070: in some deployments, IPs or ports need to be hard-coded in some specific components of the cluster. Hard-coding of NodePort typically occurs in private cloud deployments with complex hyper-converged architectures, where we cannot rely on some of the underlying infrastructure, or even need to use the services running on Kubernetes as a base service. As a concrete example, imagine we need to deploy a set of applications in an environment without DNS servers and external LBs, this set of applications includes several K8S-based deployments, but others are non-K8S-based which deploied on baremetal. At this point, CoreDNS is expected to be the only DNS server in the environment, and both need to provide resolution services not only within K8S, but also outside the K8S cluster, where a fixed IP and NodePort is necessary.
+
+During cluster initialization, some services created earlier may take up the CluserIP and NodePort ports that we expect to assign later, the ClusterIP problem is solved in and KEP 3070, while the NodePort problem may still occur.
+
+
+### Goals
+
+- allow to create a Service with a static NodePort with less risk of port conflict.
+
+
+### Non-Goals
+
+- changing the allocators implementation (bitmaps, ...).
+- add new configuration options or flags.
+
+## Proposal
+
+### Story 1
+
+"As a Kubernetes administrator, I would like to be able to assign a fixed port to my DNS Service to provide access to services outside the cluster and reduce the possibility of conflicts with existing NodePort Services within the cluster."
+
+### Story 2
+"As a Kubernetes developer I want to allocate safely fixed ports to some NodePort Services so I can automate some configuration parameters of my cluster"
+
+### Risks and Mitigations
+There is no risk. Every NodePort Service should be able to get an port from the NodePort range as long as there are free ports.
+
+This KEP only subdivides the NodePort range, and prioritize one range over other for dynamically port allocation, without adding limitations to the number of ports assigned.
+
+## Design Details
+
+### Current Services NodePort allocation model
+
+nodePort is an additional port that needs to be bound to all nodes for the service, if a port is specified manually, is in-range, and is not in use,
+it will be allocated to the nodePort; otherwise creation of the service will fail.
+The Service NodePort range is defined in the apiserver with the following flag:
+
+```
+--service-node-port-range Default: 30000-32767
+A port range to reserve for services with NodePort visibility. This must not overlap with the ephemeral port range on nodes. Example: '30000-32767'. Inclusive at both ends of the range.
+```
+
+In a multi master environment, multiple requests can arrive at same time at different apiservers, the allocation logic must
+guarantee that there are no duplicate port assigned. To solve this consensus problem in a distributed system, allocators are
+implemented using bitmaps that are serialized and stored in an ["opaque" API object](https://github.com/kubernetes/kubernetes/blob/6a16d7d31aeb0f95c4ae513311bfecef9492f30e/pkg/apis/core/types.go#L5714).
+
+```go
+// RangeAllocation is an opaque API object (not exposed to end users) that can be persisted to record
+// the global allocation state of the cluster. The schema of Range and Data generic, in that Range
+// should be a string representation of the inputs to a range (for instance, for IP allocation it
+// might be a CIDR) and Data is an opaque blob understood by an allocator which is typically a
+// binary range. Consumers should use annotations to record additional information (schema version,
+// data encoding hints). A range allocation should *ALWAYS* be recreatable at any time by observation
+// of the cluster, thus the object is less strongly typed than most.
+type RangeAllocation struct {
+ metav1.TypeMeta
+ // +optional
+ metav1.ObjectMeta
+ // A string representing a unique label for a range of resources, such as a CIDR "10.0.0.0/8" or
+ // port range "10000-30000". Range is not strongly schema'd here. The Range is expected to define
+ // a start and end unless there is an implicit end.
+ Range string
+ // A byte array representing the serialized state of a range allocation. Additional clarifiers on
+ // the type or format of data should be represented with annotations. For IP allocations, this is
+ // represented as a bit array starting at the base IP of the CIDR in Range, with each bit representing
+ // a single allocated address (the fifth bit on CIDR 10.0.0.0/8 is 10.0.0.4).
+ Data []byte
+}
+```
+
+The [apiserver Service registry contains allocators backed by those bitmaps](p[kg/registry/core/rest/storage_core.go](https://github.com/kubernetes/kubernetes/blob/2ac6a4121f5b2a94acc88d62c07d8ed1cd34ed63/pkg/registry/core/rest/storage_core.go#L96-L103)) in order to assign these ports.
+
+```go
+type LegacyRESTStorage struct {
+ ServiceClusterIPAllocator rangeallocation.RangeRegistry
+ SecondaryServiceClusterIPAllocator rangeallocation.RangeRegistry
+ ServiceNodePortAllocator rangeallocation.RangeRegistry
+}
+
+...
+func (c LegacyRESTStorageProvider) NewLegacyRESTStorage(apiResourceConfigSource serverstorage.APIResourceConfigSource, restOptionsGetter generic.RESTOptionsGetter) (LegacyRESTStorage, genericapiserver.APIGroupInfo, error) {
+ ...
+ var serviceNodePortRegistry rangeallocation.RangeRegistry
+ serviceNodePortAllocator, err := portallocator.New(c.ServiceNodePortRange, func(max int, rangeSpec string) (allocator.Interface, error) {
+ mem := allocator.NewAllocationMap(max, rangeSpec)
+ // TODO etcdallocator package to return a storage interface via the storageFactory
+ etcd, err := serviceallocator.NewEtcd(mem, "/ranges/servicenodeports", serviceStorageConfig.ForResource(api.Resource("servicenodeportallocations")))
+ if err != nil {
+ return nil, err
+ }
+ serviceNodePortRegistry = etcd
+ return etcd, nil
+ })
+```
+
+The bitmaps implement a [strategy for allocating a random free value if none is specified](https://github.com/kubernetes/kubernetes/blob/2ac6a4121f5b2a94acc88d62c07d8ed1cd34ed63/pkg/registry/core/service/allocator/bitmap.go#L186-L205):
+
+```go
+// randomScanStrategy chooses a random address from the provided big.Int, and then
+// scans forward looking for the next available address (it will wrap the range if
+// necessary).
+type randomScanStrategy struct {
+ rand *rand.Rand
+}
+
+func (rss randomScanStrategy) AllocateBit(allocated *big.Int, max, count int) (int, bool) {
+ if count >= max {
+ return 0, false
+ }
+ offset := rss.rand.Intn(max)
+ for i := 0; i < max; i++ {
+ at := (offset + i) % max
+ if allocated.Bit(at) == 0 {
+ return at, true
+ }
+ }
+ return 0, false
+}
+```
+
+### Proposed Services NodePorts allocation model
+
+The strategy proposaled here is mentioned and implemented in (KEP 3070)[https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/3070-reserved-service-ip-range/README.md#proposed-services-clusterips-allocation-model] first, with some fine-tuning. The following is a specific explanation of the strategy.
+
+the strategy following formula `min(max($min, node-range-size/$step), $max)`, described as never less than $min or more than $max, with a graduated step function between them, with $min = 16, $max = 128 and $step = 32. If node-range-size < $min the formula doesn't apply and the strategy will not change, both dynamic and static ports will be allocated from the whole range with equal probability.
+
+- lower band, used preferably for static port assignment.
+- upper band, used preferably for dynamic port assignment.
+
+Dynamically port assignment will use the upper band by default, once this has been exhausted it will use the lower range. This will allow users to define static allocations on the lower band of the predefined range with a low risk of collision, keeping the implementation backwards compatible.
+
+
+Example 1 Default service nodeport range:
+
+- Service Node Port Range: 30000-32767
+- Range Size: 32767 - 30000 = 2767
+- Band Offset: min(max(16,2767/32),128) = min(86,128) = 86
+- Static band start: 30000
+- Static band ends: 30086
+
+ ┌─────────────┬─────────────────────────────────────────────┐
+ │ static │ dynamic │
+ └─────────────┴─────────────────────────────────────────────┘
+
+ ◄────────────► ◄────────────────────────────────────────────►
+ 30000 30086 32767
+
+
+Example 2 Large service nodeport range:
+
+- Service Node Port Range: 20000-32767
+- Range Size: 32767 - 30000 = 12767
+- Band Offset: min(max(16,12767/32),128) = min(398,128) = 128
+- Static band start: 20000
+- Static band ends: 20128
+
+ ┌─────────────┬─────────────────────────────────────────────┐
+ │ static │ dynamic │
+ └─────────────┴─────────────────────────────────────────────┘
+
+ ◄────────────► ◄────────────────────────────────────────────►
+ 20000 20128 32767
+
+
+Example 3 Small service nodeport range:
+
+- Service Node Port Range: 32567-32767
+- Range Size: 32767 - 32567 = 200
+- Band Offset: min(max(16,200/32),128) = min(max(16, 6),128) = 16
+- Static band start: 32567
+- Static band ends: 32583
+
+ ┌─────────────┬─────────────────────────────────────────────┐
+ │ static │ dynamic │
+ └─────────────┴─────────────────────────────────────────────┘
+
+ ◄────────────► ◄────────────────────────────────────────────►
+ 32567 32583 32767
+
+
+
+### Test Plan
+
+[x] I/we understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+##### Prerequisite testing updates
+None
+
+##### Unit tests
+- `kubernetes/pkg/registry/core/service/portallocator`: `12/10/2022` - `83.1`
+
+##### Integration tests
+
+This feature doesn't modify the cluster behavior, only the order on which dynamic port are assigned to NodePort Services, there is no need for e2e or integration tests, unit tests are enough.
+
+##### e2e tests
+
+### Graduation Criteria
+
+#### Alpha
+
+- Feature implemented behind a feature flag
+
+#### Beta
+
+- Gather feedback from developers and community.
+- No issues reported.
+
+#### GA
+
+- No issues reported during two releases.
+
+
+### Upgrade / Downgrade Strategy
+
+The feature only changes the allocation strategy, the bitmaps and the underlay logic remain the same,
+guaranteeing full compatibility and no risk rolling back the feature.
+
+### Version Skew Strategy
+
+Same as with upgrade, there is no change in the bitmaps, keeping the feature 100% compatible.
+
+
+## Production Readiness Review Questionnaire
+
+### Feature Enablement and Rollback
+
+###### How can this feature be enabled / disabled in a live cluster?
+
+- [x] Feature gate (also fill in values in `kep.yaml`)
+ - Feature gate name: ServiceNodePortStaticSubrange
+ - Components depending on the feature gate: kube-apiserver
+- [ ] Other
+ - Describe the mechanism:
+ - Will enabling / disabling the feature require downtime of the control
+ plane?
+ - Will enabling / disabling the feature require downtime or reprovisioning
+ of a node? (Do not assume `Dynamic Kubelet Config` feature is enabled).
+
+###### Does enabling the feature change any default behavior?
+
+Dynamic allocated NodePort will not be initially chosen from the first X values of the Service Node Port range, where X is obtained from the formula
+described in [Proposed Services NodePorts allocation model](#proposed-services-nodeports-allocation-model)
+This change is transparent to the user, since the port is and was already random, it doesn't matter from which range the port is obtained.
+
+###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
+
+Yes.
+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+Nothing, this only affects new requests and how these new request obtain the dynamically assigned ports.
+
+
+###### Are there any tests for feature enablement/disablement?
+
+No need to add new test, current tests should keep working as today.
+
+### Rollout, Upgrade and Rollback Planning
+
+###### How can a rollout or rollback fail? Can it impact already running workloads?
+
+Current Services will not be affected, it doesn't matter in which range their port are allocated.
+This only affects the creation of new NodePort Services without an port specified, there is no risk of impact on running workloads.
+
+###### What specific metrics should inform a rollback?
+
+There is currently no metric available for the NodePort allocation. so 4 metric should be added.
+
+ - kube_apiserver_nodeport_allocator_allocated_ports
+ - kube_apiserver_nodeport_allocator_available_ports
+ - kube_apiserver_nodeport_allocator_allocation_total
+ - kube_apiserver_nodeport_allocator_allocation_errors_total
+
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+
+Since it is a purely in-memory feature the upgrade or downgrande doesn't have any impact.
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+no
+
+### Monitoring Requirements
+
+Following allocator metrics have a new label to each metric containing information about the type of allocation requested (dynamic or static):
+
+ - kube_apiserver_nodeport_allocator_allocation_total
+ - kube_apiserver_nodeport_allocator_allocation_errors_total
+
+The other allocator the metrics can not use the additional label because, when an port is released, there is not information on how the port was allocated.
+
+ - kube_apiserver_nodeport_allocator_allocated_ports
+ - kube_apiserver_nodeport_allocator_available_ports
+
+###### How can an operator determine if the feature is in use by workloads?
+
+An operator will only observe that the change in behavior described for dynamically assigned NodePort for Services.
+
+###### How can someone using this feature know that it is working for their instance?
+
+- [ ] Events
+ - Event Reason:
+- [ ] API .status
+ - Condition name:
+ - Other field:
+- [X] Other (treat as last resort)
+ - Details: Create services without setting the nodePort and observe that all the services ports allocated belong the upper band of the range, and once the upper band is exhausted it keep assigning ports on the lower band until the whole range is exhausted.
+
+###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
+
+N/A
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+- [ ] Metrics
+ - Metric name:
+ - [Optional] Aggregation method:
+ - Components exposing the metric:
+- [ ] Other (treat as last resort)
+ - Details:
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+
+This proposal adds 4 more metrics, and two of them is used to observe this feature. I think there is no any missing metrics should be added in feature.
+
+### Dependencies
+
+###### Does this feature depend on any specific services running in the cluster?
+
+No
+### Scalability
+
+###### Will enabling / using this feature result in any new API calls?
+
+No
+###### Will enabling / using this feature result in introducing new API types?
+
+No
+###### Will enabling / using this feature result in any new calls to the cloud provider?
+
+No
+
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+
+No
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+
+No
+###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+
+No
+
+###### Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?
+
+No
+
+### Troubleshooting
+
+###### How does this feature react if the API server and/or etcd is unavailable?
+
+N/A
+###### What are other known failure modes?
+
+N/A
+
+###### What steps should be taken if SLOs are not being met to determine the problem?
+
+## Implementation History
+
+## Drawbacks
+
+## Alternatives
diff --git a/keps/sig-network/3668-reserved-service-nodeport-range/kep.yaml b/keps/sig-network/3668-reserved-service-nodeport-range/kep.yaml
new file mode 100755
index 000000000000..7261316646a1
--- /dev/null
+++ b/keps/sig-network/3668-reserved-service-nodeport-range/kep.yaml
@@ -0,0 +1,40 @@
+title: Reserve Nodeport Ranges For Dynamic And Static Port Allocation
+kep-number: 3668
+authors:
+ - "@xuzhenglun"
+owning-sig: sig-network
+participating-sigs:
+status: implementable
+creation-date: 2022-11-30
+reviewers:
+approvers:
+ - "@thockin"
+ - "@aojea"
+
+see-also:
+ - "/keps/sig-network/3070-reserved-service-ip-range"
+
+# The target maturity stage in the current dev cycle for this KEP.
+stage: alpha
+
+# The most recent milestone for which work toward delivery of this KEP has been
+# done. This can be the current (upcoming) milestone, if it is being actively
+# worked on.
+latest-milestone: "v1.27"
+
+# The milestone at which this feature was, or is targeted to be, at each stage.
+milestone:
+ alpha: "v1.27"
+ beta: "v1.28"
+ stable: "v1.30"
+
+# The following PRR answers are required at alpha release
+# List the feature gate name and the components for which it must be enabled
+feature-gates:
+ - name: ServiceNodePortStaticSubrange
+ components:
+ - kube-apiserver
+disable-supported: true
+
+# The following PRR answers are required at beta release
+metrics:
diff --git a/keps/sig-network/3705-cloud-node-ips/README.md b/keps/sig-network/3705-cloud-node-ips/README.md
new file mode 100644
index 000000000000..7baba7274423
--- /dev/null
+++ b/keps/sig-network/3705-cloud-node-ips/README.md
@@ -0,0 +1,766 @@
+# KEP-3705: Cloud Dual-Stack --node-ip Handling
+
+
+- [Release Signoff Checklist](#release-signoff-checklist)
+- [Summary](#summary)
+- [Motivation](#motivation)
+ - [Goals](#goals)
+ - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+ - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+ - [Current behavior](#current-behavior)
+ - [Changes to --node-ip](#changes-to-)
+ - [Changes to the provided-node-ip annotation](#changes-to-the--annotation)
+ - [Changes to cloud providers](#changes-to-cloud-providers)
+ - [Example of --node-ip possibilities](#example-of--possibilities)
+ - [Test Plan](#test-plan)
+ - [Prerequisite testing updates](#prerequisite-testing-updates)
+ - [Unit tests](#unit-tests)
+ - [Integration tests](#integration-tests)
+ - [e2e tests](#e2e-tests)
+ - [Graduation Criteria](#graduation-criteria)
+ - [Alpha](#alpha)
+ - [Beta](#beta)
+ - [GA](#ga)
+ - [Deprecation](#deprecation)
+ - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+ - [Version Skew Strategy](#version-skew-strategy)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+ - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+ - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+ - [Monitoring Requirements](#monitoring-requirements)
+ - [Dependencies](#dependencies)
+ - [Scalability](#scalability)
+ - [Troubleshooting](#troubleshooting)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+
+
+## Release Signoff Checklist
+
+Items marked with (R) are required *prior to targeting to a milestone / release*.
+
+- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [ ] (R) KEP approvers have approved the KEP status as `implementable`
+- [ ] (R) Design details are appropriately documented
+- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+ - [ ] e2e Tests for all Beta API Operations (endpoints)
+ - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+ - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+- [ ] (R) Graduation criteria is in place
+ - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+- [ ] (R) Production readiness review completed
+- [ ] (R) Production readiness review approved
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://git.k8s.io/enhancements
+[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
+[kubernetes/website]: https://git.k8s.io/website
+
+## Summary
+
+Kubelet supports dual-stack `--node-ip` values for clusters with
+no cloud provider (eg, "bare metal" clusters), but not for clusters
+using a cloud provider. This KEP proposes to fix that.
+
+## Motivation
+
+### Goals
+
+- Allow administrators of clusters using external cloud providers to
+ override the cloud-selected node IPs in a more dual-stack-aware
+ manner:
+
+ - Allow administrators to override a single node IP on a node in a
+ dual-stack cluster without causing the node to become
+ single-stack.
+
+ - Allow administrators to override both node IPs on a node in a
+ dual-stack cluster.
+
+ - Allow administrators to override the order of node IPs on a node
+ in a dual-stack cluster (ie, requesting that nodes be
+ IPv6-primary dual stack rather than IPv4-primary dual stack, or
+ vice versa).
+
+ - Allow administrators to configure nodes to be single-stack when
+ the cloud provider suggests dual-stack, without needing to
+ provide a specific IPv4 or IPv6 node IP.
+
+- Define how kubelet will communicate these new intents to cloud
+ providers.
+
+- Update the code in `k8s.io/cloud-provider/node/helpers` to implement
+ the needed algorithms for the new behaviors.
+
+- Rename the `alpha.kubernetes.io/provided-node-ip` annotation, which
+ has not been alpha for a long time.
+
+### Non-Goals
+
+- Changing the behavior of nodes using legacy cloud providers.
+
+- Changing the node-IP-selection behavior in any currently-existing
+ Kubernetes cluster. This means that the default behavior when
+ `--node-ip` is not specified will remain the same, and the behavior
+ of any currently-allowed `--node-ip` value will remain the same. New
+ behavior will only be triggered by `--node-ip` values that would
+ have been rejected by kubelet in older clusters.
+
+- Adding the ability for nodes in clusters with cloud providers to use
+ node IPs that are not already known to the cloud provider. (In
+ particular, this implies that we will continue to not support
+ dual-stack nodes in clouds that do not themselves support
+ dual-stack.)
+
+- Improving the behavior of autodetecting the node IP in any other
+ ways. (Eg, being able to pass a CIDR rather than an IP.) There was
+ some discussion of ways we might do this in the abandoned [KEP-1664]
+ (which this KEP is a spiritual successor to), but if we want that,
+ it can be done later, independently of these changes. (These changes
+ were mostly desired in the context of non-cloud-provider clusters
+ anyway.)
+
+[KEP-1664]: https://github.com/kubernetes/enhancements/issues/1664
+
+## Proposal
+
+### Risks and Mitigations
+
+As the intention is to not change the user-visible behavior except in
+clusters where administrators explicitly make use of the new
+functionality, there should be no risk of breaking existing clusters,
+nor of surprising administrators by suddenly exposing node services on
+unexpected IPs.
+
+## Design Details
+
+### Current behavior
+
+Currently, when `--cloud-provider` is passed to kubelet, kubelet
+expects `--node-ip` to be either unset, or a single IP address. (If it
+is unset, that is equivalent to passing `--node-ip 0.0.0.0`, which
+means "autodetect an IPv4 address, or if there are no usable IPv4
+addresses, autodetect an IPv6 address".)
+
+If `--cloud-provider` and `--node-ip` are both specified (and
+`--node-ip` is not "`0.0.0.0`" or "`::`"), then kubelet will add an
+annotation to the node, `alpha.kubernetes.io/provided-node-ip`. Cloud
+providers expect this annotation to conform to the current expected
+`--node-ip` syntax (ie, a single value); if it does not, then they
+will log an error and not remove the
+`node.cloudprovider.kubernetes.io/uninitialized` taint from the node,
+causing the node to remain unusable until kubelet is restarted with a
+valid (or absent) `--node-ip`.
+
+When `--cloud-provider` is not passed, the `--node-ip` value can also
+be a comma-separated pair of dual-stack IP addresses. However, unlike
+in the single-stack case, the IPs in the dual-stack case are not
+currently allowed to be "unspecified" IPs (ie `0.0.0.0` or `::`); you
+can only make a (non-cloud) node be dual-stack if you explicitly
+specify both IPs that you want it to use.
+
+### Changes to `--node-ip`
+
+The most obvious required change is that we need to allow
+comma-separated dual-stack `--node-ip` values in clusters using
+external cloud providers (but _not_ in clusters using legacy cloud
+providers).
+
+Additionally, the fact that kubelet does not currently pass
+"`0.0.0.0`" and "`::`" to the cloud provider creates a compatibility
+problem: we would like for administrators to be able to say "use an
+IPv6 node IP but I don't care which one" in cloud-provider clusters
+like they can in non-cloud-provider clusters, but for compatibility
+reasons, we can't change the existing behavior of "`--cloud-provider
+external --node-ip ::`" (which doesn't do what it's "supposed to", but
+does have possibly-useful side effects that have led some users to use
+it anyway; see [kubernetes #111695].)
+
+So instead, we will add new syntax, and allow administrators to say
+"`--node-ip IPv4`" or "`--node-ip IPv6`" if they want to explicitly
+require that the cloud provider pick a node IP of a specific family.
+(This also improves on the behavior of the existing "`0.0.0.0`" and
+"`::`" options, because you almost never actually want the "or fall
+back to the other family if there are no IPs of this family" behavior
+that "`0.0.0.0`" and "`::`" imply.)
+
+Additionally, we will update the code to allow including "`IPv4`" and
+"`IPv6`" in dual-stack `--node-ip` values as well (in both cloud and
+non-cloud clusters). This code will have to check the status of the
+feature gate until the feature is GA.
+
+[kubernetes #111695]: https://github.com/kubernetes/kubernetes/issues/111695
+
+### Changes to the `provided-node-ip` annotation
+
+Currently, if the user passes an IP address to `--node-ip` which is
+not recognized by the cloud provider as being a valid IP for that
+node, kubelet will set that value in the `provided-node-ip`
+annotation, and the cloud provider will see it, realize that the node
+IP request can't be fulfilled, log an error, and leave the node in the
+tainted state.
+
+It makes sense to have the same behavior if the user passes a "new"
+(eg, dual-stack) `--node-ip` value to kubelet, but the cloud provider
+does not recognize the new syntax and thus can't fulfil the request.
+Conveniently, we can do this just by passing the dual-stack
+`--node-ip` value in the existing annotation; the old cloud provider
+will try to parse it as a single IP address, fail, log an error, and
+leave the node in the tainted state, which is exactly what we wanted
+it to do if it can't interpret the `--node-ip` value correctly.
+
+Therefore, we do not _need_ a new annotation for the new `--node-ip`
+values; we can continue to use the existing annotation, assuming
+existing cloud providers will treat unrecognized values as errors.
+
+That said, the existing annotation name is
+`alpha.kubernetes.io/provided-node-ip` but it hasn't been "alpha" for
+a long time. We should fix this. So:
+
+ 1. When `--node-ip` is unset, kubelet should delete both the legacy
+ `alpha.kubernetes.io/provided-node-ip` annotation and the new
+ `kubernetes.io/provided-node-ip` annotation (regardless of
+ whether the feature gate is enabled or not, to avoid problems
+ with rollback and skew).
+
+ 2. When the `CloudNodeIPs` feature is enabled and `--node-ip` is
+ set, kubelet will set both the legacy annotation and the new
+ annotation. (It always sets them both to the same value, even if
+ that's a value that old cloud providers won't understand).
+
+ 2. When the `CloudNodeIPs` feature is enabled, the cloud provider
+ will use the new `kubernetes.io/provided-node-ip` annotation _if
+ the legacy alpha annotation is not set_. (But if both annotations
+ are set, it will prefer the legacy annotation, so as to handle
+ rollbacks correctly.)
+
+ 3. A few releases after GA, kubelet can stop setting the legacy
+ annotation, and switch to unconditionally deleting it, and
+ setting/deleting the new annotation depending on whether
+ `--node-ip` was set or not. Cloud providers will also switch to
+ only using the new annotation, and perhaps logging a warning if
+ they see a node with the old annotation but not the new
+ annotation.
+
+Kubelet will preserve the existing behavior of _not_ passing
+"`0.0.0.0`" or "`::`" to the cloud provider, even via the new
+annotation. This is needed to preserve backward compatibility with
+current behavior in clusters using those `--node-ip` values. However,
+it _will_ pass "`IPv4`" and/or "`IPv6`" if they are passed as the
+`--node-ip`.
+
+### Changes to cloud providers
+
+Assuming that all cloud providers use the `"k8s.io/cloud-provider"`
+code to handle the node IP annotation and node address management, no
+cloud-provider-specific changes should be needed; we should be able to
+make the needed changes in the `cloud-provider` module, and then the
+individual providers just need to revendor to the new version.
+
+### Example of `--node-ip` possibilities
+
+Assume a node where the cloud has assigned the IPs `1.2.3.4`,
+`5.6.7.8`, `abcd::1234` and `abcd::5678`, in that order of preference.
+
+("SS" = "Single-Stack", "DS" = "Dual-Stack")
+
+| `--node-ip` value | New? | Annotation | Resulting node addresses |
+|----------------------|------|------------------------|--------------------------|
+| (none) | no | (unset) | `["1.2.3.4", "5.6.7.8", "abcd::1234", "abcd::5678"]` (DS IPv4-primary) |
+| `0.0.0.0` | no | (unset) | `["1.2.3.4", "5.6.7.8", "abcd::1234", "abcd::5678"]` (DS IPv4-primary) |
+| `::` | no | (unset) | `["1.2.3.4", "5.6.7.8", "abcd::1234", "abcd::5678"]` (DS IPv4-primary *) |
+| `1.2.3.4` | no | `"1.2.3.4"` | `["1.2.3.4"]` (SS IPv4) |
+| `9.10.11.12` | no | `"9.10.11.12"` | (error, because the requested IP is not available) |
+| `abcd::5678` | no | `"abcd::5678"` | `["abcd::5678"]` (SS IPv6) |
+| `1.2.3.4,abcd::1234` | yes* | `"1.2.3.4,abcd::1234"` | `["1.2.3.4", "abcd::1234"]` (DS IPv4-primary) |
+| `IPv4` | yes | `"IPv4"` | `["1.2.3.4"]` (SS IPv4) |
+| `IPv6` | yes | `"IPv6"` | `["abcd::1234"]` (SS IPv6) |
+| `IPv4,IPv6` | yes | `"IPv4,IPv6"` | `["1.2.3.4", "abcd::1234"]` (DS IPv4-primary) |
+| `IPv6,5.6.7.8` | yes | `"IPv6,5.6.7.8"` | `["abcd::1234", "5.6.7.8"]` (DS IPv6-primary) |
+| `IPv4,abcd::ef01` | yes | `"IPv4,abcd::ef01"` | (error, because the requested IPv6 IP is not available) |
+
+Notes:
+
+ - In the `--node-ip ::` case, kubelet will be expecting a
+ single-stack IPv6 or dual-stack IPv6-primary setup and so would
+ get slightly confused in this case since the cloud gave it a
+ dual-stack IPv4-primary configuration. (In particular, you would
+ have IPv4-primary nodes but IPv6-primary pods.)
+
+ - `--node-ip 1.2.3.4,abcd::ef01` was previously valid syntax when
+ using no `--cloud-provider`, but was not valid for cloud kubelets.
+
+<<[UNRESOLVED multiple-addresses ]>>
+
+In the examples above, `--node-ip IPv4` means "pick any one IPv4
+address", not "use all IPv4 addresses". This seemed to me to be
+consistent with the behavior when specifying a specific IP address (in
+which case all other IPs of the same NodeAddressType are removed from
+the list), but is inconsistent with the behavior of `--node-ip
+0.0.0.0` / `--node-ip ::` with legacy cloud providers and bare metal
+(where that affects the _sorting_ of IPs but does not do any
+_filtering_).
+
+In past discussions (eg https://kep.k8s.io/1664,
+https://issues.k8s.io/95768) no one has ever been entirely sure what
+the point of having multiple node.status.addresses values is, and
+there has never been a way to get multiple node.status.addresses
+values (of the same IP family) on "bare metal", so that would tend to
+prevent people from designing features that depended on multiple node
+addresses. So it's not clear that there's really any downside in
+filtering out the "unused" node IPs...
+
+<<[/UNRESOLVED]>>
+
+If the cloud only had IPv4 IPs for the node, then the same examples would look like:
+
+| `--node-ip` value | New? | Annotation | Resulting node addresses |
+|----------------------|------|------------------------|--------------------------|
+| (none) | no | (unset) | `["1.2.3.4", "5.6.7.8"]` (SS IPv4) |
+| `0.0.0.0` | no | (unset) | `["1.2.3.4", "5.6.7.8"]` (SS IPv4) |
+| `::` | no | (unset) | `["1.2.3.4", "5.6.7.8"]` (SS IPv4 *) |
+| `1.2.3.4` | no | `"1.2.3.4"` | `["1.2.3.4"]` (SS IPv4) |
+| `9.10.11.12` | no | `"9.10.11.12"` | (error, because the requested IP is not available) |
+| `abcd::5678` | no | `"abcd::5678"` | (error, because the requested IP is not available) |
+| `1.2.3.4,abcd::1234` | yes* | `"1.2.3.4,abcd::1234"` | (error, because the requested IPv6 IP is not available) |
+| `IPv4` | yes | `"IPv4"` | `["1.2.3.4"]` (SS IPv4) |
+| `IPv6` | yes | `"IPv6"` | (error, because no IPv6 IPs are available) |
+| `IPv4,IPv6` | yes | `"IPv4,IPv6"` | (error, because no IPv6 IPs are available) |
+| `IPv6,5.6.7.8` | yes | `"IPv6,5.6.7.8"` | (error, because no IPv6 IPs are available) |
+| `IPv4,abcd::ef01` | yes | `"IPv4,abcd::ef01"` | (error, because the requested IPv6 IP is not available) |
+
+In this case, kubelet would be even more confused in the
+`--node-ip ::` case, and some things would likely not work.
+By contrast, with `--node-ip IPv6`, the user would get a clear error.
+
+### Test Plan
+
+
+
+[X] I/we understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+##### Prerequisite testing updates
+
+
+
+##### Unit tests
+
+Most of the changes will be in `k8s.io/cloud-provider/node/helpers`.
+There will also be small changes in kubelet startup.
+
+- `k8s.io/kubernetes/pkg/kubelet`: `2023-01-30` - `66.9`
+- `k8s.io/kubernetes/pkg/kubelet/nodestatus`: `2023-01-30` - `91.2`
+- `k8s.io/kubernetes/vendor/k8s.io/cloud-provider/node/helpers`: `2023-01-30` - `31.7`
+- `k8s.io/kubernetes/vendor/k8s.io/cloud-provider/node/helpers/address.go`: `2023-01-30` - `100`
+
+##### Integration tests
+
+
+
+```
+<<[UNRESOLVED integration ]>>
+
+I don't know what existing integration testing of kubelet and cloud
+providers there is. Given unit tests for the new `--node-ip` parsing
+and `node.status.addresses`-setting code, and e2e tests of some sort,
+we probably don't need integration tests.
+
+<<[/UNRESOLVED]>>
+```
+
+- : , ,
+, ", and then we could
+pass `--node-ip ,IPv6` to kubelet. If the cloud
+provider did not receive and interpret the `--node-ip` value
+correctly, then we would end up with node IPs that didn't work and the
+test job should fail.
+
+<<[/UNRESOLVED]>>
+```
+
+- : : :
+ +CCIAddContainerResource (Pod, container, cgroup, containerid):
+ +CCIRemoveContainerResource (containerid): err
+
+`CCIAdmit` function provides information if a given container belonging to a
+Pod and having a specific CCI spec can be admitted to next stage of the allocation
+pipeline. The admission will return a reserved resource-set or error. In case of
+successful admission the resource set will be stored in the CCI Store and made
+available to the cpu manager policy. In the case of failure the error is reported
+back to user and the Pod allocation is cancelled.
+
+`CCIAddContainerResource` function is called before container start with a given
+Pod name and container name, cgroup of Pod and the container id. The driver than
+performs an allocation of the admitted resource-set. In case of failure of the
+driver and error is returned.
+
+`CCIRemoveContainerResource` function is called to free the cpu resources taken
+by a container. The functions returns nil if the operation was successful or an
+error if the operation failed on the plugin side.
+
+ /*
+ CCIDriver is a grpc service interface for CCI resource kubelet drivers.
+ The interface provides admission, allocation and cleanup entrypoints.
+ Drivers are registered as kubelet plugins and will be called by the
+ CCI resource manager for pods which are associated with this driver.
+ */
+ service CCIDriver {
+ //cciAdmit admission function to reserve resources for a given container
+ rpc CCIAdmit(AdmitRequest) returns (AdmitResponse) {}
+
+ //cciAddContainerResource allocation entry point for admitted containers
+ rpc CCIAddContainerResource (AddResourceRequest)
+ returns (AddResourceResponse) {}
+
+ //cciRemoveContainerResource clea ubelet dted resources for a given container
+ rpc CCIRemoveContainerResource (RemoveResourceRequest)
+ returns (RemoveResourceResponse) {}
+ }
+
+ message AdmitRequest{
+ // currently available cpus
+ string availablecpus = 1;
+ // pod identfier
+ string pod = 2;
+ // container identfier
+ string container = 3;
+ // cci spec for the container, subset of podspec
+ string cci_spec = 4;
+ }
+
+ message ResourceSet{
+ // admitted cpuset
+ string cpuset = 1;
+ // flag if exclisve
+ bool exclusive = 2;
+ }
+
+ message AdmitResponse {
+ // allocated resource set
+ ResourceSet rset = 1;
+ // error object
+ string err = 2;
+ }
+
+ message AddResourceRequest{
+ // pod cgroup
+ string cgroup = 1;
+ // pod identifier
+ string pod = 2;
+ // container identifier
+ string container = 3;
+ // container id
+ string container_id = 4;
+ }
+
+
+
+ message AddResourceResponse {
+ // error object
+ string err = 1;
+ }
+
+ message RemoveResourceRequest {
+ // id of the container to be removed
+ string container_id = 1;
+ }
+
+ message RemoveResourceResponse {
+ // error object
+ string err = 1;
+ }
+
+5. CCI Drivers Factory API>
+Add detailed scenarios and result here, and cc @wojtek-t.
+< >
+
###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+No.
+
### Monitoring Requirements
-Add a label `notReady` to the existing metric `pending_pods` to distinguish unschedulable Pods:
+A new "queue" type `gated` is added to the existing metric `scheduler_pending_pods` to distinguish
+unschedulable Pods:
-- `unschedulable` (existing): scheduler tried but cannot find any Node to host the Pod
-- `notReady` (new): scheduler respect the Pod's present `schedulingGates` and hence not schedule it
+- `scheduler_pending_pods{queue="unschedulable"}` (existing): scheduler tried but cannot find any
+Node to host the Pod
+- `scheduler_pending_pods{queue="gated"}` (new): scheduler respect the Pod's present `schedulingGates`
+and hence not schedule it
+
+The metric `scheduler_plugin_execution_duration_seconds{plugin="SchedulingGates"}` gives a histogram
+to show the Nth percentile value how SchedulingGates plugin is executed.
Moreover, to explicitly indicate a Pod's scheduling-unready state, a condition
-`{type:PodScheduled, reason:WaitingForGates}` is introduced.
+`{type:PodScheduled, reason:SchedulingGated}` is introduced.
###### How can an operator determine if the feature is in use by workloads?
@@ -836,6 +872,10 @@ checking if there are objects with field X set) may be a last resort. Avoid
logs or events for this purpose.
-->
+- observe non-zero value for the metric `pending_pods{queue="gated"}`
+- observe entries for the metric `scheduler_plugin_execution_duration_seconds{plugin="SchedulingGates"}`
+- observe non-empty value in a Pod's `.spec.schedulingGates` field
+
###### How can someone using this feature know that it is working for their instance?
+N/A.
+
###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
-- [ ] Metrics
- - Metric name:
- - [Optional] Aggregation method:
- - Components exposing the metric:
-- [ ] Other (treat as last resort)
- - Details:
+- [x] Metrics
+ - Metric name: scheduler_pending_pods{queue="gated"}
+ - Metric name: scheduler_plugin_execution_duration_seconds{plugin="SchedulingGates"}
+ - Components exposing the metric: kube-scheduler
###### Are there any missing metrics that would be useful to have to improve observability of this feature?
@@ -892,12 +937,16 @@ Describe the metrics themselves and the reasons why they weren't added (e.g., co
implementation difficulties, etc.).
-->
+N/A.
+
### Dependencies
+N/A.
+
###### Does this feature depend on any specific services running in the cluster?
+N/A.
+
### Scalability
+The feature itself doesn't generate API calls. But it's expected the API Server would receive
+additional update/patch requests to mutate the scheduling gates, by external controllers.
+
###### Will enabling / using this feature result in introducing new API types?
+No.
+
###### Will enabling / using this feature result in any new calls to the cloud provider?
+No.
+
###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+- No to existing API objects that doesn't use this feature.
+- For API objects that use this feature:
+ - API type: Pod
+ - Estimated increase in size: new field `.spec.schedulingGates` about ~64 bytes (in the case of 2 scheduling gates)
+
###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+No.
+
###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+No.
+
### Troubleshooting
+In a highly-available cluster, if there are skewed API Servers, some update requests
+may get accepted and some may get rejected.
+
###### What steps should be taken if SLOs are not being met to determine the problem?
+N/A.
+
## Implementation History
-- 2022-09-16 - Initial Proposal
+- 2022-09-16: Initial KEP
+- 2022-01-14: Graduate the feature to Beta
## Drawbacks
@@ -1052,11 +1131,11 @@ not need to be as detailed as the proposal, but should include enough
information to express the idea and why it was not acceptable.
-->
-- Define a boolean filed `.spec.schedulingPaused`. Its value is optionally initialized to `True` to
+Define a boolean filed `.spec.schedulingPaused`. Its value is optionally initialized to `True` to
indicate it's not scheduling-ready, and flipped to `False` (by a controller) afterwards to trigger
this Pod's scheduling cycle.
- This approach is not chosen because it cannot support multiple independent controllers to control
+This approach is not chosen because it cannot support multiple independent controllers to control
specific aspect a Pod's scheduling readiness.
## Infrastructure Needed (Optional)
diff --git a/keps/sig-scheduling/3521-pod-scheduling-readiness/kep.yaml b/keps/sig-scheduling/3521-pod-scheduling-readiness/kep.yaml
index 3c865a57e01c..b864c45495c0 100644
--- a/keps/sig-scheduling/3521-pod-scheduling-readiness/kep.yaml
+++ b/keps/sig-scheduling/3521-pod-scheduling-readiness/kep.yaml
@@ -18,12 +18,12 @@ see-also:
- "/keps/sig-scheduling/624-scheduling-framework"
# The target maturity stage in the current dev cycle for this KEP.
-stage: alpha
+stage: beta
# The most recent milestone for which work toward delivery of this KEP has been
# done. This can be the current (upcoming) milestone, if it is being actively
# worked on.
-latest-milestone: "v1.26"
+latest-milestone: "v1.27"
# The milestone at which this feature was, or is targeted to be, at each stage.
milestone:
@@ -42,4 +42,4 @@ disable-supported: true
# The following PRR answers are required at beta release
metrics:
- - TBD
+ - scheduler_pending_pods{queue="gated"}
diff --git a/keps/sig-scheduling/3633-matchlabelkeys-to-podaffinity/README.md b/keps/sig-scheduling/3633-matchlabelkeys-to-podaffinity/README.md
new file mode 100644
index 000000000000..6d00d042667e
--- /dev/null
+++ b/keps/sig-scheduling/3633-matchlabelkeys-to-podaffinity/README.md
@@ -0,0 +1,924 @@
+
+# KEP-3633: Introduce MatchLabelKeys to PodAffinity and PodAntiAffinity
+
+
+
+
+
+
+- [Release Signoff Checklist](#release-signoff-checklist)
+- [Summary](#summary)
+- [Motivation](#motivation)
+ - [Goals](#goals)
+ - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+ - [User Stories (Optional)](#user-stories-optional)
+ - [Story 1](#story-1)
+ - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional)
+ - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+ - [Test Plan](#test-plan)
+ - [Prerequisite testing updates](#prerequisite-testing-updates)
+ - [Unit tests](#unit-tests)
+ - [Integration tests](#integration-tests)
+ - [e2e tests](#e2e-tests)
+ - [Graduation Criteria](#graduation-criteria)
+ - [Alpha](#alpha)
+ - [Beta](#beta)
+ - [GA](#ga)
+ - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+ - [Version Skew Strategy](#version-skew-strategy)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+ - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+ - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
+ - [Monitoring Requirements](#monitoring-requirements)
+ - [Dependencies](#dependencies)
+ - [Scalability](#scalability)
+ - [Troubleshooting](#troubleshooting)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+- [Infrastructure Needed (Optional)](#infrastructure-needed-optional)
+
+
+## Release Signoff Checklist
+
+
+
+Items marked with (R) are required *prior to targeting to a milestone / release*.
+
+- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR)
+- [ ] (R) KEP approvers have approved the KEP status as `implementable`
+- [ ] (R) Design details are appropriately documented
+- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
+ - [ ] e2e Tests for all Beta API Operations (endpoints)
+ - [ ] (R) Ensure GA e2e tests for meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+ - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free
+- [ ] (R) Graduation criteria is in place
+ - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md)
+- [ ] (R) Production readiness review completed
+- [ ] (R) Production readiness review approved
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
+- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://git.k8s.io/enhancements
+[kubernetes/kubernetes]: https://git.k8s.io/kubernetes
+[kubernetes/website]: https://git.k8s.io/website
+
+## Summary
+
+
+
+This KEP proposes introducing a complementary field `MatchLabelKeys` to `PodAffinityTerm`.
+This enables users to finely control the scope where Pods are expected to co-exist (PodAffinity)
+or not (PodAntiAffinity), on top of the existing `LabelSelector`.
+
+## Motivation
+
+
+
+During a workload's rolling upgrade, depending on its `upgradeStrategy`, old and new versions of
+Pods may co-exist in the cluster.
+As scheduler cannot distinguish "old" from "new", it cannot properly honor the API semantics of
+PodAffinity and PodAntiAffiniy during the upgrade.
+In the worse case, new version of Pod cannot be scheduled if it's a saturated cluster: [1], [2], [3].
+
+[1]: https://github.com/kubernetes/kubernetes/issues/56539#issuecomment-449757010
+[2]: https://github.com/kubernetes/kubernetes/issues/90151
+[3]: https://github.com/kubernetes/kubernetes/issues/103584
+
+On the other hand, on an idle cluster, this can cause the scheduling result sub-optimal because
+some qualifying Nodes are filtered out incorrectly.
+
+The same issue applies to other scheduling directives as well. For example, MatchLabelKeys was introduced in topologyConstaint in [KEP-3243: Respect PodTopologySpread after rolling upgrades](https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades).
+
+### Goals
+
+- Introduce `MatchLabelKeys` in `PodAffinityTerm` to let users define the scope where Pods are evaluated in required and preferred Pod(Anti)Affinity.
+
+### Non-Goals
+
+
+
+- Apply additional internal labels when evaluating `MatchLabelKeys`
+
+## Proposal
+
+
+
+### User Stories (Optional)
+
+
+
+#### Story 1
+
+When users run a rolling update with a deployment that uses required PodAffinity,
+and they want only replicas from the same replicaset to be evaluated.
+
+The deployment controller adds [pod-template-hash](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#pod-template-hash-label) to underlying ReplicaSet and thus every Pod created from Deployment carries the hash string.
+
+Therefore, users can use `pod-template-hash` in `MatchlabelKeys` to inform the scheduler to only evaluate Pods with the same `pod-template-hash` value.
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: application-server
+...
+ affinity:
+ podAffinity:
+ requiredDuringSchedulingIgnoredDuringExecution:
+ - labelSelector:
+ matchExpressions:
+ - key: app
+ operator: In
+ values:
+ - database
+ topologyKey: topology.kubernetes.io/zone
+ matchlabelKeys: # ADDED
+ - pod-template-hash
+```
+
+### Notes/Constraints/Caveats (Optional)
+
+
+
+In most scenarios, users can use the label keyed with `pod-template-hash` added
+automatically by the Deployment controller to distinguish between different
+revisions in a single Deployment. But for more complex scenarios
+(e.g., Pod(Anti)Affinity associating two deployments at the same time), users are
+responsible for providing common labels to identify which pods should be grouped.
+
+### Risks and Mitigations
+
+
+
+In addition to using `pod-template-hash` added by the Deployment controller,
+users can also provide the customized key in `MatchLabelKeys` to identify
+which pods should be grouped. If so, the user needs to ensure that it is
+correct and not duplicated with other unrelated workloads.
+
+## Design Details
+
+
+
+A new optional field `MatchLabelKeys` is introduced to `PodAffinityTerm`.
+
+```go
+type PodAffinityTerm struct {
+ LabelSelector *metav1.LabelSelector
+ Namespaces []string
+ TopologyKey string
+ NamespaceSelector *metav1.LabelSelector
+
+ // MatchLabelKeys is a set of pod label keys to select which pods will
+ // be taken into consideration. The keys are used to lookup values from the
+ // incoming pod labels, those key-value labels are ANDed with `LabelSelector`
+ // to select the group of existing pods which pods will be taken into consideration
+ // for the incoming pod's pod (anti) affinity. Keys that don't exist in the incoming
+ // pod labels will be ignored. The default value is empty.
+ // +optional
+ MatchLabelKeys []string
+}
+```
+
+The inter-Pod Affinity plugin will obtain the labels from the pod
+labels by the keys in `MatchLabelKeys`. The obtained labels will be merged
+to `LabelSelector` of `PodAffinityTerm` to filter and group pods.
+The pods belonging to the same group will be evaluated.
+
+### Test Plan
+
+
+
+[x] I/we understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes necessary
+to implement this enhancement.
+
+##### Prerequisite testing updates
+
+
+
+##### Unit tests
+
+
+
+
+
+- `k8s.io/kubernetes/pkg/scheduler/framework/plugins/interpodaffinity/filtering.go`: `2022-11-09 14:43 JST` (The commit hash: `20a9f7786aa4ee0b6e1619c7974ea4562d2b2500`) - `91.7%`
+- `k8s.io/kubernetes/pkg/scheduler/framework/plugins/interpodaffinity/filtering.go`: `2022-11-09 14:43 JST` (The commit hash: `20a9f7786aa4ee0b6e1619c7974ea4562d2b2500`) - `87.3%`
+
+##### Integration tests
+
+
+
+- These tests will be added.
+ - `MatchLabelKeys` in `PodAffinity` (both in Filter and Score) works as expected.
+ - `MatchLabelKeys` in `PodAntiAffinity` (both in Filter and Score) works as expected.
+ - `MatchLabelKeys` with the feature gate enabled/disabled.
+
+**Filter**
+- `k8s.io/kubernetes/test/integration/scheduler/filters/filters_test.go`: https://storage.googleapis.com/k8s-triage/index.html?test=TestPodTopologySpreadFilter
+
+**Score**
+- `k8s.io/kubernetes/test/integration/scheduler/scoring/priorities_test.go`: https://storage.googleapis.com/k8s-triage/index.html?test=TestPodTopologySpreadScoring
+
+Also, we should make sure this feature brings no significant performance degradation in both Filter and Score.
+
+- `k8s.io/kubernetes/test/integration/scheduler_perf/scheduler_perf_test.go`: https://storage.googleapis.com/k8s-triage/index.html?test=BenchmarkPerfScheduling
+
+##### e2e tests
+
+
+
+- These e2e tests will be added.
+ - `MatchLabelKeys: ["pod-template-hash"]` in `PodAffinity` (both in Filter and Score) works as expected with a rolling upgrade scenario.
+ - `MatchLabelKeys: ["pod-template-hash"]` in `PodAntiAffinity` (both in Filter and Score) works as expected with a rolling upgrade scenario.
+
+### Graduation Criteria
+
+
+
+#### Alpha
+
+- Feature implemented behind a feature flag
+- Unit tests and e2e tests are implemented
+- No significant performance degradation is observed from the benchmark test
+
+#### Beta
+
+- The feature gate is enabled by default.
+
+#### GA
+
+- No negative feedback.
+- No bug issues reported.
+
+### Upgrade / Downgrade Strategy
+
+
+
+**Upgrade**
+
+The previous PodAffinity/PodAntiAffinity behavior will not be broken. Users can continue to use
+their Pod specs as it is.
+
+To use this enhancement, users need to enable the feature gate (during this feature is in the alpha.),
+and add `MatchLabelKeys` on their PodAffinity/PodAntiAffinity.
+
+**Downgrade**
+
+kube-apiserver will ignore `MatchLabelKeys` in PodAffinity/PodAntiAffinity,
+and thus, kube-scheduler will also do nothing with it.
+
+### Version Skew Strategy
+
+
+
+N/A
+
+## Production Readiness Review Questionnaire
+
+
+
+### Feature Enablement and Rollback
+
+
+
+###### How can this feature be enabled / disabled in a live cluster?
+
+
+
+- [x] Feature gate (also fill in values in `kep.yaml`)
+ - Feature gate name: `MatchLabelKeysInPodAffinityAndPodAntiAffinity`
+ - Components depending on the feature gate: `kube-scheduler`, `kube-apiserver`
+- [ ] Other
+
+###### Does enabling the feature change any default behavior?
+
+
+
+No.
+
+###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?
+
+
+
+The feature can be disabled in Alpha and Beta versions
+by restarting kube-apiserver and kube-scheduler with the feature-gate off.
+In terms of Stable versions, users can choose to opt-out by not setting the
+`MatchLabelKeys` field.
+
+
+###### What happens if we reenable the feature if it was previously rolled back?
+
+Scheduling of new Pods is affected. (Scheduled Pods aren't affected.)
+
+###### Are there any tests for feature enablement/disablement?
+
+
+
+No. But, the tests to confirm the behavior on switching the feature gate will be added.
+
+### Rollout, Upgrade and Rollback Planning
+
+
+
+###### How can a rollout or rollback fail? Can it impact already running workloads?
+
+
+
+###### What specific metrics should inform a rollback?
+
+
+
+###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+
+
+
+###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+
+
+
+### Monitoring Requirements
+
+
+
+###### How can an operator determine if the feature is in use by workloads?
+
+
+
+###### How can someone using this feature know that it is working for their instance?
+
+
+
+- [ ] Events
+ - Event Reason:
+- [ ] API .status
+ - Condition name:
+ - Other field:
+- [ ] Other (treat as last resort)
+ - Details:
+
+###### What are the reasonable SLOs (Service Level Objectives) for the enhancement?
+
+
+
+###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
+
+
+
+- [ ] Metrics
+ - Metric name:
+ - [Optional] Aggregation method:
+ - Components exposing the metric:
+- [ ] Other (treat as last resort)
+ - Details:
+
+###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+
+
+
+### Dependencies
+
+
+
+###### Does this feature depend on any specific services running in the cluster?
+
+
+
+### Scalability
+
+
+
+###### Will enabling / using this feature result in any new API calls?
+
+
+
+###### Will enabling / using this feature result in introducing new API types?
+
+
+
+###### Will enabling / using this feature result in any new calls to the cloud provider?
+
+
+
+###### Will enabling / using this feature result in increasing size or count of the existing API objects?
+
+
+
+###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
+
+
+
+###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
+
+
+
+### Troubleshooting
+
+
+
+###### How does this feature react if the API server and/or etcd is unavailable?
+
+###### What are other known failure modes?
+
+
+
+###### What steps should be taken if SLOs are not being met to determine the problem?
+
+## Implementation History
+
+
+
+ - 2022-11-09: Initial KEP PR is submitted.
+
+## Drawbacks
+
+
+
+## Alternatives
+
+
+
+## Infrastructure Needed (Optional)
+
+
diff --git a/keps/sig-scheduling/3633-matchlabelkeys-to-podaffinity/kep.yaml b/keps/sig-scheduling/3633-matchlabelkeys-to-podaffinity/kep.yaml
new file mode 100644
index 000000000000..17c2d8e713fb
--- /dev/null
+++ b/keps/sig-scheduling/3633-matchlabelkeys-to-podaffinity/kep.yaml
@@ -0,0 +1,29 @@
+title: Introduce MatchLabelKeys to PodAffinity and PodAntiAffinity
+kep-number: 3633
+authors:
+ - "@sanposhiho"
+owning-sig: sig-scheduling
+status: provisional
+creation-date: 2022-11-09
+reviewers:
+ - "@Huang-Wei"
+approvers:
+ - "@Huang-Wei"
+
+see-also:
+ - "/keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades"
+
+stage: alpha
+
+latest-milestone: "v1.27"
+
+milestone:
+ alpha: "v1.27"
+ beta: "v1.28"
+ stable: "v1.30"
+
+feature-gates:
+ - name: MatchLabelKeysInPodAffinity
+ components:
+ - kube-scheduler
+disable-supported: true
diff --git a/keps/sig-storage/2485-read-write-once-pod-pv-access-mode/README.md b/keps/sig-storage/2485-read-write-once-pod-pv-access-mode/README.md
index a327a089c87b..313655346379 100644
--- a/keps/sig-storage/2485-read-write-once-pod-pv-access-mode/README.md
+++ b/keps/sig-storage/2485-read-write-once-pod-pv-access-mode/README.md
@@ -94,17 +94,23 @@ tags, and then generate with `hack/update-toc.sh`.
- [Design Details](#design-details)
- [Kubernetes Changes, Access Mode](#kubernetes-changes-access-mode)
- [Scheduler Enforcement](#scheduler-enforcement)
+ - [Alpha](#alpha)
+ - [Beta](#beta)
- [Mount Enforcement](#mount-enforcement)
- [CSI Specification Changes, Volume Capabilities](#csi-specification-changes-volume-capabilities)
- [Test Plan](#test-plan)
+ - [Prerequisite testing updates](#prerequisite-testing-updates)
+ - [Unit tests](#unit-tests)
+ - [Integration tests](#integration-tests)
+ - [e2e tests](#e2e-tests)
- [Validation of PersistentVolumeSpec Object](#validation-of-persistentvolumespec-object)
- [Mounting and Mapping with ReadWriteOncePod](#mounting-and-mapping-with-readwriteoncepod)
- [Mounting and Mapping with ReadWriteOnce](#mounting-and-mapping-with-readwriteonce)
- [Mapping Kubernetes Access Modes to CSI Volume Capability Access Modes](#mapping-kubernetes-access-modes-to-csi-volume-capability-access-modes)
- [End to End Tests](#end-to-end-tests)
- [Graduation Criteria](#graduation-criteria)
- - [Alpha](#alpha)
- - [Beta](#beta)
+ - [Alpha](#alpha-1)
+ - [Beta](#beta-1)
- [GA](#ga)
- [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
- [Version Skew Strategy](#version-skew-strategy)
@@ -112,6 +118,9 @@ tags, and then generate with `hack/update-toc.sh`.
- [API Server Version N / Scheduler Version N-1 / Kubelet Version N-1 or N-2](#api-server-version-n--scheduler-version-n-1--kubelet-version-n-1-or-n-2)
- [API Understands ReadWriteOncePod, CSI Sidecars Do Not](#api-understands-readwriteoncepod-csi-sidecars-do-not)
- [CSI Controller Service Understands New CSI Access Modes, CSI Node Service Does Not](#csi-controller-service-understands-new-csi-access-modes-csi-node-service-does-not)
+ - [API Server Has Feature Enabled, Scheduler and Kubelet Do Not](#api-server-has-feature-enabled-scheduler-and-kubelet-do-not)
+ - [API Server Has Feature Enabled, Scheduler Does not, Kubelet Does](#api-server-has-feature-enabled-scheduler-does-not-kubelet-does)
+ - [API Server Has Feature Enabled, Scheduler Does, Kubelet Does Not](#api-server-has-feature-enabled-scheduler-does-kubelet-does-not)
- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
- [Feature Enablement and Rollback](#feature-enablement-and-rollback)
- [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning)
@@ -389,6 +398,8 @@ This access mode will be enforced in two places:
#### Scheduler Enforcement
+##### Alpha
+
First is at the time a pod is scheduled. When scheduling a pod, if another pod
is found using the same PVC and the PVC uses ReadWriteOncePod, then scheduling
will fail and the pod will be considered UnschedulableAndUnresolvable.
@@ -407,6 +418,26 @@ marked UnschedulableAndUnresolvable.
[volume restrictions plugin]: https://github.com/kubernetes/kubernetes/blob/v1.21.0/pkg/scheduler/framework/plugins/volumerestrictions/volume_restrictions.go#L29
[node info cache]: https://github.com/kubernetes/kubernetes/blob/v1.21.0/pkg/scheduler/framework/types.go#L357
+##### Beta
+
+Support for pod preemption is enforced in beta.
+
+When a pod (A) with a ReadWriteOncePod PVC is scheduled, if another pod (B) is
+found using the same PVC and pod (A) has higher priority, the scheduler will
+return an "Unschedulable" status and attempt to preempt pod (B).
+
+The implementation goes like follows:
+
+In the PreFilter phase of the volume restrictions scheduler plugin, we will
+build a cache of the ReadWriteOncePod PVCs for the pod-to-be-scheduled and the
+number of conflicting PVC references (pods already using any of these PVCs).
+This cache will be saved as part of the scheduler's cycleState and forwarded to
+the following step. During AddPod and RemovePod, if there is a conflict we will
+add or subtract from the number of conflicting references. During the Filter
+phase, if the cache contains a non-zero amount of conflicting references then
+return "Unschedulable". If the pod has a PVC that cannot be found, return
+"UnschedulableAndUnresolvable".
+
#### Mount Enforcement
As an additional precaution this will also be enforced at the time a volume is
@@ -487,14 +518,7 @@ external-attacher, and external-resizer.
+[X] I/we understand the owners of the involved components may require updates to
+existing tests to make this code solid enough prior to committing the changes
+necessary to implement this enhancement.
+
+##### Prerequisite testing updates
+
+
+
+None. New tests will be added for the transition to beta to support scheduler
+changes.
+
+##### Unit tests
+
+
+
+
+
+In alpha, the following unit tests were updated. See
+https://github.com/kubernetes/kubernetes/pull/102028 and
+https://github.com/kubernetes/kubernetes/pull/103082 for more context.
+
+- `k8s.io/kubernetes/pkg/apis/core/helper`: `09-22-2022` - `26.2`
+- `k8s.io/kubernetes/pkg/apis/core/v1/helper`: `09-22-2022` - `56.9`
+- `k8s.io/kubernetes/pkg/apis/core/validation`: `09-22-2022` - `82.3`
+- `k8s.io/kubernetes/pkg/controller/volume/persistentvolume`: `09-22-2022` - `79.4`
+- `k8s.io/kubernetes/pkg/kubelet/volumemanager/cache`: `09-22-2022` - `66.3`
+- `k8s.io/kubernetes/pkg/volume/csi/csi_client.go`: `09-22-2022` - `76.2`
+- `k8s.io/kubernetes/pkg/scheduler/apis/config/v1beta2`: `09-22-2022` - `76.8`
+- `k8s.io/kubernetes/pkg/scheduler/framework/plugins/volumerestrictions`: `09-22-2022` - `85`
+- `k8s.io/kubernetes/pkg/scheduler/framework`: `09-22-2022` - `77.1`
+
+In beta, there will be additional unit test coverage for
+`k8s.io/kubernetes/pkg/scheduler/framework/plugins/volumerestrictions` to cover
+preemption logic.
+
+##### Integration tests
+
+
+
+Integration tests for scheduler plugin behavior are available here:
+
+- [test/integration/scheduler/filters/filters_test.go] : [testgrid]
+
+
+[test/integration/scheduler/filters/filters_test.go]: https://github.com/kubernetes/kubernetes/blob/v1.25.2/test/integration/scheduler/filters/filters_test.go
+[testgrid]: https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-integration&include-filter-by-regex=k8s.io/kubernetes/test/integration/scheduler/filters.TestUnschedulablePodBecomesSchedulable/scheduled_pod_uses_read-write-once-pod_pvc
+
+##### e2e tests
+
+
+
+For alpha, to test this feature end to end, we will need to check the following cases:
+
+- A ReadWriteOncePod volume will succeed mounting when consumed by a single pod
+ on a node
+- A ReadWriteOncePod volume will fail to mount when consumed by a second pod on
+ the same node
+- A ReadWriteOncePod volume will fail to attach when consumed by a second pod on
+ a different node
+
+For testing the mapping for ReadWriteOnce, we will update the CSI hostpath
+driver to support the new volume capability access modes and cut a release. The
+existing Kubernetes end to end tests will be updated to use this version which
+will test the K8s to CSI access mode mapping behavior because most storage end
+to end tests rely on the ReadWriteOnce access mode, which now maps to the
+SINGLE_NODE_MULTI_WRITER CSI access mode.
+
+E2E tests for alpha behavior can be found here:
+
+- [test/e2e/storage/testsuites/readwriteoncepod.go] : [k8s-triage]
+
+For beta, we will want to cover the additional cases for preemption:
+
+- A high-priority pod requesting a ReadWriteOncePod volume that's already in-use
+ will result in the preemption of the pod previously using the volume
+- A low-priority (or no priority) pod requesting a ReadWriteOncePod volume
+ that's already in-use will result in it being UnschedulableAndUnresolvable
+
+[test/e2e/storage/testsuites/readwriteoncepod.go]: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/storage/testsuites/readwriteoncepod.go
+[k8s-triage]: https://storage.googleapis.com/k8s-triage/index.html?pr=1&test=read-write-once-pod
+
#### Validation of PersistentVolumeSpec Object
To test the validation logic of the PersistentVolumeSpec, we need to check the
@@ -538,20 +675,6 @@ well as in CSI sidecars.
#### End to End Tests
-To test this feature end to end, we will need to check the following cases:
-
-- A ReadWriteOncePod volume will succeed mounting when consumed by a single pod
- on a node
-- A ReadWriteOncePod volume will fail to mount when consumed by a second pod on
- the same node
-- A ReadWriteOncePod volume will fail to attach when consumed by a second pod on
- a different node
-
-For testing the mapping for ReadWriteOnce, we should update the mock CSI driver
-to support the new volume capability access modes and cut a release. The
-existing Kubernetes end to end tests will be updated to use this version which
-will test the mapping behavior because most storage end to end tests rely on the
-ReadWriteOnce access mode.
### Graduation Criteria
@@ -623,8 +746,6 @@ in back-to-back releases.
- Scheduler enforces ReadWriteOncePod access mode by marking pods as
Unschedulable, preemption logic added
- ReadWriteOncePod access mode has end to end test coverage
-- Mock CSI driver supports `SINGLE_NODE_*_WRITER` access modes, relevant end to
- end tests updated to use this driver
- Hostpath CSI driver supports `SINGLE_NODE_*_WRITER` access modes, relevant end
to end tests updated to use this driver
@@ -653,8 +774,11 @@ feature gate enabled. Additionally they will need to update their CSI drivers
and sidecars to versions that depend on the new Kubernetes API and CSI spec.
When downgrading a cluster to disable this feature, the user will need to
-restart the kube-apiserver with the ReadWriteOncePod feature gate disabled. When
-disabling this feature gate, any existing volumes with the ReadWriteOncePod
+disable the ReadWriteOncePod feature gate in kube-apiserver, kube-scheduler, and
+kubelet. They may also roll back their CSI sidecars if they are encountering
+errors.
+
+When disabling this feature gate, any existing volumes with the ReadWriteOncePod
access mode will continue to exist, but can only be deleted. An alternative is
to allow these volumes to be treated as ReadWriteOnce, however that would
violate the intent of the user and so it is not recommended.
@@ -735,6 +859,45 @@ node service does not understand these access modes, the behavior will depend on
the CSI driver and how it treats unknown access modes. The recommendation is to
upgrade the CSI drivers for the controller and node services together.
+#### API Server Has Feature Enabled, Scheduler and Kubelet Do Not
+
+In this scenario, the kube-scheduler will not enforce the ReadWriteOncePod
+access mode and proceed to schedule pods sharing the same ReadWriteOncePod PVCs.
+
+If you have two pods sharing the same ReadWriteOncePod PVC and they land on
+separate nodes, the volume will [only be able to attach to a single node]. The
+other pod will be stuck because the volume is already attached elsewhere.
+
+However, if both pods land on the same node, kubelet will not enforce the access
+mode and allow both pods to mount the same ReadWriteOncePod volume.
+
+[only be able to attach to a single node]: https://github.com/kubernetes/kubernetes/blob/v1.26.1/pkg/volume/util/util.go#L716-L718
+
+#### API Server Has Feature Enabled, Scheduler Does not, Kubelet Does
+
+In this scenario, the kube-scheduler will not enforce the ReadWriteOncePod
+access mode and proceed to schedule pods sharing the same ReadWriteOncePod PVCs.
+
+If you have two pods sharing the same ReadWriteOncePod PVC and they land on
+separate nodes, the volume will [only be able to attach to a single node]. The
+other pod will be stuck because the volume is already attached elsewhere.
+
+If both pods land on the same node, kubelet will enforce the access mode and
+only allow one pod to mount the volume.
+
+[only be able to be attached to a single node]: https://github.com/kubernetes/kubernetes/blob/v1.26.1/pkg/volume/util/util.go#L716-L718
+
+#### API Server Has Feature Enabled, Scheduler Does, Kubelet Does Not
+
+In this scenario, the kube-scheduler will enforce the ReadWriteOncePod access
+mode and ensure only a single pod may use a ReadWriteOncePod PVC.
+
+If you have two pods sharing the same ReadWriteOncePod PVC and they both have
+`spec.nodeName` set, then scheduling will be bypassed. See [the above scenario]
+on the expected behavior.
+
+[the above scenario]: #api-server-has-feature-enabled-scheduler-and-kubelet-do-not
+
## Production Readiness Review Questionnaire
+Rolling out this feature involves enabling the ReadWriteOncePod feature gate
+across kube-apiserver, kube-scheduler, kubelet, and updating CSI driver and
+sidecar versions. The order in which these are performed does not matter.
+
+The only way this rollout can fail is if a user does not update all components,
+in which case the feature will not work. See the above section on version skews
+for behavior in this scenario.
+
+Rolling out this feature does not impact any running workloads.
+
###### What specific metrics should inform a rollback?
+If pods using ReadWriteOncePod PVCs fail to schedule, you may see an increase in
+`scheduler_unschedulable_pods{plugin="VolumeRestrictions"}`.
+
+For enforcement in kubelet, if there are issues you may see changes in metrics
+for "volume_mount" operations. For example, an increase in
+`storage_operation_duration_seconds_bucket{operation_name="volume_mount"}` for
+larger buckets may indicate issues with mount.
+
###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?
+Manual tests were performed to test the whole end to end flow.
+
+Starting with the upgrade path:
+
+- Unsuccessfully create workloads using ReadWriteOncePod PVCs prior to upgrade
+- Perform the upgrade in two stages:
+ - First, update CSI sidecars
+ - Second, enable feature flag
+- Successfully create workloads (1) and (2) using ReadWriteOncePod PVCs (1) and
+ (2).
+- Unsuccessfully create workload (3) using ReadWriteOncePod PVC (2) (already
+ in-use)
+- Observe the workloads and PVCs are healthy
+
+For the downgrade path:
+
+- Perform the downgrade in two stages:
+ - First, disable feature flag
+ - Second, downgrade CSI sidecars
+- Observe the workloads and PVCs are still healthy
+- Successfully delete workload (1) and ReadWriteOncePod PVC (1)
+
+And re-upgrading the feature again:
+
+- Perform the upgrade in two stages:
+ - First, update CSI sidecars
+ - Second, enable feature flag
+- Successfully create workload (1) using ReadWriteOncePod PVC (1)
+- Unsuccessfully create workload (3) using ReadWriteOncePod PVC (2) (already
+ in-use)
+- Observe the workloads and PVCs are still healthy
+
###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
+No.
+
### Monitoring Requirements
+An operator can query for PersistentVolumeClaims and PersistentVolumes in the
+cluster with the ReadWriteOncePod access mode. If any exist then the feature is
+in use.
+
###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
-- [ ] Metrics
- - Metric name:
+- [X] Metrics
+ - Metric name: `scheduler_unschedulable_pods{plugin="VolumeRestrictions"}`
- [Optional] Aggregation method:
- Components exposing the metric:
-- [ ] Other (treat as last resort)
- - Details:
+ - kube-scheduler
###### What are the reasonable SLOs (Service Level Objectives) for the above SLIs?
@@ -892,6 +1110,23 @@ high level (needs more precise definitions) those may be things like:
- 99,9% of /health requests per day finish with 200 code
-->
+Defining an SLO for this metric is difficult because a pod may be
+"Unschedulable" due to user error; they mistakenly scheduled a workload that has
+volume conflicts. Additionally, this metric captures volume conflicts for some
+legacy in-tree drivers that do not support the ReadWriteOncePod feature but are
+part of the same scheduler plugin.
+
+Any unexpected increases in
+`scheduler_unschedulable_pods{plugin="VolumeRestrictions"}` should be
+investigated by checking the status of pods failing scheduling.
+
+If there are failures during attach, detach, mount, or unmount, you may see an
+increase in the `storage_operation_duration_seconds` metric exported by
+kubelet.
+
+You may also see an increase in the `csi_sidecar_operations_seconds_bucket`
+metric exported by CSI sidecars if there are issues performing CSI operations.
+
###### Are there any missing metrics that would be useful to have to improve observability of this feature?
+No.
+
### Dependencies
+This feature depends on the cluster having CSI drivers and sidecars that use CSI
+spec v1.5.0 at minimum.
+
+- [CSI drivers and sidecars]
+ - Usage description:
+ - Impact of its outage on the feature: Inability to perform CSI storage
+ operations on ReadWriteOncePod PVCs and PVs (for example, provisioning
+ volumes)
+ - Impact of its degraded performance or high-error rates on the feature:
+ Increase in latency performing CSI storage operations (due to repeated
+ retries)
+
### Scalability
+None.
+
###### What steps should be taken if SLOs are not being met to determine the problem?
+- Delete any unhealthy pods / PVCs using ReadWriteOncePod
+- Disable the feature gate (target the API server first to prevent creation of new PVCs)
+- Downgrade CSI sidecars and drivers if you're seeing elevated errors there
+
## Implementation History