docs: add documentation for feature gates

CHANGELOG.md

@@ -39,6 +39,17 @@
 
 ## [2.0.5] - TBD
 
+#### Added
+
+- [Feature Gates][k8s-fg] have been added to the controller manager in order to
+  enable alpha/beta/experimental features and provide documentation about those
+  features and their maturity over time. For more information see the
+  [KIC Feature Gates Documentation][kic-fg].
+  [#1970](https://github.com/Kong/kubernetes-ingress-controller/pull/1970)
+
+[k8s-fg]:https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
+[kic-fg]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/FEATURE_GATES.md
+
 #### Fixed
 
 - Fixed a bug where version reported for the controller manager was missing

CONTRIBUTING.md

@@ -86,6 +86,12 @@ In general the maintainers here feel establishing these things in a KEP should b
 
 [template]:https://github.com/kubernetes/enhancements/blob/master/keps/NNNN-kep-template/README.md
 
+## Feature Gates
+
+New features should be added to the [Feature Gates][kic-fg] documentation and `internal/manager/feature_gates.go`.
+
+[kic-fg]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/FEATURE_GATES.md
+
 ## Development environment
 
 ## Environment

FEATURE_GATES.md

@@ -0,0 +1,158 @@
+# Feature Gates
+
+Upstream [Kubernetes][k8s] includes [Feature Gates][gates] to enable or disable features with flags and track the maturity of a feature using [Feature Stages][stages]. Here in the Kubernetes Ingress Controller (KIC) we use the same definitions of `Feature Gates` and `Feature Stages` from upstream Kubernetes, but with our own list of features.
+
+Using `Feature Gates` enables contributors to add and manage new (and potentially) experimental functionality to the KIC in a controlled manner: the features will be "hidden" until generally available (GA) and the progress and maturity of features on their path to GA will be documented. Feature gates also create a clear path for deprecating features.
+
+See below for current features and their statuses, and follow the links to the relevant feature documentation.
+
+[k8s]:https://kubernetes.io
+[gates]:https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
+[stages]:https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-stages
+
+## Feature gates
+
+Below you will find the overviews of features at various maturity levels:
+
+- [Feature gates for graduated or deprecated features](/#feature-gates-for-graduated-or-deprecated-features)
+- [Feature gates for Alpha or Beta features](/#feature-gates-for-alpha-or-beta-features)
+
+Please read the [Important Notes](/#important-notes) section before using any `Alpha` or `Beta` features.
+
+### Important notes
+
+- Most features will be planned and detailed using [Kubernetes Enhancement Proposals (KEP)][k8s-kep]: If you're interested in the development side of features familiarize yourself with our [KEPs][kic-keps]
+- The `Since` and `Until` rows in below tables refer to [KIC Releases][releases]
+- For `Alpha` and `Beta` features the documentation for those features will be present here in this document
+- For `GA` features the documentation graduates to the main [Kong Documentation][kong-docs], see the [API reference][api-ref] and [Guides][kic-guides]
+- For `Deprecated` features the documentation may only be available in historical versions of the Kong Documentation (according to their `Until` version)
+- Features that have existed `Since` a version less than `2.0.0` generally have a `--enable-controller-<API>` flag associated with them on the manager, see the [Controller Manager CLI Arguments][cli-args] for more details. This is interchangeable and works the same as the equivalent feature gate.
+
+An additional **warning** for end-users who are reading this documentation and trying to enable `Alpha` or `Beta` features: it is **very important** to understand that features that are currently in an `Alpha` or `Beta` state may **become `Deprecated` at any time** and **may be removed as part of the next consecutive minor release**. This is especially true for `Alpha` maturity features. In other words, **until a feature becomes GA there are no guarantees that it's going to continue being available**. To avoid disruption to your services engage with the community and read the [CHANGELOG](/CHANGELOG.md) carefully to track progress. Alternatively **do not use features until they have reached a GA status**.
+
+[k8s-keps]:https://github.com/kubernetes/enhancements
+[kic-keps]:https://github.com/Kong/kubernetes-ingress-controller/tree/main/keps
+[releases]:https://github.com/Kong/kubernetes-ingress-controller/releases
+[kong-docs]:https://github.com/Kong/docs.konghq.com
+[api-ref]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/
+[kic-guides]:https://docs.konghq.com/kubernetes-ingress-controller/latest/guides/overview/
+[cli-args]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/cli-arguments/
+
+### Feature gates for graduated or deprecated features
+
+{{< table caption="Feature Gates for Graduated or Deprecated Features" >}}
+
+| Feature                    | Default | Stage      | Since              | Until              |
+|----------------------------|---------|------------|--------------------|--------------------|
+| [KongPlugin][plug]         | `true`  | GA         | [0.1.0][kic-0.1.0] | -                  |
+| [KongClusterPlugin][cplug] | `true`  | GA         | [0.8.0][kic-0.8.0] | -                  |
+| [KongIngress][king]        | `true`  | GA         | [0.1.0][kic-0.1.0] | -                  |
+| [TCPIngress][ting]         | `true`  | GA         | [0.8.0][kic-0.8.0] | -                  |
+| [UDPIngress][Ping]         | `true`  | GA         | [2.0.0][kic-2.0.0] | -                  |
+| [KongConsumer][cons]       | `true`  | GA         | [0.1.0][kic-0.1.0] | -                  |
+| [KongCredential][cred]     | `false` | Deprecated | [0.1.0][kic-0.1.0] | [2.0.0][kic-2.0.0] |
+
+{{< /table >}}
+
+[kic-0.1.0]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/CHANGELOG.md#010---20180817
+[kic-0.8.0]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/CHANGELOG.md#080---20200325
+[kic-2.0.0]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/CHANGELOG.md#200---20211007
+[plugin]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/#kongplugin
+[cplugin]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/#kongclusterplugin
+[king]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/#kongingress
+[ting]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/#tcpingress
+[uing]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/#udpingress
+[cons]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/#kongconsumer
+
+### Feature gates for Alpha or Beta features
+
+{{< table caption="Feature gates for features in Alpha or Beta states" >}}
+
+| Feature                      | Default | Stage | Since              | Until |
+|------------------------------|---------|-------|--------------------|-------|
+| [Knative](#Knative)          | `true`  | Alpha | [0.8.0][kic-0.8.0] | TBD   |
+| [GatewayAPIs](#Gateway-APIs) | `false` | Alpha | TBD                | TBD   |
+
+{{< /table > }}
+
+[kic-0.8.0]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/CHANGELOG.md#080---20200325
+
+#### Knative
+
+Our [Knative][knative] integration includes support for [Knative Serving][knative-serving].
+
+[knative]:https://knative.dev
+[knative-serving]:https://knative.dev/docs/serving/
+
+##### Usage Examples
+
+Before proceeding make sure you familiarize yourself with the [Knative Documentation][knative-docs].
+
+Ensure that `Knative` is enabled by adding the following [Controller Manager Flag][cm-flags]:
+
+```console
+--feature-gates="Knative=true"
+```
+
+Deploy `knative-serving` components to your cluster:
+
+```console
+$ kubectl apply -f https://github.com/knative/serving/releases/download/v0.26.0/serving-crds.yaml
+$ kubectl apply -f https://github.com/knative/serving/releases/download/v0.26.0/serving-core.yaml
+```
+
+Patch the Knative network configuration for the `ingress.class` in use by the KIC (by default, this will be `kong`):
+
+```console
+$ kubectl -n knative-serving patch configmap config-network -p '{"data":{"ingress.class":"kong"}}'
+```
+
+A basic "hello world" application can be used to test traffic from Kong:
+
+```console
+$ cat << EOF | kubectl apply -f -
+apiVersion: serving.knative.dev/v1
+kind: Service
+metadata:
+ name: helloworld-go
+ namespace: default
+spec:
+ template:
+  spec:
+   containers:
+    - image: gcr.io/knative-samples/helloworld-go
+      env:
+        - name: TARGET
+          value: "Go Sample v1"
+EOF
+```
+
+And then access the service via Kong:
+
+```
+$ curl -H "Host: helloworld-go.default.$PROXY_IP" http://$PROXY_IP/
+```
+
+[knative-docs]:https://knative.dev/docs
+[cm-flags]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/cli-arguments/
+
+##### Status
+
+Support for Knative was added to the KIC before the first `v1.x` release and before feature gates were in use, as such there are some important differences from other features at the same maturity level:
+
+- a `--enable-controller-knativeingress` flag for the controller manager exists outside of normal feature gates to enable the feature (but the effect is the same)
+- for historical reasons and due to the existing flag deprecation and removal of this feature can not be done without cutting a new major release of the KIC, as this would be backwards incompatible
+
+There are [integration tests][tests] which cover `Knative Serving` functionality on every pull request to the repository to protect against regressions and validate updates.
+
+At the time of writing there's no currently planned path to graduate Knative support to `Beta` or further due to a lack of community engagement on the matter. This is distinct from being `Deprecated` as we're considering this an evaluation period while we try and engage with the community to establish interest. If you're reading this and are interested in helping us justify and plan a path forward for Knative integrations in KIC, please feel free to open a [Discussion][discussions] or a [KEP][keps] and let us know about your use case.
+
+[discussions]:https://github.com/Kong/kubernetes-ingress-controller/discussions
+[keps]:https://github.com/Kong/kubernetes-ingress-controller/tree/main/keps
+
+#### Gateway APIs
+
+TODO: planning for [Gateway API][gwapi] support is being actively worked on, once planning is complete you'll see updates here but in the meantime check the progress on the [Gateway APIs KEP][gwkep].
+
+[gwapi]:https://github.com/kubernetes-sigs/gateway-api
+[gwkep]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/keps/0002-kong-gateway-api.md

README.md

@@ -5,7 +5,7 @@
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Kong/kong/blob/master/LICENSE)
 [![Twitter](https://img.shields.io/twitter/follow/thekonginc.svg?style=social&label=Follow)](https://twitter.com/intent/follow?screen_name=thekonginc)
 
-# Kong Ingress Controller for Kubernetes
+# Kong Ingress Controller for Kubernetes (KIC)
 
 Use [Kong][kong] for Kubernetes [Ingress][ingress].
 Configure [plugins][kong-hub], health checking,
@@ -37,6 +37,10 @@ Custom Resource Definitions(CRDs) and Kubernetes-native tooling.
   Configure all of Kong
   using CRDs in Kubernetes and manage Kong declaratively.
 
+Some features of the KIC are under development and not yet ready for general availability (GA), see the [Feature Gates][kic-fg] documentation for more information.
+
+[kic-fg]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/FEATURE_GATES.md
+
 ## Get started
 
 You can use