Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature gates #1970

Merged
merged 3 commits into from
Nov 1, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,17 @@

## [2.0.5] - TBD

#### Added

- [Feature Gates][k8s-fg] have been added to the controller manager in order to
enable alpha/beta/experimental features and provide documentation about those
features and their maturity over time. For more information see the
[KIC Feature Gates Documentation][kic-fg].
[#1970](https://github.com/Kong/kubernetes-ingress-controller/pull/1970)

[k8s-fg]:https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
[kic-fg]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/FEATURE_GATES.md

#### Fixed

- Fixed a bug where version reported for the controller manager was missing
Expand Down
6 changes: 6 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,12 @@ In general the maintainers here feel establishing these things in a KEP should b

[template]:https://github.com/kubernetes/enhancements/blob/master/keps/NNNN-kep-template/README.md

## Feature Gates

New features should be added to the [Feature Gates][kic-fg] documentation and `internal/manager/feature_gates.go`.

[kic-fg]:https://github.com/Kong/kubernetes-ingress-controller/blob/main/FEATURE_GATES.md

## Development environment

## Environment
Expand Down
61 changes: 61 additions & 0 deletions FEATURE_GATES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Feature Gates

Upstream [Kubernetes][k8s] includes [Feature Gates][gates] to enable or disable features with flags and track the maturity of a feature using [Feature Stages][stages]. Here in the Kubernetes Ingress Controller (KIC) we use the same definitions of `Feature Gates` and `Feature Stages` from upstream Kubernetes, but with our own list of features.

Using `Feature Gates` enables contributors to add and manage new (and potentially) experimental functionality to the KIC in a controlled manner: the features will be "hidden" until generally available (GA) and the progress and maturity of features on their path to GA will be documented. Feature gates also create a clear path for deprecating features.

See below for current features and their statuses, and follow the links to the relevant feature documentation.

[k8s]:https://kubernetes.io
[gates]:https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
[stages]:https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-stages

## Feature gates

Below you will find the overviews of features at various maturity levels:

- [Feature gates for graduated or deprecated features](/#feature-gates-for-graduated-or-deprecated-features)
- [Feature gates for Alpha or Beta features](/#feature-gates-for-alpha-or-beta-features)

Please read the [Important Notes](/#important-notes) section before using any `Alpha` or `Beta` features.

### Important notes

- Most features will be planned and detailed using [Kubernetes Enhancement Proposals (KEP)][k8s-kep]: If you're interested in the development side of features familiarize yourself with our [KEPs][kic-keps]
- The `Since` and `Until` rows in below tables refer to [KIC Releases][releases]
- For `GA` features the documentation exists in the main [Kong Documentation][kong-docs], see the [API reference][api-ref] and [Guides][kic-guides]

An additional **warning** for end-users who are reading this documentation and trying to enable `Alpha` or `Beta` features: it is **very important** to understand that features that are currently in an `Alpha` or `Beta` state may **become `Deprecated` at any time** and **may be removed as part of the next consecutive minor release**. This is especially true for `Alpha` maturity features. In other words, **until a feature becomes GA there are no guarantees that it's going to continue being available**. To avoid disruption to your services engage with the community and read the [CHANGELOG](/CHANGELOG.md) carefully to track progress. Alternatively **do not use features until they have reached a GA status**.

[k8s-keps]:https://github.com/kubernetes/enhancements
[kic-keps]:https://github.com/Kong/kubernetes-ingress-controller/tree/main/keps
[releases]:https://github.com/Kong/kubernetes-ingress-controller/releases
[kong-docs]:https://github.com/Kong/docs.konghq.com
[api-ref]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/
[kic-guides]:https://docs.konghq.com/kubernetes-ingress-controller/latest/guides/overview/

### Feature gates for graduated or deprecated features

{{< table caption="Feature Gates for Graduated or Deprecated Features" >}}

| Feature | Default | Stage | Since | Until |
|----------------------------|---------|------------|-------|-------|

{{< /table >}}

Features that reach GA and over time become stable will be removed from this table, they can be found in the main [KIC CRD Documentation][specs] and [Guides][guides].

[specs]:https://docs.konghq.com/kubernetes-ingress-controller/latest/references/custom-resources/
[guides]:https://docs.konghq.com/kubernetes-ingress-controller/latest/guides/overview/

Comment on lines +37 to +50
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YAGNI - suggestion: delete

### Feature gates for Alpha or Beta features

{{< table caption="Feature gates for features in Alpha or Beta states" >}}

| Feature | Default | Stage | Since | Until |
|----------------------------|---------|-------|-------|-------|
| [Knative](#Knative) | `true` | Alpha | 0.8.0 | TBD |
| [GatewayAPI](#Gateway-API) | `false` | Alpha | TBD | TBD |

{{< /table > }}

2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Kong/kong/blob/master/LICENSE)
[![Twitter](https://img.shields.io/twitter/follow/thekonginc.svg?style=social&label=Follow)](https://twitter.com/intent/follow?screen_name=thekonginc)

# Kong Ingress Controller for Kubernetes
# Kong Ingress Controller for Kubernetes (KIC)

Use [Kong][kong] for Kubernetes [Ingress][ingress].
Configure [plugins][kong-hub], health checking,
Expand Down
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ require (
k8s.io/apiextensions-apiserver v0.22.3
k8s.io/apimachinery v0.22.3
k8s.io/client-go v0.22.3
k8s.io/component-base v0.22.3
knative.dev/networking v0.0.0-20210914225408-69ad45454096
knative.dev/pkg v0.0.0-20210919202233-5ae482141474
knative.dev/serving v0.26.0
Expand Down Expand Up @@ -123,7 +124,6 @@ require (
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
k8s.io/component-base v0.22.3 // indirect
k8s.io/klog/v2 v2.9.0 // indirect
k8s.io/kube-openapi v0.0.0-20210421082810-95288971da7e // indirect
k8s.io/utils v0.0.0-20210819203725-bdf08cb9a70a // indirect
Expand Down
8 changes: 8 additions & 0 deletions internal/manager/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"github.com/spf13/pflag"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"
cliflag "k8s.io/component-base/cli/flag"
"sigs.k8s.io/controller-runtime/pkg/client"

"github.com/kong/kubernetes-ingress-controller/v2/internal/adminapi"
Expand Down Expand Up @@ -81,6 +82,9 @@ type Config struct {
EnableProfiling bool
EnableConfigDumps bool
DumpSensitiveConfig bool

// Feature Gates
FeatureGates map[string]bool
}

// -----------------------------------------------------------------------------
Expand Down Expand Up @@ -175,6 +179,10 @@ func (c *Config) FlagSet() *pflag.FlagSet {
flagSet.BoolVar(&c.EnableConfigDumps, "dump-config", false, fmt.Sprintf("Enable config dumps via web interface host:%v/debug/config", DiagnosticsPort))
flagSet.BoolVar(&c.DumpSensitiveConfig, "dump-sensitive-config", false, "Include credentials and TLS secrets in configs exposed with --dump-config")

// Feature Gates (see FEATURE_GATES.md)
flagSet.Var(cliflag.NewMapStringBool(&c.FeatureGates), "feature-gates", "A set of key=value pairs that describe feature gates for alpha/beta/experimental features. "+
fmt.Sprintf("See the Feature Gates documentation for information and available options: %s", featureGatesDocsURL))

// Deprecated (to be removed in future releases)
flagSet.Float32Var(&c.ProxySyncSeconds, "sync-rate-limit", proxy.DefaultSyncSeconds,
"Define the rate (in seconds) in which configuration updates will be applied to the Kong Admin API (DEPRECATED, use --proxy-sync-seconds instead)",
Expand Down
10 changes: 8 additions & 2 deletions internal/manager/controllerdef.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ func (c *ControllerDef) MaybeSetupWithManager(mgr ctrl.Manager) error {
// Controller Manager - Controller Setup Functions
// -----------------------------------------------------------------------------

func setupControllers(mgr manager.Manager, proxy proxy.Proxy, c *Config) ([]ControllerDef, error) {
func setupControllers(mgr manager.Manager, proxy proxy.Proxy, c *Config, featureGates map[string]bool) ([]ControllerDef, error) {
// Choose the best API version of Ingress to inform which ingress controller to enable.
var ingressPicker ingressControllerStrategy
if err := ingressPicker.Initialize(c, mgr.GetClient()); err != nil {
Expand Down Expand Up @@ -196,8 +196,14 @@ func setupControllers(mgr manager.Manager, proxy proxy.Proxy, c *Config) ([]Cont
IngressClassName: c.IngressClassName,
},
},
// ---------------------------------------------------------------------------
// Other Controllers
// ---------------------------------------------------------------------------
{
Enabled: c.KnativeIngressEnabled,
// knative is a special case because it existed before we added feature gates functionality
// for this controller (only) the existing --enable-controller-knativeingress flag overrides
// any feature gate configuration. See FEATURE_GATES.md for more information.
Enabled: featureGates["Knative"] || c.KnativeIngressEnabled,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that feature gate names deserve a dedicated enum type, and they definitely deserve consts rather than literal values

AutoHandler: crdExistsChecker{GVR: schema.GroupVersionResource{
Group: knativev1alpha1.SchemeGroupVersion.Group,
Version: knativev1alpha1.SchemeGroupVersion.Version,
Expand Down
39 changes: 39 additions & 0 deletions internal/manager/feature_gates.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package manager

import (
"fmt"

"github.com/go-logr/logr"
)

// featureGatesDocsURL provides a link to the documentation for feature gates in the KIC repository
const featureGatesDocsURL = "https://github.com/Kong/kubernetes-ingress-controller/blob/main/FEATURE_GATES.md"

// setupFeatureGates converts feature gates to controller enablement
func setupFeatureGates(setupLog logr.Logger, c *Config) (map[string]bool, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function could have a simpler and cohesive mental model: take two map[string]bools (rather than c) and have the semantics of "merging two maps key-wise but using the first in case of a conflict".

// generate a map of feature gates by string names to their controller enablement
ctrlMap := getFeatureGatesDefaults()

// override the default settings
for feature, enabled := range c.FeatureGates {
setupLog.Info("found configuration option for gated feature", "feature", feature, "enabled", enabled)
_, ok := ctrlMap[feature]
if !ok {
return ctrlMap, fmt.Errorf("%s is not a valid feature, please see the documentation: %s", feature, featureGatesDocsURL)
}
ctrlMap[feature] = enabled
}

return ctrlMap, nil
}

// getFeatureGatesDefaults initializes a feature gate map given the currently
// supported feature gates options and derives defaults for them based on
// manager configuration options if present.
//
// NOTE: if you're adding a new feature gate, it needs to be added here.
func getFeatureGatesDefaults() map[string]bool {
return map[string]bool{
"Knative": false,
}
}
39 changes: 39 additions & 0 deletions internal/manager/feature_gates_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
package manager

import (
"bytes"
"testing"

"github.com/bombsimon/logrusr"
"github.com/sirupsen/logrus"
"github.com/stretchr/testify/assert"
)

func TestFeatureGates(t *testing.T) {
t.Log("setting up configurations and logging for feature gates testing")
out := new(bytes.Buffer)
baseLogger := logrus.New()
baseLogger.SetOutput(out)
baseLogger.SetLevel(logrus.DebugLevel)
setupLog := logrusr.NewLogger(baseLogger)
config := new(Config)

t.Log("verifying feature gates setup defaults when no feature gates are configured")
fgs, err := setupFeatureGates(setupLog, config)
assert.NoError(t, err)
assert.Len(t, fgs, len(getFeatureGatesDefaults()))

t.Log("verifying feature gates setup results when valid feature gates options are present")
config.FeatureGates = map[string]bool{"Knative": true}
fgs, err = setupFeatureGates(setupLog, config)
assert.NoError(t, err)
assert.True(t, fgs["Knative"])

t.Log("configuring several invalid feature gates options")
config.FeatureGates = map[string]bool{"Batman": true}

t.Log("verifying feature gates setup results when invalid feature gates options are present")
_, err = setupFeatureGates(setupLog, config)
assert.Error(t, err)
assert.Contains(t, err.Error(), "Batman is not a valid feature")
}
10 changes: 8 additions & 2 deletions internal/manager/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ func Run(ctx context.Context, c *Config, diagnostic util.ConfigDumpDiagnostic) e
utilruntime.Must(configurationv1beta1.AddToScheme(scheme))
utilruntime.Must(knativev1alpha1.AddToScheme(scheme))

setupLog.Info("getting enabled options and features")
featureGates, err := setupFeatureGates(setupLog, c)
if err != nil {
return fmt.Errorf("failed to configure feature gates: %w", err)
}

setupLog.Info("getting the kubernetes client configuration")
kubeconfig, err := c.GetKubeconfig()
if err != nil {
Expand All @@ -68,7 +74,7 @@ func Run(ctx context.Context, c *Config, diagnostic util.ConfigDumpDiagnostic) e
}

setupLog.Info("Starting Enabled Controllers")
controllers, err := setupControllers(mgr, proxy, c)
controllers, err := setupControllers(mgr, proxy, c, featureGates)
if err != nil {
return fmt.Errorf("unable to setup controller as expected %w", err)
}
Expand Down Expand Up @@ -97,7 +103,7 @@ func Run(ctx context.Context, c *Config, diagnostic util.ConfigDumpDiagnostic) e

if c.AnonymousReports {
setupLog.Info("Starting anonymous reports")
if err := mgrutils.RunReport(ctx, kubeconfig, kongConfig, metadata.Release); err != nil {
if err := mgrutils.RunReport(ctx, kubeconfig, kongConfig, metadata.Release, featureGates); err != nil {
setupLog.Error(err, "anonymous reporting failed")
}
} else {
Expand Down
3 changes: 2 additions & 1 deletion internal/mgrutils/reports.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import (
)

// RunReport runs the anonymous data report and reports any errors that have occurred.
func RunReport(ctx context.Context, kubeCfg *rest.Config, kongCfg sendconfig.Kong, kicVersion string) error {
func RunReport(ctx context.Context, kubeCfg *rest.Config, kongCfg sendconfig.Kong, kicVersion string, featureGates map[string]bool) error {
// if anonymous reports are enabled this helps provide Kong with insights about usage of the ingress controller
// which is non-sensitive and predominantly informs us of the controller and cluster versions in use.
// This data helps inform us what versions, features, e.t.c. end-users are actively using which helps to inform
Expand Down Expand Up @@ -67,6 +67,7 @@ func RunReport(ctx context.Context, kubeCfg *rest.Config, kongCfg sendconfig.Kon
Hostname: hostname,
ID: uuid,
KongDB: kongDB,
FeatureGates: featureGates,
}

// run the reporter in the background
Expand Down
1 change: 1 addition & 0 deletions internal/util/reports.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ type Info struct {
Hostname string
KongDB string
ID string
FeatureGates map[string]bool
}

// Reporter sends anonymous reports of runtime properties and
Expand Down