feature gates

[shaneutt]

What this PR does / why we need it:

This patch documents and enables feature gates for KIC to add tooling which helps to manage the lifecycle of new and/or experimental features for the KIC.

This was meant to solve two problems:

• strongly communicate the lifecycle and trajectory of features, and give us a path to graduate as well as deprecate while significantly reducing any surprises for end-users
• provide a safe/isolated way to iteratively develop Gateway support without putting it directly in end-users hands day 1
• allow Gateway development to start quickly but without being locked in forever if the long term implementation lives partially or wholly elsewhere in time (e.g. operator code)

PR Readiness Checklist:

• the CHANGELOG.md release notes have been updated

[shaneutt]

Reporting We should have reporting for these to inform removal decisions. That doesn't strictly need to go in this PR (I'm not familiar with adding new reporting info and unsure how much additional work it is--it doesn't look like we have much anything on internals other than version at this point), but it should be a high-ish priority new feature if we merge this without it.

I was actually thinking about this already this morning before I read this, 080719a includes reporting for all feature gate options.

Existing features and GA handling I don't think we should include (most) existing GA features in this, and think we should remove feature flags once a feature is GA. Upstream indicates that they do remove graduated flags:

https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
A General Availability (GA) feature is also referred to as a stable feature. It means:
The feature is always enabled; you cannot disable it.
The corresponding feature gate is no longer needed.

That suggests that GA removes the actual flag-handling code and only keeps the documentation Status/Since/Until record around. Is that not true in practice? Can you still toggle GA features off?

We'd maybe need some stub code to indicate that it was previously a flag to play nice with validity detection and/or print a "this feature is now GA, you can remove it from the list" log, unless we want to be strict about users removing graduated flags when upgrading.

IMO the KongPlugin/TCPIngress/KongConsumer/etc. controllers should not be listed here at all--they didn't go through this process and were effectively GA immediately on release, so listing them here is a bit confusing, doubly so if they don't actually have feature flags (which, per the above GA rule, they shouldn't).

d3013a4 rips it all out as you suggest. In truth I was aware of upstream and I had intentionally gone slightly against the grain here because I was thinking about a potentially soon future where we would actually deprecate APIs like UDPIngress and TCPIngress since upstream Kubernetes will soon have (Gateway) options for these, but these are always things we can add later and at least right now I particularly want to laser focus on this PR in support of gateways.

KongCredential is particularly confusing, since it more changed shape rather than being removed entirely.

I've since removed that one.

[rainest]

LGTM with changes. I think the implication is that any other controller feature flag will have the condition be && to require both the gate and controller flag, and there's just no example because all we have now is Knative and Knative is weird.

We're probably stuck with TCP/UDPIngress for quite some time, since upstream isn't the fastest-moving thing--we're probably quite a ways out from being able to deprecate them, since we'd want Gateway APIs in at least beta (the APIs themselves rather than our implementation) and available on major providers--even once APIs reach beta providers may not actually enable them by default (or allow you to enable them at all).

[mflendrich]

YAGNI - suggestion: delete

[mflendrich]

I think that feature gate names deserve a dedicated enum type, and they definitely deserve consts rather than literal values

[mflendrich]

This function could have a simpler and cohesive mental model: take two map[string]bools (rather than c) and have the semantics of "merging two maps key-wise but using the first in case of a conflict".