Add ingress class docs

[rainest]

What this PR does / why we need it:
Adds documentation describing how the controller handles ingress.class annotations.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #677

Special notes for your reviewer:

• Changes indicate when class annotations are required, but not when they are not required. I don't think it's as important to call out the latter, since worst case a user adds an annotation and it does more or less nothing (the only point of confusion may be that adding a class annotation to those resources, e.g. KongPlugin, does not prevent their use by other controllers). The new resource-classes.md concept guide should cover that: it goes over how classless resources are loaded via references in classful resources, and provides an overview of some of the confusing <0.10 behavior.
• Not all resources were covered equally to begin with. I've updated existing docs when they cover classful resources, but not added new sections where they previously weren't covered. For example, annotations.md has a section for KongConsumer (which now mentions its class requirements, but not TCPIngress). All resources are covered in at least one location, and I list all resources that require ingress.class under https://github.com/Kong/kubernetes-ingress-controller/blob/a72ba9c03f542aa8af6e16cb7d459a30c2c26bba/docs/references/annotations.md#kubernetesioingressclass

[rainest]

As of today, this documents ingress.class in general with a concept doc. It still needs reference doc updates to indicate which resources do/do not require class info.

[mflendrich]

I'd replace this with a table. Such table would be an authoritative source as to whether an ingress class annotation is effective and/or obligatory. This would solve the problem stated in the PR description:

worst case a user adds an annotation and it does more or less nothing

| Kind | Ingress class setting |
| Ingress (networking.k8s.io/v1beta1 ) | `kubernetes.io/ingress.class` annotation must be set |
| Ingress (networking.k8s.io/v1 ) | `spec.ingressClassName` annotation must be set |
| Knative Service | ingress class must be specified by Knative installation <link> |
| SomeOtherResource | `kubernetes.io/ingress.class` annotation is optional |
| YetAnotherResource | must not be set |
| YetAnotherResource | must not be set |
...

[rainest]

I chose to write this as I did because it's brief--we require this annotation or similar only on a small set (5 total) of resources. An exhaustive list would not be brief:

kubectl api-resources | wc -l
116

For the vast majority, it simply does nothing, and there are no functional consequences to accidentally adding it elsewhere.

User confusion is a possible concern, but not one I see enough evidence for to warrant an exhaustive list: anecdotally, existing user confusion is more centered around resources that require the resource, where the user wasn't aware that was the case or didn't understand the nuances of the "empty allowed, but only if you didn't override the default controller class, and also not during updates for some things because bugs" legacy behavior. I have not seen questions around annotating a resource with a non-matching class and still seeing the controller load it; if we do start receiving those in the future, we can revisit this. Leaving as-is for now.

[mflendrich]

As discussed elsewhere, KIC operates using ~12 object kinds (judging by informers registered in the code), not 116, so it should be sufficient only to cover those 12.

there are no functional consequences to accidentally adding it elsewhere.

But there are consequences to making an effectively non-functional configuration change by mistake, and one expectation I have (and, I believe, we should have in this case - we can discuss if that seems unreasonable) from documentation is to prevent such mistakes. This one seems very simple to prevent.

I chose to write this as I did because it's brief

I do not see how a table with 12 entries is not brief, keeping in mind the complexity of the subject. Table is not a format I'll strongly enforce here. My requirement is that we have an authoritative source which answers the question whether a specific resource requires an ingress class or not (with an equally strong "yes" and "no"), and how to set it.

[rainest]

Noted, left as-is.

[mflendrich]

I request that we do (in whatever form, if the table suggested above does not work) provide an answer to the question Does resource X require an ingress class annotation? in our docs, especially given the fact, that there are at least 4 different ways for Kong to decide whether to include a resource in config or not (the annotation, the ingressv1 way, the knative way, the dependent resource way)

[mflendrich]

We could be more consistent and brief by removing this section and putting this information in the reference table, as explained in my other comment.

[mflendrich]

@rainest, resolved because duplicate of https://github.com/Kong/kubernetes-ingress-controller/pull/811/files#discussion_r479298310 ?

It would be simpler to navigate resolved comments if marking as resolved came with an reason for marking as such. At least when the author does not apply the suggestion.

[mflendrich]

Suggested change

	_This resource requires the `kubernetes.io/ingress.class` annotation. Its value
	must match the value of the controller's `--ingress-class` argument, which is
	"kong" by default._
	_This resource requires the [`kubernetes.io/ingress.class` annotation](../README.md#resource-classes)._

The definition of what an ingress class means is already present in the docs. Let's link to it instead of repeating it over and over.

This comment applies to all repetitions of such pattern.

[mflendrich]

This comment applies to all repetitions of such pattern.

Note that there are two other repetitions in this file still left.

[mflendrich]

This sounds very complicated because of the passive voice.

I suggest something along those lines:

For many resource kinds <link to table>, Kong excludes from processing all objects of those resources which do not have a matching ingress class set. For certain resource types <url or list>, you can instruct Kong to include objects with no class by setting the following flags:

[rainest]

Overruled; it isn't; diatribe explanation elsewhere.

[mflendrich]

Please at least make a short explanation as to why specifying a class is recommended, or link to such. Also, please be specific, for which resources specifying a class is not optional. The table mentioned in my other comment could help with that.

[rainest]

Such explanation follows within 4 lines. Objection noted and dismissed.

[mflendrich]

Also, please be specific, for which resources specifying a class is not optional. (...)

I don't see how this was fixed.

To further clarify my point: I request that in this situation, making a general claim about "some resources" being such as such, I request that we enable the user to navigate to the right part of the reference.

On a side note, it's a part of the reference I've asked for in another comment.

[mflendrich]

Suggested change

	For example, an Ingress is translated directly into a Kong route, and a
	For example, an Ingress is translated directly into a [Kong route](https://docs.konghq.com/latest/admin-api/#route-object), and a

[rainest]

Unnecessary here. The focus of this content isn't to discuss basic Kong proxy concepts, and we may reasonably assume many readers are familiar with them: they're part of the basic set of configuration introduced in our getting started guides.

[mflendrich]

Three arguments against not including it here:

1. It costs nothing to add (because the link just changes a term into a hyperlink, and the likelihood of the link rotting is low)

2. we may reasonably assume many readers are familiar with them

   This is a harmful and (imo unnecessary) assumption: some people will come here from a Google search, some people will be only 90% sure about the relationships between routes, services and upstreams.

3. Wikipedia would cross-link in such situation. Take for example the Molotov-Ribbentrop pact article. The reader may be reasonably assumed to be familiar with terms "Soviet Union" and "Germany", but the article has them linked nonetheless.

[mflendrich]

Let's move on with what we have now and iterate on the reference part and cross-linking as needed 👍