With multiple ingress controllers, KIC touches all ingresses, regardless of "kubernetes.io/ingress.class"

[rurus9]

Summary

Kong Ingress Controller touches all ingresses, regardless of "kubernetes.io/ingress.class".

Kong Ingress controller version
0.7.0 (with 0.7.1 is the same)

Kong or Kong Enterprise version
1.4 (I tried also 1.5.0)

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Environment

• Cloud provider or hardware configuration:
  bare metal machines
• OS (e.g. from /etc/os-release):
  PRETTY_NAME="Ubuntu 18.04.3 LTS"
• Kernel (e.g. uname -a):
  Linux kub03n01 4.15.0-66-generic Multiple SNIs on a TLS certificate are not reflected in Kong certificate object #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

What happened

I have K8s cluster deployed by Rancher, with default ingress-nginx (Ingress Controller) from Rancher. I have configured many ingresses without set any "kubernetes.io/ingress.class".
Then I installed Kong from official chart (version 1.1.1) (it's DB-less config), and with default "ingressClass: kong" in values.yaml. So it is fresh kong installation without any kong related CRD and kong ingress configurations.
I observed, that my ingresses (without any "kubernetes.io/ingress.class" annotations) are changed very frequently (pay attention to ADDRESS). I truncated logs only to two ingresses, but it touches every ingress in my cluster:

NAMESPACE              NAME                                 HOSTS                                                     ADDRESS                                  PORTS     AGE
cattle-system          rancher                              kub03.mydomain.com                                     172.17.0.137,172.17.0.139,172.17.0.140   80, 443   97d 
kube-logging           kub03-kibana-frontend-cert-manager   kub03-kibana.kube-logging.ingress.kub03.mydomain.com   172.17.0.137,172.17.0.139,172.17.0.140   80, 443   7d  

grpatron@rur:~/projects/kong/test-dokumentacji$ kubectl get ingresses.extensions --all-namespaces
NAMESPACE              NAME                                 HOSTS                                                     ADDRESS      PORTS     AGE
cattle-system          rancher                              kub03.mydomain.com                                     172.19.3.6   80, 443   97d 
kube-logging           kub03-kibana-frontend-cert-manager   kub03-kibana.kube-logging.ingress.kub03.mydomain.com   172.19.3.6   80, 443   7d 

In kong ingress controller I can observe corresponding logs:

I0206 12:55:43.769280       1 status.go:356] updating Ingress cattle-system/rancher status to [{172.19.3.6 }]
I0206 12:55:43.769356       1 status.go:356] updating Ingress kube-logging/kub03-kibana-frontend-cert-manager status to [{172.19.3.6 }]
(...)
I0206 12:55:43.778936       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:55:47.113347       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:55:50.445406       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:56:33.500774       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:56:36.834384       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:56:40.168399       1 kong.go:57] no configuration change, skipping sync to Kong
(...)
I0206 12:56:43.768662       1 status.go:356] updating Ingress kube-logging/kub03-kibana-frontend-cert-manager status to [{172.19.3.6 }]
I0206 12:56:43.768833       1 status.go:356] updating Ingress cattle-system/rancher status to [{172.19.3.6 }]
(...)
I0206 12:56:43.780881       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:56:47.113086       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:57:33.501878       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:57:36.835443       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:57:40.168330       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:57:43.766828       1 status.go:356] updating Ingress cattle-system/rancher status to [{172.19.3.6 }]
I0206 12:57:43.766833       1 status.go:356] updating Ingress kube-logging/kub03-kibana-frontend-cert-manager status to [{172.19.3.6 }]
(...)
I0206 12:57:43.778670       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:57:47.111386       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:57:50.444447       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:58:33.499531       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:58:36.832474       1 kong.go:57] no configuration change, skipping sync to Kong
I0206 12:58:40.165806       1 kong.go:57] no configuration change, skipping sync to Kong

Expected behavior

KIC shoudn't touches ingresses without his kubernetes.io/ingress.class

Steps To Reproduce

Deploy kong on K8s with other ingress controllers, check ingress addresses

[hbagdi]

That's a special behavior of the default kong ingress class where any ingress resource that is not annotated is picked up.
If you change the ingress class to anything else, let's say, kong-foo, then the controller will not pick up ingress resources without any ingress class.

[rurus9]

Oh boy, this is true.
Did I miss something? I'm pretty sure, that information (about a special behavior of the default kong ingress class) isn't included in charm values.yaml. I belive, in the docs directory also it isn't mentioned. I think, it is important for users like me, with few ingress-controllers, to have a clear information about this behavior, with big, screaming font ;) (deployment kong with proxy.type=ClusterIP is pretty useless without this wisdom - you can't redirect traffic from other ingress to kong proxy service, when you use multiple host definitions in ingresses).

I also noticed, that with different ingress class than kong, I must include kubernetes.io/ingress.class annotation (with proper value) to every kong CRD object (plugin, consumer), which I use.

[hbagdi]

Your frustration is valid and justified. PR welcome to fix it. I'll leave this issue open to track the missing documentation.

I also noticed, that with different ingress class than kong, I must include kubernetes.io/ingress.class annotation (with proper value) to every kong CRD object (plugin, consumer), which I use.

Yes, this exists because users use multiple installation of Kong Ingress Controller (in the same k8s cluster) and want to configure them differently.

[rurus9]

#534, feel free to re-edit.
Best regards

[hbagdi]

#690 changes this behavior, which will be available as part of 0.10 release (coming out within a couple weeks).