KongClusterPlugin can not be found

[Panzki]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I created a KongClusterPlugin and I want to use it with an ingress. The ingress is created in a different namespace. The plugin is not applied to my route and the log of the ingress-controller pod contains the following error message.

time="2021-12-13T12:33:26Z" level=error msg="failed to fetch KongPlugin: no KongPlugin or KongClusterPlugin was found" kongplugin_name=jwt kongplugin_namespace=bff subsystem=proxy-cache-resolver

However when I get all KongClusterPlugin resources using kubectl I can see that a KongClusterPlugin resources with the name 'jwt' exists.

$ kubectl get KongClusterPlugin
NAME   PLUGIN-TYPE   AGE
jwt    jwt           91m

---

Backgroud

The issue started to occur after upgrading to Kong Ingress Controller v2.4.0 in combination with Kubernetes v1.22.x. Prior to the upgrade we used KIC v2.2.0 and K8s v1.19.x and hand no issues.

• I tested the exact same configuration setup in a local cluster with the older versions and it worked.
• I tested with the new versions and a KongPlugin resource, instead of a KongClusterPlugin resource, and it worked.

The issue might be related to updates to the KIC CRDs.

Expected Behavior

The plugin is applied to the route. KIC is able to use the KongClusterPlugin resource across namespaces.

Steps To Reproduce

1. Install KIC v2.4.0 in a K8s cluster with v1.22.x
2. (I'm using the jwt plugin) Create a `KongClusterPlugin`, a `KongConsumer`, and a `Secret` containing the necessary configuration.
3. Create an `Ingress` that uses the `KongClusterPlugin` by adding the annotation `konghq.com/plugins=jwt`.

Kong Ingress Controller version

v2.4.0 (tested with v2.6.1 as well)

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"archive", BuildDate:"2021-10-28T06:55:39Z", GoVersion:"go1.17.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"32a137c122b0474c719988922410f4027a4b002e", GitTreeState:"clean", BuildDate:"2021-11-01T16:43:17Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

I use some other plugins (basic-auth and response-transformer) as KongPlugin resources with several different ingress resources and in several different namespace. All KongPlugin resources work fine, the issue seems to be related to the KongClusterPlugin resource.

Thanks for looking into the issue. :smile

[shaneutt]

Hi @Panzki thanks for the report.

There appears to be some problems with the versioning reported here:

The issue started to occur after upgrading to Kong Ingress Controller v2.4.0 in combination with Kubernetes v1.22.x. Prior to the upgrade we used KIC v2.2.0 and K8s v1.19.x and hand no issues.
v2.4.0 (tested with v2.6.1 as well)

These are not versions of the Kong Kubernetes Ingress Controller (KIC) but instead would appear to be versions of the Kong Gateway (packaged with KIC). The latest version of KIC currently is v2.0.6, can you please verify which KIC version is at play here? A common way to do this is by looking at the relevant image tag in your manifests, or checking the logs of the ingress-controller container in the Kong pod.

[Panzki]

Hi,
thank you very much for the fast reply. I'm sorry, I confused the the version numbers with the Helm chart. In the original post I was referring to the version of the Helm chart.
The pod runs the KIC image kong/kubernetes-ingress-controller:2.0.

[shaneutt]

Thanks for the update.

So using the 2.0 controller image with the 2.6 Kong image I was able to use a KongClusterPlugin with Ingress to supplement.

I used this deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin-deployment
  labels:
    app: httpbin
spec:
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
      - name: httpbin
        image: kennethreitz/httpbin
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: httpbin
  name: httpbin-deployment
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: httpbin
  type: ClusterIP

With this Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin-ingress
  annotations:
    httpbin.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "kong"
spec:
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin-deployment
            port:
              number: 80

The initial result was:

$ curl -vvv http://172.18.0.240/status/200
*   Trying 172.18.0.240:80...
* Connected to 172.18.0.240 (172.18.0.240) port 80 (#0)
> GET /status/200 HTTP/1.1
> Host: 172.18.0.240
> User-Agent: curl/7.80.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Date: Mon, 13 Dec 2021 16:09:33 GMT
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< demo:  injected-by-kong
< X-Kong-Upstream-Latency: 3
< X-Kong-Proxy-Latency: 1
< Via: kong/2.6.0
< 
* Connection #0 to host 172.18.0.240 left intact

Then I created this KongClusterPlugin:

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: add-response-header
  annotations:
    kubernetes.io/ingress.class: kong
config:
  add:
    headers:
    - "demo: injected-by-kong"
plugin: response-transformer

and applied it to my Ingress resource:

$ kubectl patch ingress httpbin-ingress -p '{"metadata":{"annotations":{"konghq.com/plugins":"add-response-header"}}}'
ingress.networking.k8s.io/httpbin-ingress patched

After which:

$ curl -vvv http://172.18.0.240/status/200
*   Trying 172.18.0.240:80...
* Connected to 172.18.0.240 (172.18.0.240) port 80 (#0)
> GET /status/200 HTTP/1.1
> Host: 172.18.0.240
> User-Agent: curl/7.80.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Server: gunicorn/19.9.0
< Date: Mon, 13 Dec 2021 16:09:33 GMT
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< demo:  injected-by-kong
< X-Kong-Upstream-Latency: 3
< X-Kong-Proxy-Latency: 1
< Via: kong/2.6.0
< 
* Connection #0 to host 172.18.0.240 left intact

And as we can see demo: injected-by-kong was injected using the response-transformer.

So it appears that given the versions you're using I'm not able to reproduce this problem at least with a simple example, do you have any more details you can provide? Can you please put together an example of manifests that don't work properly (you can use mine as a basis)?

Also double check that your KongClusterPlugin has the ingress.class configuration as seen in my example above:

  annotations:
    kubernetes.io/ingress.class: kong

If memory serves, some older versions of the KIC would GET KongClusterPlugin options even if they didn't have the class on them, but with v2 this is not the case as that can cause issues with multi-tenant clusters (clusters which have more than one distinct deployment of KIC). If the ingress.class is missing it may be the cause of this issue.

[Panzki]

Hi,
thanks for taking the time to help, your help was invaluable.

If the ingress.class is missing it may be the cause of this issue.

The missing annotation to the ingress resource was indeed the cause of the issue. Adding the annotation fixed the problem.

I was not aware that this annotation was necessary. With version v2.2.0 of the Helm Chart it was working without the annotation, so the behavior must have changed. Is there any documentation or change log that can help me understand why the annotation is necessary with never version?

[shaneutt]

Hi, thanks for taking the time to help, your help was invaluable.
The missing annotation to the ingress resource was indeed the cause of the issue. Adding the annotation fixed the problem.

No problem, glad that got your situation sorted out 👍

I was not aware that this annotation was necessary. With version v2.2.0 of the Helm Chart it was working without the annotation, so the behavior must have changed. Is there any documentation or change log that can help me understand why the annotation is necessary with never version?

After digging into this it seems this was was present in the documentation for the type from v1.0.x forward:

https://docs.konghq.com/kubernetes-ingress-controller/1.0.x/concepts/custom-resources/#kongclusterplugin

What version did you upgrade from?

[rainest]

This limitation has always been in place for global KongClusterPlugins: https://github.com/Kong/kubernetes-ingress-controller/pull/520/files#diff-36de13ddfa1690d7bef40e3c6e8d05adfe4bb49493b54dce77829dc288906df7R258-R260

Referenced KongClusterPlugins maybe not. I think our advice was to include the class annotation anyway as a simplification ("just add it to all of these" rather than "add this annotation if you're making a global plugin, but omit it if not"), but it's possible we supported un-annotated referenced plugins.

kubernetes-ingress-controller/internal/controllers/configuration/zz_generated_controllers.go

Line 585 in 994fa04

preds := ctrlutils.GeneratePredicateFuncsForIngressClassFilter(r.IngressClassName, false, true)

is definitely new for 2.x, but I'm unsure what was in place before. We probably don't strictly need that since

kubernetes-ingress-controller/internal/store/store.go

Line 674 in 994fa04

if ok && s.isValidIngressClass(&p.ObjectMeta, annotations.ExactClassMatch) {

still filters the global ones after, but requiring it always matches our documentation.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.