Ingress Does not Get Assigned Address

[doctorwu]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When I install Kong from the bit.ly URL in GKE (not sure about anywhere else), and create an Ingress in a script. The Ingress resource does get assigned an IP address. Doing kubectl -n kong rollout restart deployment ingress-kong jogs it into working again.

Expected Behavior

Ingress should be assigned the kong proxy IP address.

Steps To Reproduce

Attached is a script snippet:


kubectl create namespace kong --context $KUBE_CONTEXT

kubectl create secret generic kong-enterprise-license --from-file=./license -n kong --context=$KUBE_CONTEXT

kubectl apply -f https://bit.ly/k4k8s-enterprise-install --context $KUBE_CONTEXT

echo "apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kuma-control-plane
  namespace: $NS
  annotations:
    konghq.com/strip-path: 'true'
    kubernetes.io/ingress.class: kong
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: $CP_SERVICE
          servicePort: 5681
        path: /kuma" | kubectl --context $KUBE_CONTEXT apply -f -

### Kong Ingress Controller version

```shell
v2.0.2

Kubernetes version

1.19.13-gke.1200

Anything else?

No response

[shaneutt]

@doctorwu and I are working together on this in Slack, so far I've been unable to reproduce this on a local kind cluster, but we can consistently reproduce it on GKE with a very specific configuration. We're continuing to debug that environment.

We did find that when this issue is triggered, a restart of the pods is an effective mitigation:

$ kubectl -n <namespace> rollout restart deployment <kong-deployment>

Though this is potentially disruptive, and not ideal.

[shaneutt]

After digging into the reproduction environment the problem is with the status reporting components of this specifically, the actual routing works but the status updates silently fail. Also turns out we don't have good debugging output on the status updates components so in addition to a fix we should also improve debugging output there to make debugging these kinds of issues better. At this point I have high confidence this is a v2 only issue and should not affect the most recent v1.3.x releases. I'll be doing a study of that code and look for and test improvements given the GKE reproduction environment.

[shaneutt]

#1930 is related and improves the debugging logging in this area, but doesn't resolve this.

[shaneutt]

The issue ultimately ended up being a race condition with LoadBalancer type Service usage for the Kong Gateway. In GKE these IPs can provision very slowly (and thus the difficulty to reproduce this in local cluster envs). There was a mishandling of the gateway service when starting up the controller manager that is fixed with #1931, and @doctorwu and I tested this fix in GKE with success.

[shaneutt]

The fix will be in v2.0.3, which is releasing today: #1932

[doctorwu]

Seems to be back. Using GKE v1.20.11-gke.1300 and KIC v2.1.1

When I install Kong with kubectl apply -f https://bit.ly/k4k8s-enterprise-install and create an Ingress right after installation, the Ingress never gets an IP address. However, when I delete the Ingress and recreate, all is good.

[rainest]

7aaaf43 is the proposed non-blocking fix. Waiting for nightly stuff to work so that we can run E2E on it.