Consecutive // in path cause Ingress Controller to block updates

[sonujose]

Summary

Getting this error from the kong ingress controller, services are up, and ingress rues have been set, Loadbalancer IP has been assigned by Kong. All Kong components are up.

These are the error and warning we are getting from kong ingress controller.

E0423 1 controller.go:124] unexpected failure updating Kong configuration:
posting new config to /config: 400 Bad Request {"fields":{"services":[null,{"routes":[{"paths":["must not have empty segments"]}]}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {services={[2]={routes={{paths={\"must not have empty segments\"}}}}}}"}
W0423 1 queue.go:112] requeuing ven-dev/release-contact-ms-contact, err posting new config to /config: 400 Bad Request {"fields":{"services":[null,{"routes":[{"paths":["must not have empty segments"]}]}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {services={[2]={routes={{paths={\"must not have empty segments\"}}}}}}"}

Kong Ingress controller version
0.8.0

Kubernetes version

Client Version: "v1.17.0",
Server Version: "v1.15.7"

Environment

AKS

What happened

We observed a sudden change in the kong behavior. Kong was working fine before and we were able to create routes, Once we started adding more and more microservices. It went down. Now independently these ms are running, but once connected to kong, the routes are not getting created. KongIP is successfully assigned, but routes are not working. We even tested it with Konga, it was not showing any routes.

For debugging, I created another Kong Instance and deployed couple of microservices and it is working fine in the same cluster. Really wants to know if one of the microservice configuration is taking down kong??

I have kong validationwebhook installed in the cluster.

[hbagdi]

It seems like the path property of routes in Kong is malformed.
Can you share your ingress resources?

[sonujose]

This is the ingress resource I used for deploying. Created another instance of kong in the same cluster and the same ingress got properly deployed.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    configuration.konghq.com: release-v1-hexagonal-microservice-custom-kong-ingress
    kubernetes.io/ingress.class: kong-dev
  creationTimestamp: "2020-04-20T09:52:25Z"
  generation: 3
  labels:
    app.kubernetes.io/domain: core
    app.kubernetes.io/environment: dev
    app.kubernetes.io/instance: release-v1
    app.kubernetes.io/name: release-v1-hexagonal-microservice
    helm.sh/chart: core-microservice-hexagonal-microservice
  name: release-v1-hexagonal-microservice-main
  namespace: caf-dev
  resourceVersion: "8292449"
  selfLink: /apis/extensions/v1beta1/namespaces/caf-dev/ingresses/release-v1-hexagonal-microservice-main
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: release-v1-hexagonal-microservice
          servicePort: 80
        path: /core/caf/v1/

The only difference is that as I added more microservices to that specific kong instance suddenly stopped working.

[hbagdi]

Does any of your Ingress path properties have a consecutive /?
Kong throws that error if that's the case.
More here.

[sonujose]

Seems like got the culprit. Thanks @hbagdi .

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    configuration.konghq.com: wf-add-user-task-handler-kong-ingress
    kubernetes.io/ingress.class: kong-dev
  creationTimestamp: "2020-04-22T03:21:46Z"
  generation: 1
  name: wf-add-user-task-handler
  namespace: sdf-dev
  resourceVersion: "8259907"
  selfLink: /apis/extensions/v1beta1/namespaces/sdf-dev/ingresses/wf-add-user-task-handler
  uid: f23c7ed4-be1f-4faa-ac6d-a789d79c0378
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: wf-add-user-task-handler
          servicePort: 80
        path: //caf/

[sonujose]

@hbagdi Is it expected to get down entire Kong Controller, because of configuration error happened in one of the ingress resource?

Is there any way to prevent it? Because this will end up in a situation where kong can be a single point of failure for the cluster.

[hbagdi]

Is it expected to get down entire Kong Controller, because of configuration error happened in one of the ingress resource?

Certainly not.

There have been such issues before where a single problem has caused a halt. We have added necessary validations in all places to avoid it.
I acknowledge that path validation needs to be better here.
I'm going to label this as a bug to fix it.