Prometheus needs to scrape 2 addresses in a pod

[mflendrich]

Problem statement:
Kong Gateway has prom metrics on port 8100, KIC 2.0 has prom metrics on port 10255.
We want users who deploy KIC using our default methods to see metrics from both endpoints in Prometheus.

Proposed solution:

• Define an appropriate ServiceMonitor in the Helm chart that tells Prom to scrape both endpoints.
• Figure out a way for the kustomize manifests to achieve scraping both (FWIW, our kustomize manifests don't use ServiceMonitors today), and implement it.

Acceptance criteria:

• Kustomization makes it possible for Prometheus to scrape both KIC and Gateway postponed to post KIC 2.0
• An issue to cover support for Prometheus in Kustomize manifests exists
• Helm chart makes it possible for Prometheus to scrape both KIC and Gateway

[ccfishk]

Proxy is exporting prometheus format data, and Controller is reporting metrics using golang-prometheus library (#1520).
And the two mechanisms are different while providing prometheus scrape.
Do you think they are good enough ? Or, do you still think we need this for KIC2.0 beta.

[rainest]

Currently we get these from the base manifests:

kubernetes-ingress-controller/deploy/single-v2/all-in-one-dbless.yaml

Lines 1331 to 1332 in bc19341

	prometheus.io/port: "8100"
	prometheus.io/scrape: "true"

Those explicitly do not support multiple ports, however, because Kubernetes labels don't work that way. The CRD approach appears to be the officially endorsed solution: prometheus/prometheus#3756 (comment) (I thought I'd found another discussion indicating that Prometheus specifically no longer mentioned those annotations in their docs because of this, but can't find those again).

We already did provide the CRD in the chart, and should be able to add the controller's fairly easily: Kong/charts@0a8295d

However, that does not work in 1.x, and we don't currently have any explicit 1.x/2.x gate in the chart for 2.x-only features that need template changes.

[shaneutt]

Testing Notes: this would be a good time to add Prometheus as a clusters/addon in KTF so that we can write integration tests that prove the ServiceMonitor is functioning via Prometheus.

[mflendrich]

Manual acceptance tests for the Helm chart support:

1. Spin up a Kubernetes cluster
2. Install Prometheus in that cluster, with ServiceMonitor support. For example, using Prometheus Operator.
3. Install KIC using the helm chart. This installs a deployment with 2 containers in each pod: proxy and ingress controller.
4. Go to the Prometheus UI, choose the "Targets" tab, and verify that there are 2 entries for each pod: one for the proxy /metrics endpoint on port 8100, one for the ingress controller on port 10255
5. Ensure that metrics from KIC are present in Prometheus.

[rainest]

It looks like adding the new endpoint not supported in 1.x wouldn't be outright harmful if you're still running 1.x, but it will result in Prometheus spamming "hey I can't read this endpoint" errors forever, which is annoying.

We could make conditional features based on semver comparisons between the ingressController.image.tag value and a static value, but it's a bit inflexible since the sprig function just up and dies/terminates template rendering if it can't parse a value. If you use custom controller images and your tag is not semver, rendering just explodes and your command fails. Imposing a semver requirement on controller image tags is probably reasonable, but is breaking.

Kong/charts@dbfd970 demonstrates this but it doesn't quite work yet.

[shaneutt]

As per recent @Kong/team-k8s discussions the work for this repository is complete since we're passing on kustomize. All remaining work for this issue belongs in https://github.com/kong/charts so this does not need to be marked as a blocker for our first beta tag.

[ccfishk]

Synced offline, as pointed out https://github.com/Kong/kubernetes-ingress-controller/blob/next/deploy/single-v2/all-in-one-dbless.yaml#L1331

Need fix ServiceMonitor resource and adopt it into manifests.yaml

[ccfishk]

This feature has been in and verified. close as it is.