Prometheus metrics for the controller

[hbagdi]

Problem statement:

As a user I cannot monitor the performance and errors of individual operations performed by KIC.

Proposed solution:

Use the existing Prometheus endpoint exposed by KIC 2.0 (created by the controller-runtime library) to expose additional metrics:

• [type: histogram] Duration of the last sync to Kong
  • with a metric field mode="deck-sync", mode="post-config" to provide distinction between these different operations
  • with a metric field result="success", result="error"
• [type: counter] Parsing error count
• [type: counter] "sync to Kong" count
  • with a metric field mode="deck-sync", mode="post-config" to provide distinction between these different operations
  • with a metric field result="success", result="error"

Acceptance criteria:

• KIC 2.0 exposes the metrics defined above on the --metrics-bind-addr (which is already present in KIC 2.0)
• Tests for the metrics above - guaranteeing that errors are accounted for appropriately and that the distribution metrics contain valid data
• Documentation for the metrics - both in prometheus metrics metadata and in docs.konghq.com
• The solution is forward-compatible with KIC sending configuration to multiple proxies.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[shaneutt]

Double checked with Harry, this should be a post-2.0 item.

[ccfishk]

@mflendrich do you want to refer [type: distribution] as https://prometheus.io/docs/concepts/metric_types/#histogram ?
Kindly apply the prometheus term.

[mflendrich]

@mflendrich do you want to refer [type: distribution] as https://prometheus.io/docs/concepts/metric_types/#histogram ?
Kindly apply the prometheus term.

Yes, when I said "distribution" I meant "histogram".