metrics missing since upgrade to 3.7.0 from 3.6.1 at the /metrics endpoint with the prometheus plugin

[mlindes]

Discussed in #14272

^{Originally posted by mlindes February 12, 2025}
Since upgrading to 3.7.1, I no longer see kong_http_requests_total kong_kong_latency* or many of the metrics that should be visible at the /metrics endpoint with the following config:

  "plugins": [
    {
      "name": "prometheus",
      "config": {
        "bandwidth_metrics": true,
        "latency_metrics": true,
        "per_consumer": true,
        "status_code_metrics": true,
        "upstream_health_metrics": true
      },
      "enabled": true,
      "protocols": [
        "grpc",
        "grpcs",
        "http",
        "https"
      ]
    }
  ]
}

The only metrics visible at that endpoint are:

kong_data_plane_cluster_cert_expiry_timestamp
kong_datastore_reachable
kong_memory*
kong_nginx_*

This is not an enterprise installation.. it's the OSS branch, so no workspaces are enabled. I see workspaces are now a default tag on metrics and I'm wondering if that has anything to do with this issue.

Anyone have similar?

[xianghai2]

Recommend to use Kong-EE enterprise version.

[ProBrian]

@mlindes , could you try latest version 3.9.0? I tried 3.9.0 locally with the same prometheus plugin like yours, and could successfully got kong_http_requests_total and kong_kong_latency* metrics with the /metrics endpoint.

[ProBrian]

By the way, what's your deployment mode, and what's the version of kong that worked in your machine?