metrics missing since upgrade to 3.7.0 from 3.6.1 at the /metrics endpoint with the prometheus plugin

[mlindes]

Since upgrading to 3.7.1, I no longer see kong_http_requests_total kong_kong_latency* or many of the metrics that should be visible at the /metrics endpoint with the following config:

  "plugins": [
    {
      "name": "prometheus",
      "config": {
        "bandwidth_metrics": true,
        "latency_metrics": true,
        "per_consumer": true,
        "status_code_metrics": true,
        "upstream_health_metrics": true
      },
      "enabled": true,
      "protocols": [
        "grpc",
        "grpcs",
        "http",
        "https"
      ]
    }
  ]
}

The only metrics visible at that endpoint are:

kong_data_plane_cluster_cert_expiry_timestamp
kong_datastore_reachable
kong_memory*
kong_nginx_*

This is not an enterprise installation.. it's the OSS branch, so no workspaces are enabled. I see workspaces are now a default tag on metrics and I'm wondering if that has anything to do with this issue.

Anyone have similar?

[robinlZG]

First thing I'd try is using admin API to check the running config of the prometheus plugin vs. your configs. If you have access/shell to a control plane host or kong node:
curl -s localhost:8001/plugins
That gets a bit more complicated if you're running a large installation with lots of plugins and you have to paginate through several subsequent requests to the admin API to find the plugin you're interested in. If you've got the plugin ID squirreled away in a state file or other IAC data store, use that:
curl -s localhost:8001/plugins/<plugin-id>
That will dump the running config of the plugin for review. I'd wager the plugin is running the default config in which all the good stuff like status_code_metrics is set to false by default.