External Metrics metricName is always empty

[askhalil]

I have installed the latest helm chart into my cluster and configured 2 external metrics for the adapter as below:

- seriesQuery: '{__name__=~"^sidekiq_queue_latency_seconds$"}'
          resources:
            template: <<.Resource>>
          name:
            as: "sidekiq_queue_latency_seconds"
            matches: ""
          metricsQuery: "sum(sidekiq_queue_latency_seconds) by (name)"
        - seriesQuery: '{__name__=~"^sidekiq_queue_enqueued_jobs$"}'
          resources:
            template: <<.Resource>>
          name:
            as: "sidekiq_queue_enqueued_jobs"
            matches: ""
          metricsQuery: "sum(sidekiq_queue_enqueued_jobs) by (name)"

The adapter is able to fetch them and post them into the external metrics api of the k8s cluster but when I run this command kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/platform/sidekiq_queue_enqueued_jobs", I always get the metrics with metricName field empty as below

{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/external.metrics.k8s.io/v1beta1/namespaces/platform/sidekiq_queue_enqueued_jobs"
  },
  "items": [
    {
      "metricName": "",
      "metricLabels": {
        "name": "default"
      },
      "timestamp": "2019-09-01T12:32:13Z",
      "value": "0"
    },
    {
      "metricName": "",
      "metricLabels": {
        "name": "integration"
      },
      "timestamp": "2019-09-01T12:32:13Z",
      "value": "0"
    }
  ]
}

The problem I am facing now is that when I use this in HPA with label selectors, it adds both numbers and ignores the label.

Here is the external metric in HPA:

- type: External
    external:
      metricName: sidekiq_queue_enqueued_jobs
      metricSelector:
        matchLabels:
          name: default
      targetValue: 50

Any idea if I am missing something or there is an issue in the external provider?

[max-rocket-internet]

I have the same problem. Also all value are 0.

I am wanting to use the metric nginx_ingress_controller_ingress_upstream_latency_seconds_sum.

My external metric looks like this:

    - seriesQuery: '{__name__=~"^nginx_ingress_controller_ingress_upstream_latency_seconds_sum"}'
      name:
        matches: "^(.*)_sum$"
        as: "nginx_ingress_upstream_latency_avg"
      metricsQuery: 'avg(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_sum{kubernetes_name=~".*-metrics"}[5m])) by (ingress)'

If I put avg(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_sum{kubernetes_name=~".*-metrics"}[5m])) by (ingress) into prometheus I get results back like this:

Element	Value
{ingress="eu01-xxx02-yyyy"}	0
{ingress="us01-xxx01-rrrrr"}	0.0003772108843503594
{ingress="as01-xxx01-uuuuu"}	0.00006543931812772487

kubectl raw results:

$ kubectl get --raw /apis/external.metrics.k8s.io/v1beta1 | jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "nginx_ingress_upstream_latency_avg",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

All values are 0:

$ kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/nginx_ingress_upstream_latency_avg | jq
{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/nginx_ingress_upstream_latency_avg"
  },
  "items": [
    {
      "metricName": "",
      "metricLabels": {
        "ingress": "eu01-xxx02-yyyy"
      },
      "timestamp": "2019-09-17T10:44:00Z",
      "value": "0"
    },
    {
      "metricName": "",
      "metricLabels": {
        "ingress": "us01-xxx01-rrrrr"
      },
      "timestamp": "2019-09-17T10:44:00Z",
      "value": "0"
    },
...

What are we missing here?

[max-rocket-internet]

Ahh mine is working, it's just that all the metrics are getting rounded down to 0. I multiplied by 1000 and now I get numbers:

    - seriesQuery: '{__name__=~"^nginx_ingress_controller_ingress_upstream_latency_seconds_sum"}'
      name:
        matches: "^(.*)_sum$"
        as: "nginx_ingress_upstream_latency_avg_ms"
      metricsQuery: 'avg(rate(<<.Series>>{kubernetes_name=~".*-metrics"}[5m])) by (ingress) * 1000'

But my metricName attributes are still ""

[askhalil]

@max-rocket-internet were you able to use an HPA with

metricSelector:
        matchLabels:

and you were able to get the value for that particular label?

My case is that it gives me the sum of all values.

[max-rocket-internet]

were you able to use an HPA with

I haven't tried this just yet but will try to implement something in the next few weeks with an SQS queue metric.

[karanmagdani]

@max-rocket-internet Did you get a SQS queue metric working?

[max-rocket-internet]

Did you get a SQS queue metric working?

Not yet as I've been working on other tasks. I'll come back to this soon though.

[jubel-han]

I got the same issue, it looks like the metric selector in HPA object haven't been handled correctly.

This is the adapter log, as I can see, the query sends to Prometheus doesn't include the queue label.
even though, the requests sent to the adapter from the controller manager, having the metricLabelSelector parameter.

I1210 10:28:31.735775       1 api.go:74] GET http://prometheus.default.svc:9090/api/v1/query?query=sum%28celery_queue_size%7Bnamespace%3D%22staging%22%7D%29+by+%28namespace%2Cqueue%29&time=1575973711.732 200 OK
I1210 10:28:31.736651       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/staging/metrics/celery_queue_size?metricLabelSelector=queue%3Dheavy: (10.466594ms) 200 [[kube-controller-manager/v1.12.10 (linux/amd64) kubernetes/27b48e2/system:serviceaccount:kube-system:horizontal-pod-autoscaler] 10.140.0.4:33472]

metrics:

# HELP celery_queue_lengths the size of the redis broker queues
# TYPE celery_queue_lengths gauge
celery_queue_size{queue="heavy"} 806.0
celery_queue_size{queue="light"} 0.0

HPA object:

apiVersion: autoscaling/v2beta1
spec:
  metrics:
  - type: Object
    object:
      metricName: celery_queue_size
      selector:
        matchLabels: {'queue': 'heavy'}
      targetValue: 1000
      target:
        apiVersion: v1
        kind: Namespace
        name: staging

adapter rules:

seriesQuery: '{__name__="celery_queue_size",namespace!="",queue!=""}'
  resources:
    template: <<.Resource>>
    name: {"as": "celery_queue_size"}
    metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>, queue)

[yiyijin]

I think the issue could be the version of docker image is used for adapter.

See this fix: #196, I think it only available for v0.6.0 (latest), so if you are using an older version, then you will have this problem

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.