Getting empty metric data with multiple dimensions in aws-cloudwatch trigger #2760

dekelev · 2022-03-15T09:29:10Z

Report

Using the following ScaledObject config, I always get an empty metric data:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 1
  maxReplicaCount: 10
  pollingInterval: 30
  cooldownPeriod:  300
  triggers:
    - type: aws-cloudwatch
      authenticationRef:
        name: keda-auth
      metadata:
        namespace: AWS/AmazonMQ
        dimensionName: Broker;Queue
        dimensionValue: aws-production;aws-worker
        metricName: MessageCount
        metricStat: Average
        metricUnit: Count
        metricStatPeriod: '60'
        metricCollectionTime: '60'
        targetMetricValue: '10'
        minMetricValue: '0'
        awsRegion: us-east-1

Running the following query in CloudWatch UI with 1 minute period returns a number above 0:
SELECT AVG(MessageCount) FROM "AWS/AmazonMQ" WHERE Broker = 'aws-production' and Queue = 'aws-worker'

While keda-operator log repeatedly shows:
aws_cloudwatch_scaler empty metric data received, returning minMetricValue

Expected Behavior

Getting a number above 0 for the requested metric.

Actual Behavior

Getting 0 for the requested metric.

Steps to Reproduce the Problem

Deploy a managed RabbitMQ - Amazon MQ
Deploy the ScaledObject config
Check keda-operator log

Logs from KEDA operator

1.6473359324006155e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6473359324026015e+09	INFO	setup	Running on Kubernetes 1.21+	{"version": "v1.21.5-eks-bc4871b"}
1.6473359324028835e+09	INFO	setup	Starting manager
1.6473359324029562e+09	INFO	setup	KEDA Version: 2.6.1
1.64733593240299e+09	INFO	setup	Git Commit: efca71d6bc770408468a9e1a4b3984f7136c0967
1.6473359324030223e+09	INFO	setup	Go Version: go1.17.3
1.6473359324030423e+09	INFO	setup	Go OS/Arch: linux/amd64
1.6473359324034576e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6473359324034605e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
I0315 09:18:52.403596       1 leaderelection.go:248] attempting to acquire leader lease keda/operator.keda.sh...
I0315 09:19:36.449460       1 leaderelection.go:258] successfully acquired lease keda/operator.keda.sh
1.6473359764499505e+09	INFO	controller.scaledobject	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
1.647335976450036e+09	INFO	controller.scaledobject	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "source": "kind source: *v2beta2.HorizontalPodAutoscaler"}
1.6473359764500802e+09	INFO	controller.scaledobject	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject"}
1.6473359764505255e+09	INFO	controller.triggerauthentication	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
1.6473359764505723e+09	INFO	controller.triggerauthentication	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "TriggerAuthentication"}
1.6473359764506705e+09	INFO	controller.scaledjob	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
1.6473359764507174e+09	INFO	controller.scaledjob	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "ScaledJob"}
1.6473359764499505e+09	INFO	controller.scaledobject	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
1.647335976450036e+09	INFO	controller.scaledobject	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "source": "kind source: *v2beta2.HorizontalPodAutoscaler"}
1.6473359764500802e+09	INFO	controller.scaledobject	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject"}
1.6473359764505255e+09	INFO	controller.triggerauthentication	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
1.6473359764505723e+09	INFO	controller.triggerauthentication	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "TriggerAuthentication"}
1.6473359764506705e+09	INFO	controller.scaledjob	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
1.6473359764507174e+09	INFO	controller.scaledjob	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "ScaledJob"}
1.6473359764508784e+09	INFO	controller.clustertriggerauthentication	Starting EventSource	{"reconciler group": "keda.sh", "reconciler kind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}
1.647335976450922e+09	INFO	controller.clustertriggerauthentication	Starting Controller	{"reconciler group": "keda.sh", "reconciler kind": "ClusterTriggerAuthentication"}
1.6473359765509717e+09	INFO	controller.triggerauthentication	Starting workers	{"reconciler group": "keda.sh", "reconciler kind": "TriggerAuthentication", "worker count": 1}
1.647335976551054e+09	INFO	controller.scaledobject	Starting workers	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "worker count": 5}
1.6473359765512447e+09	INFO	controller.scaledobject	Reconciling ScaledObject	{"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "worker, "namespace": "default"}
1.6473359765512555e+09	INFO	controller.clustertriggerauthentication	Starting workers	{"reconciler group": "keda.sh", "reconciler kind": "ClusterTriggerAuthentication", "worker count": 1}
1.6473360069127352e+09	INFO	aws_cloudwatch_scaler	empty metric data received, returning minMetricValue
1.6473360368981402e+09	INFO	aws_cloudwatch_scaler	empty metric data received, returning minMetricValue
1.647336066911557e+09	INFO	aws_cloudwatch_scaler	empty metric data received, returning minMetricValue

KEDA Version

2.6.1

Kubernetes Version

1.21

Platform

Amazon Web Services

Scaler Details

AWS CloudWatch

More Info

I'm debugging this locally and this is what I see when I add the input variable to the log:

{"level":"debug","ts":1647427653.20592,"logger":"aws_cloudwatch_scaler","msg":"empty metric data received, returning minMetricValue","data":"{\n  EndTime: 2022-03-16 12:47:00 +0200 IST,\n  MetricDataQueries: [{\n      Id: \"c1\",\n      MetricStat: {\n        Metric: {\n          Dimensions: [{\n              Name: \"Broker\",\n              Value: \"aws-production\"\n            },{\n              Name: \"Queue\",\n              Value: \"aws-worker\"\n            }],\n          MetricName: \"MessageCount\",\n          Namespace: \"AWS/AmazonMQ\"\n        },\n        Period: 60,\n        Stat: \"Average\",\n        Unit: \"Count\"\n      },\n      ReturnData: true\n    }],\n  ScanBy: \"TimestampDescending\",\n  StartTime: 2022-03-16 12:46:00 +0200 IST\n}"}

And this log shows that the response from CloudWatch is missing the Values field:

{"level":"debug","ts":1647427772.906795,"logger":"aws_cloudwatch_scaler","msg":"Received Metric Data","data":"{\n  MetricDataResults: [{\n      Id: \"c1\",\n      Label: \"MessageCount\",\n      StatusCode: \"Complete\"\n    }]\n}"}

The text was updated successfully, but these errors were encountered:

JorTurFer · 2022-03-16T11:55:04Z

Hi @dekelev
I'm not an expert on AWS services, but the query looks good, doesn't?
Maybe is there any change in their API?

dekelev · 2022-03-16T13:31:32Z

Hi @JorTurFer , Thanks for responding!

After debugging the Scaler class and the cloudwatch-sdk, I came to conclusion that it is probably an issue with the cloudwatch-sdk, because the request params sent to CloudWatch looks fine when using multiple dimensions.

As a workaround, I ended up with a forked git repo, replacing the metric data query with an expression query:

Expression: aws.String(c.metadata.expression),
Id:         aws.String("q1"),
Period:     aws.Int64(c.metadata.metricStatPeriod),
Label:      aws.String(c.metadata.label),

dekelev@d92d23c

zroubalik · 2022-03-16T14:00:33Z

@dekelev would you mind opening a PR to fix this issue?

dekelev · 2022-03-16T14:19:31Z

My current fix is just to replace the whole aws-cloudwatch scaler's internals with an expression logic, which wasn't the author vision for this scaler.

I don't have enough time available now to create a PR, but I'll consider extending this scaler in the future if no one else will.
I know there was another issue opened regarding the missing of expression feature in this scaler, so I'm not the only one that is bumping into this issue.

It doesn't seems to be much effort to extend the scaler so it would validate and execute the expression only if it is defined in the user's config.

dekelev · 2022-03-19T11:19:25Z

From some reason, I also had to multiply the metric value returned from CloudWatch by 100, so that the external metric's quantity would be correctly consumed and displayed by the HPA.

Any idea why the metric is not consumed as is by the HPA?

I'm not sure if using the original scaler without expression won't lead to the same issue, so if I create a PR, this might be a breaking-change.

zroubalik · 2022-03-22T12:09:16Z

HPA doesn't work with float numbers, integers only.
Seems like this scaler is not correct with that regard,

keda/pkg/scalers/aws_cloudwatch_scaler.go

Lines 45 to 46 in 11f0554

    
           targetMetricValue float64 
        
           minMetricValue    float64

@JorTurFer FYI

JorTurFer · 2022-03-22T12:14:10Z

Interesting! So we are missing the info during the cast to int64 🤔
Maybe we should update all scalers to use always int64 and explain it in the docs to be clearest as possible

zroubalik · 2022-03-22T12:16:08Z

yeah, we should fix that

dekelev · 2022-04-15T12:37:41Z

I've opened this PR to add alternative expression field and also changed the scaler to use int64 instead of float64.

dekelev · 2022-04-15T17:04:58Z

HPA doesn't work with float numbers, integers only. Seems like this scaler is not correct with that regard,

keda/pkg/scalers/aws_cloudwatch_scaler.go

Lines 45 to 46 in 11f0554

targetMetricValue float64

minMetricValue float64

@JorTurFer FYI

@zroubalik I still had to multiply the metric by 100, even with int64. I have no idea why, but it only works like that.
I'm running it in production environment, it runs well and I can see the expected metric numbers in the HPA stats.

https://github.com/kedacore/keda/pull/2911/files#diff-79307abe3419ddfb44131b054ea48e8b03c304201d38ad08b386b9d64e40875fR284

stale · 2022-06-14T17:19:44Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

JorTurFer · 2022-06-14T19:04:13Z

I think this is already covered in the PR, feel free to reopen it if it's not done 😄

dekelev added the bug Something isn't working label Mar 15, 2022

tomkerkhove added this to Roadmap - KEDA Core Mar 15, 2022

tomkerkhove moved this to Proposed in Roadmap - KEDA Core Mar 15, 2022

JorTurFer added the help wanted Looking for support from community label Mar 16, 2022

JorTurFer moved this from Proposed to To Do in Roadmap - KEDA Core Mar 16, 2022

JorTurFer mentioned this issue Mar 22, 2022

Update all needed scalers to use int64 instead of float64 #2790

Closed

6 tasks

dekelev mentioned this issue Apr 15, 2022

Added alternative expression field in AWS CloudWatch scaler #2911

Closed

5 tasks

dekelev mentioned this issue May 4, 2022

Added alternative expression field in AWS CloudWatch scaler #2997

Merged

5 tasks

stale bot added the stale All issues that are marked as stale due to inactivity label Jun 14, 2022

JorTurFer closed this as not planned Won't fix, can't repro, duplicate, stale Jun 14, 2022

Repository owner moved this from To Do to Ready To Ship in Roadmap - KEDA Core Jun 14, 2022

tomkerkhove moved this from Ready To Ship to Done in Roadmap - KEDA Core Aug 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Getting empty metric data with multiple dimensions in aws-cloudwatch trigger #2760

Getting empty metric data with multiple dimensions in aws-cloudwatch trigger #2760

dekelev commented Mar 15, 2022 •

edited

Loading

JorTurFer commented Mar 16, 2022

dekelev commented Mar 16, 2022 •

edited

Loading

zroubalik commented Mar 16, 2022

dekelev commented Mar 16, 2022 •

edited

Loading

dekelev commented Mar 19, 2022

zroubalik commented Mar 22, 2022

JorTurFer commented Mar 22, 2022

zroubalik commented Mar 22, 2022

dekelev commented Apr 15, 2022

dekelev commented Apr 15, 2022 •

edited

Loading

stale bot commented Jun 14, 2022

JorTurFer commented Jun 14, 2022 •

edited

Loading

Getting empty metric data with multiple dimensions in aws-cloudwatch trigger #2760

Getting empty metric data with multiple dimensions in aws-cloudwatch trigger #2760

Comments

dekelev commented Mar 15, 2022 • edited Loading

Report

Expected Behavior

Actual Behavior

Steps to Reproduce the Problem

Logs from KEDA operator

KEDA Version

Kubernetes Version

Platform

Scaler Details

More Info

JorTurFer commented Mar 16, 2022

dekelev commented Mar 16, 2022 • edited Loading

zroubalik commented Mar 16, 2022

dekelev commented Mar 16, 2022 • edited Loading

dekelev commented Mar 19, 2022

zroubalik commented Mar 22, 2022

JorTurFer commented Mar 22, 2022

zroubalik commented Mar 22, 2022

dekelev commented Apr 15, 2022

dekelev commented Apr 15, 2022 • edited Loading

stale bot commented Jun 14, 2022

JorTurFer commented Jun 14, 2022 • edited Loading

dekelev commented Mar 15, 2022 •

edited

Loading

dekelev commented Mar 16, 2022 •

edited

Loading

dekelev commented Mar 16, 2022 •

edited

Loading

dekelev commented Apr 15, 2022 •

edited

Loading

JorTurFer commented Jun 14, 2022 •

edited

Loading