Error "Operation cannot be fulfilled on scaledobjects.keda.sh" getting on all scaledobjects in all clusters

[saurabhvagrawal]

Report

We have recently upgraded keda to keda v2.4. Since yesterday we are getting this issue on all scaledobjects across all production clusters:

E0813 11:01:27.559601 1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh "k8ssanityapp": the object has been modified; please apply your changes to the latest version and try again" "Error"=null

Expected Behavior

It should not flag any errors in metrics api server

Actual Behavior

I have checked the hpa behavior and can't find any anomaly in it. However, I can see errors in keda-operator and keda-operator-metrics-apiserver container. Below are the logs from containers:

Keda-operator:

E0813 11:48:10.829282 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for wcdproduction_wcdprdinfrasystem_ingress:services_webrequests_status_nullableaverage_namespace=wdatp-infra-system_region=neu_service=k8ssanityapp"}

keda-operator-metrics-server

E0813 11:01:27.559601 1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh "k8ssanityapp": the object has been modified; please apply your changes to the latest version and try again" "Error"=null

Steps to Reproduce the Problem

1. Deploy keda from helm chart
2. Deploy application with following keda configurations:

enabled: true
minReplicaCount: 1
maxReplicaCount: 2
triggers:

• type: prometheus
  metadata:
  serverAddress: http://infra-monitoring-prometheu-prometheus.wdatp-infra-system.svc.cluster.local:9090
  metricName: nginx_ingress_controller_request_duration_seconds_count
  query: sum(rate(nginx_ingress_controller_request_duration_seconds_count{namespace="wdatp-infra-system", service="k8ssanityapp"}[1m])) * 100
  threshold: 300
• type: cpu
  metadata:
  type: Utilization
  value: 80
• type: external
  metadata:
  scalerAddress: geneva-external-scaler.wdatp-infra-system.svc.cluster.local:9090
  metricsAccount: WCDStaging
  metricsNamespace: WCDSTGInfraSystem
  metricName: ingress:services_webrequests_status
  samplingType: NullableAverage
  targetValue: 3
  d_Region: EUS2
  d_namespace: wdatp-infra-system
  d_service: k8ssanityapp

Logs from KEDA operator

Keda-operator:

E0813 11:48:10.829282 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for wcdproduction_wcdprdinfrasystem_ingress:services_webrequests_status_nullableaverage_namespace=wdatp-infra-system_region=neu_service=k8ssanityapp"}

keda-operator-metrics-server

E0813 11:01:27.559601 1 fallback.go:96] keda_metrics_adapter/provider "msg"="Error updating ScaledObject status" "error"="Operation cannot be fulfilled on scaledobjects.keda.sh "k8ssanityapp": the object has been modified; please apply your changes to the latest version and try again" "Error"=null

KEDA Version

2.4.0

Kubernetes Version

1.20

Platform

Microsoft Azure

Scaler Details

prometheus

Anything else?

No response

[zroubalik]

Thanks for reporting, the problem is probably here:

keda/pkg/provider/fallback.go

Lines 94 to 97 in b27de0c

	err := p.client.Status().Update(context.TODO(), scaledObject)
	if err != nil {
	logger.Error(err, "Error updating ScaledObject status", "Error")
	}

We should be sure that we are modifing the status on up-to-date object, or use patch for that.

[saurabhvagrawal]

Thanks for reporting, the problem is probably here:

keda/pkg/provider/fallback.go

Lines 94 to 97 in b27de0c

	err := p.client.Status().Update(context.TODO(), scaledObject)
	if err != nil {
	logger.Error(err, "Error updating ScaledObject status", "Error")
	}

We should be sure that we are modifing the status on up-to-date object, or use patch for that.

Thank you for quick action. Let me know once it is fixed. We will pull updated helm chart and deploy in our clusters.,

[zroubalik]

@saurabhvagrawal are you able to test the patch? Here are the images that contains the patch (of course this is not an official KEDA release):
KEDA Operator: docker.io/zroubalik/keda:flbkpatch
KEDA Metrics Server: docker.io/zroubalik/keda-metrics-apiserver:flbkpatch

Thanks

[saurabhvagrawal]

@saurabhvagrawal are you able to test the patch? Here are the images that contains the patch (of course this is not an official KEDA release):
KEDA Operator: docker.io/zroubalik/keda:flbkpatch
KEDA Metrics Server: docker.io/zroubalik/keda-metrics-apiserver:flbkpatch

Thanks

@zroubalik : I am working on to deploy it in our dev cluster and post that, will test its behavior. I will keep this thread updated. Meanwhile, when can we expect to get official fix/release for keda. Appreciate if we get some timeline.