create ScaledObject when triggers type is prometheus metricType is Value err

[LY-today]

Report

when triggers type is prometheus， metricType is Value，create ScaledObject ，keda-operator is err
2024-08-07T09:33:45Z ERROR Reconciler error {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"my-helm-demo","namespace":"crane-system"}, "namespace": "crane-system", "name": "my-helm-demo", "reconcileID": "f750f79f-5a1b-4fc9-88d2-b136a11b65ff", "error": "HorizontalPodAutoscaler.autoscaling \"keda-hpa-my-helm-demo\" is invalid: spec.metrics[0].external.target.type: Invalid value: \"value\": must be either Utilization, Value, or AverageValue"}

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-helm-demo
  namespace: crane-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-helm-demo
  pollingInterval: 15
  minReplicaCount: 2
  maxReplicaCount: 50
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 0
          policies:
            - type: Percent 
              value: 100
              periodSeconds: 15
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Percent
              value: 500
              periodSeconds: 15
  triggers:
    - type: prometheus
      metricType: Value
      metadata:
        serverAddress: xxx
        metricName: portrait_pod_cpu_predict_compression_all_num
        query: portrait_pod_cpu_predict_compression_all_num{}
        threshold: "50"

Expected Behavior

hpa resources can be created normally

Actual Behavior

hpa not create

Name:         my-helm-demo
Namespace:    crane-system
Labels:       scaledobject.keda.sh/name=my-helm-demo
Annotations:  <none>
API Version:  keda.sh/v1alpha1
Kind:         ScaledObject
Metadata:
  Creation Timestamp:  2024-08-07T09:33:24Z
  Finalizers:
    finalizer.keda.sh
  Generation:        1
  Resource Version:  24908
  UID:               0d11f0d0-216e-4696-be18-51a49cad0bd3
Spec:
  Advanced:
    Horizontal Pod Autoscaler Config:
      Behavior:
        Scale Down:
          Policies:
            Period Seconds:              15
            Type:                        Percent
            Value:                       100
          Stabilization Window Seconds:  0
        Scale Up:
          Policies:
            Period Seconds:              15
            Type:                        Percent
            Value:                       500
          Stabilization Window Seconds:  0
  Max Replica Count:                     50
  Min Replica Count:                     2
  Polling Interval:                      15
  Scale Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         my-helm-demo
  Triggers:
    Metadata:
      Metric Name:     portrait_pod_cpu_predict_compression_all_num
      Query:           portrait_pod_cpu_predict_compression_all_num{}
      Server Address:  xxxx
      Threshold:       50
    Metric Type:       Value
    Type:              prometheus
Status:
  Conditions:
    Message:  Failed to ensure HPA is correctly created for ScaledObject
    Reason:   ScaledObjectCheckFailed
    Status:   False
    Type:     Ready
    Message:  ScaledObject check failed
    Reason:   UnkownState
    Status:   Unknown
    Type:     Active
    Status:   Unknown
    Type:     Fallback
  External Metric Names:
    s0-prometheus-portrait_pod_cpu_predict_compression_all_num
  Original Replica Count:  8
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type     Reason                   Age                   From           Message
  ----     ------                   ----                  ----           -------
  Warning  ScaledObjectCheckFailed  59s (x17 over 6m27s)  keda-operator  Failed to ensure HPA is correctly created for ScaledObject

Steps to Reproduce the Problem

Logs from KEDA operator

2024-08-07T09:33:45Z	ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"my-helm-demo","namespace":"crane-system"}, "namespace": "crane-system", "name": "my-helm-demo", "reconcileID": "f750f79f-5a1b-4fc9-88d2-b136a11b65ff", "error": "HorizontalPodAutoscaler.autoscaling \"keda-hpa-my-helm-demo\" is invalid: spec.metrics[0].external.target.type: Invalid value: \"value\": must be either Utilization, Value, or AverageValue"}

KEDA Version

< 2.11.0

Kubernetes Version

< 1.28

Platform

None

Scaler Details

prometheus

Anything else?

$ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.21.1
WARNING: version difference between client (1.29) and server (1.21) exceeds the supported minor version skew of +/-1

kubectl apply --server-side -f https://github.com/kedacore/keda/releases/download/v2.8.0/keda-2.8.0.yaml

[JorTurFer]

Hello,
KEDA v2.8 is quite old (almost 2 years). I don't remember any issue related but just in case, have you tried using v2.8.2 -> https://github.com/kedacore/keda/releases/tag/v2.8.2?

Sadly, that version is out of support, and we won't ship any new releases for it. I know that recent versions don't face with that issue, I'd suggest upgrading to a recent version.

[LY-today]

Hello, KEDA v2.8 is quite old (almost 2 years). I don't remember any issue related but just in case, have you tried using v2.8.2 -> https://github.com/kedacore/keda/releases/tag/v2.8.2?

Sadly, that version is out of support, and we won't ship any new releases for it. I know that recent versions don't face with that issue, I'd suggest upgrading to a recent version.

@JorTurFer hi，The problem was indeed solved by switching to v2.8.2, but isn’t this the recommended version officially provided by keda?
https://keda.sh/docs/2.15/operate/cluster/#kubernetes-compatibility

[JorTurFer]

The problem was indeed solved by switching to v2.8.2

Nice! Happy to read it 😄

I guess that you are using old k8s cluster, so probably v2.8 is the greatest version that you can use (if you are running k8s < 1.23 KEDA > 2.9 won't work). The point is that we don't ship fixes for old versions, we could ship a fix for the previous version but 2.8 is 7 versions far from v2.15 and although we will try to support with misconfigurations, if there is a real bug unresolved, we won't ship any fix for old versions.

[LY-today]

The problem was indeed solved by switching to v2.8.2

Nice! Happy to read it 😄

I guess that you are using old k8s cluster, so probably v2.8 is the greatest version that you can use (if you are running k8s < 1.23 KEDA > 2.9 won't work). The point is that we don't ship fixes for old versions, we could ship a fix for the previous version but 2.8 is 7 versions far from v2.15 and although we will try to support with misconfigurations, if there is a real bug unresolved, we won't ship any fix for old versions.

@JorTurFer
OK, I understand what you mean. There is another situation for you, that is, our latest version of keda is found to be incompatible on the cluster of version 1.21, and panic will occur. The reason is that the v2 version of hpa cannot be detected.I don’t know if the community will pay attention to this issue or solve it.

This is also the reason why I switched to 2.8.0 according to the official recommendation

[guanqinglin]

I also encountered the same problem which caused the function not to work properly？

[JorTurFer]

Yes, I said, KEDA >= v2.9 requires k8s >= 1.23. This is because k8s introduced a breaking change when they released HPA v2 and removed HPA v1beta1. v2.8 uses v1beta1 and v2.9 uses v2. There isn't any fix or action to do about this, v2.8 uses v1beta1 and v2.9 uses v2.

[LY-today]

@JorTurFer
OK, thank you for your answer and wish you a happy life

[LY-today]

@JorTurFer Excuse me, when I only configure cron, will the replicas of the workload become minReplicaCount if it is not in the cron period?

[LY-today]

@JorTurFer Excuse me, when I only configure cron, will the replicas of the workload become minReplicaCount if it is not in the cron period?

@JorTurFer The test is like this. When I manually modify replicas during non-cron periods, it will eventually become minReplicaCount, which is not very friendly to the business. Based on version 1.8.2, is there any way to allow the business to still have the right to make decisions in this situation?

[JorTurFer]

Excuse me, when I only configure cron, will the replicas of the workload become minReplicaCount if it is not in the cron period?

Yes, outside the cron period and assuming the cron as the only scaler, it'll return 0 so it'll scale to minReplicaCount

[enaguo]

@JorTurFer请问一下，当我只配置cron的时候，如果不在cron周期内，工作负载的replicas会变成minReplicaCount吗？

@JorTurFer测试是这样的，当我在非cron期间手动修改replicas时，最后会变成minReplicaCount，对业务不太友好，基于1.8.2版本，有没有办法让业务在这种情况下依然拥有决策权？

I also encountered the same problem，How to solve it