Introduce Idle Replica Mode

[zroubalik]

Signed-off-by: Zbynek Roubalik zroubali@redhat.com

Introducing optional Idle Replica Mode, which allows scaling to a different (lower) number of replicas when triggers are not active through setting idleReplicaCount. Once triggers are active the target deployment is scaled to minReplicaCount which which can be arbitrary number greeter then idleReplicaCount and less then maxReplicaCount. Then the scaling is being handled by HPA as usual. Once the triggers are again not active, the target workload is again scaled to idleReplicaCount.

1. Example ScaledObject, which allows to have 0 replicas on target workload if there's nothing to do, once triggers are active the workload is scaled immediately to 5 and then scaled up to 10 by HPA as usual.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: example-so
spec:
  scaleTargetRef:
    name:  target 
  idleReplicaCount: 0            
  minReplicaCount: 5                     
  maxReplicaCount: 10

2. Example ScaledObject, which allows to have 1 replicas on target workload if there's nothing to do, once triggers are active the workload is scaled immediately to 10 and then scaled up to 20 by HPA as usual.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: example-so
spec:
  scaleTargetRef:
    name:  target 
  idleReplicaCount: 1             
  minReplicaCount: 10                     
  maxReplicaCount: 20

Checklist

• Commits are signed with Developer Certificate of Origin (DCO - learn more)
• Tests have been added
• [N/A] A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
• A PR is opened to update the documentation on (Documentation for Idle Replica Mode keda-docs#488) (if applicable)
• Changelog has been updated

Fixes #1838