unsupported HPA GVK: autoscaling/v2 #9145

marcio-pessoa · 2022-04-19T19:53:47Z

Checklist:

I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
I've included steps to reproduce the bug.
I've pasted the output of argocd version.

When I try to create an app using HPA apiVersion: autoscaling/v2beta2 the follow error is returned:

ComparisonError

unsupported HPA GVK: autoscaling/v2, Kind=HorizontalPodAutoscaler

SyncError

Failed sync attempt to fdc308aba08fbf3fc08b6d2870fc9acb70d9f09b: ComparisonError: unsupported HPA GVK: autoscaling/v2, Kind=HorizontalPodAutoscaler (retried 5 times).

To Reproduce

I used a typical HPA definition file:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: test
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: test
  minReplicas: 1
  maxReplicas: 2
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50

If I use apiVersion: autoscaling/v2beta2 Argo CD works perfectly. But this apiVersion is deprecated in Kubernetes v1.23+ and unavailable in v1.26+.

IMPORTANT: When I apply the above definition using kubectl works fine (I'm using Kubernetes v1.23.5).

Expected behavior

I would like to apply HPA definition using apiVersion: autoscaling/v2.

Version

$ kubectl exec -it -n argocd deployment/argocd-server -- argocd version
argocd: v2.3.3+07ac038
  BuildDate: 2022-03-30T00:06:18Z
  GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
  GitTreeState: clean
  GoVersion: go1.17.6
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified           
command terminated with exit code 1

The text was updated successfully, but these errors were encountered:

oswald0071 · 2022-04-26T06:44:33Z

Is there any progress or workaround?

marcio-pessoa · 2022-04-26T14:16:04Z

The current workaround is to apply the definition files using kubectl, but it's not the Argonic way. :-(

pjaak · 2022-04-28T00:30:25Z

Currently running into the same issue

sanglt · 2022-05-06T00:58:36Z

We running into the same issue

ledroide · 2022-05-06T05:14:28Z

Here is my workaround, using a kustomization.yaml :

patches:
  - path: hpa.argocd-fix.patch.yaml
    target:
      group: autoscaling
      version: v2
      kind: HorizontalPodAutoscaler

hpa.argocd-fix.patch.yaml :

- op: replace
  path: /apiVersion
  value: autoscaling/v2beta2

The v2beta2 is very similar to final v2, so it should work without an other patch - but if it happens however, just add your additional patches to the same file.
As soon as the bug in ArgoCD is fixed, it is easy to remove the patch.
My two cents.

sanglt · 2022-05-08T22:33:05Z

Here is my workaround, using a kustomization.yaml :
patches:
  - path: hpa.argocd-fix.patch.yaml
    target:
      group: autoscaling
      version: v2
      kind: HorizontalPodAutoscaler
hpa.argocd-fix.patch.yaml :
- op: replace
  path: /apiVersion
  value: autoscaling/v2beta2
The v2beta2 is very similar to final v2, so it should work without an other patch - but if it happens however, just add your additional patches to the same file. As soon as the bug in ArgoCD is fixed, it is easy to remove the patch. My two cents.

No v2beta2 is not similar to v2 - Which is very strange. The ContainerResource is not supported on v2beta2. For our use case, we need to use ContainerResource because few difference type of sidecar container will inject to the deployment.

EppO · 2022-05-09T17:38:35Z

in 1.23, using or patching autoscaling/v2beta2 doesn't work, because when argocd gets the resource kubernetes will use the highest API version available for autoscaling which is autoscaling/v2.

Here are the manifests using argocd app manifests

---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app: myapp
    app.kubernetes.io/instance: myapp-dev
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: myapp
    app.kubernetes.io/version: 1.1.3
    argocd.argoproj.io/instance: myapp-dev
    helm.sh/chart: myapp-1.1.3
    release: myapp-dev
  name: myapp
  namespace: myapp-dev
spec:
...

Here is what you get if you don't specify any API version (kubectl get hpa -o yaml)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"autoscaling/v2beta2","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"labels":{"app":"myapp","app.kubernetes.io/instance":"myapp-dev","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"myapp","app.kubernetes.io/version":"1.1.3","argocd.argoproj.io/instance":"myapp-dev","helm.sh/chart":"myapp-dev-1.1.3","release":"myapp-dev"},"name":"myapp","namespace":"myapp-dev"},"spec":{"maxReplicas":2,"metrics":[{"resource":{"name":"cpu","target":{"averageUtilization":80,"type":"Utilization"}},"type":"Resource"}],"minReplicas":1,"scaleTargetRef":{"apiVersion":"apps/v1","kind":"Deployment","name":"myapp"}}}
  creationTimestamp: "2022-01-27T14:14:36Z"
  labels:
    app: myapp
    app.kubernetes.io/instance: myapp-dev
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: myapp
    app.kubernetes.io/version: 1.1.3
    argocd.argoproj.io/instance: myapp-dev
    helm.sh/chart: myapp-dev-1.1.3
    release: myapp-dev
  name: myapp
  namespace: myapp-dev
  resourceVersion: "103570142"
  uid: 15a38d49-1143-4e6f-92ad-67857e71b022
spec:
...

and Argo CD doesn't like that:

CONDITION        MESSAGE                                                                                                                                                                 LAST TRANSITION
ComparisonError  unsupported HPA GVK: autoscaling/v2, Kind=HorizontalPodAutoscaler                                                                                                       2022-05-08 08:03:36 -0700 PDT
SyncError        Failed sync attempt to ed1c3dfad4fd44553428029c1fbcfe3ced975b19: ComparisonError: unsupported HPA GVK: autoscaling/v2, Kind=HorizontalPodAutoscaler (retried 5 times).  2022-05-09 09:50:49 -0700 PDT

EppO · 2022-05-13T19:02:15Z

I opened a MR in gitops-engine repository: argoproj/gitops-engine#411

joskuijpers · 2022-06-09T18:10:53Z

Great that a patch has been created and merged! Thank you for your work.

What is the timeline on the release of 2.4 to contain this change?

EppO · 2022-06-13T15:59:38Z

gitops-engine v0.7.0 was released with the fix. Argo CD v2.4.0 is using gitops-engine v0.7.0: https://github.com/argoproj/argo-cd/blob/v2.4.0/go.mod#L12

So hopefully, this should be fixed with the latest version of Argo CD.

EppO · 2022-06-13T17:34:32Z

I confirm autoscaling/v2 resources are synced correctly with Argo CD v2.4.0. This ticket can be closed

marcio-pessoa · 2022-06-28T16:21:08Z

Yes!
The issue was fixed.
Congrats @EppO !

srclark213 · 2022-10-19T15:20:37Z

Hey, I'm still seeing this issue on ArgoCD version 2.4.14 and kubernetes version 1.24. We have the latest argo charts (as of writing 5.6.0) deployed to our cluster and with

apiVersionOverrides:
    autoscaling: autoscaling/v2beta2

the HPAs are happy, but if I clear this out or set it to autoscaling/v2, I receive the unsupported HPA GVK: autoscaling/v2, Kind=HorizontalPodAutoscaler error on the 2 HPAs in ArgoCD.

output from argocd version:

argocd: v2.4.14+029be59
  BuildDate: 2022-10-05T17:15:37Z
  GitCommit: 029be590bfd5003d65ddabb4d4cb8a31bff29c18
  GitTreeState: clean
  GoVersion: go1.18.7
  Compiler: gc
  Platform: linux/amd64

output from kubectl version:

clientVersion:
  buildDate: "2022-09-21T13:19:24Z"
  compiler: gc
  gitCommit: b39bf148cd654599a52e867485c02c4f9d28b312
  gitTreeState: clean
  gitVersion: v1.24.6
  goVersion: go1.18.6
  major: "1"
  minor: "24"
  platform: windows/amd64
kustomizeVersion: v4.5.4
serverVersion:
  buildDate: "2022-09-21T21:46:51Z"
  compiler: gc
  gitCommit: b39bf148cd654599a52e867485c02c4f9d28b312
  gitTreeState: clean
  gitVersion: v1.24.6
  goVersion: go1.18.6
  major: "1"
  minor: "24"
  platform: linux/amd64

Are there any pieces to this puzzle that I'm missing here?

marcio-pessoa · 2022-10-19T15:55:26Z

Wow!
Could you please kindly share the definition files and Argo CD error message?

srclark213 · 2022-10-19T16:20:52Z

Sure. So we're using the argo-helm charts on 5.6.0 here (https://github.com/argoproj/argo-helm/tree/argo-cd-5.6.0), which result in this definition file for the repo-server hpa:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    app.kubernetes.io/component: repo-server
    app.kubernetes.io/instance: argo-cd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-repo-server-hpa
    app.kubernetes.io/part-of: argocd
    argocd.argoproj.io/instance: argo-cd
    helm.sh/chart: argo-cd-5.6.0
  name: argo-cd-argocd-repo-server-hpa
  namespace: argocd
spec:
  maxReplicas: 5
  metrics:
    - resource:
        name: memory
        target:
          averageUtilization: 50
          type: Utilization
      type: Resource
    - resource:
        name: cpu
        target:
          averageUtilization: 50
          type: Utilization
      type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: argo-cd-argocd-repo-server

And the error we get with this is

ComparisonError: unsupported HPA GVK: autoscaling/v2, Kind=HorizontalPodAutoscaler

beyondbill · 2022-10-21T22:30:14Z

Should we consider reopen the issue given 2.4.0 does not seem fix it based on several comments above?

jsirianni · 2022-10-23T19:21:26Z

I am also seeing this when upgrading hpa from v2beta to v2.

argocd version
argocd: v2.4.14+029be59
  BuildDate: 2022-10-05T17:37:30Z
  GitCommit: 029be590bfd5003d65ddabb4d4cb8a31bff29c18
  GitTreeState: clean
  GoVersion: go1.18.6
  Compiler: gc
  Platform: linux/amd64

mgarstecki · 2022-10-25T14:01:47Z

We used to have the same error on v2.3, but it works now on v2.4.15 and K8S 1.23:

{
    "Version": "v2.4.15+05acf7a",
    "BuildDate": "2022-10-17T20:32:39Z",
    "GitCommit": "05acf7a52e377eacfee29c68e3e5e79a172ea013",
    "GitTreeState": "clean",
    "GoVersion": "go1.18.7",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v4.4.1 2021-11-11T23:36:27Z",
    "HelmVersion": "v3.8.1+g5cb9af4",
    "KubectlVersion": "v0.23.1",
    "JsonnetVersion": "v0.18.0"
}

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:38:05Z", GoVersion:"go1.17.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.10-eks-15b7512", GitCommit:"cd6399691d9b1fed9ec20c9c5e82f5993c3f42cb", GitTreeState:"clean", BuildDate:"2022-08-31T19:17:01Z", GoVersion:"go1.17.13", Compiler:"gc", Platform:"linux/amd64"}

EppO · 2022-10-27T13:45:18Z

Using 2.4.14, just tested with a autoscaling/v2 manifest (checked using argocd app manifests command as kubectl get hpa will give the latest available API version)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  labels:
    argocd.argoproj.io/instance: myapp
  name: myapp-hpa
  namespace: default
spec:
  maxReplicas: 2
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 80
        type: Utilization
    type: Resource
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp-deploy

argocd can sync it correctly:

...
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Synced to master (8f27b1c)
Health Status:      Healthy
...
autoscaling                 HorizontalPodAutoscaler  default  myapp-hpa                     Synced   Healthy        horizontalpodautoscaler.autoscaling/myapp-hpa unchanged
...

my setup:

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.24.3
Kustomize Version: v4.5.4
Server Version: v1.24.3
$ argocd version
argocd: v2.4.11+3d9e9f2
  BuildDate: 2022-08-22T09:35:38Z
  GitCommit: 3d9e9f2f95b7801b90377ecfc4073e5f0f07205b
  GitTreeState: clean
  GoVersion: go1.18.5
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.4.14+029be59
  BuildDate: 2022-10-05T17:15:37Z
  GitCommit: 029be590bfd5003d65ddabb4d4cb8a31bff29c18
  GitTreeState: clean
  GoVersion: go1.18.7
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.4.1 2021-11-11T23:36:27Z
  Helm Version: v3.8.1+g5cb9af4
  Kubectl Version: v0.23.1
  Jsonnet Version: v0.18.0

srclark213 · 2022-10-27T18:23:46Z

I haven't been able to recreate on a fresh cluster, I'm only seeing this issue on a server that we've recently upgraded both k8s and argocd. We've been slowly trying to narrow down our dependencies to figure out if one of these is causing this. Is there any way to check the version of the gitops engine we're running?

EppO · 2022-11-09T19:10:22Z

Use argocd app manifests command to see how the manifests look like when ArgoCD processes your Helm Chart, there might be something wrong with your Helm Chart that produces incorrect HPA CRDs. Any ArgoCD version >= 2.4.0 is using a GitOps Engine library that supports autoscaling/v2

myrondev · 2022-11-18T20:54:03Z

I have this issue with ArgoCD 2.2.2 and autoscaling/v1. Hpa works fine:

HPA object is running well.
It scales when the load is happening.

So only one issue here, in Argo in events I see that it's no healthy with the message: unsupported HPA GVK: autoscaling/v2, Kind=HorizontalPodAutoscaler. But I am using autoscaling/v1, strange...

Kubectl:
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.23.4

fleeco · 2022-11-30T14:50:12Z

I'm having the same issue here. Argo helm chart 3.35.4 app version v2.2.5

fleeco · 2022-12-02T18:43:18Z

I was dumb. The problem was HPA couldn't access metrics - check your security groups from Cluster to the node group <3

brsolomon-deloitte · 2023-01-07T19:08:11Z

Was seeing this issue with ArgoCD 2.1.x with autoscaling/v2, and upgrading to 2.5.5 via https://raw.githubusercontent.com/argoproj/argo-cd/v2.5.5/manifests/install.yaml resolved it immediately. (Kubernetes: EKS 1.24)

ashish1099 · 2023-03-23T06:13:59Z

seeing the issue on latest argocd 5.22.1 and on k8s 1.23.16

tdogsizle · 2023-04-11T00:15:14Z

Seeing this issue as well
argocd helm chart: 5.27.5
argocd app version: v2.6.7
k8s: 1.25

sergeyignatov · 2023-04-14T01:57:39Z

a workaround we use for disabling health check for the HPA

resource.customizations: |-
      autoscaling/HorizontalPodAutoscaler:
        health.lua: |
          hs = {}
          hs.status = "Healthy"
          hs.message = "Ignoring HPA Health Check"
          return hs

EppO · 2023-04-24T18:58:43Z

Are we still talking about a failing sync here? I mean ComparisonError/SyncError.
If you are experiencing OutOfSync HPAs, it's a different problem with a workaround described in the diff FAQ:

For Horizontal Pod Autoscaling (HPA) objects, the HPA controller is known to reorder spec.metrics in a specific order. See kubernetes issue #74099. To work around this, you can order spec.metrics in Git in the same order that the controller prefers.

raosharadhi · 2023-07-13T10:49:16Z

Hi Team, Anyone found the solution. I am also facing same issues. It was working fine till yesterday. Now when I sync today it says "server could not find the requested resource" and failing the sync. any help would be appreciated

AidarGatin · 2023-08-30T20:20:41Z

Hi Guys, If you can't update ArgoCD this might help you :

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  labels:
    argocd.argoproj.io/instance: myapp
  name: myapp-hpa
  namespace: default
...

v1 is the default version of HPA
But you actual HPA version will be autoscaling/v1, guess k8s force it to be v2.

Works on setup:
K8s Rev: v1.27.4-eks
Argo: v2.3.3

NiklasRosenstein · 2024-01-10T00:10:12Z

I'm also facing this issue trying to deploy the GitLab Helm chart with ArgoCD. I'm using K3s v1.27 and there's no autoscaling/v1 or autoscaling/v1beta1 that I could use instead. 👀 It basically renders ArgoCD unable to sync any changes after initial deployment.

gxpd-jjh · 2024-02-19T19:46:30Z

For anyone facing this issue:

Make sure you bump a new version of your chart using the new HPA - after you upgrade to Argo > 2.4
In your HPA spec, make sure that you are using the new metrics: syntax and your metrics has content.
(My mistake was I had an empty metrics: after some if/else)

marcio-pessoa added the bug Something isn't working label Apr 19, 2022

marcio-pessoa closed this as completed Jun 28, 2022

marcio-pessoa reopened this Oct 23, 2022

johnzeng2010 mentioned this issue Apr 13, 2023

Need backward support for autoscaling/v2beta2 nginx/kubernetes-ingress#3769

Closed

brianehlert mentioned this issue Apr 19, 2023

Swap cpu and memory in HPA template nginx/kubernetes-ingress#3773

Merged

6 tasks

Henrrypg mentioned this issue Jul 6, 2023

feat: add support to palm version eduNEXT/drydock#42

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unsupported HPA GVK: autoscaling/v2 #9145

unsupported HPA GVK: autoscaling/v2 #9145

marcio-pessoa commented Apr 19, 2022

oswald0071 commented Apr 26, 2022

marcio-pessoa commented Apr 26, 2022

pjaak commented Apr 28, 2022

sanglt commented May 6, 2022

ledroide commented May 6, 2022

sanglt commented May 8, 2022

EppO commented May 9, 2022 •

edited

Loading

EppO commented May 13, 2022

joskuijpers commented Jun 9, 2022

EppO commented Jun 13, 2022

EppO commented Jun 13, 2022

marcio-pessoa commented Jun 28, 2022

srclark213 commented Oct 19, 2022

marcio-pessoa commented Oct 19, 2022

srclark213 commented Oct 19, 2022

beyondbill commented Oct 21, 2022

jsirianni commented Oct 23, 2022 •

edited

Loading

mgarstecki commented Oct 25, 2022

EppO commented Oct 27, 2022 •

edited

Loading

srclark213 commented Oct 27, 2022

EppO commented Nov 9, 2022 •

edited

Loading

myrondev commented Nov 18, 2022 •

edited

Loading

fleeco commented Nov 30, 2022

fleeco commented Dec 2, 2022

brsolomon-deloitte commented Jan 7, 2023

ashish1099 commented Mar 23, 2023

tdogsizle commented Apr 11, 2023

sergeyignatov commented Apr 14, 2023

EppO commented Apr 24, 2023

raosharadhi commented Jul 13, 2023

AidarGatin commented Aug 30, 2023 •

edited

Loading

NiklasRosenstein commented Jan 10, 2024

gxpd-jjh commented Feb 19, 2024 •

edited

Loading

unsupported HPA GVK: autoscaling/v2 #9145

unsupported HPA GVK: autoscaling/v2 #9145

Comments

marcio-pessoa commented Apr 19, 2022

oswald0071 commented Apr 26, 2022

marcio-pessoa commented Apr 26, 2022

pjaak commented Apr 28, 2022

sanglt commented May 6, 2022

ledroide commented May 6, 2022

sanglt commented May 8, 2022

EppO commented May 9, 2022 • edited Loading

EppO commented May 13, 2022

joskuijpers commented Jun 9, 2022

EppO commented Jun 13, 2022

EppO commented Jun 13, 2022

marcio-pessoa commented Jun 28, 2022

srclark213 commented Oct 19, 2022

marcio-pessoa commented Oct 19, 2022

srclark213 commented Oct 19, 2022

beyondbill commented Oct 21, 2022

jsirianni commented Oct 23, 2022 • edited Loading

mgarstecki commented Oct 25, 2022

EppO commented Oct 27, 2022 • edited Loading

srclark213 commented Oct 27, 2022

EppO commented Nov 9, 2022 • edited Loading

myrondev commented Nov 18, 2022 • edited Loading

fleeco commented Nov 30, 2022

fleeco commented Dec 2, 2022

brsolomon-deloitte commented Jan 7, 2023

ashish1099 commented Mar 23, 2023

tdogsizle commented Apr 11, 2023

sergeyignatov commented Apr 14, 2023

EppO commented Apr 24, 2023

raosharadhi commented Jul 13, 2023

AidarGatin commented Aug 30, 2023 • edited Loading

NiklasRosenstein commented Jan 10, 2024

gxpd-jjh commented Feb 19, 2024 • edited Loading

EppO commented May 9, 2022 •

edited

Loading

jsirianni commented Oct 23, 2022 •

edited

Loading

EppO commented Oct 27, 2022 •

edited

Loading

EppO commented Nov 9, 2022 •

edited

Loading

myrondev commented Nov 18, 2022 •

edited

Loading

AidarGatin commented Aug 30, 2023 •

edited

Loading

gxpd-jjh commented Feb 19, 2024 •

edited

Loading