PodDisruptionBudget and HorizontalPodAutoscaler use unavailable apis (fix Kubernetes 1.25+)

[fbertos]

Checks

• I have checked for existing issues.
• This report is about the User-Community Airflow Helm Chart.

Motivation

Hi,

We would like to deploy Airflow v2.3.3 with Chart version 8.6.1 on Kubernetes 1.23.12 with the following worker configuration:

  podDisruptionBudget:
    enabled: true
    minAvailable: "1"

  autoscaling:
    enabled: true
    maxReplicas: 16
    metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

However it is not working correctly and we receive this warning:

W1212 17:03:04.202853    6021 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1212 17:03:08.336673    6021 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler

Implementation

To change the templates of the worker:

• worker-hpa.yaml
• worker-pdb.yaml

To use the API version policy/v1 and autoscaling/v2 for these specs.

Are you willing & able to help?

• I am able to submit a PR!
• I can help test the feature!

[thesuperzapper]

@fbertos I think we can add a new template file for each of these resources, so people are able to set the apiVersion with a helm value depending on what Kubernetes version they are using.

• PodDisruptionBudget
  • policy/v1beta1
  • policy/v1
• HorizontalPodAutoscaler
  • autoscaling/v2beta2
  • autoscaling/v2

We already have a very similar thing for Ingress, and the ingress.apiVersion value:

• https://github.com/airflow-helm/charts/blob/main/charts/airflow/templates/webserver/webserver-ingress-v1beta1.yaml
• https://github.com/airflow-helm/charts/blob/main/charts/airflow/templates/webserver/webserver-ingress.yaml

Obviously, if there is no actual difference between the apiVersions (need to confirm), this change will be very simple.
Also, our default helm values should set the apiVersion which supports the current most common version of Kubernetes in use.

[fbertos]

Hi @thesuperzapper, great, that would be perfect. Thanks!

[karakanb]

@thesuperzapper hey, I believe I have fixed this issue with #685, do you think we can get v8.7.0 released only with this fix?

I'd really like to avoid forking or overriding the templates in some other ways to get this one working on my infra, it'd be helpful for me if we could get it going with as low effort from your side as possible.

[thesuperzapper]

@karakanb I agree that we should get this fix out ASAP, I have done a review of your PR, but it needs some small changes, see: #685 (review)

[hakuno]

+1 Waiting for that.