unable to install KEDA on EKS with calico in full VXLAN/Overlay mode

[venkatamutyala]

Report

I'm trying to run keda within EKS while using calico in full VXLAN mode.

I know with calico + EKS the hostNetwork needs to be enabled so here is my values.yaml (see below). It seems like the admission webhook is trying to use port 8080 still:

2023-07-05T20:45:13Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}2023-07-05T20:45:13Z	ERROR	controller-runtime.metrics	metrics server failed to listen. You may want to disable the metrics server or use another port if it is due to conflicts	{"error": "error listening on :8080: listen tcp :8080: bind: address already in use"}sigs.k8s.io/controller-runtime/pkg/metrics.NewListener	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/metrics/listener.go:48sigs.k8s.io/controller-runtime/pkg/manager.New	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/manager/manager.go:455main.main	/workspace/cmd/webhooks/main.go:81runtime.main	/usr/local/go/src/runtime/proc.go:2502023-07-05T20:45:13Z	ERROR	setup	unable to start admission webhooks	{"error": "error listening on :8080: listen tcp :8080: bind: address already in use"}main.main	/workspace/cmd/webhooks/main.go:97runtime.main	/usr/local/go/src/runtime/proc.go:250

values.yaml:

      resources:
        operator:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 1
            memory: 1000Mi
      prometheus:
        metricServer:
          enabled: true
      prometheus:
          webhooks:
              port: 45055
              serviceMonitor:
                  targetPort: 45058
          operator:
              enabled: true
              port: 45056
          metricServer:
              port: 45057
      service:
          portHttps:  "45052"
          portHttpsTarget: "45053"
      metricsServer:
          useHostNetwork: "true"
      webhooks:
          enabled: true
          port:  "45050"
          healthProbePort: "45051"
          useHostNetwork: "true"

Expected Behavior

Given I specified custom ports (where possible) I was expecting it to just work, or alternatively throw an error using one of the custom port numbers I provided in my values.yaml

Actual Behavior

Throwing an error on port 8080 even though i don't see port 8080 defined in the spec:

spec:
  automountServiceAccountToken: true
  containers:
    - args:
        - '--zap-log-level=info'
        - '--zap-encoder=console'
        - '--zap-time-encoding=rfc3339'
        - '--cert-dir=/certs'
        - '--health-probe-bind-address=:45051'
        - '--port=45050'
      command:
        - /keda-admission-webhooks
      env:
        - name: WATCH_NAMESPACE
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
      image: 'ghcr.io/kedacore/keda-admission-webhooks:2.11.1'
      imagePullPolicy: Always
      livenessProbe:
        failureThreshold: 3
        httpGet:
          path: /healthz
          port: 45051
          scheme: HTTP
        initialDelaySeconds: 25
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      name: keda-admission-webhooks
      ports:
        - containerPort: 45050
          hostPort: 45050
          name: http
          protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          path: /readyz
          port: 45051
          scheme: HTTP
        initialDelaySeconds: 20
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 1
      resources:
        limits:
          cpu: 50m
          memory: 100Mi
        requests:
          cpu: 10m
          memory: 10Mi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        readOnlyRootFilesystem: true
        seccompProfile:
          type: RuntimeDefault
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /certs
          name: certificates
          readOnly: true
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          name: kube-api-access-vxskh
          readOnly: true

Steps to Reproduce the Problem

1. deploy EKS
2. deploy calico in full overlay/VXLAN mode
3. install keda uising helm chart 2.11.1 with values.yaml:

`values.yaml`:

  resources:
    operator:
      requests:
        cpu: 100m
        memory: 100Mi
      limits:
        cpu: 1
        memory: 1000Mi
  prometheus:
    metricServer:
      enabled: true
  prometheus:
      webhooks:
          port: 45055
          serviceMonitor:
              targetPort: 45058
      operator:
          enabled: true
          port: 45056
      metricServer:
          port: 45057
  service:
      portHttps:  "45052"
      portHttpsTarget: "45053"
  metricsServer:
      useHostNetwork: "true"
  webhooks:
      enabled: true
      port:  "45050"
      healthProbePort: "45051"
      useHostNetwork: "true"

Logs from KEDA operator

example

KEDA Version

2.11.1

Kubernetes Version

1.26

Platform

Amazon Web Services

Scaler Details

No response

Anything else?

No response

[JorTurFer]

Hi,
I have checked the chart and it looks like the Prometheus server port (default 8080) is only overrided if you explicitly set enabled in the section:
https://github.com/kedacore/charts/blob/main/keda/templates/webhooks/deployment.yaml#L80-L82

I guess that as it has a default value in values.yaml, we can just set it always to prevent these cases.
Could you try setting prometheus.webhooks.enabled: true?

[venkatamutyala]

This worked. Thank you!