Kafka scaler with SASL plaintext auth mode fails with "error parsing kafka metadata: err SASL mode plaintext"

[abmajum]

Report

Initially we were using KEDA App version 2.7.1 . After upgrading to KEDA App version 2.10.0 , we saw the error "error parsing kafka metadata: err SASL mode plaintext" in the scaled object description.

Conclusion: As per keda doc SASL mode can be defined in both in scaledobject or triggerauthentication.
However sasl defined in triggerauthentication refered from secret is however not getting picked up keda-operartor 2.10.0 image, hence there is chance this could be a bug.

The manifest we were using is as below:

---
apiVersion: v1
data:
  password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  tls: ZW5hYmxlCg==
  username: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  sasl: cGxhaW50ZXh0Cg==
kind: Secret
metadata:
  labels:
    app: example-app
  name: keda-kafka-gen20x-secrets
  namespace: cpe-logging
type: Opaque
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  labels:
    app: example-app
  name: keda-trigger-auth-kafka-gen20x-credential
  namespace: cpe-logging
spec:
  secretTargetRef:
  - key: tls
    name: keda-kafka-gen20x-secrets
    parameter: tls
  - key: sasl
    name: keda-kafka-gen20x-secrets
    parameter: sasl
  - key: username
    name: keda-kafka-gen20x-secrets
    parameter: username
  - key: password
    name: keda-kafka-gen20x-secrets
    parameter: password
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  labels:
    app: example-app
  name: slipstream-kstreams-gen20x-scaled
  namespace: cpe-logging
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          policies:
          - periodSeconds: 60
            type: Percent
            value: 20
          stabilizationWindowSeconds: 300
        scaleUp:
          policies:
          - periodSeconds: 300
            type: Percent
            value: 100
          stabilizationWindowSeconds: 300
  cooldownPeriod: 300
  maxReplicaCount: 4
  minReplicaCount: 1
  pollingInterval: 60
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-app
  triggers:
  - authenticationRef:
      name: keda-trigger-auth-kafka-gen20x-credential
    metadata:
      bootstrapServers: a.b.c.d:9092
      consumerGroup: telelog
      lagThreshold: "1000"
      topic: streaming-telelog-gen20x-raw
    type: kafka
  - authenticationRef:
      name: keda-trigger-auth-kafka-gen20x-credential
    metadata:
      bootstrapServers: a.b.c.d:9092
      consumerGroup: telelog
      lagThreshold: "10000"
      topic: streaming-telelog-gen20x-formatted
    type: kafka
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: example-app
  name: example-app
  namespace: cpe-logging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: quay.io/brancz/prometheus-example-app:v0.3.0
        ports:
        - name: web
          containerPort: 8080

We have to explicitly define sasl in scaledobject like below for KEDA App version 2.10.0 to work.

---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  labels:
    app: example-app
  name: keda-trigger-auth-kafka-gen20x-credential
  namespace: cpe-logging
spec:
  secretTargetRef:
  - key: tls
    name: keda-kafka-gen20x-secrets
    parameter: tls
#  - key: sasl
#   name: keda-kafka-gen20x-secrets
#   parameter: sasl
  - key: username
    name: keda-kafka-gen20x-secrets
    parameter: username
  - key: password
    name: keda-kafka-gen20x-secrets
    parameter: password
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  labels:
    app: example-app
  name: slipstream-kstreams-gen20x-scaled
  namespace: cpe-logging
spec:
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          policies:
          - periodSeconds: 60
            type: Percent
            value: 20
          stabilizationWindowSeconds: 300
        scaleUp:
          policies:
          - periodSeconds: 300
            type: Percent
            value: 100
          stabilizationWindowSeconds: 300
  cooldownPeriod: 300
  maxReplicaCount: 4
  minReplicaCount: 1
  pollingInterval: 60
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-app
  triggers:
  - authenticationRef:
      name: keda-trigger-auth-kafka-gen20x-credential
    metadata:
      bootstrapServers: a.b.c.d:9092
      consumerGroup: telelog
      lagThreshold: "1000"
      topic: streaming-telelog-gen20x-raw
      sasl: plaintext                                                                    # Explicitly mentioning SASL here
    type: kafka
  - authenticationRef:
      name: keda-trigger-auth-kafka-gen20x-credential
    metadata:
      bootstrapServers: a.b.c.d:9092
      consumerGroup: telelog
      lagThreshold: "10000"
      topic: streaming-telelog-gen20x-formatted
      sasl: plaintext                                                                     # Explicitly mentioning SASL here
    type: kafka

Expected Behavior

Output of scaledobject when we were using KEDA App Version 2.7.1

$ helm ls -n keda
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
keda    keda            7               2023-06-15 11:20:10.8504795 +0530 IST   deployed        keda-2.7.2      2.7.1

$ k get scaledobjects -A
NAMESPACE     NAME                                SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS   AUTHENTICATION                              READY   ACTIVE   FALLBACK   AGE
cpe-logging   slipstream-kstreams-gen20x-scaled   apps/v1.Deployment   example-app       1     4     kafka      keda-trigger-auth-kafka-gen20x-credential   True    True     False      24s

Actual Behavior

After Upgrading to KEDA App Version 2.10.0 , the scaledobject status remained in False, Unknown state.

$ helm ls -n keda
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
keda    keda            8               2023-06-15 05:56:55.495356882 +0000 UTC deployed        keda-2.10.1     2.10.0 

$ k get scaledobjects -A
NAMESPACE     NAME                                SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   TRIGGERS   AUTHENTICATION                              READY   ACTIVE    FALLBACK   AGE
cpe-logging   slipstream-kstreams-gen20x-scaled   apps/v1.Deployment   example-app       1     4     kafka      keda-trigger-auth-kafka-gen20x-credential   False   Unknown   False      6m40s

Steps to Reproduce the Problem

1. Install KEDA Helm Chart 2.10.1(App Version 2.10.0)
2. Copy the manifest which is using sasl in TriggerAuthentication, tweak username, password as per your kafka environment
3. Apply the manifest.

Logs from KEDA operator

There were no such error logs in keda-operator, After describing the scaled object

$ k describe scaledobjects slipstream-kstreams-gen20x-scaled -n keda

  Original Replica Count:  1
  Scale Target GVKR:
    Group:            apps
    Kind:             Deployment
    Resource:         deployments
    Version:          v1
  Scale Target Kind:  apps/v1.Deployment
Events:
  Type     Reason                   Age                    From           Message
  ----     ------                   ----                   ----           -------
  Normal   KEDAScalersStarted       7m35s                  keda-operator  Started scalers watch
  Normal   ScaledObjectReady        7m19s (x2 over 7m35s)  keda-operator  ScaledObject is ready for scaling
  Warning  ScaledObjectCheckFailed  113s (x12 over 2m4s)   keda-operator  Failed to ensure HPA is correctly created for ScaledObject
  Warning  KEDAScalerFailed         103s (x13 over 2m4s)   keda-operator  error parsing kafka metadata: err SASL mode plaintext
 given 

KEDA Version

2.10.0

Kubernetes Version

1.25

Platform

Microsoft Azure

Scaler Details

Kafka

Anything else?

No response

[ianebot]

Hi Abhishek,

I experienced the same issue recently when I upgraded from 2.8.x to 2.10.x. In my case my scaledobject did not change and it worked fine at the beginning, but as soon as the keda pods restarted for any reason, it started to complain about the SASL and TLS fields as you posted. I had defined both SASL and TLS as part of the TriggerAuth and as secret.

I also think this may be a bug because my setup did not changed, sometimes worked and sometimes not. And as a workaround I removed the SASL and TLS definition from the TriggerAuth and secret, and define them as part of the kafka scaler and the issue was gone.

Hopefully this information is helpful.

Cheers.

[abmajum]

Hi @ianebot , the issue is persistent in Helm Chart Version 2.10.2 also, hopefully this bug gets fixed in upcoming versions. Have you tried deploying some other lower versions ?

[ianebot]

Hi @abhi-aws15,

I did not try to downgrade the version. But it is likely that 2.10.x is the affected one:
https://github.com/kedacore/keda/releases/tag/v2.10.0 -> #4322

You can try with 2.9.x as it may not have this issue.

[zroubalik]

@dttung2905 could you please take a look?

[dttung2905]

Sure let me take a look during this weekend once i come back from sick leave

[dttung2905]

Hi everyone,

Thanks for the detail write up on the issue. The error parsing kafka metadata: err SASL mode plaintext given actually comes from this line which is strange because plaintext should pass through in the first part of the if statement and not the else section

		if mode == KafkaSASLTypePlaintext || mode == KafkaSASLTypeSCRAMSHA256 || mode == KafkaSASLTypeSCRAMSHA512 || mode == KafkaSASLTypeOAuthbearer {
...
		} else {
			return fmt.Errorf("err SASL mode %s given", mode)
                }
...

I suspected there might be an extra space/character like \n in SASL mode as we left out strings.Trimspace() for SASL in 2.10 (link). The reason I think so is because of the new line in the log provided above 🤔 . Let me add the string transformation back plus a few more test cases for validation.
wdyt @zroubalik ?

[abmajum]

Hi everyone, currently the sasl and tls is only working if we define it in scaleobjects and not in triggerauthentication. This behaviour is observed in all the 2.10.x.

Earlier in 2.7.2 version , sasl and tls only works if we define it in triggerauthentication and not in scaledobjects.
If we can make this possible to use sasl and tls can be defined either in triggerauthentication or scaledobjects would be great as said it is said in the KEDA DOC . People might upgrade their KEDA Version and the kafka scaler might not work. Keeping in mind the migration part from 2.7 ---> 2.10 it would be great if it is available for both the scaleobjects and triggerauthentication.