5.92: connector error in console.cleanForK8s

[jason-da-redpanda]

What happened?

Logs show :
{"level":"error","ts":"2024-09-03T21:55:18.770Z","msg":"Reconciler error","controller":"helmrelease","controllerGroup":"helm.toolkit.fluxcd.io","controllerKind":"HelmRelease","HelmRelease":{"name":"redpanda","namespace":"buggy"},"namespace":"buggy","name":"redpanda","reconcileID":"3cb93fba-fc84-4f8e-afbf-be5866d7c12a","error":"template: redpanda/templates/tests/test-connector-via-console.yaml:150:24: executing \"redpanda/templates/tests/test-connector-via-console.yaml\" at <include \"console.fullname\" $consoleValues>: error calling include: template: redpanda/charts/console/templates/_helpers.tpl:16:10: executing \"console.fullname\" at <include \"console.Fullname\" (dict \"a\" (list .))>: error calling include: template: redpanda/charts/console/templates/_helpers.go.tpl:22:30: executing \"console.Fullname\" at <include \"console.cleanForK8s\" (dict \"a\" (list $values.fullnameOverride))>: error calling include: template: redpanda/charts/console/templates/_helpers.go.tpl:78:48: executing \"console.cleanForK8s\" at <$s>: invalid value; expected string"}

What did you expect to happen?

It to not fail

How can we reproduce it (as minimally and precisely as possible)?. Please include values file.

repro

follow install guide for helm/operator (on kind for example see docs)

use this for the redpanda_cluster.yaml

Main points for this repro/bug being
chartVersion: 5.9.2
+

   connectors:
      enabled: true
      logging:
        level: info

apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  chartRef:
    chartVersion: 5.9.2
  clusterSpec:
    connectors:
      enabled: true
      logging:
        level: info
    external:
      domain: customredpandadomain.local
    auth:
      sasl:
        enabled: true
        users:
          - name: superuser
            password: secretpassword
            mechanism: SCRAM-SHA-512
    storage:
      persistentVolume:
        enabled: true
        storageClass: csi-driver-lvm-striped-xfs

- helm values

helm get values redpanda -n buggy --all
COMPUTED VALUES:
affinity: {}
auditLogging:
  clientMaxBufferSize: 16777216
  enabled: false
  enabledEventTypes: null
  excludedPrincipals: null
  excludedTopics: null
  listener: internal
  partitions: 12
  queueDrainIntervalMs: 500
  queueMaxBufferSizePerShard: 1048576
  replicationFactor: null
auth:
  sasl:
    enabled: true
    mechanism: SCRAM-SHA-512
    secretRef: redpanda-users
    users:
    - mechanism: SCRAM-SHA-512
      name: superuser
      password: secretpassword
clusterDomain: cluster.local
commonLabels: {}
config:
  cluster:
    default_topic_replications: 3
  node:
    crash_loop_limit: 5
  pandaproxy_client: {}
  rpk: {}
  schema_registry_client: {}
  tunable:
    compacted_log_segment_size: 67108864
    group_topic_partitions: 16
    kafka_batch_max_bytes: 1048576
    kafka_connection_rate_limit: 1000
    log_segment_size: 134217728
    log_segment_size_max: 268435456
    log_segment_size_min: 16777216
    max_compacted_log_segment_size: 536870912
    topic_partitions_per_shard: 1000
connectors:
  deployment:
    create: false
  enabled: false
  test:
    create: false
console:
  affinity: {}
  annotations: {}
  automountServiceAccountToken: true
  autoscaling:
    enabled: false
    maxReplicas: 100
    minReplicas: 1
    targetCPUUtilizationPercentage: 80
  commonLabels: {}
  config: {}
  configmap:
    create: false
  console:
    config: {}
  deployment:
    create: false
  enabled: true
  enterprise:
    licenseSecretRef:
      key: ""
      name: ""
  extraContainers: []
  extraEnv: []
  extraEnvFrom: []
  extraVolumeMounts: []
  extraVolumes: []
  fullnameOverride: ""
  global: {}
  image:
    pullPolicy: IfNotPresent
    registry: docker.redpanda.com
    repository: redpandadata/console
    tag: ""
  imagePullSecrets: []
  ingress:
    annotations: {}
    className: null
    enabled: false
    hosts:
    - host: chart-example.local
      paths:
      - path: /
        pathType: ImplementationSpecific
    tls: []
  initContainers:
    extraInitContainers: ""
  livenessProbe:
    failureThreshold: 3
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
  nameOverride: ""
  nodeSelector: {}
  podAnnotations: {}
  podLabels: {}
  podSecurityContext:
    fsGroup: 99
    runAsUser: 99
  priorityClassName: ""
  readinessProbe:
    failureThreshold: 3
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
  replicaCount: 1
  resources: {}
  secret:
    create: false
    enterprise: {}
    kafka: {}
    login:
      github: {}
      google: {}
      jwtSecret: ""
      oidc: {}
      okta: {}
    redpanda:
      adminApi: {}
  secretMounts: []
  securityContext:
    runAsNonRoot: true
  service:
    annotations: {}
    port: 8080
    targetPort: null
    type: ClusterIP
  serviceAccount:
    annotations: {}
    automountServiceAccountToken: true
    create: true
    name: ""
  strategy: {}
  tests:
    enabled: true
  tolerations: []
  topologySpreadConstraints: []
enterprise:
  license: ""
  licenseSecretRef: {}
external:
  domain: customredpandadomain.local
  enabled: true
  service:
    enabled: true
  type: NodePort
fullnameOverride: ""
image:
  pullPolicy: IfNotPresent
  repository: docker.redpanda.com/redpandadata/redpanda
  tag: ""
imagePullSecrets: []
license_key: ""
license_secret_ref: {}
listeners:
  admin:
    external:
      default:
        advertisedPorts:
        - 31644
        port: 9645
        tls:
          cert: external
    port: 9644
    tls:
      cert: default
      requireClientAuth: false
  http:
    authenticationMethod: null
    enabled: true
    external:
      default:
        advertisedPorts:
        - 30082
        authenticationMethod: null
        port: 8083
        tls:
          cert: external
          requireClientAuth: false
    kafkaEndpoint: default
    port: 8082
    tls:
      cert: default
      requireClientAuth: false
  kafka:
    authenticationMethod: null
    external:
      default:
        advertisedPorts:
        - 31092
        authenticationMethod: null
        port: 9094
        tls:
          cert: external
    port: 9093
    tls:
      cert: default
      requireClientAuth: false
  rpc:
    port: 33145
    tls:
      cert: default
      requireClientAuth: false
  schemaRegistry:
    authenticationMethod: null
    enabled: true
    external:
      default:
        advertisedPorts:
        - 30081
        authenticationMethod: null
        port: 8084
        tls:
          cert: external
          requireClientAuth: false
    kafkaEndpoint: default
    port: 8081
    tls:
      cert: default
      requireClientAuth: false
logging:
  logLevel: info
  usageStats:
    enabled: true
monitoring:
  enabled: false
  labels: {}
  scrapeInterval: 30s
nameOverride: ""
nodeSelector: {}
post_install_job:
  affinity: {}
  enabled: true
  podTemplate:
    annotations: {}
    labels: {}
    spec:
      containers:
      - env: []
        name: post-install
        securityContext: {}
      securityContext: {}
post_upgrade_job:
  affinity: {}
  enabled: true
  podTemplate:
    annotations: {}
    labels: {}
    spec:
      containers:
      - env: []
        name: post-upgrade
        securityContext: {}
      securityContext: {}
rackAwareness:
  enabled: false
  nodeAnnotation: topology.kubernetes.io/zone
rbac:
  annotations: {}
  enabled: false
resources:
  cpu:
    cores: 1
  memory:
    container:
      max: 2.5Gi
serviceAccount:
  annotations: {}
  create: false
  name: ""
statefulset:
  additionalRedpandaCmdFlags: []
  additionalSelectorLabels: {}
  annotations: {}
  budget:
    maxUnavailable: 1
  extraVolumeMounts: ""
  extraVolumes: ""
  initContainerImage:
    repository: busybox
    tag: latest
  initContainers:
    configurator:
      extraVolumeMounts: ""
      resources: {}
    extraInitContainers: ""
    fsValidator:
      enabled: false
      expectedFS: xfs
      extraVolumeMounts: ""
      resources: {}
    setDataDirOwnership:
      enabled: false
      extraVolumeMounts: ""
      resources: {}
    setTieredStorageCacheDirOwnership:
      extraVolumeMounts: ""
      resources: {}
    tuning:
      extraVolumeMounts: ""
      resources: {}
  livenessProbe:
    failureThreshold: 3
    initialDelaySeconds: 10
    periodSeconds: 10
  nodeSelector: {}
  podAffinity: {}
  podAntiAffinity:
    custom: {}
    topologyKey: kubernetes.io/hostname
    type: hard
    weight: 100
  podTemplate:
    annotations: {}
    labels: {}
    spec:
      containers:
      - env: []
        name: redpanda
        securityContext: {}
      securityContext: {}
  priorityClassName: ""
  readinessProbe:
    failureThreshold: 3
    initialDelaySeconds: 1
    periodSeconds: 10
    successThreshold: 1
  replicas: 3
  securityContext:
    fsGroup: 101
    fsGroupChangePolicy: OnRootMismatch
    runAsUser: 101
  sideCars:
    configWatcher:
      enabled: true
      extraVolumeMounts: ""
      resources: {}
      securityContext: {}
    controllers:
      createRBAC: true
      enabled: false
      healthProbeAddress: :8085
      image:
        repository: docker.redpanda.com/redpandadata/redpanda-operator
        tag: v2.1.10-23.2.18
      metricsAddress: :9082
      resources: {}
      run:
      - all
      securityContext: {}
  startupProbe:
    failureThreshold: 120
    initialDelaySeconds: 1
    periodSeconds: 10
  terminationGracePeriodSeconds: 90
  tolerations: []
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
  updateStrategy:
    type: RollingUpdate
storage:
  hostPath: ""
  persistentVolume:
    annotations: {}
    enabled: true
    labels: {}
    nameOverwrite: ""
    size: 20Gi
    storageClass: csi-driver-lvm-striped-xfs
  tiered:
    config:
      cloud_storage_access_key: ""
      cloud_storage_api_endpoint: ""
      cloud_storage_azure_container: null
      cloud_storage_azure_managed_identity_id: null
      cloud_storage_azure_shared_key: null
      cloud_storage_azure_storage_account: null
      cloud_storage_bucket: ""
      cloud_storage_cache_size: 5368709120
      cloud_storage_credentials_source: config_file
      cloud_storage_enable_remote_read: true
      cloud_storage_enable_remote_write: true
      cloud_storage_enabled: false
      cloud_storage_region: ""
      cloud_storage_secret_key: ""
    credentialsSecretRef:
      accessKey:
        configurationKey: cloud_storage_access_key
      secretKey:
        configurationKey: cloud_storage_secret_key
    hostPath: ""
    mountType: emptyDir
    persistentVolume:
      annotations: {}
      labels: {}
      storageClass: ""
tests:
  enabled: true
tls:
  certs:
    default:
      caEnabled: true
    external:
      caEnabled: true
  enabled: true
tolerations: []
tuning:
  tune_aio_events: true

Anything else we need to know?

No response

Which are the affected charts?

No response

Chart Version(s)

helm -n buggy list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION redpanda buggy 2 2024-09-03 21:48:22.351235494 +0000 UTC deployed redpanda-5.9.2 v24.2.3 redpanda-controller buggy 1 2024-08-27 06:15:31.038271 +0100 BST deployed operator-0.4.28 v2.2.0-24.2.2

Cloud provider

..

JIRA Link: K8S-339

[chrisseto]

Setting tests.enabled=false should work around this issue for now. The problem stems from helm tests as indicated in the error message: "error":"template: redpanda/templates/tests/test-connector-via-console.yaml:150:24:

For a CRD:

apiVersion: cluster.redpanda.com/v1alpha1
kind: Redpanda
metadata:
  name: redpanda
spec:
  clusterSpec:
    tests:
      enabled: false

That being said, it's entirely possible that we discover other issues after disable the test step 😅

[jason-da-redpanda]

for me(support) disabling tests:
got round the error.... and it created connectors

but for customer... it got round error ...but did not create the connectors..

and when using with "helm template" it wont generate the deployment...

Just tried the tests disable workaround, while it did fix the error, it seems to not deploy the connectors deployment at all..
Tried the same values file with "helm template" on version 5.9.1 and 5.9.2, 5.9.2 just won't generate the deployment...

[Youpiiiii]

Same way with 5.9.4.
I downgraded to 5.8.15

[OOub]

Same problem. I downgraded to 5.9.1 for it to work

[RafalKorepta]

It should be fixed in the latest Redpanda chart release v5.9.5

https://github.com/redpanda-data/helm-charts/releases/tag/redpanda-5.9.5