Query: Secret created using helm and its data persists during helm upgrade when patched by kubectl command. Is it a valid behavior of helm?

[rohitsharma382]

1. Create sample helm chart having one secret template like below:

$ cat nginx/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: data-empty-secret
  labels:
    app.kubernetes.io/name: nginx
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
  annotations:
    test.com/product-name: "Test"
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
  name: data-conditional-secret
  labels:
    app.kubernetes.io/name: nginx
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
  annotations:
    test.com/product-name: "Test"
type: Opaque
{{- $fileName := .Values.quest.testfile }}
{{- $file := .Files.Get $fileName }}
{{- if $file }}
data:
 {{ .Values.quest.testfile }}: {{ .Files.Get .Values.quest.testfile | b64enc }}
{{- end }}

2. Run helm install command to deploy secret after keeping testfile in helm directory.

$ helm install test-nginx nginx/

After helm install observe secret data in data-empty-secret is not present.

$ kubectl get secret data-empty-secret -o yaml
apiVersion: v1
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: test-nginx
    meta.helm.sh/release-namespace: test-system
    test.com/product-name: Test
  creationTimestamp: "2024-09-01T08:55:00Z"
  labels:
    app.kubernetes.io/instance: test-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
  name: data-empty-secret
  namespace: test-system
  resourceVersion: "111908034"
  uid: a8e3ff97-8644-4400-97fb-d0fd331f0f66
type: Opaque

After helm install observe secret data in data-conditional-secret.

$ kubectl get secret data-conditional-secret -o yaml
apiVersion: v1
data:
  testfile.txt: ZHVtbXlzZWNyZXRkYXRhCg==
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: test-nginx
    meta.helm.sh/release-namespace: test-system
    test.com/product-name: Test
  creationTimestamp: "2024-09-01T08:55:00Z"
  labels:
    app.kubernetes.io/instance: test-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
  name: data-conditional-secret
  namespace: test-system
  resourceVersion: "111908037"
  uid: a9a658e8-909e-4936-b500-5e2309fd8351
type: Opaque

3. Now patch secret "data-empty-secret" using kubectl patch command.

$ kubectl get secret data-empty-secret -o yaml
apiVersion: v1
data:
  testfile.txt: ZHVtbXlzZWNyZXRkYXRhCg==
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: test-nginx
    meta.helm.sh/release-namespace: test-system
    test.com/product-name: Test
  creationTimestamp: "2024-09-01T08:55:00Z"
  labels:
    app.kubernetes.io/instance: test-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
  name: data-empty-secret
  namespace: test-system
  resourceVersion: "111967090"
  uid: a8e3ff97-8644-4400-97fb-d0fd331f0f66
type: Opaque

4. Now perform helm upgrade without keeping testfile in helm directory and observe secret data.

[Output truncated ]

$ helm upgrade test-nginx nginx/ --debug

upgrade.go:155: [debug] preparing upgrade for test-nginx
upgrade.go:163: [debug] performing update for test-nginx
upgrade.go:356: [debug] creating upgraded release for test-nginx
...
client.go:684: [debug] Looks like there are no changes for Secret "data-empty-secret"
client.go:693: [debug] Patch Secret "data-conditional-secret" in namespace test-system
client.go:684: [debug] Looks like there are no changes for Role "web-access"
client.go:684: [debug] Looks like there are no changes for RoleBinding "web-view"
client.go:693: [debug] Patch StatefulSet "web" in namespace test-system

After helm upgrade observe secret data in data-empty-secret is still present.

$ kubectl get secret data-empty-secret -o yaml
apiVersion: v1
data:
  testfile.txt: ZHVtbXlzZWNyZXRkYXRhCg==
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: test-nginx
    meta.helm.sh/release-namespace: test-system
    test.com/product-name: Test
  creationTimestamp: "2024-09-01T08:55:00Z"
  labels:
    app.kubernetes.io/instance: test-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
  name: data-empty-secret
  namespace: test-system
  resourceVersion: "111967090"
  uid: a8e3ff97-8644-4400-97fb-d0fd331f0f66
type: Opaque

After helm upgrade observe secret data in data-conditional-secret is lost.

$ kubectl get secret data-conditional-secret -o yaml
kubectl get secret data-conditional-secret -o yaml
apiVersion: v1
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: test-nginx
    meta.helm.sh/release-namespace: test-system
    test.com/product-name: Test
  creationTimestamp: "2024-09-01T08:55:00Z"
  labels:
    app.kubernetes.io/instance: test-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
  name: data-conditional-secret
  namespace: test-system
  resourceVersion: "113530736"
  uid: a9a658e8-909e-4936-b500-5e2309fd8351
type: Opaque

Now in above behavior it has been observed that secret data is lost after helm upgrade if updated via helm chart ( in case data-conditional-secret ) of while it still in secret (data-empty-secret) persists if data is updated by kubectl patch command.

Also can observe from helm upgrade logs helm consider no change in secret if patched by kubectl patch command.

client.go:684: [debug] Looks like there are no changes for Secret "data-empty-secret"
client.go:693: [debug] Patch Secret "data-conditional-secret" in namespace test-system

Please can you suggest is it a valid behavior of helm chart to make data persist if data updated by kubectl patch command.