Putting chart version in spec.VolumeClaimTemplate.metadata.labels of a StatefulSet prevents ANY future upgrade of existing Release

[desaintmartin]

BUG REPORT

tl;dr: any Chart containing a StatefulSet (any version) using a chart version in a volumeClaimTemplate can't be upgraded

This is the third and hopefully the last bug regarding "I can't upgrade my Release".

Note:

• for the selectors not being updated in apps/v1beta1 / extensions/v1beta1 problem, see Putting chart version in metadata.labels in a Deployment v1beta1 without selector prevents upgrade #7726.
• for the immutability of selectors apps/v1beta2 problem, see Putting chart version in spec.selector.matchLabels in apps/v1beta2 and later prevents ANY future upgrade of existing Release #7680.

Forewords

• Kubernetes API defines that some fields are immutable. In particular:
  • For a StatefulSet, everything except spec.replicas, spec.template, and spec.updateStrategy (i.e spec.selectorand spec.VolumeClaimTemplate is immutable)

The problem

In some charts, we set the chart version (directly or through the chart template) in spec.VolumeClaimTemplate.metadata.labels.

This causes helm to attempt to change immutable fields and any upgrade (even if it ONLY change the version in Chart.yaml) will fail.

IN OTHER WORDS, SOME CHARTS OF THIS REPOSITORY ARE DEEPLY BROKEN AND CAN'T BE UPGRADED, only installed once.

Impacted charts

At least, impacted charts are:

• stable/mariadb (fixed in [stable/mariadb] Fix issue with pvc on upgrades #7966)
• stable/redis (fixed in [stable/redis] Major version bump: Fix chart not being upgradable #7686)
• stable/rabbitmq (fixed in [stable/rabbitmq] Fix chart not being upgradable by removing mutable label from VolumeClaimTemplate #7687)
• incubator/couchdb (PR in Allow statefulSet updates #8527)
• incubator/patroni (PR in [incubator/patroni] Fix chart not being upgradable because of spec.VolumeClaimTemplate.metadata.labels being immutable #8594)

How to reproduce (using stable/mariadb as an example)

Let's install the stable/mariadb chart using git, change the version in chart.yaml (example: 1.2.3 -> 1.2.4) without changing anything else, and try to upgrade. See it fail:

cd stable/mariadb
helm install --name test .
vim Chart.yaml
helm upgrade test .

Because of spec.volumeClaimTemplate.0.metadata.labels in templates/master-statefulset.yaml containing:

chart: {{ template "mariadb.chart" . }}

It will fail with:

Error: UPGRADE FAILED: StatefulSet.apps "redis-test-master" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.

Let's remove this chart label, uninstall and try to reproduce: the bug is gone.

Conclusion

• tl;dr: DO NOT put fields that may change in volumeClaimTemplate, especially chart label.
• Any fix to this is a breaking change. But any other unrelated change to already impacted chart breaks existing Releases.
• A potential workaround or "how to upgrade" is to remove the StatefulSet:

kubectl delete statefulsets.apps --cascade=false test-redis-master

and upgrade.

cc @mattfarina @scottrigby @juan131 @javsalgar @unguiculus @davidkarlsen @cpanato

[desaintmartin]

ping @kocolosk @alexeyklyukin @linki

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[desaintmartin]

#8527 for couchdb not merged yet.

[scottrigby]

#8527 is merged 🎉

[Simon3]

Can anyone confirm (or disprove) that this issue also affects incubator/cassandra and would be fixed by #12402 ? Thanks!

[desaintmartin]

Thanks ! Yes, it should solve the problem.

[schnrap]

We ran into the same issue in the stable/sonatype-nexus chart, when using the StatefulSet instead of the Deployment Option (statefulset.enabled=true).