Update / Patch to deployment during "replace" rejected by kubernetes, fails in a loop

[tmckayus]

Background: changes to selectors/labels on deployments.apps (and other objects) are not allowed by kubernetes, for the discussion behind that decision see kubernetes/kubernetes#50808.

During a "replace" operation on a CSV, the OLM will attempt to update/patch an existing deployment if the new CSV uses the same deployment name as the existing CSV. If the new CSV includes changes to the deployment(s) that are not allowed by kubernetes (ie a change to a selector/label field, for example), the replace operation will fail ("field is immutable" from kube). At this point, the OLM will periodically retry the replace, failing each time -- the old CSV will never successfully be replaced (but it will stay functional afaik)

If possible, this should be handled in a way that is more clear to an end user. Currently, it may not be clear to someone who isn't an advanced kubernetes/OLM user exactly what is happening and how to correct it.

From a user perspective, there are a couple of simple workarounds to this scenario:

1. simply respin the CSV and use a different deployment name (long term solution, covers everybody going forward). This is more of a developer solution, during the crafting of a new pull request to update the CSV.

2. delete the deployment(s) from the old CSV by hand. This will allow the current replace operation to continue. This is an immediate fix for the instance, more of an end user hot-fix.

High-level steps to reproduce

1. Create/use a CSV for any old operator
2. Create a "new" CSV to update the operator with a "replaces" value
3. In the new CSV, change the spec for one of the deployments. Leave the name the same, but change the spec.selector.matchLabels and spec.template.metadata.labels fields
4. Upload the bundle for your new CSV to quay
5. Install the "old" CSV on a kubernetes/openshift cluster through the OLM
6. Add an operatorsource (and whatever other associated objects you might need, depending on whether you're using kube/openshift) that references your new CSV
7. Use "oc get clusterserviceversion" or the analagous kubectl command to watch the update process. You should see it enter a state like this, PHASE for 0.1.2 will remain "Replacing" forever, PHASE for 0.1.3 will cycle through "Pending", "Failed", and "InstallReady"

$ oc get clusterserviceversion
NAME                            DISPLAY           VERSION   REPLACES             PHASE
packageserver.v0.9.0    Package Server    0.9.0                                      Succeeded
myoperator.v0.1.2         My Operator         0.1.2                                      Replacing
myoperator.v0.1.3         My Operator         0.1.3       myoperator.v0.1.2   Pending

8. look in the olm-operator pod in the olm namespace for errors, search for "field is immutable"
9. if you want to free it up, delete the deployment(s) from the existing install and the replace operation will succeed

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[ecordell]

A couple of things to mention here:

• In newer versions of OLM, deployments are handled similarly to how pods are managed via replicasets - we stamp a hash of the spec in the CSV on the deployment and compare that. If it needs updating, we issue an Update, not a patch.
• During upgrades, if you need to change an otherwise immutable field of a deployment (like the label selector), you can signal this to olm by changing the "name" of the deployment in the CSV spec. This will cause OLM to create an entirely new deployment and delete the old.

Please re-open or open a new issue if we need further discussion / clarification.