Upgrade to OKD 4.14 stuck with Master and Worker node in NotReady Status - rpm-ostree rebase error

[Serbinmatosh]

Hi guys, really need help trying to figure out what is going on here. We are upgrading from OKD 4.13.0-0.okd-2023-10-28-065448 to 4.14.0-0.okd-2023-11-12-042703 and upon the machine config rebooting the first master and worker node, both didn't come back to a ready state and update is stuck there.

The Machine Config Pool is showing a degraded Node with the following message:

Node master-1 is reporting: "failed to update OS to
        quay.io/openshift/okd-content@sha256:34f3d15a2a5f1a9b6e5e158e2198d077b149288ccc13cb31b31563d3cd493c48
        : error running rpm-ostree rebase --experimental
        ostree-unverified-registry:quay.io/openshift/okd-content@sha256:34f3d15a2a5f1a9b6e5e158e2198d077b149288ccc13cb31b31563d3cd493c48:
        error: Importing: Unencapsulating base: Failed to invoke skopeo proxy
        method GetBlob: remote error: fetching blob: received unexpected HTTP
        status: 502 Bad Gateway\n: exit status 1"

Does anyone know how to resolve this issue? We tried rebooting the master and worker nodes manually and it didn't change anything and we cannot ssh into the nodes anymore.

Any help is greatly appreciated!!

[Goose29]

You can always put the nodes into recovery mode, by catching the Grub menu and appending Single to the end and removing the now redundant console entries - then change the root user password and continue normal boot.

Then you can try to perform the rebase manually (and debug), if nothing solves the issue, then you can revert back, and do touch /run/machine-config-daemon-force to trick the Machine Config Daemon to think its fine while continueing to debug.