☂️ [GEP-17] Shoot Control Plane Migration "Bad Case" Scenario

[stoyanr]

How to categorize this issue?

/area control-plane-migration robustness
/kind enhancement epic
/roadmap internal
/priority 3

What would you like to be added:

This is an epic to track the implementation of GEP-17 Shoot Control Plane Migration "Bad Case" Scenario.

• Add handling of owner DNSRecord resources and ensure they are reconciled properly while avoiding rate limit issues.

  • [GEP-17] Add handling of owner DNSRecord resources #4307
  • Fix owner DNSRecords stuck in states different from Succeeded #4815
  • Ensure owner DNSRecords are optional #4876
  • Add client-side provider rate limiters to DNSRecord controller gardener-extension-provider-aws#425

• Add owner check watchdog to extension reconcilers and ensure canceling the reconciliation context works as expected.

  • [GEP-17] Add owner check watchdog to extension reconcilers #4638
  • Delete Terraformer pod even if the context was cancelled #4720
  • Fix a deadlock issue in the owner check watchdogs #4752

• Ensure all extensions have vendored g/g v1.33.1 with the owher check watchdog mechanism and the corresponding versions have been released:

  • provider-aws: Vendor gardener/gardener@1.33.1 gardener-extension-provider-aws#435, v1.30.0
  • provider-azure: Vendor gardener/gardener@1.33.1 gardener-extension-provider-azure#388, v1.23.0
  • provider-gcp: Vendor gardener/gardener@1.33.1 gardener-extension-provider-gcp#339
  • provider-openstack: Vendor gardener/gardener@1.33.1 gardener-extension-provider-openstack#347
  • provider-alicloud: Vendor gardener/gardener@1.33.1 gardener-extension-provider-alicloud#372, v1.29.0
  • provider-vsphere: Vendor gardener/gardener@1.33.1 gardener-extension-provider-vsphere#182
  • networking-calico: Vendor gardener/gardener@v1.33.1 gardener-extension-networking-calico#121, v1.21.0
  • networking-cilium: Vendor gardener/gardener@v1.33.1 gardener-extension-networking-cilium#71, v1.7.1
  • os-gardenlinux: Vendor gardener/gardener@v1.33.1 gardener-extension-os-gardenlinux#51
  • os-ubuntu: Vendor gardener/gardener@v1.33.1 gardener-extension-os-ubuntu#55
  • os-suse-chost: Vendor gardener/gardener@v1.33.1 gardener-extension-os-suse-chost#57, v1.14.0
  • runtime-gvisor: Vendor gardener/gardener@v1.33.1 gardener-extension-runtime-gvisor#28
  • shoot-cert-service: Vendor gardener/gardener@v1.33.1 gardener-extension-shoot-cert-service#97
  • shoot-dns-service: Vendor gardener/gardener@v1.33.1 gardener-extension-shoot-dns-service#89

• Adapt external-dns-mangement and the shoot-dns-service extension to not reconcile shoot DNS resources if the ownership has changed.

  • DNS based owner activation external-dns-management#192, v0.11.0
  • Support DNS Activation Records for Owner objects gardener-extension-shoot-dns-service#90

• Add owner checks and taking of final snapshots to etcd-backup-restore, introduce the corresponding configuration in the Etcd resource and controller (etcd-druid), and ensure it is set properly by gardenlet.

  • [GEP-17] Add owner checks and taking of final snapshots etcd-backup-restore#383
  • [GEP-17] Add owner check configuration to Etcd resource and controller etcd-druid#239
  • [GEP-17] Set owner check configuration in etcd-main Etcd resource #4813

• Add copying of etcd backups to the destination backup container to etcd-backup-restore, the corresponding resource and controller to etcd-druid, and the corresponding adaptations to gardenlet. This is not strictly part of the "bad case" scenario implementation, but it's a prerequisite for implementing the flow as described in GEP-17.

  • Adding copy operation and additional authentication parameters forsource and target etcd-backup-restore#369
  • Adding functionality to start copy job etcd-druid#234
  • Modify restore operation to include copy of etcd backups #4894

• Ensure all provider extensions have vendored g/g v1.37.0 with the copying of backups mechanism and the corresponding versions have been released:

  • provider-aws: Revendor gardener/gardener@v1.38 and update Golang version to 1.17 gardener-extension-provider-aws#463
  • provider-azure: Revendor gardener/gardener@v1.38 and update Golang version to 1.17 gardener-extension-provider-azure#418
  • provider-gcp: Revendor gardener/gardener@v1.38 and update Golang version to 1.17 gardener-extension-provider-gcp#364
  • provider-openstack: Revendor gardener/gardener@v1.38 and update Golang version to 1.17 gardener-extension-provider-openstack#364
  • provider-alicloud: Revendor gardener/gardener@v1.38 and update Golang version to 1.17 gardener-extension-provider-alicloud#403
  • provider-vsphere: Revendor gardener/gardener@v1.38 and update Golang version to 1.17 gardener-extension-provider-vsphere#187
  • provider-equinix-metal: Revendor gardener/gardener@v1.38 and update Golang version to 1.17 gardener-extension-provider-equinix-metal#180

• Ensure CPM "bad case" scenario mechanisms in etcd-backup-restore and etcd-druid work correctly with multi-node etcd.

  • Adapt owner checks, taking of final snapshots, and disabling the cluster in etcd-backup-restore for multi-node etcd (part of Add Leader election to backup-restore and allow only backup-restore Leader to take and uploads the snapshots. etcd-backup-restore#353, see also [Feature] To discuss the impact of PR #383(Add owner checks and taking of final snapshots) on Multi-Node Etcd etcd-druid#242)
  • Add owner check watchdog to etcd-druid to prevent reconciling Etcd resources when the seed is no longer the owner of the shoot

• Adapt gardenlet shoot reconciliation and migration flows for the "bad case" and the "good case" scenarios.

  • [GEP-17] Ensure final full snapshot is created and waited for while copying backups #5095
  • [GEP-17] Let destination gardenlet determine that operation should be restore #5101

• Add forcing the shoot's restoration to the destination seed during control plane migration if the preparation for migration in the source seed is not finished after a certain grace period and is considered unlikely to succeed (falling back to the "bad case" scenario).

  • [GEP-17] Add force restoring of shoots and backup entries #5123

• Add automatic cleanup of shoot leftover resources in source seeds if the fallback to the "bad case" scenario had taken place, see also Cleanup of shoot leftover resources #5173.

  • Add cleanup of shoot leftover resources #5150
  • Enhance the ShootLeftover resource and controller to support a grace period for the actual cleanup of shoot leftover resources (see Cleanup of shoot leftover resources #5173 (comment)).
  • Add an admission plugin for ShootLeftover resources.
  • Actually trigger the cleanup of shoot leftover resources in the new reconcilers introduced with [GEP-17] Add force restoring of shoots and backup entries #5123.

• Testing

  • Add integration test for CPM "bad case" scenario

[stoyanr]

/cc @plkokanov @kris94 @rfranzke

[stoyanr]

/assign

[MartinWeindel]

Regarding the shoot-dns-service extension: PR gardener/gardener-extension-shoot-dns-service#63 was replaced by gardener/gardener-extension-shoot-dns-service#90

[stoyanr]

/assign @plkokanov

[plkokanov]

Closing this in favour of #6302