Rolling Upgrade

[juliev0]

Summary

Ideally, we'd have something more like a "canary" rollout:
Take down 1 Pod at a time (or x% of Pods at a time) and update scale.min on the upgrading one to correspond to what was removed on the other side. Perform this in a way such that the new Pod is brought up and confirmed to be healthy prior to the old one being taken down.

This gets a little tricky given that it means making multiple assessments. We currently have the idea of the "Assessment Window" for which we purposely wait 120 seconds prior to making that first assessment and then start the assessment. If there's any single failure in that window, we deem it failed.

Could we keep that one assessment start time, and basically continue to measure for the length of the progressive rollout and deem failure if there's ever any failure? We would need to make sure that there isn't a chance of an intermittent failed condition while new Pods are coming up.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.