Re-create missing pod which has incompleted downscale operation #824
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kuvesz
previously requested changes
Jun 20, 2022
amuraru
reviewed
Jun 20, 2022
stoader
requested changes
Jun 22, 2022
ecojan
reviewed
Jun 22, 2022
stoader
reviewed
Jun 23, 2022
stoader
previously requested changes
Jun 23, 2022
|
@bartam1 managed to test this locally today, working as expected now 👍 thank you for the effort put in here! |
amuraru
reviewed
Jun 27, 2022
pregnor
previously approved these changes
Jun 28, 2022
pregnor
requested changes
Jun 29, 2022
pregnor
previously approved these changes
Jun 29, 2022
pregnor
previously approved these changes
Jun 29, 2022
stoader
reviewed
Jun 29, 2022
stoader
requested changes
Jun 29, 2022
amuraru
reviewed
Jun 29, 2022
stoader
previously approved these changes
Jun 30, 2022
stoader
approved these changes
Jun 30, 2022
Kuvesz
approved these changes
Jul 5, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What's in this PR?
There is a new field in the broker status. Its name is configurationBackup.
It stores broker configuration in gzipped base64 format.
When a broker is missing and has not been removed gracefully (incompleted downscale operation), the operator will re-create that pod based on the configurationBackup thus graceful downscale operation can be continued.
Why?
During a scale-down operation if a broker that is to be removed is deleted in a non controlled fashion (while the data is being drained from the Kafka broker to other brokers) the operator doesn't bring the broker back up to finish this and then remove it again.
This in turn will slow down new replicas getting in sync (as they will have to pull from 2 or even worse, 1 in sync replica).
Even more concerning, if there is a K8s cluster rollout restart (due to VM upgrades for examples) if the Kafka pods don't come back up, this will result in offline partitions.
Checklist