[Enhancement] Integrate Cruise Control into the Kafka broker scale up/down process

[tomncooper]

When you add Kafka brokers, the new brokers only accept new topics/partitions. The user needs to manually reassign partitions to spread the load onto the new brokers. The user can run a whole cluster rebalance manually via the KafkaRebalance CR. However, it would be nice to do this automatically (if desired).

Scaling up can be improved by integrating with Cruise Control's add brokers functionality to rebalance the load as brokers are activated.

For removing brokers, the broker to be shut down needs to be "drained" of partitions. This will require integration with Cruise Control's decommissioning functionality.

[scholzj]

Triaged on 26.5.2022: Related to #133. #133 should implement a check preventing broker scaledown from happening if there are any topics assigned still to the removed brokers. This issue should build on top of it and implement the automation -> to have the rebalance done automatically when scaling up or down. A second proposal will be needed for this.

[viktorsomogyi]

Hi @scholzj , are there any plans on moving forward with this task? We're currently evaluating Strimzi and broker scalability is an important factor for us. Would you please share more info what are the current outstanding tasks and whether you'll implement this and automatic rebalance on cluster scaling?

[scholzj]

Hi @viktorsomogyi. I guess the current state is the following:

For scale-down:

• We done the initial implementation of blocking of scale-down of un-empty brokers in Prevent broker scaledown if broker contains partition replicas - Part 1 #9042.
  This will be released in Strimzi 0.38 (expected next week).
  There are some things to improve tracked in Revert blocked scale-down and continue reconciliation #9232 but it is a start.
  I hope to get to Revert blocked scale-down and continue reconciliation #9232 before the end of this year.
• We need to do the integration with Cruise Control to clean the broker being scaled down automatically.
  We did not decide how to do it yet I think.
  Either we talk directly with Cruise Control (harder) or we just spin the KafkaRebalance resource to do it and use the existing functionality (this should be easier).

For scale-up:

• We essentially need to only do the rebalance after the scale-up is done.
  Again, I do not think we discussed in detail how to do it.
  We also need to decide on what goals should be used for this.

I think for both cases, the next steps should be to discuss the details of how it will be implemented. Some of the discussion happened in strimzi/proposals#57, but we decided to postpone it among other things because we were missing the blocking on scale-down that is now done. @ppatierno was working on this.

To be honest, right now our focus is mostly on KRaft support. So I do not think we will get to this anymore in 2023. But this is still something we plan to support and I hope it should be one of the things we will do in early 2024 once most of the KRaft work is done.

Of course, we are always interested in contributions - so if you would be interested, you could also join the work on this - that might speed it up 😉.

CC @ppatierno: Not sure if you have something to add.

[viktorsomogyi]

@scholzj thanks for the very detailed writeup, it is really insightful, also the part that you'll working full speed on KRaft. That would have been another question from me 🙂.
We are interested in contributing sometime next year when we get to it as now our hands are full with onboarding Strimzi into our product. I'll reach out to you once it becomes close.

[scholzj]

@viktorsomogyi Ok, great. Please let me know if you would have any more questions. Thanks

[ppatierno]

Proposal for this one is now approved and merged https://github.com/strimzi/proposals/blob/main/078-auto-rebalancing-cluster-scaling.md. Going to work on the implementation.

[ppatierno]

Closed by #10583