diff --git a/random-thoughts/2024-12-21.md b/random-thoughts/2024-12-21.md index a38cb52..e2a0ca1 100644 --- a/random-thoughts/2024-12-21.md +++ b/random-thoughts/2024-12-21.md @@ -929,3 +929,56 @@ tags: - The client code doesn't need to know or decide which mode is being used - it simply makes metadata requests to brokers, and the brokers handle getting the metadata from the appropriate source based on how they were configured. - This is part of Kafka's evolution to remove the ZooKeeper dependency (KIP-500), making the system simpler to deploy and maintain. + +- How brokers decide which broker is the leader for a partition: + - Leadership Selection Process: + - Initial Leader Selection: + - When a topic is created, the Controller assigns leadership + - Uses round-robin distribution by default + - Considers rack awareness if configured + - Ensures even distribution across brokers + + - Leader Election Process: + + ```mermaid + sequenceDiagram + participant Controller + participant ISR as In-Sync Replicas + participant Broker + + Note over Controller: Detect leader failure or
need for new leader + + Controller->>ISR: Get list of eligible replicas + + alt Has eligible ISR + Controller->>Controller: Select first replica from ISR + Controller->>Broker: Notify new leader + Broker-->>Controller: Accept leadership + else No eligible ISR + alt unclean.leader.election.enable=true + Controller->>Controller: Select from all replicas + else unclean.leader.election.enable=false + Controller->>Controller: Keep partition offline + end + end + + Controller->>Broker: Broadcast metadata update + ``` + + - Leadership Eligibility: + - Must be in ISR (In-Sync Replicas) list + - Must be alive and responsive + - Must have caught up with previous leader + - Configured minimum ISR size must be met + + - Failure Handling: + - Controller monitors broker health + - Detects leader failures via ZooKeeper/KRaft + - Initiates new leader election if needed + - Updates cluster metadata after changes + + - Configuration Factors: + - unclean.leader.election.enable + - min.insync.replicas + - replica.lag.time.max.ms + - leader.imbalance.check.interval.seconds