KAFKA-17048; Add docs for KIP-853 #17076

jsancio · 2024-09-03T16:18:18Z

Update the quickstart, ops section, zk to kraft migration section and security section to document the KIP-853 changes.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

ahuang98 · 2024-09-05T18:11:27Z

config/kraft/controller.properties


 ############################# Socket Server Settings #############################

 # The address the socket server listens on.
-# Note that only the controller listeners are allowed here when `process.roles=controller`, and this listener should be consistent with `controller.quorum.voters` value.
+# Note that only the controller listeners are allowed here when `process.roles=controller`


does this listener still need to be consistent with what controller.quorum.bootstrap.servers contains?

In most cases yes but it is not required. The best example is that the controller.quorum.boostrap.servers could be a domain name that resolves to any of the controller hosts.

ahuang98

Thanks for the documentation changes!

ahuang98 · 2024-09-05T18:12:44Z

docs/ops.html

@@ -3776,25 +3776,71 @@ <h5 class="anchor-heading"><a id="kraft_voter" class="anchor-link"></a><a href="

  <p>A Kafka admin will typically select 3 or 5 servers for this role, depending on factors like cost and the number of concurrent failures your system should withstand without availability impact. A majority of the controllers must be alive in order to maintain availability. With 3 controllers, the cluster can tolerate 1 controller failure; with 5 controllers, the cluster can tolerate 2 controller failures.</p>

-  <p>All of the servers in a Kafka cluster discover the quorum voters using the <code>controller.quorum.voters</code> property. This identifies the quorum controller servers that should be used. All the controllers must be enumerated. Each controller is identified with their <code>id</code>, <code>host</code> and <code>port</code> information. For example:</p>
+  <p>All of the servers in a Kafka cluster discover the active controller using the <code>controller.quorum.bootrtrap.servers</code> property. All the controllers should be enumerated in this property. Each controller is identified with their <code>host</code> and <code>port</code> information. For example:</p>


typo bootstrap.servers

ahuang98 · 2024-09-05T18:13:38Z

docs/ops.html

  <p></p>
  The <code>kafka-storage.sh random-uuid</code> command can be used to generate a cluster ID for your new cluster. This cluster ID must be used when formatting each server in the cluster with the <code>kafka-storage.sh format</code> command.

  <p>This is different from how Kafka has operated in the past. Previously, Kafka would format blank storage directories automatically, and also generate a new cluster ID automatically. One reason for the change is that auto-formatting can sometimes obscure an error condition. This is particularly important for the metadata log maintained by the controller and broker servers. If a majority of the controllers were able to start with an empty log directory, a leader might be able to be elected with missing committed data.</p>

+  <h5 class="anchor-heading"><a id="kraft_storage_standalone" class="anchor-link"></a><a href="#kraft_storage_standalone">Bootstrap a Standalone Controller</a></h5>
+  The recommended method for creating a new KRaft controller cluster is to bootstrap it with one voter and dyncamically <a href="#kraft_reconfig_add">add the rest of the controllers</a>. This can be done with the following CLI command:


typo: dynamically

ahuang98 · 2024-09-05T18:14:27Z

docs/ops.html

+
+  This command will 1) create a meta.properties file in metadata.log.dir with a randomly generated directory.id, 2) create a snapshot at 00000000000000000000-0000000000.checkpoint with the necessary control records (KRaftVersionRecord and VotersRecord) to make this Kafka node the only voter for the quorum.
+
+  <h5 class="anchor-heading"><a id="kraft_storage_voters" class="anchor-link"></a><a href="#kraft_storage_voters">Bootstrap with Multiple Controller</a></h5>


nit: Controllers

ahuang98 · 2024-09-05T18:15:05Z

docs/ops.html

+  <h5 class="anchor-heading"><a id="kraft_storage_voters" class="anchor-link"></a><a href="#kraft_storage_voters">Bootstrap with Multiple Controller</a></h5>
+  The KRaft cluster metadata partition can also be bootstrapped with more than one voter. This can be done by using the --initial-controllers flag:
+
+  <pre><code class="language-bash">kafka-storage format --cluster-id <cluster-id> --initial-controller 0@controller-0:1234:3Db5QLSqSZieL3rJBUUegA,1@controller-1:1234:L3rJBUUegA3Db5QLSqSZie,2@controller-2:1234:UegA3Db5QLSqSZieL3rJBU --config controller.properties</code></pre>


comment says --initial-controllers and the command says --initial-controller

ahuang98 · 2024-09-05T18:16:57Z

docs/ops.html

+
+  <pre><code class="language-bash">kafka-storage format --cluster-id <cluster-id> --initial-controller 0@controller-0:1234:3Db5QLSqSZieL3rJBUUegA,1@controller-1:1234:L3rJBUUegA3Db5QLSqSZie,2@controller-2:1234:UegA3Db5QLSqSZieL3rJBU --config controller.properties</code></pre>
+
+This command is similar to the standalone version but the snapshot at 00000000000000000000-0000000000.checkpoint will instead contain a VotersRecord that includes information for all of the controllers specified in --initial-controllers. Just like the controller.quorum.voters properties, it is important that the set of voters is equal in all of the controllers with the same cluster id.


did you mean controller.quorum.bootstrap.servers instead of controller.quorum.voters?

if you did mean controller.quorum.voters, it might not make sense to readers since it's not mentioned prior to this line anymore

Yes. I meant controller.quorum.voters but like you said it is better to not mention that property since we are deprecating it and we have not mentioned in the past. I reworded the sentence to not mention the property.

ahuang98 · 2024-09-05T18:43:53Z

docs/ops.html

+  <h5 class="anchor-heading"><a id="kraft_storage_standalone" class="anchor-link"></a><a href="#kraft_storage_standalone">Bootstrap a Standalone Controller</a></h5>
+  The recommended method for creating a new KRaft controller cluster is to bootstrap it with one voter and dyncamically <a href="#kraft_reconfig_add">add the rest of the controllers</a>. This can be done with the following CLI command:
+
+  <pre><code class="language-bash">kafka-storage format --cluster-id <cluster-id> --standalone --config controller.properties</code></pre>


do we want to talk at all about what information must be in controller.properties? Perhaps in a separate section?

We have a "Configuration / Controllers" section. In there we discuss the kraft configuration. I updated the example to be a bit more complete.

Have you checked <cluster-id> renders fine? We may need to use </>

@mimaison Thanks! I filed #17235 to fix this.

ahuang98 · 2024-09-05T18:48:16Z

docs/ops.html

+  <h5 class="anchor-heading"><a id="kraft_reconfig_add" class="anchor-link"></a><a href="#kraft_reconfig_add">Add New Controller</a></h5>
+  If the KRaft Controller cluster already exist, the cluster can be expanded by first provisioning a new controller using the <a href="#kraft_storage_observers">kafka-storage tool</a> and starting the controller.
+
+  After sarting the controller, the replication to the new controller can be monitored using the <code>kafka-metadata-quorum describe --replication</code> command. Once the new controller has caught up to the active controller, it can be added to the cluster using the <code>kafka-metadata-quorum add-controller</code> command.


typo: starting

ahuang98 · 2024-09-05T18:49:54Z

docs/ops.html

+
+  After sarting the controller, the replication to the new controller can be monitored using the <code>kafka-metadata-quorum describe --replication</code> command. Once the new controller has caught up to the active controller, it can be added to the cluster using the <code>kafka-metadata-quorum add-controller</code> command.
+
+  <pre><code class="language-bash">kafka-metadata-quorum --command-config controller.properties --bootstrap-server localhost:9092 add-controller</code></pre>


this must be run from a broker, should we give the bootstrap-controller alternative command as well?

ahuang98 · 2024-09-05T18:50:52Z

docs/ops.html

+  <pre><code class="language-bash">kafka-metadata-quorum --command-config controller.properties --bootstrap-server localhost:9092 add-controller</code></pre>
+
+  <h5 class="anchor-heading"><a id="kraft_reconfig_remove" class="anchor-link"></a><a href="#kraft_reconfig_remove">Remove Controller</a></h5>
+  If the KRaft Controller cluster already exist, the cluster can be shrunk using the <code>kafka-metadata-quorum remove-controller</code> command. Until KIP-996: Pre-vote has been implemented and release, it is recommended to shutdown the controller that will be removed.


suggestion to make this extremely explicit: it is recommended to shutdown the controller that will be removed before running the remove-controller command.

ahuang98 · 2024-09-05T18:54:59Z

docs/security.html

-      defined by <code>controller.quorum.voters</code> must then also use port 9093,
-      as it does here.</p>
+    <p>It is a requirement for the host and port defined in <code>controller.quorum.bootstrap.servers</code>
+      is routed to the exposed controller listeners. For example, here the <code>CONTROLLER</code>


It is a requirement that the... is routed to

ahuang98

LGTM, thanks for the detailed docs!

mimaison · 2024-09-11T09:20:04Z

docs/quickstart.html

@@ -32,8 +32,8 @@ <h4 class="anchor-heading">
            the latest Kafka release and extract it:
        </p>

-        <pre><code class="language-bash">$ tar -xzf kafka_{{scalaVersion}}-{{fullDotVersion}}.tgz
-$ cd kafka_{{scalaVersion}}-{{fullDotVersion}}</code></pre>
+        <pre><code class="language-bash">tar -xzf kafka_{{scalaVersion}}-{{fullDotVersion}}.tgz


Why are we removing the $ prompts? We use them everywhere in the documentation. It's useful to distinguish the commands from their outputs.

@jsancio : I agree, let's not make this change removing the $ prompts.

(Or if you really want to do it, let's create a separate PR where we can discuss further, and do it for all docs.)

Okay. I'll revert these changes. I may do it in another PR.

I made these changes for two reasons. It assumes that the CLI prompt is $ which is not the case in all terminals and shells. The user cannot copy and paste command and instead has to edit the begin the command before executing it.

mimaison · 2024-09-11T09:28:07Z

docs/ops.html

  <p></p>
  The <code>kafka-storage.sh random-uuid</code> command can be used to generate a cluster ID for your new cluster. This cluster ID must be used when formatting each server in the cluster with the <code>kafka-storage.sh format</code> command.

  <p>This is different from how Kafka has operated in the past. Previously, Kafka would format blank storage directories automatically, and also generate a new cluster ID automatically. One reason for the change is that auto-formatting can sometimes obscure an error condition. This is particularly important for the metadata log maintained by the controller and broker servers. If a majority of the controllers were able to start with an empty log directory, a leader might be able to be elected with missing committed data.</p>

+  <h5 class="anchor-heading"><a id="kraft_storage_standalone" class="anchor-link"></a><a href="#kraft_storage_standalone">Bootstrap a Standalone Controller</a></h5>
+  The recommended method for creating a new KRaft controller cluster is to bootstrap it with one voter and dynamically <a href="#kraft_reconfig_add">add the rest of the controllers</a>. Bootstrapping the first controller can be done with the following CLI command:


Any particular reason this is the recommended method over the alternative below?

Yes. This is the safest option and easiest to get correct by the user.

With the --standalone option the user just needs to execute the command once on one the controller and the controller cluster will be consistent.

The --initial-controllers option assumes that the user is executing the same command across all of the controllers and that the replica id and replica directory id matches in all of the controllers. The KRaft controllers don't have a way to verify this and it is possible that the user will bootstrap a controller cluster incorrectly

I think it would make sense to add this reasoning to the documentation so that users can make an informed decision, and know what the requirements are if choosing to go against the recommendation for some reason.

docs/ops.html

mimaison · 2024-09-11T09:49:25Z

config/kraft/reconfig-server.properties

@@ -0,0 +1,132 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more


It's unclear why we're adding a new file

@jsancio we should use the same approach here in server.properties as in broker.properties and controller.properties

As I commented above, the simplest approach is just to have a line for controller.quorum.voters, but have it be commented out, with an explanation about how to use the old style if desired. That would avoid the need for a separate file.

It's unclear why we're adding a new file

This file is reference in Dockerfile and a few other places. I haven't had a chance to fix all of those places to use kraft.version 0. I instead created this jira: https://issues.apache.org/jira/browse/KAFKA-17576

rg "kraft\/server.properties" docker/native/Dockerfile 66:COPY --chown=appuser:root --from=build-native-image /app/kafka/config/kraft/server.properties /etc/kafka/docker/ docker/docker_official_images/3.7.0/jvm/Dockerfile 81: cp /opt/kafka/config/kraft/server.properties /etc/kafka/docker/server.properties; \ docker/docker_official_images/3.7.0/jvm/jsa_launch 20:KAFKA_JVM_PERFORMANCE_OPTS="-XX:ArchiveClassesAtExit=storage.jsa" opt/kafka/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c opt/kafka/config/kraft/server.properties 22:KAFKA_JVM_PERFORMANCE_OPTS="-XX:ArchiveClassesAtExit=kafka.jsa" opt/kafka/bin/kafka-server-start.sh opt/kafka/config/kraft/server.properties & docker/jvm/Dockerfile 81: cp /opt/kafka/config/kraft/server.properties /etc/kafka/docker/server.properties; \ docker/jvm/jsa_launch 20:KAFKA_JVM_PERFORMANCE_OPTS="-XX:ArchiveClassesAtExit=storage.jsa" opt/kafka/bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c opt/kafka/config/kraft/server.properties 22:KAFKA_JVM_PERFORMANCE_OPTS="-XX:ArchiveClassesAtExit=kafka.jsa" opt/kafka/bin/kafka-server-start.sh opt/kafka/config/kraft/server.properties & docs/streams/quickstart.html 133:<pre><code class="language-bash">$ bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties</code></pre> 139:<pre><code class="language-bash">$ bin/kafka-server-start.sh config/kraft/server.properties</code></pre> README.md 98: ./bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties 99: ./bin/kafka-server-start.sh config/kraft/server.properties

As I commented above, the simplest approach is just to have a line for controller.quorum.voters, but have it be commented out, with an explanation about how to use the old style if desired. That would avoid the need for a separate file.

See my other comment. I am not sure if we need to document all of the options. Kafka and KRaft support a lot of configurations that are not enumerated in kraft/server.properties.

For now, I guess we can add a new file, although it's not ideal.

docs/quickstart.html

config/kraft/broker.properties

config/kraft/controller.properties

cmccabe

see comments

cmccabe

LGTM

Change the configurations under config/kraft to use controller.quorum.bootstrap.servers instead of controller.quorum.voters. Add comments explaining how to use the older static quorum configuration where appropriate. In docs/ops.html, remove the reference to "tentative timelines for ZooKeeper removal" and "Tiered storage is considered as an early access feature" since they are no longer up-to-date. Add KIP-853 information. In docs/quickstart.html, move the ZK instructions to be after the KRaft instructions. Update the KRaft instructions to use KIP-853. In docs/security.html, add an explanation of --bootstrap-controller and document controller.quorum.bootstrap.servers instead of controller.quorum.voters. Reviewers: Mickael Maison <mickael.maison@gmail.com>, Alyssa Huang <ahuang@confluent.io>, Colin P. McCabe <cmccabe@apache.org>

cmccabe · 2024-09-18T18:36:44Z

Thanks, everyone. I committed what we have so far. If there are any gaps please raise another PR. Hopefully in time for RC1, so that everyone can kick the tires :)

KAFKA-17048; Add docs for KIP-853

8a8ac12

jsancio added kraft KIP-853 KRaft Controller Membership Changes labels Sep 3, 2024

ahuang98 reviewed Sep 5, 2024

View reviewed changes

ahuang98 suggested changes Sep 5, 2024

View reviewed changes

jsancio added 2 commits September 10, 2024 22:50

KAFKA-17048; Fix documentation

29a3a95

KAFKA-17048; Add controller listener names configuration

ebf3215

ahuang98 approved these changes Sep 11, 2024

View reviewed changes

KAFKA-17048; Add new configuration for reconfig quickstart

e486fa5

mimaison reviewed Sep 11, 2024

View reviewed changes

docs/ops.html Outdated Show resolved Hide resolved

mimaison reviewed Sep 11, 2024

View reviewed changes

docs/ops.html Outdated Show resolved Hide resolved

mimaison reviewed Sep 11, 2024

View reviewed changes

cmccabe reviewed Sep 13, 2024

View reviewed changes

docs/quickstart.html Outdated Show resolved Hide resolved

cmccabe reviewed Sep 13, 2024

View reviewed changes

config/kraft/broker.properties Show resolved Hide resolved

cmccabe reviewed Sep 13, 2024

View reviewed changes

config/kraft/controller.properties Show resolved Hide resolved

cmccabe requested changes Sep 13, 2024

View reviewed changes

jsancio and others added 3 commits September 18, 2024 17:58

KAFKA-17084; Add shell prompt to examples

cff8d79

fixes

5ef5245

fix

4ae624b

cmccabe approved these changes Sep 18, 2024

View reviewed changes

cmccabe merged commit aee44ef into apache:trunk Sep 18, 2024
6 of 7 checks passed

jsancio deleted the kafka-17048-kraft-reconfig-doc branch September 18, 2024 22:57

kevin-wu24 mentioned this pull request Oct 28, 2024

KAFKA-17576: Fix all references to kraft/server.properties to use reconfig-server.properties #17567

Open

3 tasks


		This command will 1) create a meta.properties file in metadata.log.dir with a randomly generated directory.id, 2) create a snapshot at 00000000000000000000-0000000000.checkpoint with the necessary control records (KRaftVersionRecord and VotersRecord) to make this Kafka node the only voter for the quorum.

		<h5 class="anchor-heading"><a id="kraft_storage_voters" class="anchor-link"></a><a href="#kraft_storage_voters">Bootstrap with Multiple Controller</a></h5>


		<pre><code class="language-bash">kafka-storage format --cluster-id <cluster-id> --initial-controller 0@controller-0:1234:3Db5QLSqSZieL3rJBUUegA,1@controller-1:1234:L3rJBUUegA3Db5QLSqSZie,2@controller-2:1234:UegA3Db5QLSqSZieL3rJBU --config controller.properties</code></pre>

		This command is similar to the standalone version but the snapshot at 00000000000000000000-0000000000.checkpoint will instead contain a VotersRecord that includes information for all of the controllers specified in --initial-controllers. Just like the controller.quorum.voters properties, it is important that the set of voters is equal in all of the controllers with the same cluster id.


		After sarting the controller, the replication to the new controller can be monitored using the <code>kafka-metadata-quorum describe --replication</code> command. Once the new controller has caught up to the active controller, it can be added to the cluster using the <code>kafka-metadata-quorum add-controller</code> command.

		<pre><code class="language-bash">kafka-metadata-quorum --command-config controller.properties --bootstrap-server localhost:9092 add-controller</code></pre>

		@@ -0,0 +1,132 @@
		# Licensed to the Apache Software Foundation (ASF) under one or more

KAFKA-17048; Add docs for KIP-853 #17076

KAFKA-17048; Add docs for KIP-853 #17076

Conversation

jsancio commented Sep 3, 2024

Committer Checklist (excluded from commit message)

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahuang98 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahuang98 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmccabe Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmccabe left a comment

Choose a reason for hiding this comment

cmccabe left a comment

Choose a reason for hiding this comment

cmccabe commented Sep 18, 2024

cmccabe Sep 13, 2024 •

edited

Loading