-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
KAFKA-18601: Assume a baseline of 3.3 for server protocol versions #18845
base: trunk
Are you sure you want to change the base?
KAFKA-18601: Assume a baseline of 3.3 for server protocol versions #18845
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copilot reviewed 61 out of 78 changed files in this pull request and generated no comments.
Files not reviewed (17)
- clients/src/main/resources/common/message/AlterPartitionRequest.json: Language not supported
- clients/src/main/resources/common/message/AlterPartitionResponse.json: Language not supported
- core/src/main/scala/kafka/cluster/Partition.scala: Language not supported
- core/src/main/scala/kafka/server/AlterPartitionManager.scala: Language not supported
- core/src/main/scala/kafka/server/ControllerRegistrationManager.scala: Language not supported
- core/src/main/scala/kafka/server/KafkaConfig.scala: Language not supported
- core/src/main/scala/kafka/tools/StorageTool.scala: Language not supported
- core/src/main/scala/kafka/tools/TestRaftServer.scala: Language not supported
- core/src/test/scala/integration/kafka/coordinator/transaction/ProducerIntegrationTest.scala: Language not supported
- core/src/test/scala/integration/kafka/server/KRaftClusterTest.scala: Language not supported
- core/src/test/scala/integration/kafka/server/MetadataVersionIntegrationTest.scala: Language not supported
- core/src/test/scala/kafka/server/RemoteLeaderEndPointTest.scala: Language not supported
- core/src/test/scala/unit/kafka/cluster/PartitionLockTest.scala: Language not supported
- core/src/test/scala/unit/kafka/cluster/PartitionTest.scala: Language not supported
- core/src/test/scala/unit/kafka/server/AbstractApiVersionsRequestTest.scala: Language not supported
- core/src/test/scala/unit/kafka/server/AlterPartitionManagerTest.scala: Language not supported
- core/src/test/scala/unit/kafka/server/BrokerRegistrationRequestTest.scala: Language not supported
Comments suppressed due to low confidence (3)
clients/src/test/java/org/apache/kafka/common/requests/AlterPartitionRequestTest.java:46
- The removal of the topicName field might lead to incomplete test coverage if any tests rely on it. Verify if topicName is essential for any test cases.
TopicData topicData = new TopicData().setTopicId(topicId);
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java:1705
- The removal of the 'topicName' field may cause unintended side effects if the 'topicName' is required elsewhere in the codebase. Please verify if 'topicName' is needed.
.setTopicId(Uuid.randomUuid())
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java:1723
- The removal of the 'topicName' field may cause unintended side effects if the 'topicName' is required elsewhere in the codebase. Please verify if 'topicName' is needed.
.setTopicId(Uuid.randomUuid())
tools/src/test/java/org/apache/kafka/tools/reassign/ReassignPartitionsCommandTest.java
Show resolved
Hide resolved
3643409
to
a7fbe00
Compare
4b19a3d
to
890c3b2
Compare
@@ -53,7 +53,8 @@ default void print(MetadataNodePrinter printer) { | |||
for (String name : names) { | |||
printer.enterNode(name); | |||
MetadataNode child = child(name); | |||
child.print(printer); | |||
if (child != null) | |||
child.print(printer); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixes a NPE I noticed while debugging a failing test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In reading the code of child, I notice that it seems the AclsImageByIdNode
is misplaced.
private static final Map<String, Function<MetadataImage, MetadataNode>> CHILDREN = Map.of(
...
// AclsImageByIdNode should be replaced by AclsImageNode
AclsImageNode.NAME, image -> new AclsImageByIdNode(image.acls()),
ScramImageNode.NAME, image -> new ScramImageNode(image.scram()),
DelegationTokenImageNode.NAME, image -> new DelegationTokenImageNode(image.delegationTokens())
);
AclsImageNode.NAME, image -> new AclsImageByIdNode(image.acls()), |
and AclsImage#toString
has similar issue.
@Override
public String toString() {
// AclsImageByIdNode should be replaced by AclsImageNode
return new AclsImageByIdNode(this).stringify();
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. Shall we file a separate JIRA ticket for that? Perhaps we can wait for @mumrah to confirm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, it's definitely a bug, but I don't think it relates to this PR. The *Node classes are used by the metadata shell and some toString methods, so they are just informational. I think the exact side effect is that acls would appear at the wrong level of the metadata shell "tree".
I suggest we file a separate issue and fix it for 4.0.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ijuma @mumrah thanks for your response. open https://issues.apache.org/jira/browse/KAFKA-18803 for it
metadata/src/main/java/org/apache/kafka/metadata/bootstrap/BootstrapDirectory.java
Show resolved
Hide resolved
* Test downgrading to a MetadataVersion that doesn't support FeatureLevelRecord. | ||
*/ | ||
@Test | ||
public void testPremodernVersion() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We don't support these anymore, so remove it.
* Test downgrading to a MetadataVersion that doesn't support inControlledShutdown. | ||
*/ | ||
@Test | ||
public void testPreControlledShutdownStateVersion() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We now always have the controlled shutdown state.
890c3b2
to
e654cdb
Compare
9622ce9
to
86d1053
Compare
} | ||
|
||
@Test | ||
public void testKRaftVersions() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This did not seem to add much value after the changes in this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ijuma thanks for this patch. overall LGTM. some minor comments remains.
metadata/src/main/java/org/apache/kafka/metadata/bootstrap/BootstrapDirectory.java
Show resolved
Hide resolved
metadata/src/test/java/org/apache/kafka/metadata/bootstrap/BootstrapMetadataTest.java
Outdated
Show resolved
Hide resolved
metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java
Show resolved
Hide resolved
tools/src/test/java/org/apache/kafka/tools/reassign/ReassignPartitionsCommandTest.java
Show resolved
Hide resolved
metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
Show resolved
Hide resolved
metadata/src/main/java/org/apache/kafka/controller/ReplicationControlManager.java
Show resolved
Hide resolved
metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
Outdated
Show resolved
Hide resolved
metadata/src/main/java/org/apache/kafka/metadata/BrokerRegistration.java
Outdated
Show resolved
Hide resolved
…ssume-baseline-3.3-for-server * apache-github/trunk: KAFKA-18366 Remove KafkaConfig.interBrokerProtocolVersion (apache#18820) KAFKA-18658 add import control for examples module (apache#18812) MINOR: Java version and TLS documentation improvements (apache#18822) KAFKA-18743 Remove leader.imbalance.per.broker.percentage as it is not supported by Kraft (apache#18821) KAFKA-18225 ClientQuotaCallback#updateClusterMetadata is unsupported by kraft (apache#18196) KAFKA-18441: Remove flaky tag on KafkaAdminClientTest#testAdminClientApisAuthenticationFailure (apache#18847) MINOR: fix KStream#to incorrect javadoc (apache#18838) KAFKA-18763: changed the assertion statement for acknowledgements to include only successful acks (apache#18846) MINOR: Accept specifying consumer group assignors by their short names (apache#18832)
@junrao Thanks for the review. I addressed your comments and/or replied within the relevant thread. I also pushed a couple of commits for changes I had in progress: (1) Use Let's see how the test results look after this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @ijuma for this PR, left some commments
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java
Show resolved
Hide resolved
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java
Show resolved
Hide resolved
server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java
Show resolved
Hide resolved
Besides the open discussion threads, there are two additional issues to consider/resolve:
The failures have the following error:
Even though there is a log message just before saying:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ijuma : Thanks for the updated PR. Made a pass of all files. A few more comments.
singletonMap(MetadataVersion.FEATURE_NAME, MetadataVersion.IBP_3_2_IV0.featureLevel()), | ||
singletonMap(MetadataVersion.FEATURE_NAME, FeatureUpdate.UpgradeType.UNSAFE_DOWNGRADE), | ||
true)); | ||
setFeatureLevel(MetadataVersion.IBP_3_4_IV0.featureLevel())); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we use MetadataVersion.MINIMUM_VERSION?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was previously checking that the metadata version would change and using minimum would take it backwards. But I did notice an issue where we are actually using the same version versus a newer one - fixing that.
metadata/src/test/java/org/apache/kafka/controller/FeatureControlManagerTest.java
Outdated
Show resolved
Hide resolved
metadata/src/test/java/org/apache/kafka/controller/FeatureControlManagerTest.java
Outdated
Show resolved
Hide resolved
metadata/src/test/java/org/apache/kafka/controller/FeatureControlManagerTest.java
Outdated
Show resolved
Hide resolved
@@ -163,7 +163,7 @@ public void testEmpty() { | |||
assertFalse(new FeaturesImage(Collections.singletonMap("foo", (short) 1), | |||
FeaturesImage.EMPTY.metadataVersion()).isEmpty()); | |||
assertFalse(new FeaturesImage(FeaturesImage.EMPTY.finalizedVersions(), | |||
MetadataVersion.IBP_3_3_IV0).isEmpty()); | |||
MetadataVersion.IBP_3_4_IV0).isEmpty()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this can be any MV, could we just use MetadataVersion.MINIMUM_VERSION?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That causes the test to fail. This is related to your question of whether FeatuesImage.isEmpty
should be changed not to include metadataVersion.equals(MetadataVersion.MINIMUM_VERSION)
.
metadata/src/test/java/org/apache/kafka/image/loader/MetadataLoaderTest.java
Outdated
Show resolved
Hide resolved
metadata/src/test/java/org/apache/kafka/image/loader/MetadataLoaderTest.java
Outdated
Show resolved
Hide resolved
metadata/src/test/java/org/apache/kafka/image/loader/MetadataLoaderTest.java
Outdated
Show resolved
Hide resolved
metadata/src/test/java/org/apache/kafka/image/MetadataVersionChangeTest.java
Outdated
Show resolved
Hide resolved
server-common/src/test/java/org/apache/kafka/server/common/MetadataVersionTest.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, there are a few references to 3.0 in docs/ops.html that need to be changed.
Feature: metadata.version SupportedMinVersion: 3.0-IV1 SupportedMaxVersion: 3.9-IV0 FinalizedVersionLevel: 3.9-IV0 Epoch: 5
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for @ijuma update, left some comments
metadata/src/main/java/org/apache/kafka/controller/ActivationRecordsGenerator.java
Outdated
Show resolved
Hide resolved
core/src/test/scala/unit/kafka/server/AlterPartitionManagerTest.scala
Outdated
Show resolved
Hide resolved
core/src/test/scala/unit/kafka/server/ApiVersionsResponseIntegrationTest.scala
Outdated
Show resolved
Hide resolved
core/src/test/scala/unit/kafka/server/ApiVersionsResponseIntegrationTest.scala
Outdated
Show resolved
Hide resolved
metadata/src/test/java/org/apache/kafka/controller/FeatureControlManagerTest.java
Outdated
Show resolved
Hide resolved
metadata/src/test/java/org/apache/kafka/controller/FeatureControlManagerTest.java
Outdated
Show resolved
Hide resolved
…the log to try and debug CI-only test failures
… a specific range)
// We automatically fallback to IBP_3_3_IV3 in that case. We use explicit versions instead of `MINIMUM_VERSION` because | ||
// we want to force an explicit decision if we change `MetadataVersion.MINIMUM_VERSION` in the future. | ||
if (record.featureLevel() >= MINIMUM_PERSISTED_FEATURE_LEVEL && record.featureLevel() <= MetadataVersion.IBP_3_3_IV3.featureLevel()) | ||
metadataVersionChange = MetadataVersion.IBP_3_3_IV3; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@junrao @cmccabe Thoughts on this fallback? It seems like quite a rare edge case (since it requires usage of a pre-release version and for the older metadata version not to be compacted away if I understand correctly), but it seems fair to never fail while reading the log and the fallback code is pretty simple.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we only support upgrading from officially released 3.3 to 4.0, we could probably just fail MV 3_3_IV0 to 3_3_IV2 as other unsupported MVs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it guaranteed that we won't have a record with the older metadata version in the log in that case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no guarantee, but the expected MV is IBP_3_3_IV3 and above. If this is not the case, it seems easier to just fail instead of changing it silently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ijuma : Thanks for the updated PR. A few more comments.
|
||
private static final MetadataVersionChange CHANGE_3_3_IV0_TO_3_0_IV1 = | ||
new MetadataVersionChange(IBP_3_3_IV0, IBP_3_0_IV1); | ||
private static final MetadataVersionChange CHANGE_3_6_IV0_TO_3_3_IV3 = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could we change this to use MetadataVersion.MINIMUM_VERSION and MetadataVersion.latestProduction() too?
// We automatically fallback to IBP_3_3_IV3 in that case. We use explicit versions instead of `MINIMUM_VERSION` because | ||
// we want to force an explicit decision if we change `MetadataVersion.MINIMUM_VERSION` in the future. | ||
if (record.featureLevel() >= MINIMUM_PERSISTED_FEATURE_LEVEL && record.featureLevel() <= MetadataVersion.IBP_3_3_IV3.featureLevel()) | ||
metadataVersionChange = MetadataVersion.IBP_3_3_IV3; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we only support upgrading from officially released 3.3 to 4.0, we could probably just fail MV 3_3_IV0 to 3_3_IV2 as other unsupported MVs.
} | ||
return ControllerResult.of(records, false); | ||
}, | ||
() -> ControllerResult.of(List.of(new ApiMessageAndVersion(new NoOpRecord(), (short) 0)), false), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The javadoc above on if the MetadataVersion supports it.
needs to be changed too.
server-common/src/main/java/org/apache/kafka/server/common/UnitTestFeatureVersion.java
Show resolved
Hide resolved
@@ -56,7 +53,7 @@ public FeaturesImage( | |||
|
|||
public boolean isEmpty() { | |||
return finalizedVersions.isEmpty() && | |||
metadataVersion.equals(MetadataVersion.MINIMUM_KRAFT_VERSION); | |||
metadataVersion.equals(MetadataVersion.MINIMUM_VERSION); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
isEmpty()
is only used in MetadataLoaderTest
. So, we can address this in a followup jira. But I am not sure if isEmpty()
will ever be false given that metadata version is required in the metadata log and probably the snapshot.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for @ijuma update, one more comment
metadata/src/main/java/org/apache/kafka/controller/ClusterControlManager.java
Outdated
Show resolved
Hide resolved
a5d0b49
to
65a7166
Compare
…on whether image is empty
b6eabaa
to
70667a4
Compare
…ssume-baseline-3.3-for-server * apache-github/trunk: (36 commits) MINOR: Add KIP-848's metric to the doc (apache#18890) KAFKA-18772 Define share group config defaults for Docker (apache#18899) KAFKA-18733: Updating share group metrics (1/N) (apache#18826) KAFKA-18634: Fix ELR metadata version issues (apache#18680) KAFKA-18298 Fix flaky testConsumerGroupsDeprecatedConsumerGroupState and testConsumerGroups in PlaintextAdminIntegrationTest (apache#18513) MINOR: Marking testVerifyFetchAndCloseImplicit flaky (apache#18893) MINOR: Adjust javadoc to reflect the correct status of standby task TopicPartition (apache#18892) KAFKA-17182: Consumer fetch sessions are evicted too quickly with AsyncKafkaConsumer (apache#18795) KAFKA-16720: Support multiple groups in DescribeShareGroupOffsets RPC (apache#18834) KAFKA-18654[2/2]: Transction V2 retry add partitions on the server side when handling produce request. (apache#18810) MINOR: fix warn log message in Kafka Streams (apache#18878) KAFKA-18776: Fix flaky coordinator disconnect test & fix log level (apache#18866) KAFKA-17298: Update upgrade notes for 4.0 KIP-848 (apache#18756) MINOR bump setup-gradle (apache#18879) MINOR: Run javadoc as part of check task (apache#18863) MINOR: Updated share partition manager tests to close and other fixes (apache#18862) MINOR: Fix typo in ClusterControlManager (apache#18886) KAFKA-18728 Move ListOffsetsPartitionStatus to server module (apache#18807) MINOR: cleanup KStream JavaDocs (14/N) - stream-globalTable-left-join (apache#18867) MINOR: cleanup KStream JavaDocs (13/N) - stream-stream-outer-join (apache#18865) ...
…ssume-baseline-3.3-for-server * apache-github/trunk: MINOR: Add release notes for Transactions Server Side Defense (KIP-890) (apache#18896) MINOR: TransactionManager logs the epoch bump less frequently. (apache#18895) MINOR: Mark IBP_4_0_IV3 as production ready! (apache#18902)
@@ -100,7 +100,7 @@ public boolean hasSeenRecord() { | |||
*/ | |||
public final void resetToImage(MetadataImage image) { | |||
this.image = image; | |||
this.hasSeenRecord = true; | |||
this.hasSeenRecord = !image.isEmpty(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was an existing issue that somehow became easier to trigger due to these changes. Thanks to @mumrah for figuring out that this was the reason for the remaining failures.
3.3.0 was the first KRaft release that was deemed production-ready and also when KIP-778 (KRaft to KRaft upgrades) landed. Given that, it's reasonable for 4.x to only support upgrades from 3.3.0 or newer.
Noteworthy changes:
AlterPartition
no longer includes topic names, which makes it possible to simplifyAlterParitionManager
logic.IBP_3_3_IV3
have been removed andIBP_3_3_IV3
is now the minimum version.MINIMUM_BOOTSTRAP_VERSION
has been removed.isLeaderRecoverySupported
,isNoOpsRecordSupported
,isKRaftSupported
,isBrokerRegistrationChangeRecordSupported
andisInControlledShutdownStateSupported
- these are alwaystrue
now. Also removed related conditional code.TODO:
Committer Checklist (excluded from commit message)