Illustrate how max.poll.records affects buffer #865
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tests fail as the cross Scala running leaves messages from the first run on the topic. |
ennru
changed the title
ennru
force-pushed
the
illustrate-partition-buffer
branch
from
3825f5f to
7d3e6f7
Compare
|
Issue described in #872 |
ennru
added a commit
to ennru/alpakka-kafka
that referenced
this pull request
Oct 23, 2019
As the `RebalanceSpec` test in akka#865 shows, the consumer stage's buffer continues to emit messages of revoked partitions that will be re-emitted by a differnt consumer. This adds another `PartitionAssignmentHandler` which will add filtering in the stage so that messages of partitions that were revoked are not issued anymore.
ennru
added a commit
to ennru/alpakka-kafka
that referenced
this pull request
Oct 23, 2019
As the `RebalanceSpec` test in akka#865 shows, the consumer stage's buffer continues to emit messages of revoked partitions that will be re-emitted by a differnt consumer. This adds another `PartitionAssignmentHandler` which will add filtering in the stage so that messages of partitions that were revoked are not issued anymore.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Once messages are fetched from the Kafka broker even a rebalance doesn't drop them from Alpakka Kafka's internal buffer. That buffer can be reduced by setting the
max.poll.recordsto some value lower than the 500 messages.This test case illustrates this.
Background Context
This behaviour leads to duplicated messages to different Alpakka Kafka streams after a rebalance. It would be interesting to explore ways to minimise this behaviour without lowering the
max.poll.records.