Consumer stream #695
Closed
Consumer stream #695
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@hyperlink Is it normal to see sporadic test failures? Locally and via travis, we're seeing intermittent issues with brokers not being available. |
|
Thanks for the PR @lliss! Could you add some documentation and examples on using the stream? Regarding the tests I'm seeing the failures happen more often now that the tests run against different kafka versions though I haven't had a chance to investigate it. |
|
@lliss if you rebase you'll find the tests to be more stable. I will look at this PR in more detail soon. |
…multiple fetches.
The commit stream is a writable stream that you can pipe events to and it will commit them. This is useful so that you can pipe the ConsumerStream through a transform stream and be completely confident that the only way you will commit an offset is by successfully processing the message.
…stream so that the messages consumed can be observed.
…rm stream mode if and when it is piped to another source and cleaning up the _trasnform logic.
Merged
|
Thanks for your contribution. Closing this PR since this is continued in #732 any discussions can done in that PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Designed to solve #371, this change reimplements the processing of data as a stream in such a way that back pressure should not cause dropped messages. The change is designed to be fully backwards compatible.