Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consumer streams #732

Merged
merged 33 commits into from
Aug 3, 2017
Merged

Consumer streams #732

merged 33 commits into from
Aug 3, 2017

Conversation

hyperlink
Copy link
Collaborator

Continuing work from #695 originally proposed in #371

@hyperlink hyperlink mentioned this pull request Aug 2, 2017
Closed
tizzo and others added 29 commits August 2, 2017 16:46
@tizzo @hyperlink
Stubbing out a consumer stream and test.
48c012e
@tizzo @hyperlink
Creating a stream based consumer class and a simple automated test.
8c0d464
@tizzo @hyperlink
Coding standards and whitespace cleanup.
87200a0
@tizzo @hyperlink
Coding stadnards fix.
4e763fb
@tizzo @hyperlink
Set max bytes to a value small enough to ensure that we will perform …
54aa628 
…multiple fetches.
@tizzo @hyperlink
perform a fetch in the callback from the earlier fetch.
abc3e5a
@tizzo @hyperlink
Adding tests, making lots of fixes.
de73858
@tizzo @hyperlink
Cleaning up tests and comments.
8f069f4
@tizzo @hyperlink
Coding standards cleanup.
710b4d2
@tizzo @hyperlink
Testing substituting the ConsumerStream for the Consumer.
c5de309
@tizzo @hyperlink
Fixing whitespace.
d0124bf
@tizzo @hyperlink
Fix to prevent very large datasets from loading more than they should.
e1e6fa2
@tizzo @hyperlink
Imrpove commenting and
8496ee7
@tizzo @hyperlink
Creating a writable commit stream.
eeb4f10 
The commit stream is a writable stream that you can pipe events to and
it will commit them. This is useful so that you can pipe the
ConsumerStream through a transform stream and be completely confident
that the only way you will commit an offset is by successfully
processing the message.
@tizzo @hyperlink
Coding standards cleanup.
fc14be4
@tizzo @hyperlink
Moving readable event to after everything is up and working.
ef1549d
@tizzo @hyperlink
Removing broken test.
5181d3c
@tizzo @hyperlink
Removing unused constant.
70166bc
@tizzo @hyperlink
Updating consumerStream to optionally pass through data as a through …
ad17b45 
…stream so that the messages consumed can be observed.
@tizzo @hyperlink
Fixing race condition that can cause init to trip up fetches already …
125a6a5 
…in flight.
@tizzo @hyperlink
Tidying comment formatting.
0ee8706
@tizzo @hyperlink
Ensuring that the consumer stream automatically switches into transfo…
d8b02a5 
…rm stream mode if and when it is piped to another source and cleaning up the _trasnform logic.
@hyperlink
Lint cleanup.
c6b9832
@hyperlink
don't use debug directly
3f050a9
@hyperlink
Add docs for consumer streams
38814b7
@hyperlink
Add consumerGroupStream
1472835
@hyperlink
highWaterMark should be customizable for consumerStream
a0decbf
@hyperlink
Add consumerGroupStream and avoid major version change by exposing co…
f4851b7 
…nsumerStream
@hyperlink
remove yarn
828b8ab
@hyperlink hyperlink merged commit 0e68220 into master Aug 3, 2017
@hyperlink hyperlink deleted the consumer-streams branch August 3, 2017 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hyperlink @tizzo