Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backpressure support for most BufferUntilSubscriber use places. #3050

Closed

Conversation

akarnokd
Copy link
Member

I've added backpressure support to BufferUntilSubscriber, however, the internal workings of groupBy were in conflict with my 'clean' approach. Therefore, I kept the original BufferUntilSubscriber use in groupBy and added BufferUntilSubscriberV2 and made the various window() operators use it instead.

I've also added a two unbounded Spsc queue implementations from JCTools which can reduce the allocation-rate of these linked-node queues.

The problem with groupBy seems to be that it sets a producer on the Subscriber before it is actually subscribed to the BufferUntilSubscriber. The V2 however sets its on producer to support proper replay and kicks out the original coordinator producer. I don't know how to make the two work together yet. Maybe the groupBy has to be rewritten from scratch and inline the group backpressure support, request coordination and queueing more directly. An alternative might be to expose the single child request amount in BufferUntilSubscriber through a callback to the group somehow.

@benjchristensen
Copy link
Member

Holding off on this ... it's is a non-trivial change and I don't have time to review it right now.

And yes, groupBy is a very tricky one to solve. We have rewritten it a couple times already. There are some very tricky edge cases such as "time gap" and head-of-line-blocking.

@benjchristensen
Copy link
Member

@akarnokd I'm about to review this with @stevegury but before we do, do you want anything changed on this? It does need to be rebased at a minimum if it's still all valid.

@akarnokd
Copy link
Member Author

No changes planned to this. I need to manually reapply the changes to a fresh version because my Eclipse's git is broken. Maybe it is better to review a mergeable version instead of this.

@akarnokd
Copy link
Member Author

See #3150.

@akarnokd akarnokd closed this Aug 12, 2015
@akarnokd akarnokd deleted the BufferUntilSubscriberBackpressure branch September 9, 2015 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants