-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pub/Sub: messages stuck in buffer, preventing proper load balancing #17
Comments
Max D has suggested a simple fix for this: close the stream whenever the buffer is full. This would not have worked when we used the same streamingPull connection for acks as for pulls. But using a separate connection for acks makes this feasible. The cost is additional bandwidth, but at least the "stuckness" and load-balancing behavior is sane. |
@saturnism FYI -- and thanks for discovering this. |
This should have been alleviated by googleapis/google-cloud-java#3147 since we'd only open 1 connection by default and fewer messages should get stuck in buffer. |
Not quite enough, since the buffer per 1 connection is still 10MB. So in
my case of < 1000 messages, the behavior still looks ... surprising.
…On Wed, Apr 18, 2018 at 2:12 PM Michael Darakananda < ***@***.***> wrote:
This should have been alleviated by googleapis/google-cloud-java#3147
<googleapis/google-cloud-java#3147>
since we'd only open 1 connection by default and fewer messages should get
stuck in buffer.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<https://github.com/GoogleCloudPlatform/google-cloud-java/issues/3152#issuecomment-382479317>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ARrMFvh8HRkKD_6vjC14LZrd0mnzpCpIks5tp4INgaJpZM4TUbbt>
.
--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub
<https://cloud.google.com/pubsub/overview>
|
@pongad I agree with @kir-titievsky. This issue occurs in my production when there are lots of messages published to a topic. At last, I downgraded to v0.39.0-beta and used PollingSubscriberConnection |
I'm also running into this issue on my own project. Almost same exact criteria as @kir-titievsky laid out above where we only want each instance of our application to be picking up 1 message at a time because the processing time of the message is relatively long, yet variable. We're seeing situations where a single subscriber hoards messages while another is left with nothing to do even though the max element count is set to 1 with a single executor. Are there any updates on this issue? It's looking like we're going to have to change to synchronous polling instead because of this issue. Thanks for your time. |
@kir-titievsky This sounds like a serious bug, not a feature request. Or, am I missing something? |
The "hoarding" is a known limitation of streamingPull (see
https://cloud.google.com/pubsub/docs/pull -- at the very end there), which
is relevant when you have a large backlog of small tasks that take >>100ms
to process. The workaround is to use synchronous pull, documented in the
same place.
…On Thu, Sep 20, 2018 at 3:38 PM Mike Eltsufin ***@***.***> wrote:
@kir-titievsky <https://github.com/kir-titievsky> This sounds like a
serious bug, not a feature request. Or, am I missing something?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://github.com/GoogleCloudPlatform/google-cloud-java/issues/3152#issuecomment-423307245>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ARrMFo1giv9H2jvSM2UO38oYrmDo_6ORks5uc-7JgaJpZM4TUbbt>
.
--
Kir Titievsky | Product Manager | Google Cloud Pub/Sub
<https://cloud.google.com/pubsub/overview>
|
Are there any code snippets to avoid "hoarding" using sync. pull? I'm having something similar to this issue on some jobs. Do I have to take care of extending the ack deadline myself when using sync. pull? |
@roku6185 The discussion about "hoarding" was in regards to the streaming (async) pull. Can you explain your issue with the sync pull? In sync pull you can specify the limit on how many messages to retrieve at a time. So, hoarding should not be an issue, in theory. |
My bad, let my rephrase. I'm using async. pull today and I'm having some kind of problem with retries. I think this has to do with the ack deadlnes not being extended properly. So I'm looking for some code snippet on how to implement the sync. pull in order to avoid this problem. I don't know if my problem is a "hoarding" problem, but my setup is similar to what @justbaum30 described above. |
Problems with streaming pull do exist as laid out in the above discussion. |
@meltsufin @roku6185 @kir-titievsky You can use below settings to avoid getting lot of message buffered in to client.
|
@ajaaym This pretty ugly, but may be a valuable workaround for some people. I'd prefer to see this exposed a bit more conveniently in the Pub/Sub client library. |
@meltsufin yes it is, but can be used until proper solution is out. |
@ajaaym and I discussed the possibility of having a new |
@kir-titievsky and @kamalaboulhosn — is this feature request still relevant? Feels like it might be made obsolete by the recent discussions re: flow control. |
yes, this should now be solved by the flow control api changes. |
Ok, great. I'm just going to close this FR then - let me know if I've misunderstood. :) |
So anyone knows if this feature is available so that example above works as expected? |
https://cloud.google.com/pubsub/docs/pull with the included sample https://cloud.google.com/pubsub/docs/samples/pubsub-subscriber-flow-settings#pubsub_subscriber_flow_settings-java shows using flow control with a Pub/Sub subscriber. I'm not sure which portion of the thread you mean re: works as expected, but hopefully that'll get you started! |
We actually experienced same behavior mentioned in example above. So if you have 3 subscribers on same subscription/topic and you want that each subscriber is processing 1 message at the time which is kind of standard point to point workflow it is not really possible with FlowControlSettings and setting setMaxOutstandingElementCount=1 because as soon as first subscriber gets message it will buffer more than 1 message. Let me know if this is not the case and example above is actually working. |
I opened a new issue to debug the issue you are having, since the original issue was a feature request. |
Repro:
The hypothesis here is that the entire backlog is stuck in the gRPC and other buffers, between the server and the client. So the server thinks the messages are out and being processed, while the client code can't really see the messages. When new clients connect, the server does not have anything to send them. Killing the original client effectively "nacks" the messages in the buffer, by sending a stream close signal to the server. This allows the server to start sending messages to the other clients.
The text was updated successfully, but these errors were encountered: