-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pubsub acks are unsuccessful #3567
Comments
gRPC bidirectional streaming is not as straightforward as one might think. Here are some details.
[1] I'm not an expert on gRPC implementation, but this is accurate to the best of my understanding. The above problem mostly affects long-running processes. The best fix for this is probably to use polling pull, we plan to bring it back.
These two questions are related. "Expired acks" shouldn't exist. AFAIK, acks should always "work". Even if the message is expired and redelivered, an ack should still stop pubsub from sending it out in the future. Could you let us know what date and time (the graph says 10AM, but I'm not sure which timezone) you're seeing this and what your subscription name is? I'll investigate with pubsub team to see what's going on. |
Thanks for the reply. I did not realize that expired acks behaved that way. Is there documentation for those metric statuses in Stackdriver? E.g., for pull_ack_message_operation_count there's expired and success, but perhaps there's more statuses that I'm not aware of. The above screenshots are from August 16 around 10 AM Pacific Time, but the issue is still happening at the moment (August 20, ~2:45 PM Pacific Time). The subscription is Please let me know if you need anything else. |
@stetra I contacted the pubsub team with your info. I'll keep you posted. |
Hi @stetra, I am trying to debug this issue. |
@pongad you mentioned bringing polling pull back, do you have an ETA on that? |
I don't want to promise anything right now, but I'll try to work on it over the next couple of weeks if my availability opens up. |
I am seeing similar behaviour where I get a lot of expired responses to acks and the number of undelivered messages stops going down. It seems to happen in 5-20 minute intervals. I am using the java client library version 1.45.0. My subscriber is created as follows:
|
#3743 might help. Let's see where that gets us first. |
The processing of messages is also slow in our case. Handing the processing off to another thread pool and keeping the message receiver lightweight appears to have fixed the issue. |
I'm also seeing such behavior that @stetra faced, but now on node.js environment. Therefore it may be not related to exactly java implementation of the library. Here is the link on the question on SO, in case if this might help. https://stackoverflow.com/questions/54597310/google-cloud-pubsub-not-ack-messages |
@stetra This issue will be resolved once we bring back polling pull. Closing this issue and we will track in #3500 |
When I use the streaming pull subscriber through the
MessageReceiver
interface, it seems acks usually expire, and even if they don't expire, the number of backlogged messages never changes (num_undelivered_messages in Stackdriver). This only seems to happen if messages take a while to be processed, i.e., more than a few minutes.It's strange that acks are expired in the first place because I do see that the client is sending modify ack deadline requests (mod_ack_deadline_message_operation_count in Stackdriver). The subscription itself has an ack deadline of 10 minutes so these messages are processed well within the deadline.
Even when acks are successful (i.e., non-expired), num_undelivered_messages is unaffected in Stackdriver. I do see the previously acked messages being redelivered as well.
Another thing I noticed that even though the client is performing streaming pulls, in Stackdriver I see metric data coming in for
pull_ack_message_operation_count
but none forstreaming_pull_ack_message_operation_count
. Similarly I see results formod_ack_deadline_message_operation_count
but not forstreaming_pull_mod_ack_deadline_message_operation_count
. Is this expected?I am using version 1.37.1 of the google-cloud-pubsub library. I have uploaded my code here: https://gist.github.com/stetra/d757fed41cb67d4a73dd7487ca4d452e
The following Stackdriver graphs show that successful acks are apparently happening, yet the backlog is not decreasing.
In summary, these are my questions:
Despite successful acks occurring, why is the number of unacknowledged messages not decreasing?
Why are expired acks occurring in the first place?
Why is the client performing streaming pulls with non-streaming acks and modacks?
The text was updated successfully, but these errors were encountered: