messages sit in queue until GKE pod with subscriber gets reset

[stephenplusplus]

From @ShahNewazKhan on October 1, 2017 9:3

Environment details

• OS: Debian GNU/Linux 8.9 (jessie) [K8s pod based on dockerfile gcr.io/google_appengine/base]
• Node.js version: 6.11.3
• npm version: 5.4.2
• google-cloud/pubsub version: 0.14.2

Steps to reproduce

1. Spin up nodejs pubsub publisher to topic1 in GKE pod 1
2. Spin up nodejs pubsub subscriber to subscription to topic1 in GKE pod 2
3. Publish messages to topic1

I am facing an intermittent issue where pubsub messages are sitting in the queue and not being delivered to the subscriber in GKE pod 2. Only when I delete the GKE pod 2 subscriber and restart the pod does the message get delivered.

Copied from original issue: googleapis/google-cloud-node#2640

[stephenplusplus]

From @callmehiphop on October 1, 2017 15:50

We've seen a number of reports of messages not being delivered in k8s, I believe this issue is being investigated internally, although I do not know the status. @lukesneeringer have we heard any news in regards to this?

[stephenplusplus]

From @eyalse on October 1, 2017 22:33

I'm suffering from the same issue at the moment :(

[stephenplusplus]

From @ApeNox on October 2, 2017 8:5

Suffering the same issue too, please provide a fix as these are production used tools.

[stephenplusplus]

From @eyalse on October 2, 2017 9:26

@callmehiphop (@lukesneeringer) hey any update? as mentioned these tools (k8s and pubsub) are used in production.

[stephenplusplus]

From @callmehiphop on October 2, 2017 14:54

I don't have any official updates, but a new patch release was made this morning that might resolve the issues you're seeing.

[stephenplusplus]

From @ShahNewazKhan on October 2, 2017 20:25

@callmehiphop I have done some preliminary testing with the google-cloud/pubsub patch version: 0.14.3 release this morning and it looks promising so far

I have not been able to reproduce the issue yet however will need to run full end to end tests to confirm

[stephenplusplus]

From @callmehiphop on October 2, 2017 20:34

@ShahNewazKhan that's great, please keep us posted! 😃

[stephenplusplus]

From @ShahNewazKhan on October 3, 2017 1:42

@callmehiphop I have been able to replicate the issue with google-cloud/pubsub patch 0.14.3 in a slightly different use case.

Environment details

OS: Debian GNU/Linux 8.9 (jessie) [K8s pod based on dockerfile gcr.io/google_appengine/base]
Node.js version: 6.11.3
npm version: 5.4.2
google-cloud/pubsub version: 0.14.3

Steps to reproduce

1. Spin up nodejs pubsub publisher to topic1 in GKE pod 1
   

2. Spin up nodejs pubsub subscriber to subscription to topic1 in GKE pod 2
   

3. Reset GKE pod 1 [pubsub publisher app]
   

4. Publish messages to topic1
   

At this point the message remains stuck in the pubsub queue until I reset the GKE pod 2 [pubsub subscriber app]

[stephenplusplus]

From @ShahNewazKhan on October 10, 2017 21:41

Just checking in for updates on this issue.

[stephenplusplus]

From @callmehiphop on October 11, 2017 18:27

@ShahNewazKhan We believe this is a GKE issue and because of that I can't comment on if its being worked on and when it will be fixed. I'm really sorry for the inconvenience.

[stephenplusplus]

From @ehacke on October 16, 2017 1:25

We may be having similar issues, not sure. @ShahNewazKhan what version of GKE are you on?

[stephenplusplus]

From @ShahNewazKhan on October 16, 2017 2:10

@ehacke

GKE: 1.6.10-gke.1
Kubernetes: 1.5.6

[stephenplusplus]

From @kir-titievsky on October 26, 2017 18:23

Question for those who'd reported this: is there any chance you had no messages published or delivered for 10 minutes or longer before you started publishing and accumulating them in the backlog?

[stephenplusplus]

From @ShahNewazKhan on October 26, 2017 19:2

@kir-titievsky I can confirm that the published messages sit in the subscription queue only when the publisher has been inactive longer than 10 minutes.

[stephenplusplus]

From @kir-titievsky on October 26, 2017 19:47

Thanks @ShahNewazKhan . My guess here is this: by default, GCE suspends inactive connections after 10 minutes [1]. Since Pub/Sub relies on a persistent streamingPull connection, this connection would get suspended if no messages flow for 10 minutes. This condition was not properly detected by Pub/Sub. This was fixed as of 2017-10-20 by shutting down affected streamingPull connections. The server-initiated shutdown should now trigger the client library to rebuild the connection.

Can those of you affected check if the issue persists?

[1] https://cloud.google.com/compute/docs/troubleshooting#communicatewithinternet

[stephenplusplus]

From @ShahNewazKhan on October 31, 2017 22:56

@kir-titievsky Can you clarify what you mean by 'server-initiated shutdown'. Does this mean that the inactive Pub/Sub streamingPull connections are now being shutdown instead of being suspended by GCE?

I have noticed messages sitting in the queue intermittently still, do I have to update the Pub/Sub client to a latest version to handle the streamingPull connection rebuilds?

Thanks in advance!

[stephenplusplus]

I'm marking this as blocked, since it sounds like GKE is the party responsible for any progress on this. @callmehiphop does this sound right?

[stephenplusplus]

From @callmehiphop on November 27, 2017 15:37

@stephenplusplus I believe it does!

[kir-titievsky]

@ShahNewazKhan: Pub/Sub servers now close streamingPull connections regularly, with a timeout shorter than GCE's 10 minute limit. This allow the client library to quickly rebuilt the connections making sure that none are stuck in a suspended state.

[stephenplusplus]

Please let us know if there are still any issues. For now, this sounds like it's resolved.

[thomas-hilaire]

Hello,

I continue to encounter this issue regularly. As you my pod stops to receive any message after an inactive period, as soon as I restart the pod all messages are well delivered to this new pod instance.

Do I need to update anything to get the fix explained by @kir-titievsky? How can I investigate about this issue?

Thanks!

Environment details:

• google-cloud/pubsub: 0.15.0
• node: 8.4

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.5-gke.0", GitCommit:"2c2a807131fa8708abc92f3513fe167126c8cce5", GitTreeState:"clean", BuildDate:"2017-12-19T20:05:45Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"}

[janisto]

We have experienced the same issue in the past few weeks with new deployments. Older deployments seem to work fine.

A bit annoying to restart the pods daily.

Environment details:

• google-cloud/pubsub: 0.13.1
• node: 6.11

[janisto]

I think this was caused by grpc 1.8.4. I added grpc 1.7.3 as dependency and so far everything seems to be working fine.

[theacodes]

Related: googleapis/google-cloud-python#4737

[domparry]

I have also solved this issue by adding grpc 1.7.3 as a dependency.

[Alexandredc]

Hi, any news on this issue ? Because I encountered the same problem.

I move from Google compute engine to Kubernetes. I have 2 pubsub topics. One is used frequently (eg: many message pushed) and there is no problem. And another is used less frequently (message pushed every 30 minutes), after few minutes, pubsub stops receiving messages.

[callmehiphop]

@alexandreawe what version of the PubSub client are you experiencing these issues with?

[Alexandredc]

@callmehiphop i'm using v0.18.0. I have fixed this issue by looping every 15 minutes and restarting the subscription

[barrettc]

I'm also seeing this behavior with the latest version of google pubusb as of this comment - 0.22.2. We have two apps running in GKE communicating with each other via pubsub and the subscribers just stop receiving messages until the pods are restarted. At this point I guess I'm looking at looping and restarting the subscription every 15 minutes as described above but this feels very hacky.

[dinvlad]

I'd like to chime in and point out that we're experiencing a very similar issue in our on-prem service, where streaming pull connections stop "restarting every 10 min" after a few times (typically in about 30 min). That seems to be correlated with modAcks/acks not working as expected (i.e. all acks are indicated as "expired" after streaming pulls stop). This is described in #314 (comment)

Additionally, we've experienced a very similar issue with the Java client on GKE a few months ago, and it was resolved as a "server-side" problem on behalf of Google. Is there a chance we're seeing it pop up again here?

EDIT: It appears that only versions 0.23.0 and up are affected. With 0.22.2, only a fraction of acks turns up "expired", and streaming pulls don't stop.

[sduskis]

@plamut, this is a similar problem to the one you're seeing on Python.

[camsjams]

@alexandreawe @barrettc
Just for my sake and perhaps other lurkers, how are you restarting your subscriptions?

Are you doing something straightforward like close and then open within a setInterval like so:

// assuming topic
let topic = pubsub.topic('dogs');

let subscription = subscriptionTopic.subscription('my-subscription');
setInterval(async () => {
    await subscription.close();
    subscription.open();
}, 15 * 60 * 1000);

I would love to hear about your experiences ~six months later and if this has been reliable enough.

Thanks!

[barrettc]

@camsjams For our particular issue, it turns out we had a code path in which messages were not being properly acknowledged and the buildup of those messages created the perception that the subscriber had stopped working. In other words, we had stupid programmer error. I had forgotten about the comment I wrote back in December searching for answers but ultimately we did not have the problem described here in this issue.

[camsjams]

Thank you for your response. In our code we have a thin wrapper mediating PubSub activity, so it "always" acknowledges.