'Stream removed' with GRPC 1.8.0

[schmidt-sebastian]

We have gotten a number of customer complaints since GRPC updated to 1.8.0. This is for the @google-cloud/firestore SDK running on Google Cloud Functions:

https://stackoverflow.com/questions/47856259/firebase-functions-get-error-stream-removed-oncreate-firestore-event&sa=D&usg=AFQjCNHq0OnrOaORoqS0zYJzmDp-lLpAAw
https://groups.google.com/forum/#!msg/firebase-talk/oza-JZMtXlo/DpFGz9BHCQAJ
https://stackoverflow.com/questions/47870711/stream-removed-error-in-firebase-functions

We are going back to 1.7.1 for now.

[schmidt-sebastian]

Dave Gramlich was able to confirm that this issue also manifests itself with GRPC 1.7.1.

[callmehiphop]

There's a good chance I am mistaken as GCF ignores package-lock.json files.

[schmidt-sebastian]

Note that this error occurs for me on Google Cloud Functions every twenty minutes:

https://gist.github.com/schmidt-sebastian/6061f431280b270d34f2974fdee28a05

I am executing a GCF function using Firestore + GRPC 1.8 every 2 minutes and 15 seconds.

I have not seen a single occurrence of this error on GRPC 1.7.1.

[murgatroid99]

I have published version 1.8.4, which includes a change that marks some network errors with the "UNAVAILABLE" status code. I believe that that affects this error, and if so, the cloud client libraries should retry requests that fail this way.

[stephenplusplus]

I believe our libraries already do that; e.g. firestore: https://github.com/googleapis/nodejs-firestore/blob/7cdc1c529735c840b93effb81079ab7581c2f2c1/src/v1beta1/firestore_client_config.json#L4

@schmidt-sebastian how do you want to go about testing that the new gRPC release is a fix?

[murgatroid99]

Right. I meant to say that I think the libraries already retry that status code, so I think they will pick up this fix.

[schmidt-sebastian]

I can see if I can re-create my setup from last time. I basically set up a function on GCF that I triggered every two minutes (which is GCF's idle interval) and saw this error every twenty minutes. I will see if I have cycles today, but I still need to be convincd that Firestore indeed benefits from upgrading to 1.8. Due to these networking problems and serialization errors on our end, we have far exhausted our error budget (as tracked by my gut feeling) when it comes to GCF.

[schmidt-sebastian]

@murgatroid99 @stephenplusplus I have been running Firestore with GRPC 0.8.4 for a couple hours now and have not seen this error again.

[stephenplusplus]

Are you comfortable upgrading?

[schmidt-sebastian]

Are you comfortable upgrading?

I think so (-> yes).

[stephenplusplus]

👍 I'll send the PRs to get back to ^1.8.4