Frequent gRPC StatusCode.UNAVAILABLE errors

[rishsriv]

Of late, a small but significant (5-10%) of fetch operations to the Cloud Datastore seem to fail. They always succeed upon retry (which seems to be the prescribed way to solve the problem). However, the library should handle the retries IMO.

Here is the error traceback:

Traceback (most recent call last):
  File "/path/to/file.py", line 22, in get_client_id_dets
    res = list(query.fetch())
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/query.py", line 463, in __iter__
    self.next_page()
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/query.py", line 434, in next_page
    transaction_id=transaction and transaction.id,
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/connection.py", line 538, in run_query
    response = self._datastore_api.run_query(project, request)
  File "/usr/local/lib/python2.7/dist-packages/google/cloud/datastore/connection.py", line 280, in run_query
    return self._stub.RunQuery(request_pb)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 481, in __call__
    return _end_unary_response_blocking(state, False, deadline)
  File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 432, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
_Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, {"created":"@1478303817.915316389","
description":"Secure read failed","file":"src/core/lib/security/transport/secure_endpoint.c","file_line":157,"grpc_
status":14,"referenced_errors":[{"created":"@1478303817.915244674","description":"EOF","file":"src/core/lib/iomgr/t
cp_posix.c","file_line":235}]})>

I'm using Ubuntu 16.04 on Google Compute Engine, with python 2.7.12 and google cloud version 0.20. I am making the request from a compute instance in the Asia region.

[daspecster]

@rishsriv thanks for reporting!

There has been talk of adding retry with exponential backoff to the library.
Most recently, #2684 (comment).

I would suggest using exponential backoff with your favorite Python retry library for now.

There are a couple that seem to be alright.

• https://pypi.python.org/pypi/retrying
• https://pypi.python.org/pypi/retry/

Also, I created #2694 to track where retry is needed and most helpful and design the interface.

[rishsriv]

@daspecter, thanks for the update!

[dhermes]

Closing as a dupe of #2683