Datastore on GCE - Intermittent Unauthorized 401

[jmgirven]

There are a few questions similar to this, but I don't think any are the same. I apologise if I didn't recognise it.

I am using Datastore inside a docker on a Google Compute Engine instance. The docker container is based off of python:3.4-slim. The python version is Python 3.4.6. I have used pip to install google-cloud-datastore==1.0.0. My GCE instance explicitly has Cloud API access scope Cloud Datastore enabled. I have a worker that regularly (10s interval) checks my datastore for new tasks to process. This makes a query like:

    query = ds_client.query(kind=RepeatingJob.KIND)
    query.add_filter("deleted", "=", False)
    query.add_filter("next_process", "<", datetime.now())
    jobs = list(query.fetch(limit=1))

where ds_client is initialised once in the application:

from google.cloud import datastore
ds_client = datastore.Client()

If I understand correctly, because this is GCE instance and I have enabled the datastore scope, I should not have to do anything more for authorization.

This system works fine for tens of minutes at a time (40 minutes 21 seconds) in my last test. Then an error is raised:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/google/gax/retry.py", line 120, in inner
    return to_call(*args)
  File "/usr/local/lib/python3.4/site-packages/google/gax/retry.py", line 68, in inner
    return a_func(*updated_args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/grpc/_channel.py", line 507, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/usr/local/lib/python3.4/site-packages/grpc/_channel.py", line 455, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
    yield
  File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 134, in run_query
    return super(GAPICDatastoreAPI, self).run_query(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 258, in run_query
    return self._run_query(request, options)
  File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 419, in inner
    return api_caller(api_call, this_settings, request)
  File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 407, in base_caller
    return api_call(*args)
  File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 368, in inner
    return a_func(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/google/gax/retry.py", line 126, in inner
    ' classified as transient', exception)
google.gax.errors.RetryError: GaxError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.UNAUTHENTICATED, Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.)>)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/application/jobs/models/repeating_job.py", line 199, in needs_processing
    jobs = list(query.fetch(limit=1))
  File "/usr/local/lib/python3.4/site-packages/google/cloud/iterator.py", line 218, in _items_iter
    for page in self._page_iter(increment=False):
  File "/usr/local/lib/python3.4/site-packages/google/cloud/iterator.py", line 247, in _page_iter
    page = self._next_page()
  File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/query.py", line 497, in _next_page
    query=query_pb,
  File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 134, in run_query
    return super(GAPICDatastoreAPI, self).run_query(*args, **kwargs)
  File "/usr/local/lib/python3.4/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 82, in _catch_remap_gax_error
    six.reraise(error_class, new_exc, sys.exc_info()[2])
  File "/usr/local/lib/python3.4/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 74, in _catch_remap_gax_error
    yield
  File "/usr/local/lib/python3.4/site-packages/google/cloud/datastore/_gax.py", line 134, in run_query
    return super(GAPICDatastoreAPI, self).run_query(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/google/cloud/gapic/datastore/v1/datastore_client.py", line 258, in run_query
    return self._run_query(request, options)
  File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 419, in inner
    return api_caller(api_call, this_settings, request)
  File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 407, in base_caller
    return api_call(*args)
  File "/usr/local/lib/python3.4/site-packages/google/gax/api_callable.py", line 368, in inner
    return a_func(*args, **kwargs)
  File "/usr/local/lib/python3.4/site-packages/google/gax/retry.py", line 126, in inner
    ' classified as transient', exception)
google.cloud.exceptions.Unauthorized: 401 Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

At this point, if I immediately retry the request, it fails with the same error. At some point a short (maybe 1 minute) period of time later, the same command starts working again.

I also use Sentry error reporting to track issues. Perhaps interestingly, I noticed that in the tracked variables at this point (/usr/local/lib/python3.4/site-packages/google/gax/retry.py in inner at line 126) that env = RetryError('Retry total timeout exceeded before anyresponse was received',). I don't think I can share that report, but if it is relevant, let me know and I will see what I can do.

[jmgirven]

$ pip freeze
appdirs==1.4.3
blinker==1.4
cachetools==2.0.0
click==6.7
contextlib2==0.5.5
dill==0.2.6
enum34==1.1.6
Flask==0.12.1
future==0.16.0
gapic-google-cloud-datastore-v1==0.15.3
google-auth==1.0.0
google-auth-httplib2==0.0.2
google-cloud-core==0.24.0
google-cloud-datastore==1.0.0
google-gax==0.15.8
googleapis-common-protos==1.5.2
grpcio==1.2.1
gunicorn==19.7.1
honcho==1.0.1
httplib2==0.10.3
instagram-private-api==1.2.5
itsdangerous==0.24
Jinja2==2.9.6
MarkupSafe==1.0
oauth2client==3.0.0
packaging==16.8
ply==3.8
proto-google-cloud-datastore-v1==0.90.3
protobuf==3.2.0
pyasn1==0.2.3
pyasn1-modules==0.0.8
pyparsing==2.2.0
raven==6.0.0
requests==2.13.0
rsa==3.4.2
simpleflake==0.1.5
six==1.10.0
Werkzeug==0.12.1

[jmgirven]

Error stops occurring after 5 minutes of failures. Re-occurs after 55-60 minutes of success. (3 tests)

[theacodes]

This should be fixed by https://github.com/GoogleCloudPlatform/google-auth-library-python/releases/tag/v1.0.1 which was just released. Try making sure that package is up-to-date and if this is still an issue, let us know.