Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

invalid_scope: Empty or missing scope not allowed #83

Closed
rabernat opened this issue Feb 26, 2018 · 10 comments
Closed

invalid_scope: Empty or missing scope not allowed #83

rabernat opened this issue Feb 26, 2018 · 10 comments

Comments

@rabernat
Copy link
Contributor

rabernat commented Feb 26, 2018 •
edited
Loading

After overcoming #82, I am on to a new error.

I set my GOOGLE_APPLICATION_CREDENTIALS to point to an appropriate .json file. I verified that the config is working with the following code

In [10]: from google.cloud import storage

In [11]: storage_client = storage.Client()

In [12]: list(storage_client.list_buckets())
Out[12]:
[<Bucket: pangeo>,
 <Bucket: pangeo-data>,
 <Bucket: pangeo-data-private>,
 <Bucket: zarr_store_test>]

Now I am trying to do the same thing from gcsfs. I get an error invalid_scope: Empty or missing scope not allowed.

In [8]: fs = gcsfs.GCSFileSystem()

In [9]: fs.buckets
DEBUG:gcsfs.core:_list_buckets(args=(), kwargs={})
DEBUG:gcsfs.core:_call(args=('get', 'b/'), kwargs={'project': 'pangeo-181919'})
ERROR:gcsfs.core:_call exception: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
Traceback (most recent call last):
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py", line 431, in _call
    r = meth(self.base + path, params=kwargs, json=json)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request
    self._auth_request, method, url, request_headers)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/service_account.py", line 322, in refresh
    request, self._token_uri, assertion)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 145, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 111, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 61, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
ERROR:gcsfs.core:_call exception: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
Traceback (most recent call last):
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py", line 431, in _call
    r = meth(self.base + path, params=kwargs, json=json)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request
    self._auth_request, method, url, request_headers)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/service_account.py", line 322, in refresh
    request, self._token_uri, assertion)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 145, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 111, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 61, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
ERROR:gcsfs.core:_call exception: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
Traceback (most recent call last):
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py", line 431, in _call
    r = meth(self.base + path, params=kwargs, json=json)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request
    self._auth_request, method, url, request_headers)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/service_account.py", line 322, in refresh
    request, self._token_uri, assertion)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 145, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 111, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 61, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
ERROR:gcsfs.core:_call exception: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
Traceback (most recent call last):
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py", line 431, in _call
    r = meth(self.base + path, params=kwargs, json=json)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py", line 198, in request
    self._auth_request, method, url, request_headers)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py", line 121, in before_request
    self.refresh(request)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/service_account.py", line 322, in refresh
    request, self._token_uri, assertion)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 145, in jwt_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 111, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py", line 61, in _handle_error_response
    error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')
---------------------------------------------------------------------------
RefreshError                              Traceback (most recent call last)
<ipython-input-9-bb76ad6e7216> in <module>()
----> 1 fs.buckets

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in buckets(self)
    449     def buckets(self):
    450         """Return list of available project buckets."""
--> 451         return [b["name"] for b in self._list_buckets()["items"]]
    452
    453     @classmethod

<decorator-gen-122> in _list_buckets(self)

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs)
     49         logger.log(logging.DEBUG - 1, tb_io.getvalue())
     50
---> 51     return f(self, *args, **kwargs)
     52
     53

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _list_buckets(self)
    568         items = []
    569         page = self._call(
--> 570             'get', 'b/', project=self.project
    571         )
    572

<decorator-gen-117> in _call(self, method, path, *args, **kwargs)

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs)
     49         logger.log(logging.DEBUG - 1, tb_io.getvalue())
     50
---> 51     return f(self, *args, **kwargs)
     52
     53

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _call(self, method, path, *args, **kwargs)
    435                 logger.exception("_call exception: %s", e)
    436                 if retry == self.retries - 1:
--> 437                     raise e
    438                 if is_retriable(e):
    439                     # retry

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/gcsfs/core.py in _call(self, method, path, *args, **kwargs)
    429             try:
    430                 time.sleep(2**retry - 1)
--> 431                 r = meth(self.base + path, params=kwargs, json=json)
    432                 validate_response(r, path)
    433                 break

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/requests/sessions.py in get(self, url, **kwargs)
    519
    520         kwargs.setdefault('allow_redirects', True)
--> 521         return self.request('GET', url, **kwargs)
    522
    523     def options(self, url, **kwargs):

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/transport/requests.py in request(self, method, url, data, headers, **kwargs)
    196
    197         self.credentials.before_request(
--> 198             self._auth_request, method, url, request_headers)
    199
    200         response = super(AuthorizedSession, self).request(

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/auth/credentials.py in before_request(self, request, method, url, headers)
    119         # the http request.)
    120         if not self.valid:
--> 121             self.refresh(request)
    122         self.apply(headers)
    123

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/service_account.py in refresh(self, request)
    320         assertion = self._make_authorization_grant_assertion()
    321         access_token, expiry, _ = _client.jwt_grant(
--> 322             request, self._token_uri, assertion)
    323         self.token = access_token
    324         self.expiry = expiry

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py in jwt_grant(request, token_uri, assertion)
    143     }
    144
--> 145     response_data = _token_endpoint_request(request, token_uri, body)
    146
    147     try:

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py in _token_endpoint_request(request, token_uri, body)
    109
    110     if response.status != http_client.OK:
--> 111         _handle_error_response(response_body)
    112
    113     response_data = json.loads(response_body)

/nobackup/rpaberna/conda/envs/pangeo/lib/python3.6/site-packages/google/oauth2/_client.py in _handle_error_response(response_body)
     59
     60     raise exceptions.RefreshError(
---> 61         error_details, response_body)
     62
     63

RefreshError: ('invalid_scope: Empty or missing scope not allowed.', '{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')

I am using the latest gcsfs master, installed from pip + git.
@rabernat
Copy link
Contributor Author

Seems related: googleapis/google-auth-library-python#217

@martindurant
Copy link
Member

The "service" route does already have the scopes set ( https://github.com/dask/gcsfs/blob/master/gcsfs/core.py#L370 ), but - shooting in the dark here - I can add it also to the file-based auth. I wouldn't have thought that made a difference, but you could try git+https://github.com/martindurant/gcsfs@scoped. If not, it would be useful to know via pdb what fs.method ended up as.

@rabernat
Copy link
Contributor Author

Ok, I will try that branch later today or tomorrow.

Do you recommend a different method of authentication?

@martindurant
Copy link
Member

If you have a .json file, you can always pass that directly to the constructor GCSFileSystem(token='/path/to/file.json', project='my-project'), and should also explicitly give the appropriate project=.

In the past, google-default and cloud auth mechanisms have worked without problem. If you give no indication to GCSFileSystem at all, it will try the various methods in turn, and it seems that something apparently gave no error (some auth was found), but the auth was actually incomplete. Or something. I would expect token='cloud' to be the smoothest method on GCE.

@rabernat
Copy link
Contributor Author

rabernat commented Feb 27, 2018 •
edited
Loading

I am not on GCE. I am on pleiades.

I will try passing the token explicity.

@rabernat
Copy link
Contributor Author

It looks like specifying the token='/path/to/file.json' works. Sorry for the clutter.

However, it would be great if gcsfs would read the GOOGLE_APPLICATION_CREDENTIALS environment variable and use this to fill in token. I could try to put together a PR for this if you wish.

@martindurant
Copy link
Member

That's an impressive cluster!

I agree that GOOGLE_APPLICATION_CREDENTIALS should be respected, but I am surprised that gauth.default() doesn't pick this up already. Your original point about providing a more sensible error message, preferably when connecting as opposed to the first usage, still stands - but I am not sure how to achieve this.

@martindurant
Copy link
Member

OK, so what I understand is that google.auth.default() is supposed to respect GOOGLE_APPLICATION_CREDENTIALS, but "old" credentials can be missing scopes and/or project identifiers. Perhaps google.cloud.storage.Client knows how to deal with that. I can add the scopes to the default credentials, which apparently will use them "if necessary" - doesn't actually seem to make any difference to me.

@martindurant martindurant mentioned this issue Feb 27, 2018
Merged
@rabernat
Copy link
Contributor Author

rabernat commented Feb 27, 2018 •
edited
Loading

I just created / downloaded the credentials yesterday. How can they be "old" already?

@martindurant
Copy link
Member

I just created / downloaded the credentials yesterday. How can they be "old" already?

I don't know :( It's hard to try to say what the google auth library actually did for your case.

@blueandgold blueandgold mentioned this issue May 21, 2018
Closed
@JacobHayes JacobHayes mentioned this issue Dec 14, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rabernat @martindurant