RefreshError with config.load_kube_config()

[dizcology]

gcloud container clusters get-credentials my-cluster

Confirmed with gcloud container clusters get-credentials my-cluster, and ./kube/config was created.

Then in python:

from kubernetes import client, config
config.load_kube_config()

returns:

RefreshError: ('invalid_scope: Empty or missing scope not allowed.', u'{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')

What did I miss?

---

stack trace:

---------------------------------------------------------------------------
RefreshError                              Traceback (most recent call last)
<ipython-input-1-267adcf41323> in <module>()
      1 from kubernetes import client, config
----> 2 config.load_kube_config()

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/kubernetes/config/kube_config.pyc in load_kube_config(config_file, context, client_configuration, persist_config)
    359         config_file, active_context=context,
    360         client_configuration=client_configuration,
--> 361         config_persister=config_persister).load_and_set()
    362 
    363 

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/kubernetes/config/kube_config.pyc in load_and_set(self)
    251 
    252     def load_and_set(self):
--> 253         self._load_authentication()
    254         self._load_cluster_info()
    255         self._set_config()

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/kubernetes/config/kube_config.pyc in _load_authentication(self)
    174         if not self._user:
    175             return
--> 176         if self._load_gcp_token():
    177             return
    178         if self._load_user_token():

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/kubernetes/config/kube_config.pyc in _load_gcp_token(self)
    194                  _is_expired(provider['config']['expiry']))):
    195             # token is not available or expired, refresh it
--> 196             self._refresh_gcp_token()
    197 
    198         self.token = "Bearer %s" % provider['config']['access-token']

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/kubernetes/config/kube_config.pyc in _refresh_gcp_token(self)
    203             self._user['auth-provider'].value['config'] = {}
    204         provider = self._user['auth-provider']['config']
--> 205         credentials = self._get_google_credentials()
    206         provider.value['access-token'] = credentials.token
    207         provider.value['expiry'] = format_rfc3339(credentials.expiry)

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/kubernetes/config/kube_config.pyc in _refresh_credentials()
    133             credentials, project_id = google.auth.default()
    134             request = google.auth.transport.requests.Request()
--> 135             credentials.refresh(request)
    136             return credentials
    137 

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/google/oauth2/service_account.pyc in refresh(self, request)
    308         assertion = self._make_authorization_grant_assertion()
    309         access_token, expiry, _ = _client.jwt_grant(
--> 310             request, self._token_uri, assertion)
    311         self.token = access_token
    312         self.expiry = expiry

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/google/oauth2/_client.pyc in jwt_grant(request, token_uri, assertion)
    141     }
    142 
--> 143     response_data = _token_endpoint_request(request, token_uri, body)
    144 
    145     try:

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/google/oauth2/_client.pyc in _token_endpoint_request(request, token_uri, body)
    107 
    108     if response.status != http_client.OK:
--> 109         _handle_error_response(response_body)
    110 
    111     response_data = json.loads(response_body)

/Users/yuhanliu/.virtualenvs/hpsearch/lib/python2.7/site-packages/google/oauth2/_client.pyc in _handle_error_response(response_body)
     57 
     58     raise exceptions.RefreshError(
---> 59         error_details, response_body)
     60 
     61 

RefreshError: ('invalid_scope: Empty or missing scope not allowed.', u'{\n  "error" : "invalid_scope",\n  "error_description" : "Empty or missing scope not allowed."\n}')

[mbohlool]

Can you provide these information:
1- gcloud version
2- keep the content of ~/.kube/config when the python client is not working
3- run kubectl (it should work?)
4- run the client to see if it is working
5- see the changes in ~/.kube/config

If kubeconfig succeeded, most probably we need to change our code to run the command line code gcloud provides instead of using google credentials api.

[dizcology]

1- with gcloud -v:

Google Cloud SDK 171.0.0
alpha 2017.03.24
beta 2017.03.24
bq 2.0.25
core 2017.09.11
datalab 20170818
gcloud 
gsutil 4.27
kubectl 

2- removed current ~/.kube/config (since I had already shut down all clusters from GCP console anyways), then ran gcloud container clusters create test-cluster. Resulting ~/.kube/config:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [redacted]
    server: https://[redacted]
  name: [redacted]
contexts:
- context:
    cluster: [redacted]
    user: [redacted]
  name: [redacted]
current-context: [redacted]
kind: Config
preferences: {}
users:
- name: [redacted]
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: [redacted]
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

And indeed config.load_kube_config() raised the same error as I reported above.

3- ran kubectl get nodes and it worked. Now ~/.kube/config became:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [redacted]
    server: https://[redacted]
  name: [redacted]
contexts:
- context:
    cluster: [redacted]
    user: [redacted]
  name: [redacted]
current-context: [redacted]
kind: Config
preferences: {}
users:
- name: [redacted]
  user:
    auth-provider:
      config:
        access-token: [redacted]
        cmd-args: config config-helper --format=json
        cmd-path: [redacted]
        expiry: 2017-09-20T02:38:48Z
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

Note that the access-token and expiry fields were added.

4- python client worked.

5- the only difference in the two config files are indeed those two fields access-token and expiry.

Thanks!

[mbohlool]

It is strange that I did the same steps, even created a new python environment and installed latest version of kubernetes package but couldn't reproduce this. first run of the client adds access-token and expiry for me. Can you do the same to make sure we have the same environment? This is the chain of commands I ran:

virtualenv /tmp
source /tmp/bin/activate
pip install kubernetes
rm ~/.kube/config
gcloud container clusters create test-cluster
cp ~/.kube/config ~/.kube/config.before
python example1.py
diff ~/.kube/config.before ~/.kube/config

if you get different results, please also include all pypi package versions so we can compare.

[dizcology]

When creating a new virtualenv to follow your steps I remembered that I had previously installed python 2.7.13 with homebrew to resolve an issue with OpenSSL causing the kubernetes client not being able to deploy containers to clusters with macOS's stock python 2.7.10.

Looking in brew info python it suggested adding the following to my PATH, which I did not do:

export PATH="/usr/local/opt/python/libexec/bin:$PATH"

So I added it to my .zshrc, and everything works now (config.load_kube_config() correctly connected to the cluster and updated ~/.kube/config). So perhaps an incorrect version of python was used when trying to connect to the cluster without the above export, even when the virtualenv points to the correct python version?

Anyways now I can continue with my project. Thanks!

[mbohlool]

Nice. You are welcome.

[dizcology]

Running into the same issue again even with the setup that used to work (as far as I remember). Below is the print out from pip freeze, in case it helps:

appnope==0.1.0
backports-abc==0.5
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.6.0
bleach==2.0.0
bs4==0.0.1
cachetools==2.0.1
certifi==2017.7.27.1
chardet==3.0.4
configparser==3.5.0
cycler==0.10.0
decorator==4.1.2
entrypoints==0.2.3
enum34==1.1.6
functools32==3.2.3.post2
futures==3.1.1
google-api-python-client==1.6.3
google-auth==1.1.1
google-cloud-core==0.27.1
google-cloud-storage==1.4.0
google-resumable-media==0.2.3
googleapis-common-protos==1.5.2
html5lib==0.999999999
httplib2==0.10.3
idna==2.6
ipaddress==1.0.18
ipykernel==4.6.1
ipython==5.5.0
ipython-genutils==0.2.0
ipywidgets==7.0.1
Jinja2==2.9.6
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.2.0
jupyter-core==4.3.0
kubernetes==3.0.0
MarkupSafe==1.0
matplotlib==2.0.2
mistune==0.7.4
nbconvert==5.3.1
nbformat==4.4.0
notebook==5.1.0
numpy==1.13.1
oauth2client==4.1.2
olefile==0.44
pandocfilters==1.4.2
pathlib2==2.3.0
pexpect==4.2.1
pickleshare==0.7.4
Pillow==4.2.1
prompt-toolkit==1.0.15
protobuf==3.4.0
ptyprocess==0.5.2
pyasn1==0.3.6
pyasn1-modules==0.1.4
PyGithub==1.35
Pygments==2.2.0
PyJWT==1.5.3
pyparsing==2.2.0
python-dateutil==2.6.1
pytz==2017.2
PyYAML==3.12
pyzmq==16.0.2
qtconsole==4.3.1
requests==2.18.4
rsa==3.4.2
scandir==1.5
scikit-learn==0.19.0
scipy==0.19.1
simplegeneric==0.8.1
singledispatch==3.4.0.3
six==1.11.0
sklearn==0.0
subprocess32==3.2.7
termcolor==1.1.0
terminado==0.6
testpath==0.3.1
tornado==4.5.2
traitlets==4.3.2
uritemplate==3.0.0
urllib3==1.22
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.40.0
widgetsnbextension==3.0.3

[jlewi]

I hit the same issue when using a service account (by setting environment variable GOOGLE_APPLICATION_CREDENTIALS).

I suspect this is an issue with google-auth-library-python and not the Kubernetes Python Client library

[jlewi]

I opened up googleapis/google-auth-library-python#217

[jlewi]

The response to googleapis/google-auth-library-python#217 was that this is working as intended.

I suspect https://www.googleapis.com/auth/userinfo.email is needed in order to obtain an identity token to use to authenticate to the K8s master. I wouldn't expect the Kubernetes client talks to any other GCP services so userinfo.email might be sufficient.

I actually need cloud-platform scope otherwise requests to K8s master returned unauthorized.

I was able to work around this by piping in my own credential refresher so I could set the scopes see here.

Some options for better supporting this would be

1. Modifying kube_config.py to always set the scope e.g.

    def _refresh_credentials():
          credentials, project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform"])
          request = google.auth.transport.requests.Request()
          credentials.refresh(request)
          return credentials
   

2. Modify load_kube_config so it passes through unknown keyword args to _get_kube_config_loader_for_yaml_file

3. In combination with the previous option, add an optional scopes argument to KubeConfigLoader and use it in _refresh_credentials

[kevinmu]

Hm I seem to be running in to this same issue in #386. I agree that the google auth side is working as expected; isn't this a bug with the kubernetes python client?

[mbohlool]

does running the command line tool in .kube/config file refresh the right way? What kubectl is doing is running the command specified in .kube/config. If kubectl is working for you, then the command line tool should work, can you guys test this?

[chrish42]

I'm running into what looks like the same bug, and it's reproducible here. Starting with no kubeconfig, I run gcloud ... get-credentials. When I try to access it using the Python API, I get a RefreshError exception. But it I run a kubectl command (e.g. get pods), it works fine and then the Python APi works fine too. And looking at the kubeconfig, the access-token and expiry fields were added.

I can help debug this. Let me know if you prefer to reopen this, or if I should file another bug report. Thanks!

[tania-python-dev]

I also faced with a problem with refreshing token. Python client reads token from ~./kube/config, if it is already expired, creates new token and sends next request with a new one. It looks like a usual flow but the issue is that kubectl config remains the same (with old expired token) and API responds me with 403 Forbidden error since I come with unknown for him token.
The only way I can refresh token in ~./kube/config is to execute any kubectl command from console (e.g. kubectl get namespaces). After this token refreshes and API responds 200 OK. Does anybody know how to fix it?