The kubeconfig loader should run refresh command to update token when it is expired #233

nhumrich · 2017-05-22T21:43:04Z

I am using google container engine, and trying to use this to access the k8s api. Trying to follow the example on the readme

from kubernetes import client, config

config.load_kube_config()
api = client.CoreV1Api()
pods = api.list_pod_for_all_namespaces(watch=False)


for p in pods.items:
    print(p.metadata.name, p.status.phase)

which gives me the following error:

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 6, in <module>
    config.load_kube_config()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 311, in load_kube_config
    client_configuration=client_configuration).load_and_set()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 216, in load_and_set
    self._load_authentication()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 155, in _load_authentication
    if self._load_gcp_token():
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 171, in _load_gcp_token
    self.token = "Bearer %s" % self._get_google_credentials()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 124, in <lambda>
    GoogleCredentials.get_application_default()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1271, in get_application_default
    return GoogleCredentials._get_implicit_credentials()
  File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1261, in _get_implicit_credentials
    raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

If I add the GOOGLE_APPLICATION_CREDENTIALS env-var and download a google json credential file, I then get a generic 401.

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in <module>
    pods = api.list_pod_for_all_namespaces(watch=False)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:25:24 GMT', 'Content-Length': '13'})
HTTP response body: Unauthorized

If I try to add an api key (client.configuration.api_key['authorization'] = 'AbX.....SYh' I get another error.

Traceback (most recent call last):
  File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in <module>
    pods = api.list_pod_for_all_namespaces(watch=False)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces
    (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info
    collection_formats=collection_formats)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request
    headers=headers)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET
    query_params=query_params)
  File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:40:29 GMT', 'Content-Length': '119'})
HTTP response body: User "system:anonymous" cannot list pods at the cluster scope.: "No policy matched.\nUnknown user \"system:anonymous\""

Are there any examples of how I authenticate with kubernetes/google container engine so that I can get this working?

Note: one possible solution is to run gcloud auth application-default login but that isn't automated and only works locally.

The text was updated successfully, but these errors were encountered:

jonathan-kosgei · 2017-05-23T05:48:00Z

Hi Nick, seems you're trying to use gcloud credentials to auth to the Kubernetes cluster. The two are very different, instead of using a gce serviceaccount as the credential source in load_config you need to use your kube config file. If you can run kubectl commands locally then you have one under ~/.kube/config

…

---- On Tue, 23 May 2017 00:43:06 +0300 notifications@github.com wrote ---- I am using google container engine, and trying to use this to access the k8s api. Trying to follow the example on the readme from kubernetes import client, config config.load_kube_config() api = client.CoreV1Api() pods = api.list_pod_for_all_namespaces(watch=False) for p in pods.items: print(p.metadata.name, p.status.phase) which gives me the following error: Traceback (most recent call last): File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 6, in <module> config.load_kube_config() File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 311, in load_kube_config client_configuration=client_configuration).load_and_set() File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 216, in load_and_set self._load_authentication() File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 155, in _load_authentication if self._load_gcp_token(): File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 171, in _load_gcp_token self.token = "Bearer %s" % self._get_google_credentials() File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/config/kube_config.py", line 124, in <lambda> GoogleCredentials.get_application_default() File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1271, in get_application_default return GoogleCredentials._get_implicit_credentials() File "/home/nhumrich/.local/lib/python3.6/site-packages/oauth2client/client.py", line 1261, in _get_implicit_credentials raise ApplicationDefaultCredentialsError(ADC_HELP_MSG) oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. If I add the GOOGLE_APPLICATION_CREDENTIALS env-var and download a google json credential file, I then get a generic 401. Traceback (most recent call last): File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in <module> pods = api.list_pod_for_all_namespaces(watch=False) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info collection_formats=collection_formats) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request headers=headers) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET query_params=query_params) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request raise ApiException(http_resp=r) kubernetes.client.rest.ApiException: (401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain; charset=utf-8', 'Www-Authenticate': 'Basic realm="kubernetes-master"', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:25:24 GMT', 'Content-Length': '13'}) HTTP response body: Unauthorized If I try to add an api key (client.configuration.api_key['authorization'] = 'AIz.....SYU' I get another error. ``` Traceback (most recent call last): File "/home/nhumrich/devops/containers/deployment/scripts/kube-deploy.py", line 19, in pods = api.list_pod_for_all_namespaces(watch=False) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13650, in list_pod_for_all_namespaces (data) = self.list_pod_for_all_namespaces_with_http_info(**kwargs) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py", line 13743, in list_pod_for_all_namespaces_with_http_info collection_formats=collection_formats) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 329, in call_api _return_http_data_only, collection_formats, _preload_content, _request_timeout) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 153, in __call_api _request_timeout=_request_timeout) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 361, in request headers=headers) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 240, in GET query_params=query_params) File "/home/nhumrich/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request raise ApiException(http_resp=r) kubernetes.client.rest.ApiException: (403) Reason: Forbidden HTTP response headers: HTTPHeaderDict({'Content-Type': 'text/plain', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 22 May 2017 21:40:29 GMT', 'Content-Length': '119'}) HTTP response body: User "system:anonymous" cannot list pods at the cluster scope.: "No policy matched.\nUnknown user "system:anonymous"" Are there any examples of how I authenticate with kubernetes/google container engine so that I can get this working? Note: one possible solution is to run `gcloud auth application-default login` but that isn't automated and only works locally. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

nhumrich · 2017-05-23T20:54:28Z

I tried doing that which is the first option. I also tried passing in the config file location, but that doesnt change anything, as that is the default anyways.

mbohlool · 2017-05-24T03:57:29Z

Does kubectl works? if yes, can you provide content of ~/.kube/config file (remove any key/token/etc. from the file, only "current context" should be enough. e.g. if your current-context is "x" provide user "x" and cluster "x".

nhumrich · 2017-05-24T19:50:20Z

Here is my kube config file

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ...Something goes here...
    server: https://some.url
  name: gke_infastructure-111111_us-central1-a_my-cluster
contexts:
- context:
    cluster: gke_infastructure-111111_us-central1-a_my-cluster
    user: gke_infastructure-111111_us-central1-a_my-cluster
  name: gke_infastructure-111111_us-central1-a_my-cluster
current-context: gke_infastructure-111111_us-central1-a_my-cluster
kind: Config
preferences: {}
users:
- name: gke_infastructure-111111_us-central1-a_my-cluster
  user:
    auth-provider:
      config:
        access-token: zzzzzz.some-token-here.zzzzz
        cmd-args: config config-helper --format=json
        cmd-path: /opt/google-cloud-sdk/bin/gcloud
        expiry: 2017-05-24T19:57:52Z
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

rafitadiaz · 2017-06-12T12:42:26Z

+1 On this one. We are having the same problem. kubectl is working fine, but we get the 401 when using the library. And our config file look like this one above.

Anyone can send an example of how to do it properly or is it impossible make it works with GKE clusters?

mbohlool · 2017-06-12T15:24:21Z

The kubeconfig loader should run refresh command to update token when it is expired. To solve this, we can find the corresponding code in go client (which kubectl uses) and port it to python. Added it to milestone to make sure it will be looked at for the next release.

zweizeichen · 2017-06-12T17:32:48Z

Just ran into this issue myself. You can use this dirty hack to get auth on GKE in the meantime. Please note that the token retrieved will expire within a rather short timeframe. This should not be used in production.

In kube_config.py add these imports:

import subprocess
import json

modify KubeConfigLoader's __init__ method like this:

if get_google_credentials:
        self._get_google_credentials = get_google_credentials
else:
        self._get_google_credentials = lambda: (
            self._get_gcp_cmd_credentials()
        )

and add the following method to the class:

def _get_gcp_cmd_credentials(self):
    cmd_path = self._user.value['auth-provider']['config']['cmd-path']
    cmd_args = self._user.value['auth-provider']['config']['cmd-args'].split(' ')
    output = subprocess.run([cmd_path]+cmd_args, stdout=subprocess.PIPE, check=False).stdout.decode('utf-8')

    return json.loads(output)['credential']['access_token']

Done!

nhumrich · 2017-06-12T17:44:01Z

Update:
I was able to work around this issue by creating a serviceaccount in kubernetes.

I then ran kubectl describe serviceaccount myserviceaccount and that will give you a secret name, then use that secret name to run: kubectl describe secrets [secret-name] and then copy the token field. One you have the token field, all you need to do is set the api token in the client:

config.load_kube_config()
client.configuration.api_key['authorization'] = 'your token goes here'
client.configuration.api_key_prefix['authorization'] = 'Bearer'

This worked great for me. If you dont want to use the kube config file at all, you can also set the host and cert yourself:

client.configuration.api_key['authorization'] = 'your token goes here'
client.configuration.api_key_prefix['authorization'] = 'Bearer'
client.configuration.host = 'https://some.domain-or-ip.example'
client.configuration.ssl_ca_cert = 'cert/location.crt'

zweizeichen · 2017-06-12T17:53:06Z

Hah I'm using the API to set up the service accounts - so it is a kind of chicken and egg problem. Of course using a service account is the preferred and more durable solution.

nhumrich · 2017-06-13T15:14:16Z

@zweizeichen you could always use kubectl to create a single serviceaccount, then go from there.

jeremad · 2018-03-29T11:34:57Z

I am still experiencing that issue (version 5.0.0 on mac)

OraserZhang · 2018-05-18T11:53:32Z

I am still experiencing that issue (version 5.0.0 on mac)

richerlariviere · 2018-05-22T18:18:08Z

I am still experiencing that issue (version 6.0.0 on mac and I also tried HEAD version) in Kubernetes 1.10.1

efim-a-efim · 2018-08-13T08:20:28Z

Same issue here - version 6.0.0 on Windows/WSL (Ubuntu). kubernetes 1.10.5

tomplus · 2018-08-13T09:02:53Z

The current implementation refreshes a token when configuration is loaded. It doesn't refresh it for long living applications. PTAL kubernetes-client/python-base#59

mooperd · 2018-11-24T10:31:29Z

+1 I'm confused why the library isn't able to use the Kubeconfig properly.

tellet · 2019-02-15T11:06:37Z

I'm facing this issue with version 7.0.1 (on mac), kubernetes 1.10.11. Workarounds don't help.

load_kube_config_from_dict() support define custom temp files path

sidpremkumar · 2022-04-21T13:57:13Z

We are also currently experiencing the same issue +1

OraserZhang · 2022-10-11T08:03:15Z

您的来信收到，谢谢！

emdete · 2023-02-27T06:17:49Z

why is this issue closed, the bug still exists:

  File "python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 15205, in list_namespaced_event
    return self.list_namespaced_event_with_http_info(namespace, **kwargs)  # noqa: E501
  File "python3.9/site-packages/kubernetes/client/api/core_v1_api.py", line 15320, in list_namespaced_event_with_http_info
    return self.api_client.call_api(
  File "python3.9/site-packages/kubernetes/client/api_client.py", line 348, in call_api
    return self.__call_api(resource_path, method,
  File "python3.9/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
    response_data = self.request(
  File "python3.9/site-packages/kubernetes/client/api_client.py", line 373, in request
    return self.rest_client.GET(url,
  File "python3.9/site-packages/kubernetes/client/rest.py", line 241, in GET
    return self.request("GET", url,
  File "python3.9/site-packages/kubernetes/client/rest.py", line 235, in request
    raise ApiException(http_resp=r)
kubernetes.client.exceptions.ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '...', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/
json', 'Date': 'Thu, 23 Feb 2023 10:15:09 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}


running kubectl after that refreshes the token and the api works again.

OraserZhang · 2023-02-27T06:18:09Z

您的来信收到，谢谢！

akaihola · 2023-03-04T12:11:10Z

@OraserZhang wrote twice:

您的来信收到，谢谢！

This seems to be an automated reply, it machine-translates to "Your letter was received, thank you!". Could you configure your e-mail to not auto-respond to GitHub notifications? Thanks!

这似乎是一个自动回复。你能不能把你的电子邮件配置为不自动回复GitHub的通知？谢谢!

emdete · 2023-03-09T09:25:06Z

can this ticket be reopened because the bug still exists?

mbohlool added the kind/bug Categorizes issue or PR as related to a bug. label Jun 12, 2017

mbohlool added this to the v3.0 milestone Jun 12, 2017

vogxn mentioned this issue Jul 7, 2017

kubectl get <object> <space> throws exception cloudnativelabs/kube-shell#26

Closed

mbohlool added the help-needed label Jul 7, 2017

szpotona mentioned this issue Jul 10, 2017

Support Google Authentication for GKE cloudify-cosmo/cloudify-kubernetes-plugin#20

Closed

mbohlool mentioned this issue Jul 14, 2017

Move this repo to kubernetes-client org #243

Closed

mbohlool changed the title ~~How do I authenticate correctly?~~ The kubeconfig loader should run refresh command to update token when it is expired Jul 14, 2017

mbohlool mentioned this issue Jul 25, 2017

Update base submodule #305

Merged

mbohlool closed this as completed in #305 Jul 25, 2017

mbohlool mentioned this issue Nov 11, 2017

Kubernetes python client unable to refresh access token #386

Closed

OlafSzmidt mentioned this issue Feb 5, 2018

Investigate potential kubernetes bug ocadotechnology/aimmo#463

Closed

edaniszewski mentioned this issue Jul 26, 2019

Fix "Long-lasting job fails on EKS" (#124) vapor-ware/kubetest#125

Open

srinivasreddych mentioned this issue Oct 11, 2021

[BUG] - CodeBuild for lake-admin regression stuck with k8s dynamic client unauthorized error. awslabs/aws-orbit-workbench#1078

Closed

yliaog pushed a commit to yliaog/client-python that referenced this issue Jan 8, 2022

Merge pull request kubernetes-client#233 from onecer/master

f38221c

load_kube_config_from_dict() support define custom temp files path

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The kubeconfig loader should run refresh command to update token when it is expired #233

The kubeconfig loader should run refresh command to update token when it is expired #233

nhumrich commented May 22, 2017 •

edited

Loading

jonathan-kosgei commented May 23, 2017 via email

nhumrich commented May 23, 2017

mbohlool commented May 24, 2017

nhumrich commented May 24, 2017

rafitadiaz commented Jun 12, 2017

mbohlool commented Jun 12, 2017

zweizeichen commented Jun 12, 2017

nhumrich commented Jun 12, 2017 •

edited

Loading

zweizeichen commented Jun 12, 2017

nhumrich commented Jun 13, 2017

jeremad commented Mar 29, 2018 •

edited

Loading

OraserZhang commented May 18, 2018

richerlariviere commented May 22, 2018

efim-a-efim commented Aug 13, 2018

tomplus commented Aug 13, 2018

mooperd commented Nov 24, 2018

tellet commented Feb 15, 2019

sidpremkumar commented Apr 21, 2022

OraserZhang commented Oct 11, 2022 via email

emdete commented Feb 27, 2023 •

edited

Loading

OraserZhang commented Feb 27, 2023 via email

akaihola commented Mar 4, 2023

emdete commented Mar 9, 2023

The kubeconfig loader should run refresh command to update token when it is expired #233

The kubeconfig loader should run refresh command to update token when it is expired #233

Comments

nhumrich commented May 22, 2017 • edited Loading

jonathan-kosgei commented May 23, 2017 via email

nhumrich commented May 23, 2017

mbohlool commented May 24, 2017

nhumrich commented May 24, 2017

rafitadiaz commented Jun 12, 2017

mbohlool commented Jun 12, 2017

zweizeichen commented Jun 12, 2017

nhumrich commented Jun 12, 2017 • edited Loading

zweizeichen commented Jun 12, 2017

nhumrich commented Jun 13, 2017

jeremad commented Mar 29, 2018 • edited Loading

OraserZhang commented May 18, 2018

richerlariviere commented May 22, 2018

efim-a-efim commented Aug 13, 2018

tomplus commented Aug 13, 2018

mooperd commented Nov 24, 2018

tellet commented Feb 15, 2019

sidpremkumar commented Apr 21, 2022

OraserZhang commented Oct 11, 2022 via email

emdete commented Feb 27, 2023 • edited Loading

OraserZhang commented Feb 27, 2023 via email

akaihola commented Mar 4, 2023

emdete commented Mar 9, 2023

nhumrich commented May 22, 2017 •

edited

Loading

nhumrich commented Jun 12, 2017 •

edited

Loading

jeremad commented Mar 29, 2018 •

edited

Loading

emdete commented Feb 27, 2023 •

edited

Loading