GCP Service Account key unable to retrieve token

[lbennett-stacki]

Expected behavior

Using only a GCP service account key, build using cloud build.

Actual behavior

Never able to auth service account key, always tries to fallback to gcloud application default credentials

Information

I am using skaffold in GitLab CI. Running in debug mode, I see that when skaffold starts the cloud build, it tries to auth after grabbing the gcloud credentials from gcloud auth print-access-token.

Unfortunately, no matter the permissions of the service account, my debug log shows
unable to retrieve token: oauth2: token expired and refresh token is not set.

It then continues, tried to request information about the logging bucket, which times out.

Related PR: #2731

• Skaffold version: v1.7.1 (and latest from gcloud components on CI)
• Operating system: Mac (and Debian on CI using gcloud docker image)
• Installed via: gcloud sdk
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta8
kind: Config
metadata:
  name: aaaa
build:
  googleCloudBuild:
    projectId: aaaa-bbbb
    machineType: E2_HIGHCPU_8
  artifacts:
    - image: gcr.io/aaaa-bbbb/cccc
      sync:
        infer:
          - '**/*.json'
      kaniko:
        cache: {}
  tagPolicy:
    sha256: {}
deploy:
  kustomize:
    paths:
      - k8s/overlays/lb-dev
profiles:
  - name: lb-dev
    patches:
      - op: replace
        path: /deploy/kustomize/paths/0
        value: k8s/overlays/lb-dev
      - op: replace
        path: /build/tagPolicy
        value:
          sha256: {}
  - name: development
    patches:
      - op: replace
        path: /deploy/kustomize/paths/0
        value: k8s/overlays/development
      - op: replace
        path: /build/tagPolicy
        value:
          gitCommit: {}

Steps to reproduce the behavior

1. create GCP service account, assign "Owner" perms
2. gcloud auth activate-service-account .... && gcloud container clusters get-credentials ....
3. skaffold run -pdevelopment -vdebug

 - gcr.io/xxxx/xxxxx: Not found. Building
time="2021-05-06T02:05:58Z" level=info msg="Cache check completed in 1.244 second"
Starting build...
Building [gcr.io/xxxx/xxxx]...
time="2021-05-06T02:05:58Z" level=debug msg="Running command: [gcloud auth print-access-token --format=json]"
time="2021-05-06T02:05:59Z" level=debug msg="Command output: [{\n  \"token\": \"xxxx\"\n}\n]"
time="2021-05-06T02:05:59Z" level=info msg="unable to retrieve token: oauth2: token expired and refresh token is not set"
time="2021-05-06T02:05:59Z" level=info msg="falling back to application default credentials"
creating bucket if not exists: getting bucket "xxxx_cloudbuild": Get "https://storage.googleapis.com/storage/v1/b/xxxx?alt=json&prettyPrint=false&projection=full": Get "http://xxx.xxx.xxx.xxx/computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdevstorage.full_control": dial tcp xxx.xxx.xxx.xxx:80: i/o timeout

[aaron-prindle]

Can you clarify how the service account is setup to be used for gcloud? What is the full command used here gcloud auth activate-service-account ... (w/ any private/important details removed). Do you download the service-account key locally and then use that in the cloud command, ex from docs here:

gcloud auth activate-service-account test-service-account@google.com  --key-file=/path/key.json --project=testproject

If you have the key file locally (key.json) you should be able to get this to work correctly using the GOOGLE_APPLICATION_CREDENTIALS env var and set that to point to the the json key for the new service account.

Example:

export GOOGLE_APPLICATION_CREDENTIALS=path/to/service-account-key/key.json
skaffold run -pdevelopment -vdebug

Additionally if skaffold is correctly pointing to the key (as the logs suggest), can you verify the key is not expired or re-create the key and try again?

[aaron-prindle]

I saw a similar error to what is posted when the projectId used:

build:
  googleCloudBuild:
    projectId: foo

and the project inferred from the default repo:

export SKAFFOLD_DEFAULT_REPO=gcr.io/foo

were not the same

[lbennett-stacki]

Interesting. I did indeed get this going with GOOGLE_APPLICATION_CREDENTIALS so I haven't paid much attention to it today. I'll have a look if projectId is an offender when I'm next active.

[aaron-prindle]

Great, I'm going to close this for now as it seems using GOOGLE_APPLICATION_CREDENTIALS works for skaffold users w/ a service account. Feel free to open this if there is more work to be done or something not addressed, thanks!