Cannot use impersonate with domain wide delegation

[damir-dezeljin]

TL;DR

Hello,

I setup a Federated identity between my GCP and a GitHub workflow.

The workload identity is set properly. I've tested it by installing gcloud SDK and running gcloud auth list. The later command outputs the correct SA being impersonated.

Also the google-github-actions/auth@v0 without impersonalization, using token_format: 'access_token' passes correctly.

The impersonated SA has both Service Account Token Creator and Workload Identity User roles granted on the tools-iac-example GCP project.
Also, the SA has Domain Wide Delegation granted for all 4 specified scopes.

Am I missing something?

Thanks,
Damir Dezeljin

Expected behavior

The impersonation should work.

Observed behavior

The impersonalization fails with the following error:

##[debug]Evaluating condition for step: 'Authenticate to Google Cloud'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Authenticate to Google Cloud
##[debug]Register post job cleanup for action: google-github-actions/auth@v0
##[debug]Loading inputs
##[debug]Loading env
Run google-github-actions/auth@v0
##[debug]Using workload identity provider "projects/73[1](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:1)01467[2](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:2)9[3](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:3)/locations/global/workloadIdentityPools/iac-gh-idpool/providers/iac-gh-idpool-provider"
##[debug]ID token url is https://pipelines.actions.githubusercontent.com/RoKEYgiuTaqbqfrcBb7sGLAs9VLI3QvF5bqVk65EcSqciIZUoT/00000000-0000-0000-0000-000000000000/_apis/distributedtask/hubs/Actions/plans/c22e10bb-b0d9-[4](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:4)918-9a71-014dd3[5](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:5)f8[6](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:6)01/jobs/5d0864fc-c269-5600-cd0[7](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:7)-7aaf32b0e293/idtoken?api-version=2.0&audience=https%3A%2F%2Fiam.googleapis.com%2Fprojects%2F23101461234%2Flocations%2Fglobal%2FworkloadIdentityPools%2Fiac-gh-idpool%2Fproviders%2Fiac-gh-idpool-provider
::add-mask::***
##[debug]Creating credentials file
Created credentials file at "/home/runner/work/tf-mgmt/tf-mgmt/gha-creds-baf7b4b41cd[8](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:8)aa52.json"
##[debug]Creating access token
Error: google-github-actions/auth failed with: retry function failed after 1 attempt: failed to sign JWT using tf-management@tools-iac-example.iam.gserviceaccount.com: (403) {
  "error": {
    "code": [40](https://github.com/example-io/tf-mgmt/actions/runs/3285540817/jobs/5412743142#step:3:41)3,
    "message": "The caller does not have permission",
    "status": "PERMISSION_DENIED"
  }
}
##[debug]Node Action run completed with exit code 1

Action YAML

steps:
  - uses: actions/checkout@v3

  - name: Authenticate to Google Cloud
    id: "auth"
    uses: google-github-actions/auth@v0
    with:
      workload_identity_provider: projects/23101461234/locations/global/workloadIdentityPools/iac-gh-idpool/providers/iac-gh-idpool-provider
      service_account: tf-management@tools-iac-example.iam.gserviceaccount.com
      token_format: 'access_token'
      access_token_lifetime: 1800s
      access_token_scopes: https://www.googleapis.com/auth/cloud-platform, https://www.googleapis.com/auth/admin.directory.user, https://www.googleapis.com/auth/admin.directory.group, https://www.googleapis.com/auth/apps.groups.settings
      access_token_subject: workspace-user@example.io
      project_id: tools-iac-example

Log output

No response

Additional information

No response

[github-actions]

Hi there @damir-dezeljin 👋!

Thank you for opening an issue. Our team will triage this as soon as we can. Please take a moment to review the troubleshooting steps which lists common error messages and their resolution steps.

[sethvargo]

Hi @damir-dezeljin

In order to support Domain-Wide Delegation via Workload Identity Federation, you must grant the external identity ("principalSet") roles/iam.serviceAccountTokenCreator in addition to roles/iam.workloadIdentityUser.

What are your current IAM permissions for both tf-management and the principalSet?

[kbroughton]

I got bit by an upper-cased github org name. Github doesn't distinguish case on the gh CLI and url paths, but the principalSet mapping is case sensitive.

[damir-dezeljin]

Thank you @sethvargo , I overlooked the point I had to add IAM roles to the principalSet.

Still, this alone didn't solve my problem, but this had to be done as well:

• GitHub workflow GCP authentication step and related:

  ...
  env:
   GOOGLE_ADMIN_SCOPES: https://www.googleapis.com/auth/cloud-platform, https://www.googleapis.com/auth/admin.directory.user, https://www.googleapis.com/auth/admin.directory.group, https://www.googleapis.com/auth/apps.groups.settings
  ...
   permissions:
     contents: "read"
     id-token: "write"
  ...
     - name: Authenticate to Google Cloud
       id: "auth"
       uses: google-github-actions/auth@v0
       with:
         workload_identity_provider: ${{ env.GCP_WORKLOAD_IDPOOL_PROVIDER }}
         service_account: ${{ env.CICD_GCP_SA }}
         token_format: "access_token"
         access_token_lifetime: 1800s
         access_token_scopes: ${{ env.GOOGLE_ADMIN_SCOPES }}
         access_token_subject: ${{ env.GOOGLE_ADMIN_ACCOUNT }}
         create_credentials_file: true
  

• TF Google Workspace provider had to be specified as follows:

  variable "googleworkspace_access_token" {
    type        = string
    description = "Oauth access token for workspace user"
  }
  ...
  provider "googleworkspace" {
    customer_id     = var.customer_id
    access_token    = var.googleworkspace_access_token
    service_account = var.super_admin_account
    oauth_scopes = [
      "https://www.googleapis.com/auth/admin.directory.user",
      "https://www.googleapis.com/auth/admin.directory.group",
      "https://www.googleapis.com/auth/apps.groups.settings"
    ]
  }
  

  ^^^
  Please note I had to specify this env var in the GitHub workflow for the terraform plan step to work:

  env:
    TF_VAR_googleworkspace_access_token: ${{ steps.auth.outputs.access_token }}