google_service_account read after creation issue - apply fails with 403 right after creation

[sbocinec]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.8.5
on amd64

• provider registry.terraform.io/hashicorp/google v5.39.1

Affected Resource(s)

google_service_account

Terraform Configuration

locals {
  agent_id = "cba1234567890xyz"
}

resource "google_service_account" "agent" {
  count = 1
  account_id                   = "agent-${local.agent_id}"
  display_name                 = "Agent ${local.agent_id} Service Account"
  description                  = "Agent ${local.agent_id} Service Account"
  create_ignore_already_exists = true
}

Debug Output

2024-09-20T10:25:44.332Z	google_service_account.agent[0]: Creating...
2024-09-20T10:25:45.052Z	
2024-09-20T10:25:45.052Z	Error: googleapi: Error 403: Permission 'iam.serviceAccounts.get' denied on resource (or it may not exist).
2024-09-20T10:25:45.052Z	Details:
2024-09-20T10:25:45.052Z	[
2024-09-20T10:25:45.052Z	  {
2024-09-20T10:25:45.052Z	    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
2024-09-20T10:25:45.052Z	    "domain": "iam.googleapis.com",
2024-09-20T10:25:45.052Z	    "metadata": {
2024-09-20T10:25:45.052Z	      "permission": "iam.serviceAccounts.get"
2024-09-20T10:25:45.052Z	    },
2024-09-20T10:25:45.052Z	    "reason": "IAM_PERMISSION_DENIED"
2024-09-20T10:25:45.052Z	  }
2024-09-20T10:25:45.052Z	]
2024-09-20T10:25:45.052Z	, forbidden
2024-09-20T10:25:45.052Z	
2024-09-20T10:25:45.052Z	  with google_service_account.agent[0],
2024-09-20T10:25:45.052Z	  on main.tf line 119, in resource "google_service_account" "agent":
2024-09-20T10:25:45.052Z	 119: resource "google_service_account" "agent" {
2024-09-20T10:25:45.052Z	
2024-09-20T10:25:45.129Z	

Expected Behavior

Due to the eventual consistency of the GCP IAM API, the provider is supposed to pollread the SA after creation unless the SA is found for the defined timeout.

As the GCP IAM API returns 403 IAM_PERMISSION_DENIED on a SA read even when the SA is not found, the pollread must handle the permission denied error as it does in the first read after it's created a few lines before.

The pollRead actually throws the error on the first 403 occurrence, as the poller only handles 404 errors thus the polling is not effective and instead it errors out quickly.

I prepared a fix PR for this GoogleCloudPlatform/magic-modules#11811 by using PollCheckForExistenceWith403 poller that continues polling even when 403 is thrown.

Actual Behavior

As you can see, the SA creation fails in 1s, shortly after the creation. It should handle 403 errors in the poller doing post-creation reads.

Steps to reproduce

1. terraform apply

Important Factoids

• Previous attempt to handle 403 errors on SA creation Retry Service Account post-creation reads if 403 is returned GoogleCloudPlatform/magic-modules#8815

References

• Retry Service Account post-creation reads if 403 is returned GoogleCloudPlatform/magic-modules#8815
• Add a sleep to google_service_account to reduce the likelihood of eventual consistency errors GoogleCloudPlatform/magic-modules#10813

[ggtisc]

Hi @sbocinec!

After executing your code with the specified terraform and Google provider versions everything was successful without errors. A 403 is an unauthorized issue and it is usually related to account permissions. I suggest you check your account permissions and try it again.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.