Eliminate retries in the inner loop for errors that will not succeed.

[nat-henderson]

Fixes hashicorp/terraform-provider-google#7853
Fixes hashicorp/terraform-provider-google#7940
Fixes hashicorp/terraform-provider-google#6833

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

filestore: updated retry logic to fail fast on quota error which cannot succeed on retry.

storage: updated IAM resources to refresh etag sooner on an IAM conflict error, which will make applications of multiple IAM resources much faster.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 19 insertions(+), 8 deletions(-))
Terraform Beta: Diff ( 5 files changed, 19 insertions(+), 8 deletions(-))
TF Conversion: Diff ( 1 file changed, 11 insertions(+))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=163828"

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 5 files changed, 19 insertions(+), 15 deletions(-))
Terraform Beta: Diff ( 5 files changed, 19 insertions(+), 15 deletions(-))
TF Conversion: Diff ( 1 file changed, 11 insertions(+), 7 deletions(-))

[modular-magician]

I have triggered VCR tests based on this PR's diffs. See the results here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=164235"

[modular-magician]

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceStorageBucketObjectContent_Basic|TestAccActiveDirectoryDomainTrust_activeDirectoryDomainTrustBasicExample|TestAccComputeRegionUrlMap_defaultUrlRedirectWithinPathMatcher|TestAccContainerCluster_withConfidentialNodes|TestAccContainerCluster_withPrivateClusterConfigMissingCidrBlock|TestAccContainerCluster_withIPv4Error|TestAccDataprocJob_Spark You can view the result here: "https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=164240"

[nat-henderson]

Cool - i tested this with the following config:

resource "google_service_account" "test1" {
  account_id   = "concurrent-iam-test1"
}

resource "google_service_account" "test2" {
  account_id   = "concurrent-iam-test2"
}

resource "google_storage_bucket_iam_member" "concurrent_iam" {
  bucket = var.bucket
  role   = "roles/storage.objectAdmin"
  member = "serviceAccount:${google_service_account.test1.email}"
}

resource "google_storage_bucket_iam_member" "concurrent_iam_2" {
  bucket = var.bucket
  role   = "roles/storage.objectAdmin"
  member = "serviceAccount:${google_service_account.test2.email}"
}

Completes in a few seconds, seems to be doing it with parallelism, as expected.

google_storage_bucket_iam_member.concurrent_iam: Creating...
google_storage_bucket_iam_member.concurrent_iam_2: Creating...
google_storage_bucket_iam_member.concurrent_iam: Creation complete after 9s [id=b/personal-graphite-testing-ml/roles/storage.objectAdmin/serviceaccount:concurrent-iam-test1@personal-graphite-testing.iam.gserviceaccount.com]
google_storage_bucket_iam_member.concurrent_iam_2: Creation complete after 9s [id=b/personal-graphite-testing-ml/roles/storage.objectAdmin/serviceaccount:concurrent-iam-test2@personal-graphite-testing.iam.gserviceaccount.com]