Implement isResourceTimeoutError() Conditionals After All resource.Retry() Usage

[bflad]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

When implementing resource.Retry() to allow Terraform AWS Provider resources to retry AWS API requests either for eventual consistency errors or for known concurrency errors, the code immediately following should include a conditional check for isResourceTimeoutError() to retry the request outside the timeout.

If the AWS Go SDK encounters an error such as 5XX AWS API responses, networking related errors, or certain throttling errors, it will automatically retry the AWS API requests itself without returning to the calling code. If these automatic AWS Go SDK retries last longer than the resource.Retry() timeout and we do not retry the request outside the resource.Retry() block before returning the error, operators will receive an "empty" timeout error without any context about the underlying AWS Go SDK error:

timeout while waiting for state to become 'success' (timeout: 1m0s)

Debugging these with Terraform is only possible by enabling debug logging, e.g. TF_LOG=debug terraform plan/apply.

Example incorrect implementation in pseudocode:

err := resource.Retry(1 * time.Minute, func() *resource.RetryError {
  _, err := conn.SomeRequest(input)
  // ... other logic including resource.RetryableError() ...
})

if err != nil {
  return err
}

Fixed implementation:

err := resource.Retry(1 * time.Minute, func() *resource.RetryError {
  _, err := conn.SomeRequest(input)
  // ... other logic including resource.RetryableError() ...
})

if isResourceTimeoutError(err) {
  // Using = here intentionally to overwrite the err
  _, err = conn.SomeRequest(input)
}

if err != nil {
  return err
}

References

There are numerous bugs and pull requests relating to this behavior, but most recently: #7871

[ryndaniels]

HURRAY WE DID IT

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!