Terraform plan/apply stuck in loop when getting access denied

[hashibot]

This issue was originally opened by @rhyspowell as hashicorp/terraform#16458. It was migrated here as a result of the provider split. The original body of the issue is below.

---

While the cause of the access denied is within our configuration, there is no indication that there is an issue with getting access. On an original apply the process was still running after an hour. Now, when running plan, it appears to freeze even ctrl+c not allowing the program to stop

Terraform version

../../terraform -v
2017/10/26 11:44:21 [INFO] Terraform version: 0.10.7  5f9bf20ba6aaa174600109a95cc77f7788d4959b+CHANGES
2017/10/26 11:44:21 [INFO] Go runtime version: go1.9
2017/10/26 11:44:21 [INFO] CLI args: []string{"/opt/terraform/terraform", "-v"}
2017/10/26 11:44:21 [DEBUG] Attempting to open CLI config file: /home/ec2-user/.terraformrc
2017/10/26 11:44:21 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2017/10/26 11:44:21 [DEBUG] CLI Config is main.Config{Providers:map[string]string{}, Provisioners:map[string]string{}, DisableCheckpoint:false, DisableCheckpointSignature:false, PluginCacheDir:""}
2017/10/26 11:44:21 [INFO] CLI command args: []string{"version", "-v"}
2017/10/26 11:44:21 [DEBUG] plugin: waiting for all plugin processes to complete...
Terraform v0.10.7

Error being reported

2017-10-26T11:42:20.726Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: X-Amz-Target: AnyScaleFrontendService.DescribeScalingPolicies
2017-10-26T11:42:20.726Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: Accept-Encoding: gzip
2017-10-26T11:42:20.726Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4:
2017-10-26T11:42:20.726Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: -----------------------------------------------------
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: 2017/10/26 11:42:20 [DEBUG] [aws-sdk-go] DEBUG: Response autoscaling/DescribeScalingPolicies Details:
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: ---[ RESPONSE ]--------------------------------------
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: HTTP/1.1 400 Bad Request
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: Connection: close
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: Content-Length: 565
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: Content-Type: application/x-amz-json-1.1
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: Date: Thu, 26 Oct 2017 11:43:26 GMT
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: X-Amzn-Requestid: e1033176-ba42-11e7-ac0a-71a73ccd5044
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4:
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4:
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: -----------------------------------------------------
2017-10-26T11:42:20.834Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: 2017/10/26 11:42:20 [DEBUG] [aws-sdk-go] {"__type":"FailedResourceAccessException","Message":"Unable to retrieve alarms for scalin
g policy arn:aws:autoscaling:us-west-2:XXXXXXXXX:scalingPolicy:3443302c-9b6c-4b77-94b2-eca72bb7dfda:resource/dynamodb/table/name_policies:policyName/table/name_policies due to reason: U
ser: arn:aws:sts::XXXXXXXXX:assumed-role/DynamoDBAutoscale/AutoScaling-DescribeAlarms is not authorized to perform: cloudwatch:DescribeAlarms (Service: AmazonCloudWatch; Status Code: 403; Error Cod
e: AccessDenied; Request ID: e10d9132-ba42-11e7-8a29-e1025d574ea4)"}
2017/10/26 11:42:26 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"
2017/10/26 11:42:29 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.config_api.aws_appautoscaling_policy.dynamodb_write_policy"
2017/10/26 11:42:31 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"
2017/10/26 11:42:34 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.config_api.aws_appautoscaling_policy.dynamodb_write_policy"
2017/10/26 11:42:36 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"
2017/10/26 11:42:39 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.config_api.aws_appautoscaling_policy.dynamodb_write_policy"
2017/10/26 11:42:41 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"
2017/10/26 11:42:44 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.config_api.aws_appautoscaling_policy.dynamodb_write_policy"
2017/10/26 11:42:46 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"
2017/10/26 11:42:49 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.config_api.aws_appautoscaling_policy.dynamodb_write_policy"
2017/10/26 11:42:51 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"

When ctrl+c is pressed

2017/10/26 11:43:19 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.config_api.aws_appautoscaling_policy.dynamodb_write_policy"
2017/10/26 11:43:21 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"
^C2017-10-26T11:43:23.157Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: 2017/10/26 11:43:23 [DEBUG] plugin: received interrupt signal (count: 20). Ignoring.
2017-10-26T11:43:23.158Z [DEBUG] plugin.terraform: plugin received interrupt signal, ignoring: timestamp=2017-10-26T11:43:23.157Z count=20
^C2017-10-26T11:43:23.775Z [DEBUG] plugin.terraform: plugin received interrupt signal, ignoring: timestamp=2017-10-26T11:43:23.775Z count=21
2017-10-26T11:43:23.775Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: 2017/10/26 11:43:23 [DEBUG] plugin: received interrupt signal (count: 21). Ignoring.
^C2017-10-26T11:43:24.101Z [DEBUG] plugin.terraform: plugin received interrupt signal, ignoring: timestamp=2017-10-26T11:43:24.101Z count=22
2017-10-26T11:43:24.102Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: 2017/10/26 11:43:24 [DEBUG] plugin: received interrupt signal (count: 22). Ignoring.
^C2017-10-26T11:43:24.357Z [DEBUG] plugin.terraform-provider-aws_v1.1.0_x4: 2017/10/26 11:43:24 [DEBUG] plugin: received interrupt signal (count: 23). Ignoring.
2017-10-26T11:43:24.358Z [DEBUG] plugin.terraform: plugin received interrupt signal, ignoring: timestamp=2017-10-26T11:43:24.357Z count=23
2017/10/26 11:43:24 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "module.config_api.aws_appautoscaling_policy.dynamodb_write_policy"
2017/10/26 11:43:26 [TRACE] dag/walk: vertex "root", waiting for: "provider.aws (close)"

[FireballDWF]

I've observed the same behavior, usually on first terraform plan of the day, as my consol sts based tokens expired from the previous workday. Terraform 0.11.8 and aws_provider 1.6

[smilin-stan]

I see exactly the same behaviour as @FireballDWF. When my STS token has expired, terraform plan and apply hang and not even CTRL-C will save them.

[lvh]

This is really the same bug as #1351 and #1307 IIUC. It got automatically tagged autoscaling but AFAICT that's irrelevant.

[rhyspowell]

Yes, certinaly looks to be the same issue

[rrijkse]

This issue is happening for us with the latest version:

Terraform v0.12.13
+ provider.aws v2.34.0

We found we are running into this when using vault (using the aws assume_role secrets engine) on TFC and the plan/apply have some time in between (15 min) the apply will never finish it just hangs.

[mpechner]

TF 0.12.18 and AWS provider 2.43
Just trying to run plan.

[github-actions]

Marking this issue as stale due to inactivity. This helps our maintainers find and focus on the active issues. If this issue receives no comments in the next 30 days it will automatically be closed. Maintainers can also remove the stale label.

If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

[YakDriver]

This is closed by events. See hashicorp/aws-sdk-go-base#362 for more details.