[Bug]: Acceptance tests fail with Failed state verification, resource with ID ... not found

[bblommers]

Terraform Core Version

1.3.0,1.3.1

AWS Provider Version

latest

Affected Resource(s)

No response

Expected Behavior

When running the acceptance tests in this repository, the tests should pass.

Actual Behavior

Some tests fail with message Failed state verification, resource with ID .. not found.

Some example tests that are failing:
TestAccQuickSightGroup_withDescription
TestAccVPC_DefaultTagsProviderAndResource_nonOverlappingTag
TestAccECRRegistryPolicy_serial/basic

All these tests are passing with Terraform 1.2.9 installed, but fail with 1.3.0/1.3.1.

The exact ID specified in the error message looks to be one of [aws, {account_id}, {region}] - not an actual resource ID.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

N/A

Steps to Reproduce

These errors where found when running the tests against Moto/Localstack as an alternative to AWS.

To reproduce against Moto:

git clone git@github.com:spulec/moto.git
# start a virtualenv if desired
make init
moto_server -p 4566

# In a separate console
make terraformtests SERVICE_NAME=ec2 TEST_NAMES=TestAccVPC_DefaultTags_providerOnly

Note that Moto contains terraform-provider-aws as a submodule. The current commit is 3 months old, but the same behaviour occurs when updating terraform-provider-aws to the latest commit on main.

Debug Output

No response

Panic Output

No response

Important Factoids

The errors were discovered as part of our CI process, that runs the acceptance tests against Moto. An example output that shows the failing tests can be found here: https://github.com/spulec/moto/actions/runs/3161022833

Note that this is run on Terraform 1.3.0. Some tests are already fixed in 1.3.1, but all tests that fail with Failed state verification.. still fail against 1.3.1.

Some background info on how/why Moto runs the tests from this repository can be found here:
http://docs.getmoto.org/en/latest/docs/contributing/development_tips/tests.html#terraform-tests

References

No response

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[christophetd]

Was able to reproduce from my side FWIW:

$ go version
go version go1.19.1 darwin/amd64
$ terraform version
Terraform v1.3.1

$ make testacc TESTS=TestAccEC2AMI_tpmSupport PKG=ec2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/ec2/... -v -count 1 -parallel 20 -run='TestAccEC2AMI_tpmSupport'  -timeout 180m
=== RUN   TestAccEC2AMI_tpmSupport
=== PAUSE TestAccEC2AMI_tpmSupport
=== CONT  TestAccEC2AMI_tpmSupport
    ec2_ami_test.go:528: Failed state verification, resource with ID us-east-1 not found
--- FAIL: TestAccEC2AMI_tpmSupport (79.26s)
FAIL
FAIL	github.com/hashicorp/terraform-provider-aws/internal/service/ec2	82.278s
FAIL
make: *** [testacc] Error 1


$  tfenv use 1.2.9
$ make testacc TESTS=TestAccEC2AMI_tpmSupport PKG=ec2
TF_ACC=1 go test ./internal/service/ec2/... -v -count 1 -parallel 20 -run='TestAccEC2AMI_tpmSupport'  -timeout 180m
=== RUN   TestAccEC2AMI_tpmSupport
=== PAUSE TestAccEC2AMI_tpmSupport
=== CONT  TestAccEC2AMI_tpmSupport
--- PASS: TestAccEC2AMI_tpmSupport (76.88s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/ec2	79.783s

[drewmullen]

I was able to reproduce on 1.3.2:

$ make testacc TESTS=TestAccIPAM_basic PKG=ec2

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/ec2/... -v -count 1 -parallel 20 -run='TestAccIPAM_basic'  -timeout 180m
=== RUN   TestAccIPAM_basic
=== PAUSE TestAccIPAM_basic
=== CONT  TestAccIPAM_basic
    ipam_test.go:25: Failed state verification, resource with ID us-east-1 not found
--- FAIL: TestAccIPAM_basic (38.95s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-aws/internal/service/ec2        42.908s
FAIL
make: *** [testacc] Error 1

[bschaatsbergen]

reproducible on 1.3.2 core:

$ make testacc TESTS=TestAccElastiCacheSubnetGroup_basic PKG=elasticache
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/elasticache/... -v -count 1 -parallel 20 -run='TestAccElastiCacheSubnetGroup_basic'  -timeout 180m
=== RUN   TestAccElastiCacheSubnetGroup_basic
=== PAUSE TestAccElastiCacheSubnetGroup_basic
=== CONT  TestAccElastiCacheSubnetGroup_basic
    subnet_group_test.go:21: Failed state verification, resource with ID us-west-2 not found
--- FAIL: TestAccElastiCacheSubnetGroup_basic (40.88s)
FAIL
FAIL    github.com/hashicorp/terraform-provider-aws/internal/service/elasticache        42.657s
FAIL
make: *** [testacc] Error 1

$ terraform -version
Terraform v1.3.2
on darwin_arm64

[YakDriver]

As of 4 days ago, we are using terraform-plugin-sdk/v2, which resolves this issue. Please let us know right away if not but on our side it looks to work. If your branch is older, rebase to pickup new plugin sdk version.

• acctests: Ignore data sources in imported state in ImportState steps terraform-plugin-sdk#1074
• https://github.com/hashicorp/terraform-plugin-sdk/releases/tag/v2.24.0

[bblommers]

Thanks for letting me know @YakDriver - I can confirm that solves the issue for me.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.