The aws_appmesh_virtual_service throws "produced an unexpected new value for was present, but now absent" error.

[mkielar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.12.29

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.3.0...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.70.0...
- Downloading plugin for provider "external" (hashicorp/external) 1.2.0...
- Downloading plugin for provider "null" (hashicorp/null) 2.1.2...

Affected Resource(s)

• aws_appmesh_virtual_service

Terraform Configuration Files

resource "aws_appmesh_virtual_service" "vs" {

  mesh_name = var.appmesh.name

  name = "${local.name}.${var.service_discovery.namespace_name}"

  spec {
    provider {
      virtual_router {
        virtual_router_name = aws_appmesh_virtual_router.vr.name
      }
    }
  }
}

Debug Output

On first run, terraform fails like this:

Error: Provider produced inconsistent result after apply
When applying changes to
module.[MASKED].module.template.aws_appmesh_virtual_service.vs,
provider "registry.terraform.io/-/aws" produced an unexpected new value for
was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Then on retry, it fails like this:

Error: error creating App Mesh virtual service: ConflictException: VirtualService with name dev-i270-sst-template.dev-i270.[MASKED].local already exists with different clientRequestToken C1833BA9-42A4-48D3-B855-243582FC4B62
  on .terraform/modules/[MASKED].template/terraform/modules/ecs/service_blue_green/appmesh.tf line 105, in resource "aws_appmesh_virtual_service" "vs":
 105: resource "aws_appmesh_virtual_service" "vs" {

It appears as if terraform (and AWS) create the resource properly, but terraform fails to acknowledge the fact and does not reflect that fact the statefile. Then, another run of terraform apply does not see that resource in the statefile, and tries to create it again - failing, because it already exists.

Panic Output

Expected Behavior

The resources should've been created and terraform should not fail.

Actual Behavior

Terraform fails (ther first time) because of what it says is a bug in the provider. Then if fails the second time because the resources actually gets created but it's not reflected in the state file.

Steps to Reproduce

1. terraform apply

Important Factoids

The error appears randomly. We're frequently creating and destroying resources as part of out CICD pipeline (entire dev environments live for just a few days and are destroyed together with git branches being merged) and this happens once or twice a month with no distinguishable pattern. Therefore, we have not been able to reproduce this issue with debug logging enabled and only have the logs as attached.

[ewbankkit]

This is most likely an "eventual consistency" issue - After creation the virtual service is read back but the API returns a NotFoundException error code.
The solution would be to add a retry in resourceAwsAppmeshVirtualServiceCreate() to ensure that the virtual service can be read back.

[mkielar]

Just today, again, this time Virtual Node, not Service:

Error: Provider produced inconsistent result after apply
When applying changes to
module.[MASKED].module.[MASKED].aws_appmesh_virtual_node.vn["blue"],
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.
This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Terraform v0.13.3
- Using hashicorp/local v1.4.0 from the shared cache directory
- Using hashicorp/null v2.1.2 from the shared cache directory
- Using hashicorp/random v2.3.1 from the shared cache directory
- Using hashicorp/tls v3.0.0 from the shared cache directory
- Using hashicorp/aws v3.25.0 from the shared cache directory
- Using hashicorp/external v1.2.0 from the shared cache directory

[ghost]

This has been released in version 3.36.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!