Provider produced inconsistent result after apply - aws_vpc_endpoint_route_table_association.s3_vpc_endpoint #12449

falgofrancis · 2020-03-18T17:47:24Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version 0.12.23 aws provider version 2.51.0

Affected Resource(s)

aws_vpc_endpoint_route_table_association.s3_vpc_endpoint

Terraform Configuration Files

resource "aws_route_table" "private_subnet_rtb" {
  count  = length(var.availability_zones) * var.flag
  vpc_id = aws_vpc.vpc[0].id
}

resource "aws_vpc_endpoint_route_table_association" "s3_vpc_endpoint" {
  vpc_endpoint_id = aws_vpc_endpoint.s3[0].id
  count           = length(var.availability_zones) * var.flag
  route_table_id  = element(aws_route_table.private_subnet_rtb.*.id, count.index)
}

resource "aws_vpc_endpoint" "s3" {
  count        = var.flag
  vpc_id       = aws_vpc.vpc[0].id
  service_name = "com.amazonaws.${data.aws_region.current.name}.s3"
}

Debug Output

Error: Provider produced inconsistent result after apply vpc.aws_vpc_endpoint_route_table_association.s3_vpc_endpoint[0], provider "registry.terraform.io/-/aws" produced an unexpected new value for was present, but now absent. This is a bug in the provider, which should be reported in the provider's own issue tracker..

Expected Behavior

Terraform apply should have been successful

Actual Behavior

Terraform apply fails

Steps to Reproduce

This does not happen all the time, running apply multiple times

ewbankkit · 2020-03-18T20:11:49Z

@ffran09 If you add count = var.flag to resource "aws_vpc_endpoint_route_table_association" "s3_vpc_endpoint" do you still get the problem?

falgofrancis · 2020-03-18T20:18:44Z

Yes, we still get the same error with the count variable.
I have updated the tf code, we are using var.flag in resource "aws_vpc_endpoint_route_table_association" "s3_vpc_endpoint" as well.

tbugfinder · 2020-04-15T16:48:11Z

I just run into this error message, too.
I'm using the terraform-registry module terraform-aws-vpc.


       Error: Provider produced inconsistent result after apply

       When applying changes to
       module.aws_vpc_test.aws_vpc_endpoint_route_table_association.private_s3[0],
       provider "registry.terraform.io/-/aws" produced an unexpected new value for
       was present, but now absent.

       This is a bug in the provider, which should be reported in the provider's own
       issue tracker.

Terraform v0.12.24

       Initializing provider plugins...
       - Checking for available provider plugins...
       - Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
       - Downloading plugin for provider "template" (hashicorp/template) 2.1.2...
       - Downloading plugin for provider "aws" (hashicorp/aws) 2.54.0...
       - Downloading plugin for provider "random" (hashicorp/random) 2.2.1...
       - Downloading plugin for provider "local" (hashicorp/local) 1.4.0...

ewbankkit · 2020-05-14T12:58:29Z

Relevant:

Clarify Terraform 0.12 Resource Missing Error Messaging terraform#20688

This is most likely an eventual consistency issue. The association is successfully created but when the provider then turns around and reads back the state, the AWS EC2 service can't find it.
The fix will be to add a retry loop in resourceAwsVpcEndpointRouteTableAssociationCreate.

queglay · 2020-05-27T22:36:17Z

I also see it with aws_network_interface_sg_attachment

3776 [23:29:14] Error: timeout - last error: ssh: rejected: connect failed (Connection timed out)
3777 [23:29:14] 
3778 [23:29:14] 
3779 [23:29:14] 
3780 [23:29:14] Error: Provider produced inconsistent result after apply
3781 [23:29:14] 
3782 [23:29:14] When applying changes to
3783 [23:29:14] module.softnas.aws_network_interface_sg_attachment.sg_attachment_vpn[0],
3784 [23:29:14] provider "registry.terraform.io/-/aws" produced an unexpected new value for
3785 [23:29:14] was present, but now absent.
3786 [23:29:14] 
3787 [23:29:14] This is a bug in the provider, which should be reported in the provider's own
3788 [23:29:14] issue tracker.

spicysomtam · 2020-06-25T16:19:14Z

I just suffered this; it seemed to have setup the route associations. I re-ran the apply and it needed to add the network associations which worked (even though they are already there). Then another plan showed all was good. So I guess the answer is just do another apply?

tbugfinder · 2020-06-25T17:53:19Z

Well, yes and no.
In most cases it applies changes in a single step. In few cases this error is raised....

bashims · 2020-07-30T19:22:39Z

Seeing this problem with Terraform v0.12.27 and AWS provider v2.70.0

This issue seems to be non-deterministic.

drexler · 2020-08-13T16:17:38Z

Same issue. Re-applying the plan does resolve it.

Terraform v0.12.24
provider.aws v3.1.0

mwheeler-aicradle · 2020-10-20T16:51:42Z

Same issue. Encountered with TF v0.12.28, and provider v3.11.0.

fractos · 2020-10-28T15:27:32Z

Seeing this for TF v0.12.29 and provider 2.70.0. Generally seems to be the s3 or dynamoDB endpoints. Non-deterministic, probably happens 1 time in 20.

lifelofranco · 2020-11-11T05:23:18Z

Same issue with TF 0.13.5 and provider v3.12.0.

Error: Provider produced inconsistent result after apply

When applying changes to
module.vpc.aws_vpc_endpoint_route_table_association.public_dynamodb[0],
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

atsalolikhin-spokeo · 2020-11-12T23:02:17Z

We are on Terraform 0.13.3 and on hashicorp/aws v2.70.0.

Apply failed with

Error: Provider produced inconsistent result after apply

When applying changes to
module.my_module_name.aws_network_interface_sg_attachment.my_server[2],
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

The next apply failed with

Error: security group [id redacted] already attached to interface ID [id redacted]

  on [filename] line 22, in resource "aws_network_interface_sg_attachment" "my_server":
  22: resource "aws_network_interface_sg_attachment" "my_server" {

So did the one after it.

We had to manually detach the Security Group, then apply again. That resolved it.

ewbankkit · 2020-11-14T11:44:20Z

Given that this is happening for Gateway VPC endpoints it is likely best to use the same logic as used in aws_route and retry after creating the association to ensure it can be read back correctly.

ewbankkit · 2020-11-14T22:20:21Z

aws_vpc_endpoint_route_table_association issue addressed in #16200.
aws_network_interface_sg_attachment issue raised in #16201.

dohoangkhiem · 2020-12-16T04:08:15Z

Got it (though rarely, but causing serious issue when it happened) with aws_eip_association , is there any PR for that resource yet?
Also, any good scenario for re-produce?

eistrati · 2021-02-04T15:06:11Z

I was able to reproduce this issue with terraform vpc module and dynamodb gateway enabled:

module.vpc.aws_vpc_endpoint_route_table_association.private_dynamodb[0]: Creating...

Error: Provider produced inconsistent result after apply

When applying changes to
module.vpc.aws_vpc_endpoint_route_table_association.private_dynamodb[0],
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

My terraform configuration looks like this:

data "aws_region" "current" {}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.70.0"

  name = var.resource["name"]
  cidr = lookup(var.cidr, data.aws_region.current.name, "")
  azs  = keys(lookup(var.azs, data.aws_region.current.name, {}))

  private_subnets = [
    for i in values(lookup(var.azs, data.aws_region.current.name, {})):
    cidrsubnet(lookup(var.cidr, data.aws_region.current.name, ""), var.resource["subcidr_newbits"], var.resource["subcidr_netnum_private"] + i)
  ]

  #public_subnets = [
  #  for i in values(lookup(var.azs, data.aws_region.current.name, {})):
  #  cidrsubnet(lookup(var.cidr, data.aws_region.current.name, ""), var.resource["subcidr_newbits"], var.resource["subcidr_netnum_public"] + i)
  #]

  manage_default_security_group  = true
  default_security_group_ingress = [{}]
  default_security_group_egress  = [{}]

  enable_dns_hostnames = true
  enable_dns_support   = true

  enable_s3_endpoint       = false
  enable_dynamodb_endpoint = true
}

variable "resource" {
  type    = map(string)
  default = {
    name                       = "devops"
    subcidr_newbits            = 3
    subcidr_netnum_private     = 0
    subcidr_netnum_public      = 10
  }
}

variable "cidr" {
  type    = map(string)
  default = {
    "us-east-1" = "10.8.64.0/21",
    "us-east-2" = "10.10.64.0/21"
  }
}

variable "azs" {
  type    = map(map(string))
  default = {
    "us-east-1" = {
      "us-east-1a" = 0
      "us-east-1b" = 1
      "us-east-1d" = 2
    },
    "us-east-2" = {
      "us-east-2b" = 0
      "us-east-2c" = 1
      "us-east-2d" = 2
    }
  }
}

For reference:

ewbankkit · 2021-02-04T16:32:10Z

Proposal to consistently deal with read-after-create failing:

Implement d.IsNewResource() Checks In Resource Read Functions #16796

…create eventual consistency Reference: #12449 Reference: #16796 Output from acceptance testing in AWS Commercial: ``` --- PASS: TestAccAWSVpcEndpointRouteTableAssociation_basic (36.76s) ``` Output from acceptance testing in AWS GovCloud (US): ``` --- PASS: TestAccAWSVpcEndpointRouteTableAssociation_basic (40.65s) ```

…create eventual consistency (#18465) Reference: #12449 Reference: #16796 Output from acceptance testing in AWS Commercial: ``` --- PASS: TestAccAWSVpcEndpointRouteTableAssociation_basic (36.76s) ``` Output from acceptance testing in AWS GovCloud (US): ``` --- PASS: TestAccAWSVpcEndpointRouteTableAssociation_basic (40.65s) ```

skeswani · 2021-04-04T17:03:26Z

I was wondering if anyone was able to work around this issue by adding a null resource with a small 5-30 second sleep on it.
if its a read-after-create consistency issue, and if "eventually" means a few seconds , then this should be a simple workaround.

did anyone attempt to and/or succeed to resolve it in this manner?

ghost · 2021-05-02T17:10:49Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

ghost added the service/ec2 Issues and PRs that pertain to the ec2 service. label Mar 18, 2020

github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Mar 18, 2020

ewbankkit mentioned this issue May 15, 2020

resource/aws_subnet: an unexpected new value for was present, but now absent. #12829

Closed

This comment has been minimized.

Sign in to view

This was referenced Nov 14, 2020

r/aws_vpc_endpoint_route_table_association: Ensure that VPC endpoint to route table association can be read before Create is successful #16200

Closed

r/aws_network_interface_sg_attachment: "Error: Provider produced inconsistent result after apply" #16201

Closed

ewbankkit added bug Addresses a defect in current functionality. and removed needs-triage Waiting for first response or review from a maintainer. labels Nov 19, 2020

molecule-z mentioned this issue Nov 27, 2020

[BUG] Failed to Create the security group rules aliyun/terraform-provider-alicloud#3079

Open

dohoangkhiem mentioned this issue Dec 17, 2020

resource/aws_eip_association: fix eventual consistency issue when associating EIP #16808

Merged

bflad mentioned this issue Mar 18, 2021

Implement d.IsNewResource() Checks In Resource Read Functions #16796

Closed

bflad self-assigned this Mar 29, 2021

bflad mentioned this issue Mar 29, 2021

resource/aws_vpc_endpoint_route_table_association: Handle read-after-create eventual consistency #18465

Merged

bflad closed this as completed in #18465 Apr 2, 2021

ghost locked as resolved and limited conversation to collaborators May 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Provider produced inconsistent result after apply - aws_vpc_endpoint_route_table_association.s3_vpc_endpoint #12449

Provider produced inconsistent result after apply - aws_vpc_endpoint_route_table_association.s3_vpc_endpoint #12449

falgofrancis commented Mar 18, 2020 •

edited

Loading

ewbankkit commented Mar 18, 2020

falgofrancis commented Mar 18, 2020 •

edited

Loading

tbugfinder commented Apr 15, 2020

ewbankkit commented May 14, 2020 •

edited

Loading

queglay commented May 27, 2020

spicysomtam commented Jun 25, 2020

tbugfinder commented Jun 25, 2020

bashims commented Jul 30, 2020

drexler commented Aug 13, 2020

mwheeler-aicradle commented Oct 20, 2020

fractos commented Oct 28, 2020 •

edited

Loading

lifelofranco commented Nov 11, 2020

atsalolikhin-spokeo commented Nov 12, 2020

This comment has been minimized.

ewbankkit commented Nov 14, 2020

ewbankkit commented Nov 14, 2020 •

edited

Loading

dohoangkhiem commented Dec 16, 2020

eistrati commented Feb 4, 2021

ewbankkit commented Feb 4, 2021

skeswani commented Apr 4, 2021 •

edited

Loading

ghost commented May 2, 2021

Provider produced inconsistent result after apply - aws_vpc_endpoint_route_table_association.s3_vpc_endpoint #12449

Provider produced inconsistent result after apply - aws_vpc_endpoint_route_table_association.s3_vpc_endpoint #12449

Comments

falgofrancis commented Mar 18, 2020 • edited Loading

Community Note

Terraform Version 0.12.23 aws provider version 2.51.0

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

ewbankkit commented Mar 18, 2020

falgofrancis commented Mar 18, 2020 • edited Loading

tbugfinder commented Apr 15, 2020

ewbankkit commented May 14, 2020 • edited Loading

queglay commented May 27, 2020

spicysomtam commented Jun 25, 2020

tbugfinder commented Jun 25, 2020

bashims commented Jul 30, 2020

drexler commented Aug 13, 2020

mwheeler-aicradle commented Oct 20, 2020

fractos commented Oct 28, 2020 • edited Loading

lifelofranco commented Nov 11, 2020

atsalolikhin-spokeo commented Nov 12, 2020

This comment has been minimized.

ewbankkit commented Nov 14, 2020

ewbankkit commented Nov 14, 2020 • edited Loading

dohoangkhiem commented Dec 16, 2020

eistrati commented Feb 4, 2021

ewbankkit commented Feb 4, 2021

skeswani commented Apr 4, 2021 • edited Loading

ghost commented May 2, 2021

falgofrancis commented Mar 18, 2020 •

edited

Loading

falgofrancis commented Mar 18, 2020 •

edited

Loading

ewbankkit commented May 14, 2020 •

edited

Loading

fractos commented Oct 28, 2020 •

edited

Loading

ewbankkit commented Nov 14, 2020 •

edited

Loading

skeswani commented Apr 4, 2021 •

edited

Loading