apply default tags to wafv2 or cloudfront_distribution and error

[watarukura]

Related:

• aws_wafv2_web_acl : Provider produced inconsistent final plan #23390
• aws_wafv2_web_acl apply fails when default action changes #23423
• Error: Provider produced inconsistent final plan - invalid new value for .rule: planned set element #23936
• aws_wafv2_web_acl - Error: Provider produced inconsistent final plan #23992
• 👉 apply default tags to wafv2 or cloudfront_distribution and error #24386
• [Bug]: Error "Provider produced inconsistent final plan" when adding a default tag to a aws_wafv2_web_acl #27175
• [Bug]: Provider produced inconsistent final plan for complex aws_wafv2_web_acl configurations #27273
• [Bug]: aws_wafv2_web_acl description/tag changes result in inconsistent final plan #27479
• [Bug]: "registry.terraform.io/hashicorp/aws" produced an invalid new value for .rule: planned set element #28191
• [Bug]: aws_wafv2_web_acl is unable to use dynamic block to manage "rule_action_override" #28672
• [Bug]: Error: Provider produced inconsistent final plan #29012
• [Bug]: Error: Provider produced inconsistent final plan #29304
• [Bug]: aws_wafv2_web_acl Error: Provider produced inconsistent final plan #30858

Hi all,

We meet a bug with add default_tags to cloudfront_distribution or wafv2.
(Attach wafv2 to cloudfront_distribution)

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request * Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request * If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

❯ terraform -v
Terraform v1.1.7
on darwin_amd64
+ provider registry.terraform.io/datadog/datadog v3.9.0
+ provider registry.terraform.io/hashicorp/aws v4.8.0

Affected Resource(s)

• aws_cloudfront_distribution
• aws_wafv2_web_acl

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

hcl # Copy-paste your Terraform configurations here - for large Terraform configs, # please use a service like Dropbox and share a link to the ZIP file. For # security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

provider "aws" {
  region = "ap-northeast-1"

  default_tags {
    tags = {
      Env            = "stg"
      Project        = "our_project_name"
    }
  }
}

resource "aws_cloudfront_distribution" "api" {
  aliases = [local.full_domain]

  web_acl_id = aws_wafv2_web_acl.api.arn

  custom_error_response {
    error_caching_min_ttl = 300
    error_code            = 500
    response_code         = 200
    response_page_path    = "/custom_404.html"
  }

  default_cache_behavior {
    allowed_methods = ["HEAD", "OPTIONS", "GET", "PUT", "POST", "DELETE", "PATCH"]
    cached_methods  = ["HEAD", "OPTIONS", "GET"]
    compress        = true

    default_ttl = 0

    max_ttl = 31536000
    min_ttl = 0

    forwarded_values {
      query_string = true
      cookies {
        forward = "all"
      }
    }
    smooth_streaming       = false
    target_origin_id       = var.storage.api_bucket.id
    trusted_signers        = []
    viewer_protocol_policy = "redirect-to-https"
  }

  default_root_object = local.default_root_object

  enabled         = true
  is_ipv6_enabled = false
  http_version    = "http2"

  logging_config {
    bucket          = aws_s3_bucket.logging_bucket.bucket_domain_name
    include_cookies = true
    prefix          = local.logging_prefix
  }

  origin {
    domain_name = "${aws_route53_record.alb.name}.${var.domain.name}"
    origin_id   = "${aws_route53_record.alb.name}.${var.domain.name}"

    custom_header {
      name  = "x-pre-shared-key"
      value = var.custom_header
    }

    custom_header {
      name  = "User-agent"
      value = "Amazon CloudFront"
    }

    custom_origin_config {
      http_port              = 80
      origin_protocol_policy = "https-only"
      origin_read_timeout    = 60
      https_port             = 443
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  origin {
    domain_name = var.storage.api_bucket.bucket_regional_domain_name
    origin_id   = var.storage.api_bucket.id

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.api.cloudfront_access_identity_path
    }
  }

  price_class = "PriceClass_All"

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  viewer_certificate {
    cloudfront_default_certificate = false
    acm_certificate_arn            = var.domain.virginia_cert_arn
    minimum_protocol_version       = "TLSv1.2_2021"
    ssl_support_method             = "sni-only"
  }

  retain_on_delete    = false
  wait_for_deployment = true

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_cloudfront_origin_access_identity" "api" {
  comment = "api"
}

resource "aws_route53_record" "api" {
  zone_id = data.aws_route53_zone.route53_zone.zone_id

  allow_overwrite = true

  name = local.full_domain
  type = "A"

  alias {
    name                   = aws_cloudfront_distribution.api.domain_name
    zone_id                = aws_cloudfront_distribution.api.hosted_zone_id
    evaluate_target_health = false
  }

  lifecycle {
    prevent_destroy = true
    ignore_changes  = [alias]
  }
}

resource "aws_wafv2_web_acl" "api" {
  name        = "managed-rule-api-${var.env}"
  description = "managed rules"
  scope       = "CLOUDFRONT"
  provider    = aws.virginia

  default_action {
    allow {}
  }

  rule {
    name     = "DenyListIP"
    priority = 1

    action {
      block {}
    }

    statement {
      ip_set_reference_statement {
        arn = aws_wafv2_ip_set.deny.arn
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "DenyListIpMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AllowListIP"
    priority = 2

    action {
      allow {}
    }

    statement {
      ip_set_reference_statement {
        arn = aws_wafv2_ip_set.allow.arn
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AllowListIpMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AWSRateBasedRule"
    priority = 5

    action {
      count {}
    }

    statement {
      rate_based_statement {
        limit              = 1000
        aggregate_key_type = "IP"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AWSRateBasedRuleMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AWSManagedRulesCommonRuleSet"
    priority = 10

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesCommonRuleSet"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AWSManagedRulesCommonRuleSetMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AWSManagedRulesKnownBadInputsRuleSet"
    priority = 20

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesKnownBadInputsRuleSet"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AWSManagedRulesKnownBadInputsRuleSetMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AWSManagedRulesAmazonIpReputationList"
    priority = 30

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesAmazonIpReputationList"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AWSManagedRulesAmazonIpReputationListMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AWSManagedRulesAnonymousIpList"
    priority = 40

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesAnonymousIpList"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AWSManagedRulesAnonymousIpListMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AWSManagedRulesSQLiRuleSet"
    priority = 50

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesSQLiRuleSet"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AWSManagedRulesSQLiRuleSetMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AWSManagedRulesLinuxRuleSet"
    priority = 60

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesLinuxRuleSet"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AWSManagedRulesLinuxRuleSetMetric"
      sampled_requests_enabled   = false
    }
  }

  rule {
    name     = "AWSManagedRulesUnixRuleSet"
    priority = 70

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesUnixRuleSet"
        vendor_name = "AWS"
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = false
      metric_name                = "AWSManagedRulesUnixRuleSetMetric"
      sampled_requests_enabled   = false
    }
  }

  visibility_config {
    cloudwatch_metrics_enabled = false
    metric_name                = "AWSWAFRulesMetric"
    sampled_requests_enabled   = false
  }
}

Debug Output

Panic Output

│ "body":cty.List(cty.EmptyObject),
│ "method":cty.List(cty.EmptyObject),
│ "query_string":cty.List(cty.EmptyObject),
│ "single_header":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
│ "single_query_argument":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
│ "uri_path":cty.List(cty.EmptyObject)})),
│ "text_transformation":cty.Set(cty.Object(map[string]cty.Type{"priority":cty.Number,
│ "type":cty.String}))})),
│ "rule_group_reference_statement":cty.ListValEmpty(cty.Object(map[string]cty.Type{"arn":cty.String,
│ "excluded_rule":cty.List(cty.Object(map[string]cty.Type{"name":cty.String}))})),
│ "size_constraint_statement":cty.ListValEmpty(cty.Object(map[string]cty.Type{"comparison_operator":cty.String,
│ "field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
│ "body":cty.List(cty.EmptyObject),
│ "method":cty.List(cty.EmptyObject),
│ "query_string":cty.List(cty.EmptyObject),
│ "single_header":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
│ "single_query_argument":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
│ "uri_path":cty.List(cty.EmptyObject)})), "size":cty.Number,
│ "text_transformation":cty.Set(cty.Object(map[string]cty.Type{"priority":cty.Number,
│ "type":cty.String}))})),
│ "sqli_match_statement":cty.ListValEmpty(cty.Object(map[string]cty.Type{"field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
│ "body":cty.List(cty.EmptyObject),
│ "method":cty.List(cty.EmptyObject),
│ "query_string":cty.List(cty.EmptyObject),
│ "single_header":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
│ "single_query_argument":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
│ "uri_path":cty.List(cty.EmptyObject)})),
│ "text_transformation":cty.Set(cty.Object(map[string]cty.Type{"priority":cty.Number,
│ "type":cty.String}))})),
│ "xss_match_statement":cty.ListValEmpty(cty.Object(map[string]cty.Type{"field_to_match":cty.List(cty.Object(map[string]cty.Type{"all_query_arguments":cty.List(cty.EmptyObject),
│ "body":cty.List(cty.EmptyObject),
│ "method":cty.List(cty.EmptyObject),
│ "query_string":cty.List(cty.EmptyObject),
│ "single_header":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
│ "single_query_argument":cty.List(cty.Object(map[string]cty.Type{"name":cty.String})),
│ "uri_path":cty.List(cty.EmptyObject)})),
│ "text_transformation":cty.Set(cty.Object(map[string]cty.Type{"priority":cty.Number,
│ "type":cty.String}))}))})}),
│ "visibility_config":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"cloudwatch_metrics_enabled":cty.False,
│ "metric_name":cty.StringVal("AWSManagedRulesUnixRuleSetMetric"),
│ "sampled_requests_enabled":cty.False})})}) does not correlate with
│ any element in actual.
│ 
│ This is a bug in the provider, which should be reported in the
│ provider's own issue tracker.

Expected Behavior

add default_tags

Actual Behavior

cannot add default_tags

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[watarukura]

Add ignore_changes = [tags_all] to resource aws_wafv2_web_acl, and applyable.

[gforien]

Same problem here. Note that

• I got an error for each rule with a managed_rule_group_statement that I was adding to the aws_wafv2_web_acl.
• I did not get any error for the rules with a rule_group_reference_statement
• I did not get any error for the rule with a statement

Error: Provider produced inconsistent final plan

When expanding the plan for aws_wafv2_web_acl.wafv2[0] to include new values
learned so far during apply, provider "registry.terraform.io/hashicorp/aws"
produced an invalid new value for .rule: planned set element
cty.ObjectVal(map[string]cty.Value{"action":cty.ListValEmpty(cty.Object(map[string]cty.Type{"allow":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
...
...

Error: Provider produced inconsistent final plan

When expanding the plan for aws_wafv2_web_acl.wafv2[0] to include new values
learned so far during apply, provider "registry.terraform.io/hashicorp/aws"
produced an invalid new value for .rule: planned set element
cty.ObjectVal(map[string]cty.Value{"action":cty.ListValEmpty(cty.Object(map[string]cty.Type{"allow":cty.List(cty.Object(map[string]cty.Type{"custom_request_handling":cty.List(cty.Object(map[string]cty.Type{"insert_header":cty.Set(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String}))}))})),
...
...
(n times)

Thank you @watarukura for sharing your workaround. It didn't work for me, but it was similar.

I had a conflict between the default_tags on the provider and the tags on the aws_wafv2_web_acl resource.

Terraform version 1.2.4 on darwin_amd64.
Provider registry.terraform.io/hashicorp/aws v4.24.0

[limitusus]

Faced the same prob.
I could not mitigate with ignore_changes = [tags_all], and tagged WAFv2 ACL resources with aws-cli (wafv2 tag-resource).

[mmaetzler]

FYI: This issues with WAFv2 has been reported multiple times:

• aws_wafv2_web_acl : Provider produced inconsistent final plan #23390
• aws_wafv2_web_acl apply fails when default action changes #23423
• Error: Provider produced inconsistent final plan - invalid new value for .rule: planned set element #23936
• aws_wafv2_web_acl - Error: Provider produced inconsistent final plan #23992
• apply default tags to wafv2 or cloudfront_distribution and error #24386
• [Bug]: Error "Provider produced inconsistent final plan" when adding a default tag to a aws_wafv2_web_acl #27175
• [Bug]: Provider produced inconsistent final plan for complex aws_wafv2_web_acl configurations #27273
• [Bug]: aws_wafv2_web_acl description/tag changes result in inconsistent final plan #27479

[YakDriver]

NOTE: I cannot reproduce this error using Terraform v1.5+/AWS provider v5.7+ after trying various configurations. Retry using a minimum of Terraform v1.4.2/AWS provider v4.67.0 but preferably Terraform v1.5.3+/AWS provider v5.8.0+ and let us know if this is still a problem! If we don't hear back and can't reproduce, we plan to close this on or around July 20, 2023. The evidence suggests this is OBE (ie, fixed in the interim).

For more details see #23992 (comment) and #28672 (comment).

[justinretzolk]

Hi all 👋 As was mentioned above, this issue appears to be fixed when using a minimum Terraform version of 1.4.2 and a minimum AWS Provider version of 4.67.0 (preferably Terraform 1.5.3 or later and AWS Provider 5.8.0 or later). If you experience additional unexpected behaviors with versions that meet these parameters, please open a new issue so that we can investigate further.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.