Handle semantically equivalent changes from the legacy SDK when planning data sources

[jbardin]

The legacy SDK can cause changes to some empty containers types when the data source configuration is round-tripped through the ReadDataSource call. This can cause a planned data source configurations to appear to have changes when in fact they do not. These changes most manifest as empty container values being swapped for null container values.

These alone do not cause an issue, as the data source is usually read again transparently, and the planned value end up being identical. However, when this behavior is matched with data sources that do not have stable attributes, the combination will cause a new data source value to appear in each plan. This can bee seen with data sources like data.aws_availability_zones as referenced in hashicorp/terraform-provider-aws#14579.

While these extra reads are usually benign, the plan output can get quite busy when there are a lot of data source being used in large configurations, as noted in #25805.

In order to mitigate this, we can possibly implement an imprecise comparison using heuristics specifically for data sources, so that extra reads are not performed when there are no semantically significant changes. Unfortunately we don't have an indication at that point when the data source was using the legacy SDK, so any changes here need to take into account that they will apply to any new SDK versions as well.

[lorengordon]

Further, this causes terraform plan -detailed-exitcode to exit non-zero, which makes it less than useful for detecting drift in scheduled CI plans.

[lorengordon]

@jbardin I think I've found another case where this is still occurring in terraform 0.13.1. I am seeing a persistent diff when the data source references a "conditional" resource (e.g. count = 0), such that the value is empty. Here is a pretty simple, contrived repro:

$ cat main.tf:

locals {
  template = "$${random}"
  vars = {
    random = join("", random_pet.this.*.id)
    empty  = join("", random_pet.empty.*.id)
  }
}

resource random_pet this {
  count = 1
}

resource random_pet empty {
  count = 0
}

data template_file this {
  template = local.template
  vars     = local.vars
}

data external this {
  program = ["jq", "-r", "."]

  query = {
    template = local.template
    vars     = jsonencode(local.vars)
  }
}

output external_result {
  value = data.external.this.result
}

output template_rendered {
  value = data.template_file.this.rendered
}

Run the apply once:

$ terraform apply
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

external_result = {
  "template" = "${random}"
  "vars" = "{\"empty\":\"\",\"random\":\"star-tuna\"}"
}
template_rendered = star-tuna

Then run it again:

$ terraform apply
random_pet.this[0]: Refreshing state... [id=star-tuna]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 <= read (data resources)

Terraform will perform the following actions:

  # data.external.this will be read during apply
  # (config refers to values not yet known)
 <= data "external" "this"  {
      + id      = "-"
      + program = [
          + "jq",
          + "-r",
          + ".",
        ]
      + query   = {
          + "template" = "${random}"
          + "vars"     = jsonencode(
                {
                  + empty  = ""
                  + random = "star-tuna"
                }
            )
        }
      + result  = {
          + "template" = "${random}"
          + "vars"     = jsonencode(
                {
                  + empty  = ""
                  + random = "star-tuna"
                }
            )
        }
    }

  # data.template_file.this will be read during apply
  # (config refers to values not yet known)
 <= data "template_file" "this"  {
      + id       = "228051b9820bc0592514d4f122899016cdb24beccdba197c4a4d9f495e9e5d36"
      + rendered = "star-tuna"
      + template = "${random}"
      + vars     = {
          + "empty"  = ""
          + "random" = "star-tuna"
        }
    }

Plan: 0 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

[lorengordon]

In the repro, the persistent diff only happens when vars contains the empty key/value. Remove that value and it works fine.

[jbardin]

Hi @lorengordon,

That example is unrelated to this PR. Please file a separate issue.

Thanks!

[lorengordon]

Oh ok, the symptom is the same in our case anyway. Thanks for following up @jbardin!

[lorengordon]

Submitted #26100

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.