google_bigquery_table configured as a Biglake table show permadiff on partition column

[nickozilla]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

hashicorp/google v5.8.0
hashicorp/google-beta v5.8.0

Affected Resource(s)

google_bigquery_table

Terraform Configuration Files

resource "google_bigquery_table" "inference_archive_table" {
  for_each = local.biglake_tables

  dataset_id          = module.dataset[each.value.dataset].google_bigquery_dataset.dataset_id
  table_id            = "${each.value.model}_${each.value.inference_type}_v${each.value.serving_version}"
  deletion_protection = false
  schema = templatefile("schemas/${var.environment}/${each.value.filename}.tpl", {
    no_access_policy_tag_id                 = data.terraform_remote_state.taxonomies.outputs.catalog["no_access"].id
  })

  external_data_configuration {
    autodetect    = false
    source_format = "NEWLINE_DELIMITED_JSON"
    connection_id = "${var.project_id}.eu.connection"

    metadata_cache_mode = "MANUAL"
    source_uris         = ["gs://${var.bucket}/thingone=${each.value.thingone}/thingtwo=${each.value.thingtwo}/*"]

    hive_partitioning_options {
      mode                     = "CUSTOM"
      source_uri_prefix        = "gs://${var.bucket}/thingone=${each.value.thingone}/thingtwo=${each.value.thingtwo}/{date:STRING}"
      require_partition_filter = false
    }
  }
}

Debug Output

Panic Output

Expected Behavior

Terraform should show the diff when rows are changed in the table schema, if rows are not changed, there should be no diff.

Actual Behavior

Permadiff of the tables, due to the partition column not being declared in the schema, forcing a replacement.

              - {
                  - mode = "NULLABLE"
                  - name = "date"
                  - type = "STRING"
                },
            ] # forces replacement

Steps to Reproduce

1. terraform apply successfully
2. Re run terraform plan, see that the partition column is triggering a recreate of the table.

Important Factoids

Functionally this isn't blocking our use of biglake tables via terraform - as the recreate of the tables doesn't affect the underlying data which is in GCS. But it does make reviewing the plan output for future releases that much more noisy, and generally recreating tables where not required is a bad thing.

The workaround is to use the lifecycle meta argument like below, but this means we'd be unable to make any schema changes to tables once created, which is less desirable.

lifecycle {
    ignore_changes = [
      schema
    ]
  }

References

b/318873211

[wj-chen]

I believe this is the same issue as reported in #12465. Could you take a look and confirm?