google_bigquery_table: external_data_configuration.connection_id is detected as having changed when it's not

[flovouin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.2.7

Affected Resource(s)

• google_bigquery_table

Terraform Configuration Files

resource "google_bigquery_table" "table" {
  dataset_id    = google_bigquery_dataset.dataset.dataset_id
  table_id      = "table"
  friendly_name = "table"

  schema = "..."

  external_data_configuration {
    autodetect    = false
    source_format = "NEWLINE_DELIMITED_JSON"
    compression   = "GZIP"
    source_uris   = ["gs://some-path"]

    # This causes a change to be incorrectly detected.
    connection_id = google_bigquery_connection.connection.name

    # This is needed instead of simply using `google_bigquery_connection.connection.name` like the documentation [1] suggests.
    # connection_id = "${google_bigquery_connection.connection.project}.${lower(google_bigquery_connection.connection.location)}.${google_bigquery_connection.connection.connection_id}"

    ignore_unknown_values = true
  }

  deletion_protection = false
}

Expected Behavior

After the initial creation, no change should be detected in the google_bigquery_table.table resource in subsequent plans.

Actual Behavior

The following change is detected:

  # google_bigquery_table.table will be updated in-place
  ~ resource "google_bigquery_table" "table" {
        id                  = "projects/project/datasets/dataset/tables/table"
        # (18 unchanged attributes hidden)

      ~ external_data_configuration {
          ~ connection_id         = "project.eu.connection" -> "projects/project/locations/eu/connections/connection"
            # (5 unchanged attributes hidden)
        }
    }

Note that the connection ID returned by BigQuery has a specific format, project.location.connection, which is not the usual "path" format for resources. This is what is causing the incorrect change detection.

Steps to Reproduce

1. Define a BigQuery table using a BigQuery connection.
2. Run terraform apply: the table is created.
3. Run terraform plan: a change is detected even though the resource has not been modified.

References

[1] https://cloud.google.com/bigquery/docs/biglake-quickstart#create-table-storage

b/300616598

[edwardmedia]

@flovouin it appears fine to me when I assign the connection_id with below value. No permdiff is found in the subsequent plan. Can you share your full debug log so I can take a close look?

connection_id         = "389245411234.us.issue12386"

[flovouin]

Hi @edwardmedia, thanks for your reply.
I'm not sure I understand. Are you setting the connection_id as this hardcoded string directly? In which case, yes, this is indeed the expected format.

My point is that the documentation shows an example of how to reference a connection managed by Terraform:

However when using the output of a google_bigquery_connection (like shown in the GCP doc), the format of the returned connection ID is different. The ID looks like a regular GCP resource URL, which is indeed what is documented in the Google provider:

The result is that the output of google_bigquery_connection is incompatible with what google_bigquery_table expects as input. When following the instructions from the GCP doc, a change will be detected in the google_bigquery_table resource every time.

I haven't found an easy way to get the expected format (project.location.connection) from the google_bigquery_connection resource. However my original post already provides a workaround:

connection_id = "${google_bigquery_connection.connection.project}.${lower(google_bigquery_connection.connection.location)}.${google_bigquery_connection.connection.connection_id}"

[edwardmedia]

@flovouin you are right, I have not found a direct id available we can use. This is what I followed in my testing https://github.com/hashicorp/terraform-provider-google/blob/main/google/resource_bigquery_table_test.go#L1968

[flovouin]

Thanks for confirming the issue @edwardmedia. I guess the easiest solution would be to at least update the documentation, because the current example will result in this issue.
However the most convenient fix would be to either expose the connection ID in its expected format in google_bigquery_connection, or have google_bigquery_table handle the conversion or support both format.
It seems that google_bigquery_table already supports the projects/project/locations/eu/connections/connection format at creation time, it's just that it will result in a permdiff in later plans.

[psvaiter]

Same issue here. I like the idea of "google_bigquery_table handle the conversion" as everything else is already in place.

[edwardmedia]

@melinath what do you think about this issue?

[melinath]

Wasn't able to get to this

[melinath]

it seems like we should probably suppress the diff in this case, since using a standard TF reference works (apart from the fact that the API returns it in a different format.) And we probably would rather let folks use the standard TF reference.

[edwardmedia]

b/245412495

[nevzheng]

resource "google_bigquery_connection" "connection" {
   connection_id = "my-connection-connecn"
   location = "US"
   cloud_resource {}
}

resource "google_bigquery_dataset" "dataset" {
  dataset_id = "my_dataset_dat"
}

resource "google_storage_bucket" "test" {
  name          = "my-bucket-bucketcddfdfdfd"
  location      = "US"
  force_destroy = true
}

resource "google_bigquery_table" "table" {
  dataset_id    = google_bigquery_dataset.dataset.dataset_id
  table_id      = "tablek_fdfdfdfdfddfdfdfdfd"
  

  schema = jsonencode([{"name":"id","type":"INTEGER"}])

  external_data_configuration {
    autodetect    = false
    source_format = "NEWLINE_DELIMITED_JSON"
    compression   = "GZIP"
    source_uris = [
      "gs://${google_storage_bucket.test.name}/*",
    ]

    # This causes a change to be incorrectly detected.
    connection_id = google_bigquery_connection.connection.name

    # This is needed instead of simply using `google_bigquery_connection.connection.name` like the documentation [1] suggests.
    # connection_id = "${google_bigquery_connection.connection.project}.${lower(google_bigquery_connection.connection.location)}.${google_bigquery_connection.connection.connection_id}"

    ignore_unknown_values = true
  }

  deletion_protection = false
}

Here's a minimal example we can run.

I can use the name field for connection id without issue. If we are able to validate this on someone else's setup, I would suggest we close this issue.

I mailed a PR to fix the ID issue GoogleCloudPlatform/magic-modules#9053

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.