Terraform Support for Dataproc Batches.

[animeshnandanwar]

Release Note Template for Downstream PRs (will be copied)

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 10 files changed, 3784 insertions(+), 3 deletions(-))
google-beta provider: Diff ( 10 files changed, 3784 insertions(+), 3 deletions(-))
terraform-google-conversion: Diff ( 4 files changed, 1030 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_batch (8 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_batch" "primary" {
  environment_config {
    execution_config {
      idle_ttl        = # value needed
      kms_key         = # value needed
      network_tags    = # value needed
      service_account = # value needed
      staging_bucket  = # value needed
      subnetwork_uri  = # value needed
      ttl             = # value needed
    }
    peripherals_config {
      metastore_service = # value needed
      spark_history_server_config {
        dataproc_cluster = # value needed
      }
    }
  }
  labels = # value needed
  pyspark_batch {
    archive_uris     = # value needed
    args             = # value needed
    file_uris        = # value needed
    jar_file_uris    = # value needed
    python_file_uris = # value needed
  }
  runtime_config {
    container_image = # value needed
    repository_config {
      pypi_repository_config {
        pypi_repository = # value needed
      }
    }
    version = # value needed
  }
  spark_batch {
    archive_uris      = # value needed
    file_uris         = # value needed
    main_jar_file_uri = # value needed
  }
  spark_r_batch {
    archive_uris    = # value needed
    args            = # value needed
    file_uris       = # value needed
    main_r_file_uri = # value needed
  }
  spark_sql_batch {
    jar_file_uris   = # value needed
    query_variables = # value needed
  }
  state_history {
    state         = # value needed
    state_message = # value needed
  }
}

Resource: google_dataproc_batch_iam_binding (2 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_batch_iam_binding" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

Resource: google_dataproc_batch_iam_member (1 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_batch_iam_member" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 63
Passed tests: 52
Skipped tests: 5
Affected tests: 6

Click here to see the affected service packages

• dataproc

Action taken

Found 6 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataprocBatchIamBindingGenerated|TestAccDataprocBatchIamMemberGenerated|TestAccDataprocBatchIamPolicyGenerated|TestAccDataprocBatch_dataprocBatchPysparkExample|TestAccDataprocBatch_dataprocBatchSparkExample|TestAccDataprocBatch_dataprocBatchSparksqlExample

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataprocBatchIamBindingGenerated[Error message] [Debug log]
TestAccDataprocBatchIamMemberGenerated[Error message] [Debug log]
TestAccDataprocBatchIamPolicyGenerated[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchPysparkExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparkExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparksqlExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 10 files changed, 3786 insertions(+), 3 deletions(-))
google-beta provider: Diff ( 10 files changed, 3786 insertions(+), 3 deletions(-))
terraform-google-conversion: Diff ( 4 files changed, 1030 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_batch (8 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_batch" "primary" {
  environment_config {
    execution_config {
      idle_ttl        = # value needed
      kms_key         = # value needed
      network_tags    = # value needed
      service_account = # value needed
      staging_bucket  = # value needed
      subnetwork_uri  = # value needed
      ttl             = # value needed
    }
    peripherals_config {
      metastore_service = # value needed
      spark_history_server_config {
        dataproc_cluster = # value needed
      }
    }
  }
  labels = # value needed
  pyspark_batch {
    archive_uris     = # value needed
    args             = # value needed
    file_uris        = # value needed
    jar_file_uris    = # value needed
    python_file_uris = # value needed
  }
  runtime_config {
    container_image = # value needed
    repository_config {
      pypi_repository_config {
        pypi_repository = # value needed
      }
    }
    version = # value needed
  }
  spark_batch {
    archive_uris      = # value needed
    file_uris         = # value needed
    main_jar_file_uri = # value needed
  }
  spark_r_batch {
    archive_uris    = # value needed
    args            = # value needed
    file_uris       = # value needed
    main_r_file_uri = # value needed
  }
  spark_sql_batch {
    jar_file_uris   = # value needed
    query_variables = # value needed
  }
  state_history {
    state         = # value needed
    state_message = # value needed
  }
}

Resource: google_dataproc_batch_iam_binding (2 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_batch_iam_binding" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

Resource: google_dataproc_batch_iam_member (1 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_batch_iam_member" "primary" {
  condition {
    description = # value needed
    expression  = # value needed
    title       = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 63
Passed tests: 52
Skipped tests: 5
Affected tests: 6

Click here to see the affected service packages

• dataproc

Action taken

Found 6 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataprocBatchIamBindingGenerated|TestAccDataprocBatchIamMemberGenerated|TestAccDataprocBatchIamPolicyGenerated|TestAccDataprocBatch_dataprocBatchPysparkExample|TestAccDataprocBatch_dataprocBatchSparkExample|TestAccDataprocBatch_dataprocBatchSparksqlExample

Get to know how VCR tests work

[modular-magician]

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataprocBatchIamBindingGenerated[Error message] [Debug log]
TestAccDataprocBatchIamMemberGenerated[Error message] [Debug log]
TestAccDataprocBatchIamPolicyGenerated[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchPysparkExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparkExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparksqlExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 5 files changed, 2919 insertions(+), 2 deletions(-))
google-beta provider: Diff ( 5 files changed, 2919 insertions(+), 2 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 656 insertions(+))

Errors

google provider:

• The diff processor failed to build. This is usually due to the downstream provider failing to compile.

google-beta provider:

• The diff processor failed to build. This is usually due to the downstream provider failing to compile.

[modular-magician]

Tests analytics

Total tests: 0
Passed tests: 0
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

• dataproc

Non-exercised tests

Tests were added that are skipped in VCR:

• TestAccDataprocBatch_dataprocBatchPysparkExample
• TestAccDataprocBatch_dataprocBatchSparkExample
• TestAccDataprocBatch_dataprocBatchSparksqlExample

$\textcolor{red}{\textsf{Errors occurred during REPLAYING mode. Please fix them to complete your PR.}}$
View the build log

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 6 files changed, 2900 insertions(+), 3 deletions(-))
google-beta provider: Diff ( 6 files changed, 2900 insertions(+), 3 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 656 insertions(+))

Errors

google provider:

• The diff processor failed to build. This is usually due to the downstream provider failing to compile.

google-beta provider:

• The diff processor failed to build. This is usually due to the downstream provider failing to compile.

[modular-magician]

Tests analytics

Total tests: 0
Passed tests: 0
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

all service packages are affected

Non-exercised tests

Tests were added that are skipped in VCR:

• TestAccDataprocBatch_dataprocBatchPysparkExample
• TestAccDataprocBatch_dataprocBatchSparkExample
• TestAccDataprocBatch_dataprocBatchSparksqlExample

$\textcolor{red}{\textsf{Errors occurred during REPLAYING mode. Please fix them to complete your PR.}}$
View the build log

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 6 files changed, 2907 insertions(+), 3 deletions(-))
google-beta provider: Diff ( 6 files changed, 2907 insertions(+), 3 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 656 insertions(+))

Errors

google provider:

• The diff processor failed to build. This is usually due to the downstream provider failing to compile.

google-beta provider:

• The diff processor failed to build. This is usually due to the downstream provider failing to compile.

[modular-magician]

Tests analytics

Total tests: 0
Passed tests: 0
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

all service packages are affected

Non-exercised tests

Tests were added that are skipped in VCR:

• TestAccDataprocBatch_dataprocBatchPysparkExample
• TestAccDataprocBatch_dataprocBatchSparkExample
• TestAccDataprocBatch_dataprocBatchSparksqlExample

$\textcolor{red}{\textsf{Errors occurred during REPLAYING mode. Please fix them to complete your PR.}}$
View the build log

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 6 files changed, 2907 insertions(+), 3 deletions(-))
google-beta provider: Diff ( 6 files changed, 2907 insertions(+), 3 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 656 insertions(+))

Errors

google provider:

• The diff processor failed to build. This is usually due to the downstream provider failing to compile.

google-beta provider:

• The diff processor failed to build. This is usually due to the downstream provider failing to compile.

[modular-magician]

Tests analytics

Total tests: 0
Passed tests: 0
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

all service packages are affected

Non-exercised tests

Tests were added that are skipped in VCR:

• TestAccDataprocBatch_dataprocBatchPysparkExample
• TestAccDataprocBatch_dataprocBatchSparkExample
• TestAccDataprocBatch_dataprocBatchSparksqlExample

$\textcolor{red}{\textsf{Errors occurred during REPLAYING mode. Please fix them to complete your PR.}}$
View the build log

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 7 files changed, 3165 insertions(+), 3 deletions(-))
google-beta provider: Diff ( 7 files changed, 3165 insertions(+), 3 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 698 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_batch (4 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_batch" "primary" {
  environment_config {
    execution_config {
      idle_ttl        = # value needed
      kms_key         = # value needed
      network_uri     = # value needed
      service_account = # value needed
      staging_bucket  = # value needed
    }
    peripherals_config {
      metastore_service = # value needed
      spark_history_server_config {
        dataproc_cluster = # value needed
      }
    }
  }
  pyspark_batch {
    archive_uris     = # value needed
    args             = # value needed
    file_uris        = # value needed
    jar_file_uris    = # value needed
    python_file_uris = # value needed
  }
  runtime_config {
    container_image = # value needed
    repository_config {
      pypi_repository_config {
        pypi_repository = # value needed
      }
    }
    version = # value needed
  }
  spark_batch {
    archive_uris      = # value needed
    file_uris         = # value needed
    main_jar_file_uri = # value needed
  }
  spark_r_batch {
    archive_uris = # value needed
    file_uris    = # value needed
  }
  spark_sql_batch {
    jar_file_uris   = # value needed
    query_variables = # value needed
  }
  state_history {
    state         = # value needed
    state_message = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 3924
Passed tests: 3507
Skipped tests: 411
Affected tests: 6

Click here to see the affected service packages

All service packages are affected

Action taken

Found 6 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccBigQueryDatasetAccess_groupByEmailWithMixedCase
• TestAccBigQueryDatasetAccess_userByEmailWithMixedCase
• TestAccDataprocBatch_dataprocBatchPysparkExample
• TestAccDataprocBatch_dataprocBatchSparkExample
• TestAccDataprocBatch_dataprocBatchSparkrExample
• TestAccDataprocBatch_dataprocBatchSparksqlExample

Get to know how VCR tests work

[modular-magician]

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBigQueryDatasetAccess_groupByEmailWithMixedCase[Debug log]
TestAccBigQueryDatasetAccess_userByEmailWithMixedCase[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$

---

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataprocBatch_dataprocBatchPysparkExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparkExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparkrExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparksqlExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

[melinath]

I'm marking this as ready for review so it will show up in my review queue.

[melinath]

The tests are failing in beta because they get a 404 from the create endpoint. For example:

POST /v1beta2/projects/analog-ace-309318/locations/us-central1/batches?alt=json&batchId=tf-test-batch28n9wsuk9x

They're passing when I run them locally in GA, though. Possibly this resource doesn't exist in the beta API?

We would need this resource to be supported in the beta API, or for the beta API to be removed from provider.yaml (if that wouldn't have negative impacts on AutoscalingPolicy - it looks like there aren't any beta-only features on that resource so it should be fine?)

It would be great to also address the missing tests identified by the CI pipeline.

[melinath]

I made a few minor tweaks to clean up the previous compilation errors - hope you don't mind! I'm waiting for the acceptance tests to run, but in the meantime some additional test cases would be great.

The TestAccComputeInstanceTemplate_withNamePrefix failures are unrelated to this PR.

[melinath]

Please add additional cases

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 8 files changed, 3250 insertions(+), 4 deletions(-))
google-beta provider: Diff ( 8 files changed, 3250 insertions(+), 4 deletions(-))
terraform-google-conversion: Diff ( 1 file changed, 698 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_dataproc_batch (4 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_dataproc_batch" "primary" {
  environment_config {
    execution_config {
      idle_ttl        = # value needed
      kms_key         = # value needed
      network_uri     = # value needed
      service_account = # value needed
      staging_bucket  = # value needed
    }
    peripherals_config {
      metastore_service = # value needed
      spark_history_server_config {
        dataproc_cluster = # value needed
      }
    }
  }
  pyspark_batch {
    archive_uris     = # value needed
    args             = # value needed
    file_uris        = # value needed
    jar_file_uris    = # value needed
    python_file_uris = # value needed
  }
  runtime_config {
    container_image = # value needed
    repository_config {
      pypi_repository_config {
        pypi_repository = # value needed
      }
    }
    version = # value needed
  }
  spark_batch {
    archive_uris      = # value needed
    file_uris         = # value needed
    main_jar_file_uri = # value needed
  }
  spark_r_batch {
    archive_uris = # value needed
    file_uris    = # value needed
  }
  spark_sql_batch {
    jar_file_uris   = # value needed
    query_variables = # value needed
  }
}

[modular-magician]

Tests analytics

Total tests: 3905
Passed tests: 3488
Skipped tests: 411
Affected tests: 6

Click here to see the affected service packages

All service packages are affected

#### Non-exercised tests

Tests were added that are skipped in VCR:

• TestCloudDataprocBatchRuntimeConfigVersionDiffSuppress

Action taken

Found 6 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

• TestAccBackupDRBackupVault_fullUpdate
• TestAccComputeInstanceTemplate_withNamePrefix
• TestAccDataprocBatch_dataprocBatchPysparkExample
• TestAccDataprocBatch_dataprocBatchSparkExample
• TestAccDataprocBatch_dataprocBatchSparkrExample
• TestAccDataprocBatch_dataprocBatchSparksqlExample

Get to know how VCR tests work

[modular-magician]

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBackupDRBackupVault_fullUpdate[Debug log]
TestAccComputeInstanceTemplate_withNamePrefix[Debug log]
$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccComputeInstanceTemplate_withNamePrefix[Error message] [Debug log]

Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.

Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.

---

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccDataprocBatch_dataprocBatchPysparkExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparkExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparkrExample[Error message] [Debug log]
TestAccDataprocBatch_dataprocBatchSparksqlExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$

View the build log or the debug log for each test

[melinath]

A couple additional things from the acceptance test run! The tests are failing in beta because they get a 404 from the create endpoint. For example:

POST /v1beta2/projects/analog-ace-309318/locations/us-central1/batches?alt=json&batchId=tf-test-batch28n9wsuk9x

They're passing when I run them locally in GA, though. Possibly this resource doesn't exist in the beta API?

We would need this resource to be supported in the beta API, or for the beta API to be removed from provider.yaml (if that wouldn't have negative impacts on AutoscalingPolicy - it looks like there aren't any beta-only features on that resource so it should be fine?)

It would be great to also address the missing tests identified by the CI pipeline (if possible)

[zli82016]

A couple additional things from the acceptance test run! The tests are failing in beta because they get a 404 from the create endpoint. For example:

POST /v1beta2/projects/analog-ace-309318/locations/us-central1/batches?alt=json&batchId=tf-test-batch28n9wsuk9x

They're passing when I run them locally in GA, though. Possibly this resource doesn't exist in the beta API?

We would need this resource to be supported in the beta API, or for the beta API to be removed from provider.yaml (if that wouldn't have negative impacts on AutoscalingPolicy - it looks like there aren't any beta-only features on that resource so it should be fine?)

@melinath , is it a breaking change to remove the beta API for AutoscalingPolicy resource?

[melinath]

is it a breaking change to remove the beta API for AutoscalingPolicy resource?

I believe that if the GA API behaves the same way it's not something we consider a breaking change. We could call it out in the release notes though.

[github-actions]

@animeshnandanwar, this PR is waiting for action from you. Please address any comments or change requests, or re-request review from a core reviewer if no action is required.

If no action is taken, this PR will be closed in 28 days.

This notification can be disabled with the disable-automatic-closure label.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 358 insertions(+))
google-beta provider: Diff ( 3 files changed, 358 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 105 insertions(+))

Missing test report

Your PR includes resource fields which are not covered by any test.

Resource: google_looker_instance (8 total tests)
Please add an acceptance test which includes these fields. The test should include the following:

resource "google_looker_instance" "primary" {
  psc_config {
    service_attachments {
      local_fqdn                    = # value needed
      target_service_attachment_uri = # value needed
    }
  }
}

[modular-magician]

Tests analytics

Total tests: 7
Passed tests: 7
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

• looker

$\textcolor{green}{\textsf{All tests passed!}}$

View the build log

[zli82016]

I opened a new PR
#11750