add lifecycle_config to dataproc_cluster.cluster_config (#1602)

* Allow domain mapping to succeed if DNS is pending Signed-off-by: Modular Magician <magic-modules@google.com> * Update google_folder import description (#1592) Signed-off-by: Modular Magician <magic-modules@google.com> Co-authored-by: Dana Hoffman <danahoffman@google.com> * add google_kms_secret_ciphertext resource, deprecate datasource (#1586) Signed-off-by: Modular Magician <magic-modules@google.com> Co-authored-by: Dana Hoffman <danahoffman@google.com> * Allow add/removing Bigtable clusters (#1589) Signed-off-by: Modular Magician <magic-modules@google.com> Co-authored-by: Riley Karson <rileykarson@google.com> * Add bootstrapped test networks for service networking tests (#1588) Signed-off-by: Modular Magician <magic-modules@google.com> Co-authored-by: emily <emilyye@google.com> * Update CHANGELOG.md * fix docs for google_bigquery_default_service_account (#1596) Signed-off-by: Modular Magician <magic-modules@google.com> Co-authored-by: Martin Nowak <code@dawg.eu> * Nil return for absent Bigtable resources (#1597) Signed-off-by: Modular Magician <magic-modules@google.com> Co-authored-by: Brian Hildebrandt <brian3.14159@gmail.com> * add lifecycle_config to dataproc_cluster.cluster_config Signed-off-by: Modular Magician <magic-modules@google.com> * add lifecycle_config to dataproc_cluster.cluster_config (#2926) Merged PR #2926. Co-authored-by: Chris Stephens <chrisst@google.com> Co-authored-by: Dana Hoffman <danahoffman@google.com> Co-authored-by: Riley Karson <rileykarson@google.com> Co-authored-by: emily <emilyye@google.com> Co-authored-by: Paddy <paddy@hashicorp.com> Co-authored-by: Martin Nowak <code@dawg.eu> Co-authored-by: Brian Hildebrandt <brian3.14159@gmail.com> Co-authored-by: megan07 <mbang@hashicorp.com>
hashicorp · Jan 8, 2020 · eb202dc · eb202dc
1 parent 0caac89
commit eb202dc
Show file tree

Hide file tree

Showing 10 changed files with 396 additions and 49 deletions.
diff --git a/.changelog/2926.txt b/.changelog/2926.txt
@@ -0,0 +1,3 @@
+```release-note:enhancement
+dataproc: added 'lifecycle_config` to `google_dataproc_cluster.cluster_config`
+```
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,4 +1,56 @@
-## 3.3.1 (Unreleased)
+## 3.4.1 (Unreleased)
+## 3.4.0 (January 07, 2020)
+
+DEPRECATIONS:
+* kms: deprecated `data.google_kms_secret_ciphertext` as there was no way to make it idempotent. Instead, use the `google_kms_secret_ciphertext` resource. ([#1586](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1586))
+
+BREAKING CHANGES:
+* `google_iap_web_iam_*`, `google_iap_web_type_compute_iam_*`, `google_iap_web_type_app_engine_*`,  and `google_iap_app_engine_service_iam_*` resources now support IAM Conditions (beta provider only). If any conditions had been created out of band before this release, take extra care to ensure they are present in your Terraform config so the provider doesn't try to create new bindings with no conditions. Terraform will show a diff that it is adding the condition to the resource, which is safe to apply. ([#1527](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1527))
+* `google_kms_key_ring_iam_*` and `google_kms_crypto_key_iam_*` resources now support IAM Conditions (beta provider only). If any conditions had been created out of band before this release, take extra care to ensure they are present in your Terraform config so the provider doesn't try to create new bindings with no conditions. Terraform will show a diff that it is adding the condition to the resource, which is safe to apply. ([#1524](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1524))
+* cloudrun: Changed `google_cloud_run_domain_mapping` to correctly match Cloud Run API expected format for `spec.route_name`, {serviceName}, instead of invalid projects/{project}/global/services/{serviceName} ([#1563](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1563))
+* compute: Added back ConflictsWith restrictions for ExactlyOneOf restrictions that were removed in v3.3.0 for `google_compute_firewall`, `google_compute_health_check`, and `google_compute_region_health_check`. This effectively changes an API-side failure that was only accessible in v3.3.0 to a plan-time one. ([#1534](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1534))
+* logging: Changed `google_logging_metric.metric_descriptors.labels` from a list to a set ([#1559](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1559))
+* resourcemanager: Added back ConflictsWith restrictions for ExactlyOneOf restrictions that were removed in v3.3.0 for `google_organization_policy`, `google_folder_organization_policy`, and `google_project_organization_policy`. This effectively changes an API-side failure that was only accessible in v3.3.0 to a plan-time one. ([#1534](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1534))
+
+FEATURES:
+* **New Data Source:** `google_sql_ca_certs` ([#1580](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1580))
+* **New Resource:** `google_identity_platform_default_supported_idp_config` ([#1523](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1523))
+* **New Resource:** `google_identity_platform_inbound_saml_config` ([#1523](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1523))
+* **New Resource:** `google_identity_platform_oauth_idp_config` ([#1523](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1523))
+* **New Resource:** `google_identity_platform_tenant_default_supported_idp_config` ([#1523](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1523))
+* **New Resource:** `google_identity_platform_tenant_inbound_saml_config` ([#1523](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1523))
+* **New Resource:** `google_identity_platform_tenant_oauth_idp_config` ([#1523](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1523))
+* **New Resource:** `google_identity_platform_tenant` ([#1523](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1523))
+* **New Resource:** `google_kms_crypto_key_iam_policy` ([#1554](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1554))
+* **New Resource:** `google_kms_secret_ciphertext` ([#1586](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1586))
+
+IMPROVEMENTS:
+* composer: Increased default timeouts for `google_composer_environment` ([#1539](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1539))
+* compute: Added graceful termination to `container_cluster` create calls so that partially created clusters will resume the original operation if the Terraform process is killed mid create. ([#1533](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1533))
+* compute: Fixed `google_compute_disk_resource_policy_attachment` parsing of region from zone to allow for provider-level zone and make error message more accurate` ([#1557](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1557))
+* datafusion: Increased default timeouts for `google_data_fusion_instance` ([#1545](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1545))
+* datafusion: Increased update timeout for updating `google_data_fusion_instance` ([#1538](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1538))
+* healthcare: Enabled request batching for (beta-only) Healthcare API IAM resources `google_healthcare_*_iam_*` to reduce likelihood of errors from very low default write quota. ([#1558](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1558))
+* iap: added support for IAM Conditions to the `google_iap_web_iam_*`, `google_iap_web_type_compute_iam_*`, `google_iap_web_type_app_engine_*`,  and `google_iap_app_engine_service_iam_*` resources (beta provider only) ([#1527](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1527))
+* kms: added support for IAM Conditions to the `google_kms_key_ring_iam_*` and `google_kms_crypto_key_iam_*` resources (beta provider only) ([#1524](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1524))
+* provider: Reduced default `send_after` controlling the time interval after which a batched request sends. ([#1565](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1565))
+
+BUG FIXES:
+* all: fixed issue where many fields that were removed in 3.0.0 would show a diff when they were removed from config ([#1585](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1585))
+* bigquery: fixed `bigquery_table.encryption_configuration` to correctly recreate the table when modified ([#1591](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1591))
+* cloudrun:  Changed `google_cloud_run_domain_mapping` to correctly match Cloud Run API expected format for `spec.route_name`, {serviceName}, instead of invalid projects/{project}/global/services/{serviceName} ([#1563](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1563))
+* cloudrun: Changed `cloud_run_domain_mapping` to poll for success or failure and throw an appropriate error when ready status returns as false. ([#1564](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1564))
+* cloudrun: Fixed `google_cloudrun_service` to allow update instead of force-recreation for changes in `spec` `env` and `command` fields ([#1566](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1566))
+* cloudrun: Removed unsupported update for `google_cloud_run_domain_mapping` to allow force-recreation. ([#1556](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1556))
+* cloudrun: Stopped returning an error when a `cloud_run_domain_mapping` was waiting on DNS verification. ([#1587](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1587))
+* compute: Fixed `google_compute_backend_service` to allow updating `cdn_policy.cache_key_policy.*` fields to false or empty. ([#1569](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1569))
+* compute: Fixed behaviour where `google_compute_subnetwork` did not record a value for `name` when `self_link` was specified. ([#1579](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1579))
+* container: fixed issue where an empty variable in `tags` would cause a crash ([#1543](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1543))
+* endpoints: Added operation wait for `google_endpoints_service` to fix 403 "Service not found" errors during initial creation ([#1560](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1560))
+* logging: Made `google_logging_metric.metric_descriptors.labels` a set to prevent diff from ordering ([#1559](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1559))
+* resourcemanager: added retries for `data.google_organization` ([#1553](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1553))
+* vpcaccess: marked `network` field as required in order to fail invalid configs at plan-time instead of at apply-time ([#1577](https://github.com/terraform-providers/terraform-provider-google-beta/pull/1577))
+
 ## 3.3.0 (December 17, 2019)
 
 BREAKING CHANGES:

diff --git a/google-beta/resource_bigtable_gc_policy.go b/google-beta/resource_bigtable_gc_policy.go
@@ -151,7 +151,7 @@ func resourceBigtableGCPolicyRead(d *schema.ResourceData, meta interface{}) erro
 	if err != nil {
 		log.Printf("[WARN] Removing %s because it's gone", name)
 		d.SetId("")
-		return fmt.Errorf("Error retrieving table. Could not find %s in %s. %s", name, instanceName, err)
+		return nil
 	}
 
 	for _, fi := range ti.FamilyInfos {

diff --git a/google-beta/resource_bigtable_instance.go b/google-beta/resource_bigtable_instance.go
@@ -44,12 +44,10 @@ func resourceBigtableInstance() *schema.Resource {
 						"cluster_id": {
 							Type:     schema.TypeString,
 							Required: true,
-							ForceNew: true,
 						},
 						"zone": {
 							Type:     schema.TypeString,
 							Required: true,
-							ForceNew: true,
 						},
 						"num_nodes": {
 							Type:         schema.TypeInt,
@@ -60,7 +58,6 @@ func resourceBigtableInstance() *schema.Resource {
 							Type:         schema.TypeString,
 							Optional:     true,
 							Default:      "SSD",
-							ForceNew:     true,
 							ValidateFunc: validation.StringInSlice([]string{"SSD", "HDD"}, false),
 						},
 					},
@@ -162,7 +159,7 @@ func resourceBigtableInstanceRead(d *schema.ResourceData, meta interface{}) erro
 	if err != nil {
 		log.Printf("[WARN] Removing %s because it's gone", instanceName)
 		d.SetId("")
-		return fmt.Errorf("Error retrieving instance. Could not find %s. %s", instanceName, err)
+		return nil
 	}
 
 	d.Set("project", project)
@@ -212,27 +209,28 @@ func resourceBigtableInstanceUpdate(d *schema.ResourceData, meta interface{}) er
 	}
 	defer c.Close()
 
-	clusters, err := c.Clusters(ctx, d.Get("name").(string))
-	if err != nil {
-		return fmt.Errorf("Error retrieving clusters for instance %s", err.Error())
+	conf := &bigtable.InstanceWithClustersConfig{
+		InstanceID: d.Get("name").(string),
 	}
 
-	clusterMap := make(map[string]*bigtable.ClusterInfo, len(clusters))
-	for _, cluster := range clusters {
-		clusterMap[cluster.Name] = cluster
-	}
-
-	for _, cluster := range d.Get("cluster").([]interface{}) {
-		config := cluster.(map[string]interface{})
-		cluster_id := config["cluster_id"].(string)
-		if cluster, ok := clusterMap[cluster_id]; ok {
-			if cluster.ServeNodes != config["num_nodes"].(int) {
-				err = c.UpdateCluster(ctx, d.Get("name").(string), cluster.Name, int32(config["num_nodes"].(int)))
-				if err != nil {
-					return fmt.Errorf("Error updating cluster %s for instance %s", cluster.Name, d.Get("name").(string))
-				}
-			}
-		}
+	displayName, ok := d.GetOk("display_name")
+	if !ok {
+		displayName = conf.InstanceID
+	}
+	conf.DisplayName = displayName.(string)
+
+	switch d.Get("instance_type").(string) {
+	case "DEVELOPMENT":
+		conf.InstanceType = bigtable.DEVELOPMENT
+	case "PRODUCTION":
+		conf.InstanceType = bigtable.PRODUCTION
+	}
+
+	conf.Clusters = expandBigtableClusters(d.Get("cluster").([]interface{}), conf.InstanceID)
+
+	_, err = bigtable.UpdateInstanceAndSyncClusters(ctx, c, conf)
+	if err != nil {
+		return fmt.Errorf("Error updating instance. %s", err)
 	}
 
 	return resourceBigtableInstanceRead(d, meta)
@@ -305,6 +303,7 @@ func expandBigtableClusters(clusters []interface{}, instanceID string) []bigtabl
 	return results
 }
 
+// resourceBigtableInstanceValidateDevelopment validates restrictions specific to DEVELOPMENT clusters
 func resourceBigtableInstanceValidateDevelopment(diff *schema.ResourceDiff, meta interface{}) error {
 	if diff.Get("instance_type").(string) != "DEVELOPMENT" {
 		return nil
@@ -318,46 +317,115 @@ func resourceBigtableInstanceValidateDevelopment(diff *schema.ResourceDiff, meta
 	return nil
 }
 
+// resourceBigtableInstanceClusterReorderTypeList causes the cluster block to
+// act like a TypeSet while it's a TypeList underneath. It preserves state
+// ordering on updates, and causes the resource to get recreated if it would
+// attempt to perform an impossible change.
+// This doesn't use the standard unordered list utility (https://github.com/GoogleCloudPlatform/magic-modules/blob/master/templates/terraform/unordered_list_customize_diff.erb)
+// because some fields can't be modified using the API and we recreate the instance
+// when they're changed.
 func resourceBigtableInstanceClusterReorderTypeList(diff *schema.ResourceDiff, meta interface{}) error {
-	old_count, new_count := diff.GetChange("cluster.#")
+	oldCount, newCount := diff.GetChange("cluster.#")
 
 	// simulate Required:true, MinItems:1, MaxItems:4 for "cluster"
-	if new_count.(int) < 1 {
+	if newCount.(int) < 1 {
 		return fmt.Errorf("config is invalid: Too few cluster blocks: Should have at least 1 \"cluster\" block")
 	}
-	if new_count.(int) > 4 {
+	if newCount.(int) > 4 {
 		return fmt.Errorf("config is invalid: Too many cluster blocks: No more than 4 \"cluster\" blocks are allowed")
 	}
 
-	if old_count.(int) != new_count.(int) {
+	// exit early if we're in create (name's old value is nil)
+	n, _ := diff.GetChange("name")
+	if n == nil || n == "" {
 		return nil
 	}
 
-	var old_ids []string
-	clusters := make(map[string]interface{}, new_count.(int))
+	oldIds := []string{}
+	clusters := make(map[string]interface{}, newCount.(int))
 
-	for i := 0; i < new_count.(int); i++ {
-		old_id, new_id := diff.GetChange(fmt.Sprintf("cluster.%d.cluster_id", i))
-		if old_id != nil && old_id != "" {
-			old_ids = append(old_ids, old_id.(string))
+	for i := 0; i < oldCount.(int); i++ {
+		oldId, _ := diff.GetChange(fmt.Sprintf("cluster.%d.cluster_id", i))
+		if oldId != nil && oldId != "" {
+			oldIds = append(oldIds, oldId.(string))
 		}
+	}
+	log.Printf("[DEBUG] Saw old ids: %#v", oldIds)
+
+	for i := 0; i < newCount.(int); i++ {
+		_, newId := diff.GetChange(fmt.Sprintf("cluster.%d.cluster_id", i))
 		_, c := diff.GetChange(fmt.Sprintf("cluster.%d", i))
-		clusters[new_id.(string)] = c
+		clusters[newId.(string)] = c
+	}
+
+	// create a list of clusters using the old order when possible to minimise
+	// diffs
+	// initially, add matching clusters to their index by id (nil otherwise)
+	// then, fill in nils with new clusters.
+	// [a, b, c, e] -> [c, a, d] becomes [a, nil, c] followed by [a, d, c]
+	var orderedClusters []interface{}
+	for i := 0; i < newCount.(int); i++ {
+		// when i is out of range of old, all values are nil
+		if i >= len(oldIds) {
+			orderedClusters = append(orderedClusters, nil)
+			continue
+		}
+
+		oldId := oldIds[i]
+		if c, ok := clusters[oldId]; ok {
+			log.Printf("[DEBUG] Matched: %#v", oldId)
+			orderedClusters = append(orderedClusters, c)
+			delete(clusters, oldId)
+		} else {
+			orderedClusters = append(orderedClusters, nil)
+		}
 	}
 
-	// reorder clusters according to the old cluster order
-	var old_cluster_order []interface{}
-	for _, id := range old_ids {
-		if c, ok := clusters[id]; ok {
-			old_cluster_order = append(old_cluster_order, c)
+	log.Printf("[DEBUG] Remaining clusters: %#v", clusters)
+	for _, elem := range clusters {
+		for i, e := range orderedClusters {
+			if e == nil {
+				orderedClusters[i] = elem
+			}
 		}
 	}
 
-	err := diff.SetNew("cluster", old_cluster_order)
+	err := diff.SetNew("cluster", orderedClusters)
 	if err != nil {
 		return fmt.Errorf("Error setting cluster diff: %s", err)
 	}
 
+	// Clusters can't have their zone / storage_type updated, ForceNew if it's
+	// changed. This will show a diff with the old state on the left side and
+	// the unmodified new state on the right and the ForceNew attributed to the
+	// _old state index_ even if the diff appears to have moved.
+	// This depends on the clusters having been reordered already by the prior
+	// SetNew call.
+	// We've implemented it here because it doesn't return an error in the
+	// client and silently fails.
+	for i := 0; i < newCount.(int); i++ {
+		oldId, newId := diff.GetChange(fmt.Sprintf("cluster.%d.cluster_id", i))
+		if oldId != newId {
+			continue
+		}
+
+		oZone, nZone := diff.GetChange(fmt.Sprintf("cluster.%d.zone", i))
+		if oZone != nZone {
+			err := diff.ForceNew(fmt.Sprintf("cluster.%d.zone", i))
+			if err != nil {
+				return fmt.Errorf("Error setting cluster diff: %s", err)
+			}
+		}
+
+		oST, nST := diff.GetChange(fmt.Sprintf("cluster.%d.storage_type", i))
+		if oST != nST {
+			err := diff.ForceNew(fmt.Sprintf("cluster.%d.storage_type", i))
+			if err != nil {
+				return fmt.Errorf("Error setting cluster diff: %s", err)
+			}
+		}
+	}
+
 	return nil
 }
 

diff --git a/google-beta/resource_bigtable_instance_test.go b/google-beta/resource_bigtable_instance_test.go
@@ -68,7 +68,23 @@ func TestAccBigtableInstance_cluster(t *testing.T) {
 				ImportStateVerify: true,
 			},
 			{
-				Config: testAccBigtableInstance_cluster_reordered(instanceName, 5),
+				Config: testAccBigtableInstance_clusterReordered(instanceName, 5),
+			},
+			{
+				ResourceName:      "google_bigtable_instance.instance",
+				ImportState:       true,
+				ImportStateVerify: true,
+			},
+			{
+				Config: testAccBigtableInstance_clusterModified(instanceName, 5),
+			},
+			{
+				ResourceName:      "google_bigtable_instance.instance",
+				ImportState:       true,
+				ImportStateVerify: true,
+			},
+			{
+				Config: testAccBigtableInstance_clusterReordered(instanceName, 5),
 			},
 			{
 				ResourceName:      "google_bigtable_instance.instance",
@@ -225,7 +241,7 @@ resource "google_bigtable_instance" "instance" {
 `, instanceName, instanceName, instanceName, instanceName, instanceName, instanceName)
 }
 
-func testAccBigtableInstance_cluster_reordered(instanceName string, numNodes int) string {
+func testAccBigtableInstance_clusterReordered(instanceName string, numNodes int) string {
 	return fmt.Sprintf(`
 resource "google_bigtable_instance" "instance" {
   name = "%s"
@@ -257,6 +273,32 @@ resource "google_bigtable_instance" "instance" {
 `, instanceName, instanceName, numNodes, instanceName, numNodes, instanceName, numNodes, instanceName, numNodes)
 }
 
+func testAccBigtableInstance_clusterModified(instanceName string, numNodes int) string {
+	return fmt.Sprintf(`
+resource "google_bigtable_instance" "instance" {
+  name = "%s"
+  cluster {
+    cluster_id   = "%s-c"
+    zone         = "us-central1-c"
+    num_nodes    = %d
+    storage_type = "HDD"
+  }
+  cluster {
+    cluster_id   = "%s-a"
+    zone         = "us-central1-a"
+    num_nodes    = %d
+    storage_type = "HDD"
+  }
+  cluster {
+    cluster_id   = "%s-b"
+    zone         = "us-central1-b"
+    num_nodes    = %d
+    storage_type = "HDD"
+  }
+}
+`, instanceName, instanceName, numNodes, instanceName, numNodes, instanceName, numNodes)
+}
+
 func testAccBigtableInstance_development(instanceName string) string {
 	return fmt.Sprintf(`
 resource "google_bigtable_instance" "instance" {

diff --git a/google-beta/resource_bigtable_table.go b/google-beta/resource_bigtable_table.go
@@ -142,7 +142,7 @@ func resourceBigtableTableRead(d *schema.ResourceData, meta interface{}) error {
 	if err != nil {
 		log.Printf("[WARN] Removing %s because it's gone", name)
 		d.SetId("")
-		return fmt.Errorf("Error retrieving table. Could not find %s in %s. %s", name, instanceName, err)
+		return nil
 	}
 
 	d.Set("project", project)