Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ec_deployment: add support for data tiers #253

Merged
merged 11 commits into from
Mar 11, 2021
Merged
Show file tree
Hide file tree
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ examples/*/*.tfstate.backup
.apikey
dist
.ci/.gpg_*
.ci/.github_*
.ci/.github_*
.terraform.lock.hcl
57 changes: 47 additions & 10 deletions docs/resources/ec_deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ Provides an Elastic Cloud deployment resource, which allows deployments to be cr

-> **Note on regions and deployment templates** Before you start, you might want to read about [Elastic Cloud deployments](https://www.elastic.co/guide/en/cloud/current/ec-create-deployment.html) and check the [full list](https://www.elastic.co/guide/en/cloud/current/ec-regions-templates-instances.html) of regions and deployment templates available in Elasticsearch Service (ESS).

-> **Note on Elasticsearch topology IDs** Since the addition of data tiers, each Elasticsearch topology block requires the `"id"` field to be set. The accepted values are set in the deployment template that you have chosen, but values are closely related to the Elasticsearch data tiers. [Learn more abut Elasticsearch data tiers](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html). For a complete list of all the supported values, refer to the deployment template definition used by your deployment.

## Example Usage

### Basic
Expand All @@ -25,7 +27,7 @@ resource "ec_deployment" "example_minimal" {

# Mandatory fields
region = "us-east-1"
version = "7.9.2"
version = "7.11.1"
deployment_template_id = "aws-io-optimized-v2"

elasticsearch {}
Expand All @@ -38,6 +40,39 @@ resource "ec_deployment" "example_minimal" {
}
```

### Hot warm cold tiered deployment

```hcl
resource "ec_deployment" "example_minimal" {
region = "us-east-1"
version = "7.11.1"
deployment_template_id = "aws-io-optimized-v2"

elasticsearch {
topology {
id = "hot_content"
size = "8g"
}

topology {
id = "warm"
size = "16g"
}

topology {
id = "cold"
size = "8g"
}
}

kibana {}

apm {}

enterprise_search {}
}
```

### With observability settings

```hcl
Expand All @@ -47,7 +82,7 @@ resource "ec_deployment" "example_observability" {

# Mandatory fields
region = "us-east-1"
version = "7.9.2"
version = "7.11.1"
deployment_template_id = "aws-io-optimized-v2"

elasticsearch {}
Expand All @@ -68,11 +103,12 @@ resource "ec_deployment" "source_deployment" {
name = "my_ccs_source"

region = "us-east-1"
version = "7.9.2"
version = "7.11.1"
deployment_template_id = "aws-io-optimized-v2"

elasticsearch {
topology {
id = "hot_content"
size = "1g"
}
}
Expand All @@ -82,7 +118,7 @@ resource "ec_deployment" "ccs" {
name = "ccs deployment"

region = "us-east-1"
version = "7.9.2"
version = "7.11.1"
deployment_template_id = "aws-cross-cluster-search-v2"

elasticsearch {
Expand All @@ -106,7 +142,7 @@ resource "ec_deployment" "with_tags" {

# Mandatory fields
region = "us-east-1"
version = "7.9.2"
version = "7.11.1"
deployment_template_id = "aws-io-optimized-v2"

elasticsearch {}
Expand Down Expand Up @@ -166,18 +202,17 @@ To set up multi-node Elasticsearch clusters, you can set single or multiple topo

The optional `elasticsearch.topology` block supports the following arguments:

* `instance_configuration_id` - (Optional) Default instance configuration of the deployment template. To change it, use the [full list](https://www.elastic.co/guide/en/cloud/current/ec-regions-templates-instances.html) of regions and deployment templates available in ESS.

-> Before you get started with instance configurations, read the [ESS hardware and Instance Configurations](https://www.elastic.co/guide/en/cloud/current/ec-reference-hardware.html#ec-instance-configuration-names) documentation.

* `size` - (Optional) Amount of memory (RAM) per topology element in the `"<size in GB>g"` notation. When omitted, it defaults to the deployment template value.
* `id` - (Required) Unique topology identifier. It generally refers to an Elasticsearch data tier, such as `hot_content`, `warm` or `cold`.
* `size` - (Optional) Amount in Gigabytes per topology element in the `"<size in GB>g"` notation. When omitted, it defaults to the deployment template value.
* `size_resource` - (Optional) Type of resource to which the size is assigned. Defaults to `"memory"`.
* `zone_count` - (Optional) Number of zones the instance type of the Elasticsearch cluster will span. This is used to set or unset HA on an Elasticsearch node type. When omitted, it defaults to the deployment template value.
* `node_type_data` - (Optional) The node type for the Elasticsearch cluster (data node).
* `node_type_master` - (Optional) The node type for the Elasticsearch cluster (master node).
* `node_type_ingest` - (Optional) The node type for the Elasticsearch cluster (ingest node).
* `node_type_ml` - (Optional) The node type for the Elasticsearch cluster (machine learning node).

~> **Note when node_type_* fields set** After upgrading to a version that supports data tiers (7.10.0 or above), the `node_type_*` has no effect even if specified. The provider automatically migrates the `node_type_*` fields to the appropriate `node_roles` as set by the deployment template. After having upgraded to `7.10.0` or above, the fields should be removed from the terraform configuration, if explicitly configured.

##### Config

The optional `elasticsearch.config` block supports the following arguments:
Expand Down Expand Up @@ -308,10 +343,12 @@ In addition to all the arguments above, the following attributes are exported:
* `elasticsearch.#.cloud_id` - Encoded Elasticsearch credentials to use in Beats or Logstash. For more information, see [Configure Beats and Logstash with Cloud ID](https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html).
* `elasticsearch.#.http_endpoint` - Elasticsearch resource HTTP endpoint.
* `elasticsearch.#.https_endpoint` - Elasticsearch resource HTTPs endpoint.
* `elasticsearch.#.topology.#.instance_configuration_id` - instance configuration of the deployment topology element.
* `elasticsearch.#.topology.#.node_type_data` - Node type (data) for the Elasticsearch topology element.
* `elasticsearch.#.topology.#.node_type_master` - Node type (master) for the Elasticsearch topology element.
* `elasticsearch.#.topology.#.node_type_ingest` - Node type (ingest) for the Elasticsearch topology element.
* `elasticsearch.#.topology.#.node_type_ml` - Node type (machine learning) for the Elasticsearch topology element.
* `elasticsearch.#.topology.#.node_roles` - List of roles for the topology element. They are inferred from the deployment template.
* `elasticsearch.#.snapshot_source.#.source_elasticsearch_cluster_id` - ID of the Elasticsearch cluster that will be used as the source of the snapshot.
* `elasticsearch.#.snapshot_source.#.snapshot_name` - Name of the snapshot to restore.
* `kibana.#.resource_id` - Kibana resource unique identifier.
Expand Down
8 changes: 2 additions & 6 deletions ec/acc/datasource_deployment_basic_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,15 +55,12 @@ func TestAccDatasourceDeployment_basic(t *testing.T) {
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.resource_id", resourceName, "elasticsearch.0.resource_id"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.http_endpoint_id", resourceName, "elasticsearch.0.http_endpoint_id"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.https_endpoint_id", resourceName, "elasticsearch.0.https_endpoint_id"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.version", resourceName, "elasticsearch.0.version"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.topology.0.instance_configuration_id", resourceName, "elasticsearch.0.topology.0.instance_configuration_id"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.topology.0.size", resourceName, "elasticsearch.0.topology.0.size"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.topology.0.size_resource", resourceName, "elasticsearch.0.topology.0.size_resource"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.topology.0.zone_count", resourceName, "elasticsearch.0.topology.0.zone_count"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.topology.0.node_type_data", resourceName, "elasticsearch.0.topology.0.node_type_data"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.topology.0.node_type_master", resourceName, "elasticsearch.0.topology.0.node_type_master"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.topology.0.node_type_ingest", resourceName, "elasticsearch.0.topology.0.node_type_ingest"),
resource.TestCheckResourceAttrPair(datasourceName, "elasticsearch.0.topology.0.node_type_ml", resourceName, "elasticsearch.0.topology.0.node_type_ml"),
// Pending the adddition of the `node_roles` field to the ec_deployment resource.
// resource.TestCheckTypeSetElemAttrPair(datasourceName, "elasticsearch.0.topology.0.node_roles.*", resourceName, "elasticsearch.0.topology.0.node_roles.*"),

// Kibana
resource.TestCheckResourceAttrPair(datasourceName, "kibana.0.elasticsearch_cluster_ref_id", resourceName, "kibana.0.elasticsearch_cluster_ref_id"),
Expand Down Expand Up @@ -115,7 +112,6 @@ func TestAccDatasourceDeployment_basic(t *testing.T) {
resource.TestCheckResourceAttrPair(depsDatasourceName, "deployment_template_id", resourceName, "deployment_template_id"),

// Deployment resources
resource.TestCheckResourceAttrPair(depsDatasourceName, "elasticsearch.0.version", resourceName, "elasticsearch.0.version"),
resource.TestCheckResourceAttrPair(datasourceName, "kibana.0.version", resourceName, "kibana.0.version"),
resource.TestCheckResourceAttrPair(datasourceName, "apm.0.version", resourceName, "apm.0.version"),
resource.TestCheckResourceAttrPair(datasourceName, "enterprise_search.0.version", resourceName, "enterprise_search.0.version"),
Expand Down
64 changes: 37 additions & 27 deletions ec/acc/deployment_basic_defaults_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,12 @@ func TestAccDeployment_basic_defaults(t *testing.T) {
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.instance_configuration_id"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size", "8g"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size_resource", "memory"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", "false"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.id", "hot_content"),
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.node_roles.#"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.zone_count", "2"),
resource.TestCheckResourceAttr(resName, "kibana.#", "1"),
resource.TestCheckResourceAttr(resName, "kibana.0.topology.#", "1"),
Expand Down Expand Up @@ -89,10 +91,12 @@ func TestAccDeployment_basic_defaults(t *testing.T) {
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size", "8g"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size_resource", "memory"),
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.instance_configuration_id"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", "false"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.id", "hot_content"),
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.node_roles.#"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.zone_count", "2"),
resource.TestCheckResourceAttr(resName, "kibana.#", "1"),
resource.TestCheckResourceAttr(resName, "kibana.0.topology.#", "1"),
Expand Down Expand Up @@ -121,10 +125,12 @@ func TestAccDeployment_basic_defaults(t *testing.T) {
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.instance_configuration_id"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size", "1g"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size_resource", "memory"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", "false"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", "false"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.id", "hot_content"),
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.node_roles.#"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.zone_count", "2"),
resource.TestCheckResourceAttr(resName, "kibana.#", "1"),
resource.TestCheckResourceAttr(resName, "kibana.0.topology.#", "1"),
Expand Down Expand Up @@ -168,10 +174,12 @@ func TestAccDeployment_basic_defaults_hw(t *testing.T) {
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.instance_configuration_id"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size", "1g"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size_resource", "memory"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", "false"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.id", "hot_content"),
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.node_roles.#"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.zone_count", "2"),
resource.TestCheckResourceAttr(resName, "kibana.#", "0"),
resource.TestCheckResourceAttr(resName, "apm.#", "0"),
Expand All @@ -192,15 +200,19 @@ func TestAccDeployment_basic_defaults_hw(t *testing.T) {
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.size_resource", "memory"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.size", "4g"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.size_resource", "memory"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", "false"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_data", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ingest", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_master", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.node_type_ml", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.id", "hot_content"),
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.0.node_roles.#"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.0.zone_count", "2"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.node_type_data", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.node_type_ingest", "true"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.node_type_master", "false"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.node_type_ml", "false"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.node_type_data", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.node_type_ingest", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.node_type_master", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.node_type_ml", ""),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.id", "warm"),
resource.TestCheckResourceAttrSet(resName, "elasticsearch.0.topology.1.node_roles.#"),
resource.TestCheckResourceAttr(resName, "elasticsearch.0.topology.1.zone_count", "2"),
resource.TestCheckResourceAttr(resName, "kibana.#", "1"),
resource.TestCheckResourceAttr(resName, "kibana.0.topology.#", "1"),
Expand All @@ -224,7 +236,5 @@ func fixtureAccDeploymentResourceBasicDefaults(t *testing.T, fileName, name, reg
if err != nil {
t.Fatal(err)
}
return fmt.Sprintf(string(b),
region, name, region, deploymentTpl,
)
return fmt.Sprintf(string(b), region, name, region, deploymentTpl)
}
Loading