ADF terraform extension

[stemons]

Issue #, if available:

Description of changes:

This PR enables ADF to run terraform pipelines multi-accounts/OUs and multi-regions and manages in a structured way terraform state file and lock.

Terraform module includes:

• adf_terraform.sh: this script:
  • runs terraform plan and apply
  • centralize the terraform state file in the S3 bucket of the deployment account
    • e.g. main region adf-global-base-deployment-pipelinebucketxyz/ProjectName/accountID.tfstate
    • e.g. secondary region adf-regional-base-deploy-deploymentframeworkregio-jsm/ProjectName/accountID.tfstate
  • centralize lock of the state file in regional DynamoDB tables in deployment account
• sample-terraform: this folder contains an example of terraform pipelines definition and repository
• adf_terraform_role: additional role added during the bootstrap of an account to deploy terraform resources
• adf_locktable: regional DynamoDB to manage the lock of the state file
• minor changes to adf core roles to enable terraform pipelines

An overview of the terraform template components and functionalities

Overview

This repository contains a module that manage the deployment of terraform code to multiple accounts and regions.
The module consists of three build stages defined in the following file:

• buildspec.yml: install the version of terraform specified in the pipeline configuration
• tf_scan.yml: (optional) returns any vulnerabilities in terraform code according with terrascan utilitiy
• tf_plan.yml: get the list of accounts from the organization and run a terraform plan
• tf_apply.yml: run a terraform apply after the manual step approval

Parameters

• TERRAFORM_VERSION: the terraform version used to deploy the resource
• TARGET_ACCOUNTS: comma separated list of target accounts
• TARGET_OUS: comma separated list of target leaf OUs (parent OUs are supported)
• REGIONS: comma separated list of target region

Deployment procedure

1. Add a sample-terraform pipeline in ADF deployment-map.yml as in the example:

- name: sample-terraform
  default_providers:
    source:
      provider: codecommit
      properties:
        account_id: 111111111111 # source account id
    build:
      provider: codebuild
      properties:
        environment_variables:
          TERRAFORM_VERSION: "0.14.10" # terraform version
    deploy:
      provider: codebuild
      properties:
        image: "STANDARD_5_0"
        environment_variables:
          TARGET_ACCOUNTS: 111111111111,222222222222 # target accounts
          TARGET_OUS: /core/infrastructure,/sandbox # target OUs
          MASTER_ACCOUNT_ID: 333333333333 # master account
          REGIONS: eu-west-1 # target regions
  params:
    restart_execution_on_update: true
  targets:
    - name: terraform-scan # optional
      properties:
        spec_filename: tf_scan.yml # terraform scan
    - name: terraform-plan
      properties:
        spec_filename: tf_plan.yml # terraform plan
    - approval # manual approval
    - name: terraform-apply
      properties:
        spec_filename: tf_apply.yml # terraform apply

2. Add the project name in params/global.yml file

3. Add terraform code to the tf folder. Do not make changes to backend.tf file and main.tf.

4. Add variable definition to tf\variables.tf file and variable values to tfvars/global.auto.tfvars

   • Local variables (per account) can be configured using the following naming convention

     tfvars <-- This folder contains the structure to define terraform variables
     │
     └───global.auto.tfvars <-- this file contains global variables applied to all the target accounts
     │
     └───111111111111 <-- this folders contains variable files related to account 111111111111
     │   └──────│   local.auto.tfvars <-- this file contains variables related to account 111111111111
     │
     └───222222222222 <-- this folders contains variable files related to account 222222222222
         └──────│   local.auto.tfvars <-- this file contains variables related to account 222222222222
     

5. Push to sample-terraform ADF repository

6. Pipeline contains a manual step approval between terraform plan and terraform apply. Confirm to proceed.

Terraform state files are stored in the regional S3 buckets in the deployment account. One state file per account/region/module is created
e.g. Project name: sample-tf-module
Target accounts: 111111111111, 222222222222
Target regions: eu-west-1 (main ADF region), us-east-1
The following state files are created

• 111111111111 main region (eu-west-1) adf-global-base-deployment-pipelinebucketxyz/sample-tf-module/111111111111.tfstate
• 111111111111 secondary region (us-east-1) adf-regional-base-deploy-deploymentframeworkregio-jsm/sample-tf-module/111111111111.tfstate
• 222222222222 main region (eu-west-1) adf-global-base-deployment-pipelinebucketxyz/sample-tf-module/222222222222.tfstate
• 222222222222 secondary region (us-east-1) adf-regional-base-deploy-deploymentframeworkregio-jsm/sample-tf-module/222222222222.tfstate

A DynamoDB table manage the lock of the state file. It is deployed in every ADF regions named adf_locktable

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[StewartW]

This looks good, thanks!
I have one main observation though.
In the deployment map we define the targets of the pipeline as a property of the deploy provider

   environment_variables:
     TARGET_ACCOUNTS: 111111111111,222222222222 # target accounts
    TARGET_OUS: /core/infrastructure,/sandbox # target OUs
    MASTER_ACCOUNT_ID: 333333333333 # master account
     REGIONS: eu-west-1 # target regions

And configure the deployment provider with the use of the target configuration

targets:
 - name: terraform-scan # optional
    properties:
      spec_filename: tf_scan.yml # terraform scan
  - name: terraform-plan
    properties:
      spec_filename: tf_plan.yml # terraform plan
  - approval # manual approval
  - name: terraform-apply
    properties:
      spec_filename: tf_apply.yml # terraform apply

This means that you've had to write duplicate logic for getting accounts etc.
Would it not make more sense to actually make a new terraform pipeline type that reuses the existing code for resolving targets and passes that information into your deploy provider?

[sbkok]

Thanks for contributing these changes.
There are some comments in here that need to be fixed before we can merge it.

Please reach out directly if anything is unclear or if help is needed.

[sbkok]

LGTM, thanks for contributing!