terraform-ecs-fargate-airflow

A Terraform template for provisioning Apache Airflow workflows on AWS ECS Fargate.

This template assumes you already have an airflow cluster up and running in AWS with the FargateEcsOperator operator installed (with fargate support). The template will output a deploy-airflow.sh script that does the following:

• builds your code into a container image (assumes you have a Dockerfile in the root of your project)
• pushes your container image to ECR
• copies your DAG (referencing your container) to an S3 location to be deployed (assumes you have a process for copying the DAGs from your S3 location into your airflow directory)

The templates are designed to be customized. The optional components can be removed by simply deleting the .tf file.

Components

base

These components are shared by all environments.

Name	Description	Optional
main.tf	AWS provider, output
state.tf	S3 bucket backend for storing Terraform remote state
ecr.tf	ECR repository for application (all environments share)

env/dev

These components are for a specific environment. There should be a corresponding directory for each environment that is needed.

Name	Description	Optional
main.tf	Terrform remote state, AWS provider, output
ecs.tf	ECS Cluster, Service, Task Definition, ecsTaskExecutionRole, CloudWatch Log Group
nsg.tf	NSG for Task
role-airflow.tf	Attaches policies to the airflow role that allow it to integrate with fargate
airflow.tf	Generates the deploy-airflow.sh deployment script
dag.tf	Generates a default DAG py file
dashboard.tf	CloudWatch dashboard: CPU, memory, and HTTP-related metrics
role.tf	Application Role for container
cicd.tf	IAM user that can be used by CI/CD systems	Yes
secretsmanager.tf	Add a base secret to Secretsmanager	Yes
ecs-event-stream.tf	Add an ECS event log dashboard	Yes

Usage

Typically, the base Terraform will only need to be run once, and then should only need changes very infrequently. After the base is built, each environment can be built.

# Move into the base directory
$ cd base

# Sets up Terraform to run
$ terraform init

# Executes the Terraform run
$ terraform apply

# Now, move into the dev environment
$ cd ../env/dev

# Sets up Terraform to run
$ terraform init

# Executes the Terraform run
$ terraform apply

Important (after initial terraform apply)

The generated base .tfstate is not stored in the remote state S3 bucket. Ensure the base .tfstate is checked into your infrastructure repo. The default Terraform .gitignore generated by GitHub will ignore all .tfstate files; you'll need to modify this!

fargate-create

Alternatively you can use the fargate-create CLI to scaffold new projects based on this template.

install

curl -s get-fargate-create.turnerlabs.io | sh

create an input vars file (terraform.tfvars)

# app/env to scaffold
app = "my-app"
environment = "dev"

schedule_expression = "@hourly"
airflow_dag_s3_bucket = "s3://my-dags/"
airflow_role = "airflow_role"
region = "us-east-1"
aws_profile = "default"
saml_role = "admin"
vpc = "vpc-123"
private_subnets = "subnet-123,subnet-456"
public_subnets = "subnet-789,subnet-012"
tags = {
  application   = "my-app"
  environment   = "dev"
  team          = "my-team"
  customer      = "my-customer"
  contact-email = "me@example.com"
}

$ fargate-create -f terraform.tfvars -t git@github.com:turnerlabs/terraform-ecs-fargate-airflow

Deploy DAG to airflow

cd iac/env/dev
./deploy-airflow.sh

Additional Information

• Base README

• Environment dev README