Airflow Migration: Build out Infrastructure needed to schedule DAGs and convert to IRSA

[julialawrence]

User Story

We would like to implement Airflow-related infrastructure that is meant to be hosted in the Analytical Platform accounts in terraform in our Data Platform monorepo.
The whole infrastructure for Airflow is currently managed in Pulumi and the authentication of workloads to the AWS environment is accomplished with kube2iam.

We need to refactor it into terraform as well as convert from the outdated K2IAM plug-in to the AWS-endorsed IRSA approach.

User Type(s)

Ops Engineers, Airflow Engineers

Value

• Alignment to tooling standards
• Refactor code
• Improve security and manageability of the infra

Definition of done

• DAG definitions converted to use IRSA
• Infra developed in data-platform in terraform
• Service accounts provisioned
• IRSA tested
• tests are green
• Follow-on stories raised

Reference

How to write good user stories

[jhpyke]

@jhpyke to decompose epic into tickets

[github-actions]

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.

[github-actions]

This issue is being closed because it has been open for a further 7 days with no activity. If this is still a valid issue, please reopen it, Thank you!