CI/CD for data processing workflow

This repository contains source code based on a guide on how to use Cloud Build and Cloud Composer to create a CI/CD pipeline for building, testing, and deploying a data processing workflow. Please refer to Setting up a CI/CD pipeline for your data-processing workflow for the overall steps to run the code.

DISCLAIMER: the present repo is a work in progress, and some steps may fail. Feel free to file an issue if you find any errors when using it!

Bear in mind the source-code/workflow folder structure in this fork is different from the original source-code/workflow-dag. The changes were inspired by additional thoughts from the below examples.

Useful Links & Examples

1. Setting up a CI/CD pipeline for your data-processing workflow, forked repo with more examples

2. Building CI/CD with Airflow, GitLab and Terraform in GCP

3. Developing a CI/CD Practice for Google Cloud Composer

4. Testing in Airflow

   • Part 1: DAG Validation Tests, DAG Definition Tests and Unit Tests
   • Part 2: Integration Tests and End-To-End Pipeline Tests