Batch ETL using Cloud Environment which is GCP by utilizing Cloud Composer + Google Cloud Storage + Dataflow + Cloud Build
An ETL pipeline to move an uploaded flat file ffrom GCS, mask PII, store Big Query, and Create a report in Looker.
A simple dag for triggering the Cloud Data Fusion Pipeline using Apache Airflow.
This project leverages GCS, Composer, Dataflow, BigQuery, and Looker on Google Cloud Platform (GCP) to build a robust data engineering solution for processing, storing, and reporting daily transaction data in the online food delivery industry.
Add a description, image, and links to the cloudcomposer topic page so that developers can more easily learn about it.
To associate your repository with the cloudcomposer topic, visit your repo's landing page and select "manage topics."