Pinned Loading

1. ETL_fintech-data-migration ETL_fintech-data-migration Public

   Data Migration with Azure Synapse, PySpark, and Delta Tables

   Jupyter Notebook

2. EL_gcs-to-bigquery EL_gcs-to-bigquery Public

   An EL pipeline built with Apache Airflow that downloads a file from the web uploads it to Google Cloud Storage, and creates an external table in BigQuery for data storage and analysis.

   Python

3. ETL_spark-on-dataproc ETL_spark-on-dataproc Public

   A Pyspark project that performs ETL on a Dataproc cluster and writes data to Google Cloud Storage/BigQuery.

   Jupyter Notebook

4. CDC_SCD-1_Pipeline CDC_SCD-1_Pipeline Public

   Data ingestion and transformation pipeline for AirBnB using Azure Data Factory, Synapse, CosmosDB, and ADLS for automated CDC and analytical reporting.

   Python

5. ETL_Airlines_CICD_Incremental_Processing ETL_Airlines_CICD_Incremental_Processing Public

   Automated ETL pipeline for incremental airline data processing using Azure Data Factory and DevOps with CI/CD integration.