Automation framework to catalog AWS data sources using Glue

This project repo 📺 offers a robust solution meticulously crafted to efficiently manage, process, and analyze YouTube video data leveraging the power of AWS services. Whether you're diving into structured statistics or exploring the nuances of trending key metrics, this pipeline is engineered to handle it all with finesse.

It is a project build using ETL(Extract, Transform, Load) pipeline using Spotify API on AWS.

Unveiling job market trends with Scrapy and AWS

Creating an audit table for a DynamoDB table using CloudTrail, Kinesis Data Stream, Lambda, S3, Glue and Athena and CloudFormation

Working with Glue Data Catalog and Running the Glue Crawler On Demand

Smart City Realtime Data Engineering Project

Engaging, interactive visualizations crafted with Streamlit, seamlessly powered by Apache Flink in batch mode to reveal deep insights from data.

Developed an ETL pipeline for real-time ingestion of stock market data from the stock-market-data-manage.onrender.com API. Engineered the system to store data in Parquet format for optimized query processing and incorporated data quality checks to ensure accuracy prior to visualization.

This repository demonstrates the creation of a robust, 3-tier data lake using AWS resources.

Example using the Iceberg register_table command with AWS Glue and Glue Data Catalog

These are the handwritten notes on Coursera's Practical data science specialization course.