Apache Spark - From installation to performing awesome operations in Apache Spark Stack

This is my fourth project of Data Engineering Nanodegree from Udacity. In this project, I have created an EMR cluster with Spark. Extracted the data from S3 into Spark, and transformed them and written them into parquet files.

Group 10 Project, Fall 2020, CS 6240: Large-Scale Parallel Data Processing, Khoury College of Computer Sciences, Northeastern University

Assignment 2 of the course 'Distributed Systems Programming' by Meni Adler. In the assignment we build an application that calculates the probabilities for any word to come after a couple of words, for ANY couple of words in the n-gram corpus (google).

Udacity's Data Engineering Nanodegree project: Data Lake with Spark.

Riffl deployment

cfn resource provider package for AWS EMRWAL

Frontend for a distributed electronic health records system

Developing a Flow with EMR and Airflow

A powerful CLI tool and API for managing Spark jobs on Amazon EMR clusters.

Data Engineering Batch Project

Streaming pipeline using AWS MSK and AWS EMR with Spark, retrieving the data from Twitter Streams API

Scripts for provisioning data science tools

Criar um Ecossistema de Big Data na AWS e rodar um programa em python para ordenar as palavras por ocorrência, apresentando as mais citadas no texto.

The EMR Helper library tries to help when setting up and managing an EMR cluster.

This repository contains the final project for the course Big Data Systems Engineering with Scala.