Developing Spark with sbt

A modal REPL protocol for Neovim, e.g. to use the Scala REPL or spark-shell.

Starting spark locally

This is a self-documentation of learning distributed data storage, parallel processing, and Linux OS using Apache Hadoop, Apache Spark and Raspbian OS. In this project, 3-node cluster will be setup using Raspberry Pi 4, install HDFS and run Spark processing jobs via YARN.

This Big Data project consists of obtaining data on vehicle theft in the city of São Paulo and consolidating it in a counting and heat map, in order to show areas with a higher index of this type of crime. All applicable in AWS Resources.

The idea with this project is to learn the basis about how to use spark with generated ROOT files from LHC experiment.

A collection of Spark/Scala example programs

Scripts, libraries and more utilities for spark-shell and spark-submit testing.

An example of installation Apache Spark on AWS

Library for remote JVM ExecutorService with only dependency being password-less SSH -- Run clustered Hadoop/Spark jobs from IDE -- IDE-pimped Spark shell with full auto-completion!