GitHub - raktim00/Hadoop-Multi-Node-Cluster-on-Multi-Cloud-using-Terraform-Ansible: Deploying Hadoop HDFS & MR cluster on Multi Node across AWS, Azure & GCP using Terraform & Ansible Scripting Automation. Finally settling up Hadoop Client on local system and installing Pig & Hive to connect with the cluster.

raktim00 / Hadoop-Multi-Node-Cluster-on-Multi-Cloud-using-Terraform-Ansible Public

Notifications You must be signed in to change notification settings
Fork 3
Star 1

Deploying Hadoop HDFS & MR cluster on Multi Node across AWS, Azure & GCP using Terraform & Ansible Scripting Automation. Finally settling up Hadoop Client on local system and installing Pig & Hive to connect with the cluster.

1 star 3 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
ansible-ws		ansible-ws
terraform-ws		terraform-ws
Hadoop_Multi_Node_Setup.png		Hadoop_Multi_Node_Setup.png
LICENSE		LICENSE
README.md		README.md
deploy.sh		deploy.sh
hadoop-multi-cloud-key-private.pem		hadoop-multi-cloud-key-private.pem
hadoop-multi-cloud-key-public.pub		hadoop-multi-cloud-key-public.pub
remove.sh		remove.sh

Repository files navigation

This project is an integration of different tools & technologies like AWS, Azure, GCP, Hadoop, Terraform, Ansible etc.

To see the practical demonstration follow the link : https://youtu.be/VB1jECOcJAk

Project Description :

Setting up Multi Node Hadoop HDFS & MR cluster on Multi Cloud. Here we are setting up total 6 instances across all three cloud. On AWS we are setting up NameNode & JobTracker. On GCP & Azure we are setting up DataNode & TaskTracker. On local node we are setting up Hadoop Client & then installing pig & hive software to work on our cluster. Refer to the below diagram to understand the infrastructure.

About

Deploying Hadoop HDFS & MR cluster on Multi Node across AWS, Azure & GCP using Terraform & Ansible Scripting Automation. Finally settling up Hadoop Client on local system and installing Pig & Hive to connect with the cluster.

aws ansible automation hadoop azure terraform gcp

Report repository

Releases

No releases published

Packages

No packages published

Languages