Deployment of non-production Amundsen on AWS ECS using aws-cli (#224)

Documentation on the deployment of non-production Amundsen on AWS ECS using aws-cli

frontend/docs/installation-aws-ecs/aws-ecs-deployment.md

@@ -0,0 +1,79 @@
+# Deployment of non-production Amundsen on AWS ECS using aws-cli 
+
+The following is a set of intructions to run Amundsen on AWS Elastic Container Service. The current configuration is very basic but it is working. It is a migration of the docker-amundsen.yml to run on AWS ECS.
+
+## Install ECS CLI
+
+The first step is to install ECS CLI, please follow the instructions from AWS [documentation](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_installation.html)
+
+### Get your access and secret keys from IAM
+
+```bash
+# in ~/<your-path-to-cloned-repo>/amundsenfrontendlibrary/docs/instalation-aws-ecs
+$ export AWS_ACCESS_KEY_ID=xxxxxxxx
+$ export AWS_SECRET_ACCESS_KEY=xxxxxx
+$ export AWS_PROFILE=profilename
+```
+
+For the purpose of this instruction we used the [tutorial](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-ec2.html#ECS_CLI_tutorial_compose_create) on AWS documentation
+
+
+## STEP 1: Create a cluster configuration:
+
+```bash
+# in ~/<your-path-to-cloned-repo>/amundsenfrontendlibrary/docs/instalation-aws-ecs
+$ ecs-cli configure --cluster amundsen --region us-west-2 --default-launch-type EC2 --config-name amundsen
+```
+
+### STEP 2: Create a profile using your access key and secret key:
+
+```bash
+# in ~/<your-path-to-cloned-repo>/amundsenfrontendlibrary/docs/instalation-aws-ecs
+$ ecs-cli configure profile --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --profile-name amundsen 
+```
+
+### STEP 3: Create the Cluster Use profile name from \~/.aws/credentials
+
+```bash
+# in ~/<your-path-to-cloned-repo>/amundsenfrontendlibrary/docs/instalation-aws-ecs
+$ ecs-cli up --keypair JoaoCorreia --extra-user-data userData.sh --capability-iam --size 1 --instance-type t2.large --cluster-config amundsen --verbose --force --aws-profile $AWS_PROFILE 
+```
+
+### STEP 4: Deploy the Compose File to a Cluster
+
+```bash
+# in ~/<your-path-to-cloned-repo>/amundsenfrontendlibrary/docs/instalation-aws-ecs
+$ ecs-cli compose --cluster-config amundsen --file docker-ecs-amundsen.yml up --create-log-groups
+```
+
+You can use the ECS CLI to see what tasks are running.
+
+```bash
+$ ecs-cli ps
+```
+
+### STEP 5 Open the EC2 Instance 
+
+Edit the Security Group to allow traffic to your IP, you should be able to see the frontend, elasticsearch and neo4j by visiting the URLs:
+
+- http://xxxxxxx:5000/
+- http://xxxxxxx:9200/
+- http://xxxxxxx:7474/browser/
+
+## TODO
+
+- Configuration sent to services not working properly (amunsen.db vs graph.db) 
+- Create a persistent volume for graph/metadata storage. [See this](https://aws.amazon.com/blogs/compute/amazon-ecs-and-docker-volume-drivers-amazon-ebs/)
+- Refactor the VPC and default security group permissions
+
+
+
+
+
+
+
+
+
+
+
+

frontend/docs/installation-aws-ecs/docker-ecs-amundsen.yml

@@ -0,0 +1,95 @@
+version: '3'
+services:
+  neo4j:
+      image: neo4j:3.3.0
+      container_name: neo4j_amundsen
+      environment:
+        - NEO4J_AUTH=neo4j/test
+        # These dont seem to be working though!
+        - NEO4J_dbms.active_database=amundsen.db
+        - NEO4J_dbms.directories.data=/neo4j/data
+        - NEO4J_dbms.directories.logs=/var/log/neo4j
+        - NEO4J_dbms.directories.import=/var/lib/neo4j/import
+        - NEO4J_dbms.security.auth_enabled=false
+        - NEO4J_dbms.connectors.default_listen_address=0.0.0.0
+      ulimits:
+        nofile:
+          soft: 40000
+          hard: 40000
+      ports:
+          - 7474:7474
+          - 7687:7687
+      logging:
+        driver: awslogs
+        options: 
+          awslogs-group: amundsen-neo4j
+          awslogs-region: us-west-2
+          awslogs-stream-prefix: amundsen-neo4j
+
+  elasticsearch:
+      image: elasticsearch:6.7.0
+      container_name: es_amundsen
+      ports:
+          - 9200:9200
+      ulimits:
+        nofile:
+           soft: 65536
+           hard: 65536
+      logging:
+        driver: awslogs
+        options: 
+          awslogs-group: amundsen-elasticsearch
+          awslogs-region: us-west-2
+          awslogs-stream-prefix: amundsen-elasticsearch
+
+  amundsensearch:
+      image: amundsendev/amundsen-search:1.1.1
+      container_name: amundsensearch
+      ports:
+        - 5001:5000
+      depends_on:
+        - elasticsearch
+      environment:
+        - PROXY_ENDPOINT=es_amundsen
+      logging:
+        driver: awslogs
+        options: 
+          awslogs-group: amundsensearch
+          awslogs-region: us-west-2
+          awslogs-stream-prefix: amundsensearch
+
+  amundsenmetadata:
+      image: amundsendev/amundsen-metadata:1.0.7
+      container_name: amundsenmetadata
+      depends_on:
+        - neo4j
+      ports:
+        - 5002:5000
+      environment:
+         - PROXY_HOST=bolt://neo4j_amundsen
+      logging:
+        driver: awslogs
+        options: 
+          awslogs-group: amundsenmetadata
+          awslogs-region: us-west-2
+          awslogs-stream-prefix: amundsenmetadata
+
+  amundsenfrontend:
+      image: amundsendev/amundsen-frontend:1.0.5
+      container_name: amundsenfrontend
+      depends_on:
+        - amundsenmetadata
+        - amundsensearch
+      ports:
+        - 5000:5000
+      environment:
+        - SEARCHSERVICE_BASE=http://amundsensearch:5000
+        - METADATASERVICE_BASE=http://amundsenmetadata:5000
+      logging:
+        driver: awslogs
+        options: 
+          awslogs-group: amundsenfrontend
+          awslogs-region: us-west-2
+          awslogs-stream-prefix: amundsenfrontend
+
+

frontend/docs/installation-aws-ecs/ecs-params.yml

@@ -0,0 +1,18 @@
+version: 1
+task_definition:
+  services:
+    neo4j:
+      cpu_shares: 100
+      mem_limit: 3GB
+    elasticsearch:
+      cpu_shares: 100
+      mem_limit: 3GB
+    amundsensearch:
+      cpu_shares: 100
+      mem_limit: 500MB
+    amundsenmetadata:
+      cpu_shares: 100
+      mem_limit: 500MB   
+    amundsenfrontend:
+      cpu_shares: 100
+      mem_limit: 500MB

frontend/docs/installation-aws-ecs/userData.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+# For ElasticSearch
+sysctl -w vm.max_map_count=262144
+