Skip to content

Home

Jump to bottom
Naga Anjaneyulu edited this page Mar 2, 2020 · 22 revisions

Welcome to the Noodle wiki!

System Architecture

System Architecture

Getting Started

This document is a step-by-step guide to have the system up and running on your machines.

Clone the project from integration branch

git clone -b integration https://github.com/airavata-courses/Noodle.git

Pre-requisites

Few Points to note :

    1. Please ensure your python version is greater than 3.7.0.
    2. Please ensure your java version is 13.
    3. Please ensure your PostgreSQL version is 12.
    4. Please set your PostgreSQL password as idontknow.3
    5. You can easily find the installation instructions at the provided link.
    6. You can find the instructions for the same in the links provided below.
    7. After all installations, please add the appropriate folder to the path variable
       of your system as per requirements.
  1. NodeJS
  2. Python
  3. Java
  4. Spring Boot
  5. Maven / Maven Ubuntu Guide
  6. MongoDB / MongoDB Ubuntu guide
  7. PostgreSQL / PostgreSQL Ubuntu Guide

Kindly install all libraries mentioned below :

Windows Users follow the following series of steps :

> pip install flask
> pip install flask_sqlalchemy
> pip install flask_script
> pip install flask_migrate
> pip install jsons
> pip install kafka-python
> pip install boto3
> pip install metpy
> pip install psycopg2 / pip install psycopg2-binary

Ubuntu Users follow the following series of steps :

$ sudo pip install virtualenv
$ sudo virtualenv  microservices
$ cd microcervices
$ microcervices > sudo pip install flask
$ microcervices > sudo pip install flask_script
$ microcervices > sudo pip install flask_migrate
$ microcervices > sudo pip install flask_sqlalchemy
$ microcervices > sudo pip install jsons
$ microcervices > sudo pip install kafka-python
$ microcervices > sudo pip install boto3
$ microcervices > sudo pip install metpy
$ microcervices > sudo pip install psycopg2 / sudo pip install psycopg2-binary

Create Databases

Windows Users

Open pgAdmin and create the following databases by the following series of steps :

> Start Menu  -> pgAdmin
> Web interactive tool opens in your browser , enter your credentials.
> Expand Servers -> Expand PostgreSQL-> Right click on databases
> Click on create database -> Create the following databases.
> retrievedb
> processdb
> sessiondb
> modeldb

Ubuntu Users

Open the terminal , Follow the series of steps for creating databases :
$ psql

> postgres=# \conninfo
> postgres=# CREATE DATABASE retrievedb;
> postgres=# CREATE DATABASE processdb;
> postgres=# CREATE DATABASE sessiondb;
> postgres=# CREATE DATABASE modeldb;

Setting up kafka and zookeeper

Set up Zookeeper and kafka :

Kafka version - kafka_2.13-2.4.0 , Zookerper version - zookeeper-3.5.7
Windows Users
Zookeeper Ubuntu Users
Kafka Ubuntu Users

Please make the following changes in /config/server.properties by adding the following line 

listeners=PLAINTEXT://127.0.0.1:9092

Create the following topics in kafka by following the series of steps:

$ cd <Kafka-Home>
$ cd bin
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic data-retrieve
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic retrieved-data
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic processed-data
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic processed-data
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic retrieve-session
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic process-session
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic model-session
$ kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic current-status

Setting up each microservice

 Note : Ensure all databases, zookeeper, kafka are up and running before proceeding further.

1. User Management

Execute the service

Note: make sure port 5000 is unoccupied on the system

 1. open cmd/terminal in usermanagement directory
 2. run 'npm install' or 'npm i' command to install required packages
 3. run 'node server.js' command to start the service

2. Data Retrieval

Execute the service

Windows users

 1. open a new cmd/terminal in dataretrieval directory
 2. run 'python app.py db init' 
 3. run 'python app.py db migrate'
 4. run 'python app.py db upgrade'
 5. run 'python app.py retrieve_data'

Ubuntu users

 Move the dataretrieval directory to the environment - microservices.
 $ cd microservices
 $ microservices > cd dataretrieval
 $ microservices > dataretrieval > sudo python app.py db init
 $ microservices > dataretrieval > sudo python app.py db migrate
 $ microservices > dataretrieval > sudo python app.py db upgrade
 $ microservices > dataretrieval > sudo python app.py retrieve_data

3. Data Processing

Execute the service

Windows users

 1. open a new cmd/terminal in dataprocessing directory
 2. run 'python app.py db init' 
 3. run 'python app.py db migrate'
 4. run 'python app.py db upgrade'
 5. run 'python app.py process_data'

Ubuntu users

 Move the dataprocessing directory to the environment - microservices.
 $ cd microservices
 $ microservices > cd dataprocessing
 $ microservices > dataprocessing > sudo python app.py db init
 $ microservices > dataprocessing > sudo python app.py db migrate
 $ microservices > dataprocessing > sudo python app.py db upgrade
 $ microservices > dataprocessing > sudo python app.py process_data

4. Data Modeling

Execute the service

Windows users

 1. open a new cmd/terminal in datamodeling directory
 2. run 'python app.py db init' 
 3. run 'python app.py db migrate'
 4. run 'python app.py db upgrade'
 5. run 'python app.py execute_model'

Ubuntu users

 Move the datamodeling directory to the environment - microservices.
 $ cd microservices
 $ microservices > cd datamodeling
 $ microservices > datamodeling > sudo python app.py db init
 $ microservices > datamodeling > sudo python app.py db migrate
 $ microservices > datamodeling > sudo python app.py db upgrade
 $ microservices > datamodeling > sudo python app.py execute_model

5. Session Management

Execute the service

Windows users

Note: make sure port 5010 is unoccupied on the system
 1. open the directory in Spring Tool Suite.
 2. Click on run -maven clean Install .
 3. Once build is success run it as Spring Boot Application.

Ubuntu 16+ users

 $ cd sessionmanagement
 $ sessionmanagement > mvn clean install
 $ sessionmanagement > cd target
 $ sessionmanagement > target > java -jar session-management-0.0.1-SNAPSHOT.jar

6. API Gateway

Execute the service

Note: make sure port 5000 is unoccupied on the system
 1. open cmd/terminal in apigateway directory
 2. run 'npm install' command to install required packages
 3. run 'node server.js' command to start the service

7. User Interface

Execute the service

Note: make sure port 3000 is unoccupied on the system
 1. open cmd/terminal in frontend directory
 2. run 'npm install' command to install required packages
 3. run 'npm run start' command to start the service

Accessing the service after the setup is done

  • Now you can go to localhost:3000 to access the services from the User Interface
  • You can create an account by clicking the Register link and providing the necessary details
  • You can login with your credentials
  • You can submit a task by selecting a provided radar stations on the 'Run command' button
  • After your task is submitted, you can view your results and pasts sessions on 'Sessions' button
  • Feel free to create issues here related to errors and fixes with appropriate labels and brief description
Clone this wiki locally