WTC-Console

This project uses Django React/Redux Base Project from Seedstarts as a boilerplate. Please refer to their github for the complete list of technologies used.

Here are the main tools whose knowledge is useful to contribute:

Frontend

React
React Router Declarative routing for React
Webpack for bundling
Redux Predictable state container for JavaScript apps
React Router Redux Ruthlessly simple bindings to keep react-router and redux in sync
styled-components Keeping styles and components in one place
font-awesome-webpack to customize FontAwesome

Backend

Django
Django REST framework Django REST framework is a powerful and flexible toolkit for building Web APIs
MongoEngine Python ODM for MongoDB
Celery Distributed tasks queue

Architecture

This app has a bit complicated architecture due to the nature of other systems it has to communicate.

MongoDB - a NoSql database where we keep application data in json like documents. Here we also store actions for Unified.
Oracle - we only read failed workflows names which are stored here by Unified. It would be nice if Unified provided api or at least stored it in MongoDB.
RequestManager - rest api used for fetching related workflows PrepID name. To access this api you need grid certificate.
WmStats - rest api from which we get all the info about workflow and errors. To access this api you need grid certificate.

Installing on a new machine (prod, dev)

Production/development setup uses nginx as reverse proxy and Gunicorn as an application server. Below are the steps needed to setup environment on RHEL from scratch. Instructions are based on this article How To Set Up Django with Postgres, Nginx, and Gunicorn on CentOS 7

Setting up environment

Use these instructions to setup a new production environment from scratch. By following these instructions you will create a dedicated user wtc-console for running this application with Gunicorn, update proxy config for Nginx and setup firewall to allow traffic on port 80.

Prerequisites:

Python >=2.7

Install Node and NPM

If you are using CERN managed machine then ask administrator to install latest Node version. If you are managing this machine, then follow this guide for RHEL

Install Oracle Instant Client

Go to Oracle client site and download Oracle Instant Client 12.2 Basic. Note: you will need to have Oracle account and download it to your machine.
Upload it to the server in your preferred way
Run sudo yum localinstall oracle-instantclient* --nogpgcheck

Install RabbitMQ

Add RabbitMQ repo according to RabbitMQ installation guide. Then run these commands:

sudo yum install rabbitmq-server
sudo chkconfig rabbitmq-server on
sudo /sbin/service rabbitmq-server start

Create user and login with it

sudo useradd wtc-console
sudo su - wtc-console

Get the sources

Clone this project to wtc-console users home directory.

git clone https://github.com/CMSCompOps/wtc-console.git
cd wtc-console/

Setup oracle client

Open wtc-console users .bashrc file:

vim ~/.bashrc

Add these lines to it:

export ORACLE_HOME=/usr/lib/oracle/12.2/client64
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export PATH=$PATH:$ORACLE_HOME/bin

And apply these changes:

. ~/.bashrc

Create virtual python environment

pip install --upgrade pip
pip install virtualenv
virtualenv wtc-console-env

Install and configure nginx

Log out of wtc-console users session and with your user install nginx and dependencies

sudo yum install epel-release
sudo yum install nginx

Add following lines to /etc/nginx/nginx.conf as a first server entry in http section and change domain_name to the actual domain name probably in format of node_name.cern.ch

server {
    listen 80;
    server_name domain_name;
    location = /favicon.ico { access_log off; log_not_found off; }
    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://0.0.0.0:8000;
    }
}

Test if configuration is valid

sudo nginx -t

Start nginx and set it to run on startup

On RHEL and CentOS:

sudo systemctl start nginx
sudo systemctl enable nginx

On Scientific Linux CERN:

sudo service nginx start
sudo chkconfig nginx on

For RHEL, Fedora, CentOS

If when opening domain.cern.ch you see this error in /var/log/nginx/error.log:

*2 connect() to 127.0.0.1:8000 failed (13: Permission denied) while connecting to upstream, client: some_ip, server: some_domain, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "some_domain"

Then use this command to solve it:

sudo setsebool -P httpd_can_network_connect 1

It turns on httpd connections and -P makes it persistent.

Give Nginx group rights to the project directory

sudo chown -R wtc-console:nginx /home/wtc-console/wtc-console
sudo chmod 770 /home/wtc-console/wtc-console

Firewall config

Ask system administrators to include port 80 to puppet config. If puppet is not used, then you can configure firewall yourself to bypass traffic on this port with these commands:

sudo iptables -I INPUT 1 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -I OUTPUT 1 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo service iptables save
sudo service iptables restart

You can see current config with sudo iptables --line -vnL or sudo less /etc/sysconfig/iptables

Update settings

For development node

Copy src/djangoreactredux/settings/local_template.py setting file to src/djangoreactredux/settings/local.py and modify oracle db credentials through UNIFIED_DB and MONGODB_DATABASES to reflect your development environment. Errors and debug information concerning celery workers and oracle db configuration will not be rendered through Django, instead always check logs/celery.log for errors: grep ERROR logs/celery.log.
Modify TNS_ADMIN located at bin/start_dev.sh, at cern use: /afs/cern.ch/project/oracle/admin/.

For production node

Create prod.py in src/djangoreactredux/settings/ directory by using prod_template.py settings template file and update the oracle and mongo db fields with prod values.

cp src/djangoreactredux/settings/prod_template.py src/djangoreactredux/settings/prod.py
vim src/djangoreactredux/settings/prod.py

Create certificates for this machine and copy them to cert/ directory.

Next steps

Proceed to Development on dev machine or Production deployment steps depending on the machine purpose.

Development on remote machine

Development on remote node requires to run three sessions (terminals). First one will have frontent watch running, second will have backend running. And third one is for developent with editor of your choise.

Add your hostname to the list of allowed sites

Modify src/djangoreactredux/settings/base.py to include the assigned hostname:

ALLOWED_HOSTS = ['localhost','myDevTestMachine.cern.ch']

For all terminals

Become wtc-console user:

sudo su - wtc-console

Go to project dir

cd wtc-console

In terminal 1

Start frontend resources watching:

npm install && npm run dev

In terminal 2

Start Django backend and Celery workers:

./bin/start_dev.sh

In terminal 3

Edit sources with an editor of you choise.

Certificates placement

To install CERN Certification Authority and User certificates:

mkdir src/djangoreactredux/cert && cd wtc-console/src/djangoreactredux/cert
wget -c "https://cafiles.cern.ch/cafiles/certificates/CERN%20Root%20Certification%20Authority%202.crt" -O CERNRootCertificationAuthority2.crt

And, depending on the location of your Key/Cert files:

cp ~/.globus/cert.pem crt.pem 
cp ~/.globus/key.pem .

The location of these files is specified via src/djangoreactredux/settings/dev.py

Production deployment

Become wtc-console user:

sudo su - wtc-console

Go to project dir

cd wtc-console

Deployment is done with one bash command. It will:

shutdown celery workers
shutdown current application if running
pull latest changes from repository master branch
install missing dependencies
build frontend app
start the application
start celery workers

./bin/deploy_prod.sh

Stopping server

If for some reason application should be stoppet then use this script:

./src/bin/stop_prod.sh

It will stop Gunicorn and Celery tasks

Maintenance

Logs are in /home/wtc-console/wtc-console/logs

Development guidelines

When developing a new feature create your own branch and push your changes at least daily.

Do not push directly to master. Create pull requests and assign someone to approve it. Go through your pull request your self, it helps to see if there is unwanted or commented-out code.

Adding new dependencies

While working on project you might encounter situations where you want to add functionality from third parties. This is done by adding dependecies to external libraries.

Frontend

Intall dependency with yarn

yarn add dependency-name

Backend

Install dependency with pypi

pip install dependency-name

Name		Name	Last commit message	Last commit date
Latest commit History 281 Commits
bin		bin
cert		cert
docker		docker
logs		logs
oracle-admin		oracle-admin
py-requirements		py-requirements
scripts @ 8e671b2		scripts @ 8e671b2
src		src
tests		tests
webpack		webpack
.babelrc		.babelrc
.bootstraprc		.bootstraprc
.env		.env
.gitignore		.gitignore
.gitmodules		.gitmodules
.travis.yml		.travis.yml
AUTHORS.md		AUTHORS.md
LICENSE		LICENSE
README.md		README.md
docker-compose-all.yml		docker-compose-all.yml
docker-compose.yml		docker-compose.yml
package.json		package.json
pytest.ini		pytest.ini
pytest_ci.ini		pytest_ci.ini

License

CMSCompOps/wtc-console

Folders and files

Latest commit

History

Repository files navigation