-
Notifications
You must be signed in to change notification settings - Fork 1
Deploying on UQCloud Zones
Given that the intention is for these applications to be used on UQCloud Zones, a guide to deploying these applications is outlined in this section. This guide works for the UQ-Ubuntu image (running Ubuntu 16.04.6 LTS, GNU/Linux 4.3.0 x86_64) of size at least large. This guide also assumes a familiarity with the system architecture (an outline of which can be found here), and the development environment (here).
The first step is to ensure that all of the relevant tools are installed and at the correct version. The following code first updates the apt package manager.
sudo apt update
Next, we install Redis, and start the Redis service for use in the applications.
sudo apt install redis-server
sudo systemctl restart redis.service
A Redis Python package also needs to be installed. To do this, both python3 and pip3 are required. The lines below can be run to install the correct version of pip, and use it to install Redis.
sudo apt install software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.7
sudo apt install python3-pip
python3.7 -m pip install redis
To allow the Allocation backend to operate, Google's OR-Tools are required. These can be installed by running the following (as per these instructions ).
python3.7 -m pip install --upgrade --user ortools
Following on from this, we install the relevant PostgreSQL packages, as well as NodeJS. The code below installs version 8 of NodeJS.
sudo apt install postgresql postgresql-contrib
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
In addition to these packages, Git also needs to be installed in order to retrieve the code from this repository, as it is not present by default on the UQ-Ubuntu image.
sudo apt install git
Once Git has been installed, this repository can be cloned into a folder called project, and the permissions on this folder changed appropriately. This folder can, in theory, be created anywhere, however this guide will place it within the /var/www/ directory, so we cd there first, as shown below.
cd /var/www/
sudo git clone https://github.com/BraeWebb/special project
sudo chown -R root:sysadmin project/
sudo chmod -R g+w project/
Currently, the branch which needs to be checked out is the moss branch, which can be done by running git checkout moss.
Additionally, once Git has been installed, another Python package called Mossum is needed. Mossum is a tool for summarising results from MoSS, and is used to generate graphs in Integrity. Similarly to before, this can be installed by running:
python3.7 -m pip install git+https://github.com/hjalti/mossum@master
To set up the database, we need to modify some of the standard PostgreSQL configurations. Firstly, however, we need to change the PostgreSQL password. To do this, we first run the following command:
sudo -u postgres psql
This will prompt you for a password, and you should type in your UQ zone password. Upon successful entry, the PostgreSQL prompt (postgres=#) should pop up. In this prompt, we can now type the following:
ALTER USER postgres PASSWORD '<POSTGRES_PASSWORD>';
where <POSTGRES_PASSWORD> is the password you have chosen for the postgres user.
Upon successfully completing this step, we need to modify the PostgreSQL settings, found in the pg_hba.conf file. To do this, we run:
sudo vim /etc/postgresql/9.5/main/pg_hba.conf
In this file, a line which looks similar to the following should exist:
local all postgres peer
This needs to be modified from having peer authentication (which uses the Linux authentication set up for the zone), to having md5 authentication, which allows a separate password (the one we just created) to be specified. The line should now look like this:
local all postgres md5
The file can now be saved and exited (by running :wq). To ensure that PostgreSQL updates itself to be aware of these changes, we should now run:
sudo systemctl restart postgresql
Once this has been completed, we can cd into the relevant directory.
cd /var/www/project/db/init
We now need to run the commands contained within the various sql files in this folder. This is done by running the following commands. A prompt to enter the newly created postgres password will pop up after each command.
psql -U postgres --password -f 00-database.sql
psql -U postgres --password -f 01-tables.sql
psql -U postgres --password -f 02-dummy.sql
A note that this is now handled by the Sequelize setup in all applications but Allocate. If, at some point, Allocate is switched over to using Sequelize, then this step will no longer be required.
Next, we can install postgraphile, necessary for using GraphQL.
sudo npm install -g postgraphile
To launch the GraphQL server, the following line can be run, where the URL is taken directly from the docker-compose.yml file, and POSTGRES_PASSWORD is once again the password we created previously for the postgres user.
postgraphile -c "postgres://postgres:<POSTGRES_PASSWORD>@0.0.0.0:5432/special" --host 0.0.0.0 --port 5000 --schema special
This line should be run any time the GraphQL server needs to be started up.
Also in the project, due to environment variables not yet being configured, the server/schema/model.js file should be changed from containing:
const db = new Sequelize("postgres://postgres:bVawU6etXvWwQgsR@localhost:5432/special", {
logging: false
});
to containing the following:
const db = new Sequelize("postgres://postgres:<POSTGRES_PASSWORD>@localhost:5432/special", {
logging: false
});
The next step is to install all the relevant Node modules for the project. This requires running npm install in a handful of folders. From the init directory, the following commands should be run:
cd ../../server
npm install
cd ../middleware/moss
npm install
cd ../allocate
npm install
cd ../queue
npm install
cd ../../frontend
npm install
npm run build
The npm run build command generates the static files needed for our applications and places them into a newly created dist folder. If the permissions of this folder are not currently set up correctly (as the command run to change permissions was done so before the dist folder was created), then the following command should be run:
sudo chmod -R 755 /var/www/project/frontend/dist/
These static files should now be copied across to the htdocs directory which Nginx uses by default to serve static files. This can be achieved by running:
sudo cp -R /var/www/project/frontend/dist/* /var/www/htdocs/
In addition, files within some of these generated dist folders need to be moved into a public folder (also created by dist). This can be done by running:
sudo cp -r /var/www/htdocs/scripts/ /var/www/htdocs/public/
sudo cp -r /var/www/htdocs/css/ /var/www/htdocs/public/
sudo cp -r /var/www/htdocs/js/ /var/www/htdocs/public/
Other configuration which needs to be changed within the project for deploying the queue is in the frontend/src/queue.js file, which can be seen here. Currently, there should be a section in this file which looks like the following:
const httpLink = new HttpLink({
uri: 'http://0.0.0.0:4000/graphql'
});
const wsLink = new WebSocketLink({
uri: 'ws://0.0.0.0:4000/graphql',
options: {
reconnect: true,
},
});
These URLs need to be changed from 0.0.0.0:4000 to the relevant URL of the UQCloud Zone being deployed to. As an example, if we were deploying to integrity.uqcloud.net, we would modify this section of the file to be:
const httpLink = new HttpLink({
uri: 'https://integrity.uqcloud.net/graphql'
});
const wsLink = new WebSocketLink({
uri: 'wss://integrity.uqcloud.net/graphql',
options: {
reconnect: true,
},
});
Please also note the changes from http to https and from ws to wss - these are important!
The Nginx configuration also needs to be modified to handle routing for Integrity. This should be placed within the section of the /etc/nginx/nginx.conf configuration file which looks like the following:
...
http {
...
server {
listen 443 default_server;
...
// new configuration code here
...
}
}
The configurations which need to be added are as follows:
location /integrity {
try_files '' /integrity/index.html =404;
}
location /queue {
try_files '' /queue/index.html =404;
}
location /moss {
proxy_pass http://localhost:3050;
}
location /graphql {
proxy_pass http://localhost:4000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
Once these lines have been added, the Nginx server should be reloaded by running
sudo nginx -s reload
To start up the Investigate middleware, the following commands should be run:
cd /var/www/project/middleware/moss
npm start
To start up the Investigate backend, the following commands should be run:
cd /var/www/project/backend
python3.7 entry.py
To start up the Allocate middleware, the following commands should be run:
cd /var/www/project/middleware/allocate
npm start
To start up the Allocate backend, the following commands should be run:
cd /var/www/project/backend/allocation
python3.7 entry.py
Similarly, to start up Floodgate (the queue), the following commands should be run:
cd /var/www/project/server
mkdir uploaded
npm start
The services should now be up at <YOUR_ZONE>.uqcloud.net/queue, /integrity, and /allocate!