Skip to content

Deployment

Jump to bottom
Stian Håklev edited this page Nov 10, 2017 · 6 revisions

Introduction

If you are serving up to 40-50 students simultaneously logged in, I think that deploying FROG to a single server should be sufficient. We still need a better understanding of performance limits, and the code could probably be made a lot more efficient.

To serve 250+ students over https, this is how we currently do it.

Architecture

We use Digital Ocean droplets, as well as two CHILI servers for Nginx load-balancing and SSL reverse proxying, because Let's Encrypt requires domain names, which we don't get from Digital Ocean. In the future, we want to investigate how to do the setup automatically, and possibly use Kubernetes.

Our infrastructure

  • icchilisrv4: Will load-balance and SSL terminate the Meteor servers
  • icchilisrv3: Will load-balance and SSL terminate the ShareDB servers
  • icchilisrv1: SSL terminates file upload servers (if needed)
  • frog-head: DO 1GB, runs Meteor, and Mongo for Meteor. Used by teacher
  • frog1-3: DO 512MB, additional Meteor servers
  • sharedb-head: DO 1GB, runs ShareDB, Mongo for ShareDB, Redis for ShareDB
  • sharedb1-3: DO 512MB, additional ShareDB servers
  • minio1-2: Optional file upload servers

Steps

Creating droplets

  • create all the DO clusters listed above, selecting any SSH keys needed, and Monitoring. Use most recent Ubuntu (16.04) image, location: Frankfurt
  • a good idea is to add the servers to /etc/hosts, so make it easier to do ssh sharedb-head etc

Install packages, generate keys

  • ssh into all of the new servers in different iTerm panes. Note that if you have previously used frog1 etc for different servers, you might have to remove these from ~/.ssh/known_hosts
  • turn on send input to all servers
  • run apt get and apt upgrade
  • apt install autossh
  • ssh-keygen -t rsa -b 4096 (no password etc)
  • cat ~/.ssh/id_rsa.pub
  • now you need to copy all the SSH keys from frog* and sharedb1-3 to sharedb-head, and from frog1-3 to frog-head - they should be added to ~/.ssh/authorized_keys, this will allow SSH login and autossh for port forwarding
  • install mongodb on sharedb-head and frog-head, and redis-server on sharedb-head
  • change port for mongodb on sharedb-head to 27300 (edit /etc/mongodb.conf, uncomment and set port, then service mongodb restart

Set up autossh links

  • send input to all servers, then tmux
  • find the ip of frog-head and sharedb-head, I'm going to use frog-head and sharedb-head below, but it must be replaced by the actual IPs
  • for frog* and sharedb1-3: /usr/lib/autossh/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -p 22 root@sharedb-head -L 27300:127.0.0.1:27300 -L 6379:127.0.0.1:6379 (this forwards Redis and Mongo for ShareDB)
    • make sure to type yes. It should not print anything, but also not return to the command prompt
  • for frog1-3: Ctrl+b c for new tmux window, then /usr/lib/autossh/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -p 22 root@frog-head -L 27017:127.0.0.1:27017
  • for sharedb1-3:

Starting ShareDB (in all sharedb*)

  • ctrl+b c for new tmux window
  • git clone https://github.com/chili-epfl/frog
  • install node8: 
    curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
  • npm install -g forever
  • cd frog/sharedb; npm install
  • forever index.js - you should see 'FROG ShareDB server started'
  • to test that ShareDB is functional, run node test_sharedb_instance.js ws://sharedb1:3002 etc from your local computer

Deploying Meteor

  • git checkout (or update) keybase://team/unilwip/frog-deploy
  • update mup.js with the relevant ip addresses
  • run mup setup the first time you are dealing with new servers
  • run mup deploy every time you have made a code change
  • run mup deploy --cached-build if something went wrong with the deploy, but you don't want to rebuild the app
  • run mup reconfig if you only changed mup.js or settings.json
  • Note: if you try to do mup deploy before having set up up the autossh, the Docker container will fail to start, and be caught in a restart-loop`
  • Note: it is not unusual for mup deploy to take 10-15 minutes, so try to avoid changing code just before you are planning to run a graph`

Updating Nginx configs

  • on icchilisrv4.epfl.ch, edit /etc/nginx/sites-available/default, and update the ip addresses in upstream meteor (listing all the frog* servers)
  • on icchilisrv3.epfl.ch, edit /etc/nginx/sites-available/default, and update the ip addresses in upstream sharedb (listing all the sharedb* servers, make sure to include the port 3002)
  • on both, run nginx -t to verify the config, and then service nginx reload
  • to test that ShareDB is functional, run node test_sharedb_instance.js wss://icchilisrv3.epfl.ch etc from your local computer

Verifying startup

  • you can check docker ps on any of the frog* servers, and you can tell whether the image is constantly rebooting or it's up for however many minutes since you deployed
  • you can visit individual Meteor servers (if you added the ips to your /etc/hosts, you can go to frog-head/ etc in your browser)
  • you can visit https://icchilisrv4.epfl.ch
  • when visiting, look at the websockets in dev console, to see that both (Meteor and ShareDB) initialize, and there is no reloading or timeouts

Updating code

  • when you update code, you should only have to mup deploy, everything else should keep working

Backing up data after a run

  • export session from FROG teacher view
  • ssh to frog-head, run 
    apt install -y zip
mongodump -o frog
mongodump --port 27300 -o sharebd
zip -r export.zip *
    and then scp root@frog-head:~/export.zip . from your local computer
  • if you use a file server, ssh to each file server, zip all the files in /mnt/data, and copy them over in the same way
Clone this wiki locally