-
Notifications
You must be signed in to change notification settings - Fork 1
Development Notes
A virtual machine has been created at STFC to host the database and run the web server.
TODO: notes about why technologies we are using.
We firstly create a basic Django app then add authentication using GitHub.
This is based on https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-18-04. Information on setting up nginx with SSL is based on https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04.
Firstly install nginx and Certbot, which we will use for obtaining SSL certificates from Let's Encrypt. As root run:
apt update
apt-get install nginx python-certbot-nginx
On some machines ports 80 and 443 are blocked by default by iptables. Run the following commmands as root to open these ports if necessary:
iptables -I INPUT 5 -i ens3 -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -I INPUT 5 -i ens3 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
Note that the network interface above (in this case ens3) may need to be adjusted as appropriate. Use ifconfig to determine the name of the network interface.
Create a server block for your domain in /etc/nginx/sites-available, for example /etc/nginx/sites-available/test.scrc.uk. Example contents:
server {
server_name test.scrc.uk;
listen 80 default_server;
listen [::]:80 default_server;
location = /favicon.ico { access_log off; log_not_found off; }
location /static/ {
root /home/ubuntu/scrc;
}
location / {
include proxy_params;
proxy_pass http://unix:/run/gunicorn.sock;
}
}
In the above the server name has to be adjusted as appropriate and the path for the location of static files needs to be set. Here it is the path to the directory created in a later step.
Create a symbolic link to enable this site in nginx:
ln -s /etc/nginx/sites-available/test.scrc.uk /etc/nginx/sites-enabled/
and start nginx:
systemctl start nginx
Obtain a certificate from Let's Encrypt, e.g.
certbot --nginx -d test.scrc.uk
where the DNS name will have to be adjusted as appropriate.
Install PostgreSQL and some required dependencies:
apt install python3-pip python3-dev libpq-dev postgresql postgresql-contrib
Unless otherwise specified, all commands below should be run as a normal user and not as root.
Create a directory for storing the virtual environment and Djanjo project:
mkdir scrc
cd scrc
Create a virtual environment and activate it:
virtualenv dataregistry_env
source dataregistry_env/bin/activate
Install required modules:
pip install django gunicorn psycopg2-binary
Create a data registry project in Django:
django-admin.py startproject dataregistry ~/scrc
Now we setup the PostgreSQL database. Firstly:
$ sudo -u postgres psql
psql (10.12 (Ubuntu 10.12-0ubuntu0.18.04.1))
Type "help" for help.
Create a database:
postgres=# CREATE DATABASE scrc;
CREATE DATABASE
Create a user:
postgres=# CREATE USER scrc WITH PASSWORD 'password';
CREATE ROLE
Give the user the required permissions and exit:
postgres=# ALTER ROLE scrc SET client_encoding TO 'utf8';
ALTER ROLE
postgres=# ALTER ROLE scrc SET default_transaction_isolation TO 'read committed';
ALTER ROLE
postgres=# ALTER ROLE scrc SET timezone TO 'UTC';
ALTER ROLE
postgres=# GRANT ALL PRIVILEGES ON DATABASE scrc TO scrc;
GRANT
postgres=# \q
Update the file ~/scrc/dataregistry/settings.py to include details about the database. DATABASES should be defined as:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'scrc',
'USER': 'scrc',
'PASSWORD': 'password',
'HOST': 'localhost',
'PORT': '',
}
}
Also update ALLOWED_HOSTS as necessary. This should contain localhost in addition to the external DNS name of the host, e.g.
ALLOWED_HOSTS = ['localhost', 'test.scrc.uk']
Finally, at the end of the file ~/scrc/dataregistry/settings.py add the following line:
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')
Run the following commands (make sure you have the virtual environment activated):
~/scrc/manage.py makemigrations
~/scrc/manage.py migrate
Create a superuser account by running:
~/scrc/manage.py createsuperuser
You will be asked to enter a usernamd and password. Then run:
~/scrc/manage.py collectstatic
This ensure all static content is placed in the directory we specifed in ~/scrc/dataregistry/settings.py enabling nginx to handle them correctly.
We now want to run gunicorn as a service. As root, create the file /etc/systemd/system/gunicorn.socket containing:
[Unit]
Description=gunicorn socket
[Socket]
ListenStream=/run/gunicorn.sock
[Install]
WantedBy=sockets.target
and create the file /etc/systemd/system/gunicorn.service containing:
[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target
[Service]
User=ubuntu
Group=ubuntu
WorkingDirectory=/home/ubuntu/scrc
ExecStart=/home/ubuntu/scrc/dataregistry_env/bin/gunicorn \
--access-logfile - \
--workers 3 \
--bind unix:/run/gunicorn.sock \
dataregistry.wsgi:application
[Install]
WantedBy=multi-user.target
As root, start the gunicorn socket service. The gunicorn service will automatically be started as necessary.
systemctl start gunicorn.socket
systemctl enable gunicorn.socket
Go to the external URL of the service (e.g. https://test.scrc.uk/) to see the default Django screen.
We now need to create an app within the Django project:
python manage.py startapp dataregistry_app
This creates a directory containing a number of files:
$ tree
.
├── __init__.py
├── admin.py
├── apps.py
├── migrations
│ └── __init__.py
├── models.py
├── tests.py
└── views.py
1 directory, 7 files
Update the file ~/scrc/dataregistry_app/views.py to contain the content:
from django.http import HttpResponse
def homePageView(request):
return HttpResponse('Hello, World!')
Add a file ~/scrc/dataregistry_app/urls.py containing:
from django.urls import path
from .views import homePageView
urlpatterns = [
path('', homePageView, name='home')
]
Now update the file ~/scrc/dataregistry/urls.py to contain:
from django.contrib import admin
from django.urls import path, include
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('dataregistry_app.urls')),
]
After making these changes we need to restart gunicorn for the changes to take affect. As root run:
systemctl restart gunicorn
Now if you go to the external URL of the service (e.g. https://test.scrc.uk/) you should see a page displaying Hello, World!.
This is based on https://learndjango.com/tutorials/django-allauth-tutorial.
Firstly, install the django-allauth package in the virtual environment:
cd ~/scrc ; source dataregistry_env/bin/activate
pip install django-allauth
In the file ~/scrc/dataregistry/settings.py add the following lines to the INSTALLED_APPS list:
'django.contrib.sites',
'allauth',
'allauth.account',
'allauth.socialaccount',
'allauth.socialaccount.providers.github',
At the bottom of the same file add the following lines:
AUTHENTICATION_BACKENDS = (
"django.contrib.auth.backends.ModelBackend",
"allauth.account.auth_backends.AuthenticationBackend",
)
SITE_ID = 1
# We don't need email verification upon signup as we're using GitHub
ACCOUNT_EMAIL_VERIFICATION = 'none'
# Redirect authenticated users to this URL
LOGIN_REDIRECT_URL = 'home'
# Specify required scopes
SOCIALACCOUNT_PROVIDERS = {
'github': {
'SCOPE': [
'read:user',
'user:email',
],
}
}
In the file ~/scrc/dataregistry/urls.py we need to add the URLs provided by the django-allauth packge. The content of this file should now look like:
from django.contrib import admin
from django.urls import path, include
urlpatterns = [
path('admin/', admin.site.urls),
path('accounts/', include('allauth.urls')),
path('', include('dataregistry_app.urls')),
]
Now run migrate to update the database as necessary (make sure you have the virtual environment activated):
python manage.py migrate
Go to GitHub to register an OAuth app. This is explained in https://learndjango.com/tutorials/django-allauth-tutorial. The main two settings are:
-
Homepage URL: I used https://test.scrc.uk/ for this -
Authorization callback URL: I used https://test.scrc.uk/accounts/github/login/callback/ for this
For testing this can be done within a normal user account. This will allow anyone with a GitHub account to access the Django app. If you want to restrict access of the Django app to a specific GitHub organization you must register an OAuth app within that organization (this must be done by an adminstrator of the organization). Keep a copy of the Client ID and Client Secret as this will be needed.
Login to the Django admin site, e.g. at https://test.scrc.uk/admin/. Click sites, then click on example.com and change botht the Domain name and Display name to test.scrc.uk. Click Save.
Click on Home at the top left-hand side of the page. Click Social applications and click on ADD SOCIAL APPLICATION. Select GitHub as the Provider. Specify an arbitrary name, for example Github. Set the Client id and Secret key to the Client ID and Client Secret from GitHub, obtained in the previous step. Set test.scrc.uk in the list of Chosen site. Finally click Save.
Now we will create a new home page which will allow unauthenticated users to login.
Create a templates directory:
mkdir ~/scrc/dataregistry_app/templates
then create a file ~/scrc/dataregistry_app/templates/home.html containing:
{% load socialaccount %}
<h1>SCRC Data Registry</h1>
{% if user.is_authenticated %}
<p>Welcome {{ user.username }} !!!</p>
{% else %}
<a href="{% provider_login_url 'github' %}">Sign Up using GitHub</a>
{% endif %}
We need to make Django aware of this template by updating TEMPLATES in ~/scrc/dataregistry/settings.py. In the definition of TEMPLATES replace the DIRS line with:
'DIRS': [os.path.join(BASE_DIR, 'templates')],
In the same file add the following line to INSTALLED_APPS:
'dataregistry_app.apps.DataregistryAppConfig',
Update the file ~/scrc/dataregistry_app/urls.py to contain:
from django.urls import path
from .views import Home
urlpatterns = [
path('', Home.as_view(), name='home')
]
and update the file ~/scrc/dataregistry_app/views.py to contain:
from django.views.generic import TemplateView
class Home(TemplateView):
template_name = 'home.html'
Restart gunicorn by running as root:
systemctl restart gunicorn
Now if you go to https://test.scrc.uk/ you should see a link Sign Up using GitHub. If you click on this you will be redirected to GitHub and be asked to give permission for the Django app to access your email addresses and profile. Once authorized you will be redirected back to Django and should see a message Welcome <GitHub username> !!!.
Currently django-allauth does not have a simple way to prevent users from being able to sign up by creating a username and password. However, there is a workaround. To do this we need to update the file ~/scrc/dataregistry/urls.py to contain:
from django.contrib import admin
from django.urls import path, include
from django.conf.urls import url
from allauth.account.views import confirm_email, login, logout
urlpatterns = [
url(r'^accounts/email/', login, name='disable_email'),
url(r'^accounts/password/', login, name='disable_password'),
url(r'^accounts/inactive/', login, name='disable_inactive'),
url(r'^accounts/confirm-email/', login, name='disable_confirm_email'),
path('accounts/', include('allauth.urls')),
url(r'^login/$', login, name='account_login'),
url(r'^logout/$', logout, name='account_logout'),
url(r'^signup/$', login, name='account_signup'),
path('admin/', admin.site.urls),
path('', include('dataregistry_app.urls')),
]
This will cause all of the default django-allauth URLs to be directed to the standard login page, preventing users from being able to sign up by creating an account directly in Djando. As usual, gunicorn should be restarted in order for the changes to take effect.
To logout, go to the URL https://test.scrc.uk/logout/.