Opwen cloudserver

What's this?

This repository contains the source code for the Opwen cloud server. Its purpose is to connect the application running on the Opwen Lokole devices to the rest of the world.

The server has two main responsibilities:

1. Receive emails from the internet that are addressed to Lokole users and forward them to the appropriate Lokole device.
2. Send new emails created by Lokole users to the rest of the internet.

System overview

Data exchange format

In order to communicate between the Opwen cloud server and the Opwen web-application (aka Lokole), a protocol based on gzipped jsonl files uploaded to Azure Blob Storage is used. The files contains a JSON object per line. Each JSON object describes an email, using the following schema:

{
  "sent_at": "yyyy-mm-dd HH:MM",
  "to": ["email"],
  "cc": ["email"],
  "bcc": ["email"],
  "from": "email",
  "subject": "string",
  "body": "html",
  "attachments": [{"filename": "string", "content": "base64"}]
}

Development setup

First, get the source code.

git clone git@github.com:ascoderu/opwen-cloudserver.git
cd opwen-cloudserver

Second, install the system-level dependencies using your package manager, e.g. on Ubuntu:

sudo apt-get install -y make python3-venv shellcheck

You can use the makefile to verify your checkout by running the tests and other CI steps such as linting. The makefile will automatically install all required dependencies into a virtual environment.

make tests
make lint

This project consists of a number of microservices and background jobs. You can run all the pieces via the makefile, however, it's easiest to run and manage all of the moving pieces via Docker, so install Docker on your machine by following the Docker setup instructions for your platform.

The project uses Sendgrid, so to emulate a full production environment, follow these Sendgrid setup instructions to create a free account and take note of you API key for sending emails.

The project also makes use of a number of Azure services such as Blobs, Tables, Queues, Application Insights, and so forth. To set up all the required cloud resources programmatically, you'll need to create a service principal by following these Service Principal instructions. After you created the service principal, you can run the Docker setup script to initialize the required cloud resources.

docker build -t opwenserversetup -f docker/setup/Dockerfile .

docker run \
  -e SP_APPID={appId field of your service principal} \
  -e SP_PASSWORD={password field of your service principal} \
  -e SP_TENANT={tenant field of your service principal} \
  -e SUBSCRIPTION_ID={subscription id of your service principal} \
  -e LOCATION={an azure location like eastus} \
  -e RESOURCE_GROUP_NAME={the name of the resource group to create or reuse} \
  -e SENDGRID_KEY={the sendgrid key you created earlier} \
  -v ${PWD}/secrets:/secrets \
  opwenserversetup

The secrets to access the Azure resources created by the setup script will be stored in files in the secrets directory. Other parts of the project's tooling (e.g. docker-compose) depend on these files so make sure to not delete them.

Finall, run the application stack via Docker:

docker-compose up --build

There are OpenAPI specifications that document the functionality of the application and provide references to the entry points into the code (look for "some-api-name-spec.yaml" files in the repository).

Production setup

To set up a production-ready deployment of the system, follow the development setup scripts described above, but additionally also pass the following environment variables to the Docker setup script:

• KUBERNETES_RESOURCE_GROUP_NAME: The resource group into which to provision the Azure Kubernetes Service cluster.
• KUBERNETES_NODE_COUNT: The number of VMs to provision into the cluster. This should be an odd number and can be dynamically changed later via the Azure CLI.
• KUBERNETES_NODE_SKU: The type of VMs to provision into the cluster. This should be one of the supported Linux VM sizes.

The script will then provision a cluster in Azure Kubernetes Service and install the project via Helm. The secrets to connect to the provisioned cluster will be stored in the secrets directory.