-
Notifications
You must be signed in to change notification settings - Fork 189
/
Copy patharchitecture.md
100 lines (63 loc) · 9.58 KB
/
architecture.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
---
sidebar_position: 2
---
# Architecture
Telescope is both a product and a project. It is designed to meet a number of practical requirements (e.g., blog aggregation), but also to give students the opportunity to work on a real-world open source project using a wide range of technologies. We could build something simpler; but our design is also about creating a safe space for technical exploration. As a result, Telescope is architected like many modern distributed services.
## Initial Design
Our initial design was captured in this rough diagram:
It shows RSS/Atom blog feeds (on the left) being downloaded (1) and then queued (2) in order to be downloaded and parsed into a database. Parsing each feed also included the idea of analysis, to look at word count, reading time, etc. This database of parsed feeds is then exposed to various clients via a REST API. We imagined being able to provide multiple clients, written for the web or mobile, but also connecting to Slack and other systems.
## Current Design
The following gives an overview of the current (i.e. 2.4.0) design of Telescope. There are a number of major systems that make up the app:
- legacy monolithic back-end (1.0)
- back-end microservices (2.0)
- front-end web app
- SAML2-based authentication and authorization
- autodeployment server
- test, build, and other automation tooling
### Back-end Microservices (2.0)
The legacy back-end has been split into a series of microservices. Each microservice follows a similar architectural pattern and has a single focuses. The current microservices include:
- SSO Auth Service (`src/api/sso`) - authentication and authorization with Seneca's SSO service
- Feed-Discovery Service (`src/api/feed-discovery`) - blog feed discovery and validation
- Image Service (`src/api/image`) - dynamic image generation for the front-end hero banner
- Planet Service (`src/api/planet`) - recreation of the original CDOT Planet using Telescope's back-end
- Posts Service (`src/api/posts`) - API for accessing Post and Feed data in Redis (probably not well named at this point)
- Search Service (`src/api/search`) - API for doing searches against Elasticsearch
- Status Service (`src/api/status`) - API for accessing Telescope status information, as well as providing the Dashboards
- Parser Service (`src/api/parser`) - feed and post parsing.
All microservices are built on a common foundation, the [Satellite module](https://github.com/Seneca-CDOT/telescope/tree/master/src/satellite). Satellite provides a common set of features for building Express-based microservices, with proper logging, health checks, headers, authorization middleware, as well as connections to Redis and Elasticsearch. It saves us having to manage the same set of dependencies a dozen times, and repeat the same boilerplate code.
The microservices are deployed as Docker containers. Each defines its own `Dockerfile`, and the entire system is managed via `docker-compose`, see `src/docker`, where we have a base service definition in `src/docker/docker-compose.yml` as well as overrides for various environments: `development.yml` (local dev), `production.yml` (staging and production), and `gitpod.yml` (running Telescope in GitPod). Each of these service definitions uses an associated environment file, which are defined in `src/config`. For example, Telescope's production server uses the `src/docker/production.yml` file and `src/config/env.production` environment.
When deployed, the services are not directly accessible via the network (i.e. you can't go to localhost:5555 to access the server). Instead, you have to access them via an API gateway called Traefik, which also runs in its own container. Traefik lives at api.telescope.cdot.systems (production), dev.api.telescope.cdot.systems (staging), and localhost:80 (development). It provides routing to each of the service containers via labels defined in the `src/docker/docker-compose.yml` file. For example, the Image Service is available at `/v1/image`. In production you access it at <https://api.telescope.cdot.systems/v1/image>, in staging at <https://dev.api.telescope.cdot.systems/v1/image>, and locally at <http://localhost/v1/image>. The other services are similar. Traefik makes the different services look like one thing, but they are actually deployed and running separately.
The microservices can be controlled either using the `docker-compose` command directly, or with the following `npm` scripts:
- `pnpm run services:start` starts all services. You can also list one or more services to start a subset, `npm run services:start image planet`
- `pnpm run services:stop` stops all services, or those specified
- `pnpm run logs image` shows logs for the image service (specify whichever service you want to view)
On Windows and macOS, Docker Desktop must be running before you can call these scripts, see the [Docker documentation](../docs/tools-and-technologies/docker.md).
### Front-end Web App
The front-end web app is written in TypeScript and uses next.js and Material UI for routing, component system, etc. The code lives in `src/web/app`.
The web app accesses many of the microservices at runtime, and we depend on a build step to export and include the necessary runtime variables. In the front-end app, they are available at runtime via `src/web/app/src/config.ts`. These are populated via the `src/web/app/next.config.js` during build time. See [Environment Setup](../docs/getting-started/environment-setup.md) for details on how to populate these with different values for local, CI, staging, or production environments. See also [Front End](../docs/tools-and-technologies/nextjs.md) for more details about the front-end.
The front-end is run via its own container, and hosted statically by nginx. The container is defined in `src/web/app/Dockerfile`. Locally, it is available at <http://localhost:8000>, on staging at <https://dev.telescope.cdot.systems/>, and on production at <https://telescope.cdot.systems/>.
### SAML2-based authentication and authorization
Telescope uses SAML2-based authentication via Seneca's Single Sign-On in combination with token-based authorization via the SSO Auth and Users service. A good discussion of the main ideas and implementation is available in [this blog post](https://blog.humphd.org/not-so-simple-saml/) and [this Pull Request](https://github.com/Seneca-CDOT/telescope/pull/1796#issue-812477759).
In development, we use a test PHP-based SAML Identity Provider to test against. See [Login](../docs/tools-and-technologies/login.md).
### Autodeployment Server
Our release infrastructure is built primarily on GitHub Actions, see `.github/workflows/release.yml`. However, we also host our own build service at `tools/autodeployment`. When a [Release is created](../docs/contributing/release.md), the autodeployment server receives a webhook event from GitHub. We use this to trigger a new build and deploy it when complete. The build script that gets run is at `tools/autodeployment/deploy.sh`, which does a green/blue style of deployment (i.e., keep one version running while you build a new one, swap them when ready).
The autodeployment server is run outside of Docker, since it manages our docker containers. It has an API available at <https://telescope.cdot.systems/deploy/status> (production) and <https://dev.telescope.cdot.systems/deploy/status> (staging), and build logs are available at `/deploy/status/log`.
### Dashboards
The Status service provides a number of real-time dashboards. These are available at <https://api.telescope.cdot.systems/v1/status> (production) and <https://dev.api.telescope.cdot.systems/v1/status> (staging).
The Dashboards intentionally do not use React, next.js, or other technologies from the normal front-end. This was done in order to provide students who don't want to work on React a place to contribute. Instead, it uses plain HTML, SCSS/CSS, and JavaScript, and uses server-side rendering with Express. It also uses a variety of front-end DOM code written in JavaScript.
The Status service uses this Autodeployment API to show a live build log, for example: <https://api.telescope.cdot.systems/v1/status/build> on production, <https://dev.api.telescope.cdot.systems/v1/status/build> on staging.
### Test, Build, and other Automation Tooling
We use the following tools:
- pnpm to manage packages within our monorepo, see `pnpm-workspace.yaml`
- jest (unit, snapshot) and Playwright (e2e) to write and run tests. We define configs (see `./jest*.js`) in the root of the project, and use/override them in sub-folders. Our tests are in `test/` and also in `test/` folders deeper in the tree
- eslint to lint our code, see `.eslintrc.js` and `.eslintignore`
- prettier to format our code, with Husky managing a precommit hook to automatically apply it on every commit, see `.prettierrc`, `.prettierignore`, and `.husky/`
- various VSCode settings and extensions, defined in `.vscode/`
- GitHub Actions Workflows for testing, see `.github/`
- Renovatebot to manage dependency updates, see [Renovate](../docs/tools-and-technologies/renovate.md)
- GitPod for cloud development, see [Gitpod](../docs/contributing/git-workflow.md), `.gitpod.yml` and `gitpod.Dockerfile`
We also have a number of tools running in containers at runtime (most only in staging/production) (see `docker/production.yml`):
- nginx, which we use to host static web sites and also as a reverse proxy for node APIs. Our nginx configs are built from a template that lives in `./config/nginx.conf.template`
- certbot, which manages our Let's Encrypt SSL Certificates (i.e., automatically renewing them)
- portainer, to provide a user-friendly front-end to our main containers at run-time