build.md

# Riju build system

Riju's build system is complex and takes some time to explain. Bear
with me. (If you just want to add or modify a language, you can read
the [tutorial](tutorial.md) instead.)

## Depgraph

The high level interface to Riju's build system is a tool called
Depgraph, which knows about all the build artifacts and has advanced
mechanisms for determining which of them need to rebuild based on
content hashes. Normally you can just use Depgraph to build the
artifacts you need. However, in some cases you may want to interact
with the lower level for more precise operations. This is done via
Makefile. (Furthermore, there are a few one-off artifacts such as the
admin image which are not part of the main build system, which means
that they are managed directly by Makefile.)

### Available build artifacts

* `image:ubuntu`: A fixed revision of the upstream `ubuntu:rolling`
  image that is used as a base for all Riju images.
* `image:packaging`: Provides an environment to build Debian packages.
  Depends on `image:ubuntu`.
* `image:runtime`: Provides an environment to run the Riju server and
  the test suite. Depends on `image:ubuntu`.
* `image:base`: Provides a base image upon which per-language images
  can be derived. Depends on `image:ubuntu`.
* `deb:lang-xyz`: For each language `xyz`, the Debian package that
  installs that language into the base image. Depends on
  `image:packaging` and the build script for the language (generated
  from the `install` clause of `langs/xyz.yaml`).
* `deb:shared-pqr`: Same but for shared dependencies, which are also
  archived as Debian packages.
* `image:lang-xyz`: For each language `xyz`, the per-language image
  used for user sessions in that language. Depends on `image:base`,
  `deb:lang-xyz`, and possibly one or more `deb:shared-pqr`.
* `test:lang-xyz`: An artifact certifying that the `xyz` language
  tests passed. Depends on `image:runtime`, `image:lang-xyz`, the test
  suite and API protocol code, and the `xyz` language configuration.
* `image:app`: Built on top of `image:runtime` but including the Riju
  server code, so that it can be run standalone. Depends on
  `image:runtime` and the application code.
* `deploy:ready`: Deployment configuration, ready to upload. Depends
  on `image:app` and all `test:lang-xyz`.
* `deploy:live`: Pseudo-artifact corresponding to actually running the
  deployment, which is a blue/green cutover in which all languages and
  the application server are updated at once.

### Depgraph abstractions

Each artifact has:

* A list of zero or more dependencies on other artifacts.
* A recipe to build it locally assuming that its dependencies are also
  available locally.
* A recipe to upload the local build artifact to a remote registry.
* A recipe to download the artifact from a remote registry to
  overwrite the local version.
* A way to compute a content-based hash of the artifact's dependencies
  and non-artifact inputs (data files and code). Crucially this does
  not require the dependencies to actually be built (the hash for each
  artifact is based only on its dependencies *hashes*), so it's
  possible to compute hashes for the entire dependency tree before
  doing anything.
* A way to check the hash currently attached to a local artifact,
  which can be compared to the desired hash to see if it needs to be
  rebuilt.
* A way to check the hash currently attached to an artifact in a
  remote registry, which can also be compared to the desired hash.

There are special types of artifacts:

* *Manual* artifacts do not have a hash until they are generated.
  Therefore, they must be built manually before the rest of the
  dependency calculations can proceed. `image:ubuntu` is a manual
  artifact since its hash depends on what we download from
  `ubuntu:rolling`.
* *Publish* artifacts do not have a hash after they are generated.
  Therefore, nothing can declare a dependency on them. `deploy:live`
  is a publish artifact. (Actually `deploy:ready` is a publish
  artifact too, but that is an implementation detail because I was
  lazy about my abstractions.)

### Usage of Depgraph

```
$ dep --help
Usage: dep <target>...

Options:
  --list         list available artifacts; ignore other arguments
  --manual       operate explicitly on manual artifacts
  --hold-manual  prefer local versions of manual artifacts
  --all          do not skip unneeded intermediate artifacts
  --registry     interface with remote registry for caching
  --publish      publish artifacts to remote registries
  --yes          execute plan without confirmation
  -h, --help     display help for command
```

You can run `dep --list` to list all the available artifacts. Then
`dep name-of-artifact [names-of-more-artifacts...]` will generate
those artifacts. Depgraph is like Terraform in that it will compute a
plan and then ask you to confirm before proceeding.

By default Depgraph will generate artifacts locally only, although it
can be instructed to also interact with a remote registry if you've
set up the infrastructure appropriately. Pass `--registry` to enable
this mode, in which case Depgraph will download remote artifacts when
appropriate versions exist in the registry. Pass `--publish` along
with `--registry` to also cache generated artifacts in the remote
registries. Of course `--publish` is required to build `deploy:live`.

For dealing with `image:ubuntu` specifically, you probably just want
to fetch Riju's version (available in a public ECR repository) using
`make sync-ubuntu` to keep in sync. However if you do want to update
to the latest `ubuntu:rolling`, it's `dep image:ubuntu --manual`.

The other options (aside from `--yes`) are mostly not too useful.
Depgraph is very sophisticated and should always compute the minimum
necessary build plan based on any changes you have made. So, you don't
need to worry about the details! (Except when the hashing isn't
working properly. Then you cry.)

## Makefile

To get a "quick" overview, run `make help`.

### Preliminary targets

There are a couple of targets that are independent of Depgraph and
need to be run just to make sure various bits of state and generated
files are up to date. Depgraph and/or `ci-run.bash` take care of this.

* `make ecr`: Authenticate to ECR, needed to push and pull. The
  authentication only lasts for 12 hours unfortunately, although it
  does survive an admin shell restart.
* `make all-scripts`: Generate packaging scripts (`build.bash` and
  `install.bash` in `build/{lang,shared}`) from YAML configuration.
* `make system`: Compile setuid binary used for spinning up and
  tearing down user containers. This is needed early because we use
  real containers in the test suite.

### Building Depgraph artifacts

First let's go through each of the Depgraph-enabled artifacts above.
For each one, there's:

* a way to build it locally
* a way to publish the local version to a remote registry
* a way to download the remote version locally

#### Docker images

Generally you build a Docker image named `image:foobar` using `make
image I=foobar`, you publish it with `make push I=foobar`, and you
download it with `make pull I=foobar`. Pass `NC=1` to `make image` to
disable Docker cache (although this is fairly rarely useful, and in
general for this to work with Depgraph we need a more sophisticated
mechanism).

There are one or two exceptions to this, unfortunately:

* For language images (`image:lang-foobar`), it's `make image I=lang
  L=foobar`.
* For `image:ubuntu`, you likely don't want to "build" it yourself
  (meaning take the latest `ubuntu:rolling` from Docker Hub). You can
  synchronize with upstream Riju using `make sync-ubuntu`.

For any Docker image `image:foobar`, you can jump into a shell using
`make shell I=foobar`. This has some optional arguments:

* `E=1`: Expose Riju ports outside the container. Most likely used as
  `make shell I=runtime E=1` inside the admin shell.
* `EE=1`: Same as `E=1`, but expose ports on `0.0.0.0` outside the
  container. This is helpful if you're running on the dev server and
  want to be able to access the development version of Riju in your
  browser.
* `CMD="make something"`: Instead of launching an interactive Bash
  shell inside the container, run the specified shell command (using
  Bash) and exit.

Riju source code and build directories are typically cross-mounted at
`/src` inside all non-user containers, so there is generally no need
to rebuild and/or restart containers when making changes.

Note that all of this section applies also to the `admin` and `ci`
images, which are not otherwise involved with Depgraph (and are based
directly on upstream `ubuntu:rolling`).

Note also that `admin` uses `--network=host` and maps a number of
directories such as `~/.ssh` and `~/.aws`, plus the Docker socket,
inside the container, so you can treat an admin shell more or less the
same as your external development environment.

#### Debian packages

Build a language package using `make pkg T=lang L=xyz` (where there
exists `langs/xyz.yaml`). Build a shared dependency package using
`make pkg T=shared L=pqr` (where there exists `shared/pqr.yaml`).

This has to be done in the packaging image, and will abort otherwise.
So, for short, `make shell I=packaging CMD="make pkg T=lang L=xyz"`.

To debug package installation, you can jump into a persistent
packaging shell (`make shell I=packaging`) and break down the process
into three steps:

* `make pkg-clean T=lang L=xyz`: Delete and recreate packaging
  directories for `xyz`.
* `make pkg-build T=lang L=xyz`: Run the packaging script. Substitute
  `build` for `debug` to instead start a shell in the environment of
  the packaging script, where you can operate manually.
* `make pkg-deb T=lang L=xyz`: Compress the results of the packaging
  script into a file `build/lang/xyz/riju-lang-xyz.deb`. You can pass
  `Z=(gzip|xz)` to enable compression, which is disabled by default to
  save on time during development. Otherwise, packages are
  automatically recompressed before registry upload time.

Uploading a package to the registry is `make upload T=lang L=xyz`, and
download is `make download T=lang L=xyz`.

#### Tests

You can run tests for a specific language (inside the `runtime` image
only, otherwise it will abort) using `make test L=xyz`. `L` can also
be a comma-separated list of languages. You can additionally (or
instead) filter by test type, e.g. `make test L=python T=lsp`.
Uploading and downloading test hashes is only implemented at the
Depgraph layer.

#### Final deployment

* `make deploy-config`: Build the deployment configuration JSON that
  will be pushed to S3.
* `make deploy-latest`: Push it to S3.
* `make deploy`: Combination of the above.

### Application build

We have three compiled parts of Riju:

* Frontend assets (compiled with Webpack)
* Setuid binary used for privilege deescalation (compiled with LLVM)
* Supervisor binary used on deployed images (compiled with Go tooling)

For development:

* `make frontend-dev` (compile frontend assets, auto recompile on
  change)
* `make system-dev` (compile setuid binary, auto recompile on change)
* `make supervisor-dev` (compile supervisor binary, auto recompile on
  change)
* `make server-dev` (run server, auto restart on change)
* `make dev` (all four of the above)

For production:

* `make frontend` (compile frontend assets)
* `make system` (compile setuid binary)
* `make supervisor` (compile supervisor binary)
* `make build` (all three of the above)
* `make server` (run server)

### Miscellaneous utilities

* `make sandbox`: Bash shell emulating a user session at the command
  line, with many useful functions in scope for executing various
  commands from the language configuration YAML.
* `make lsp`: LSP REPL. This is not working currently as it needs to
  be updated for the new build system that uses per-language images.
* `make dockerignore`: Update `.dockerignore` from `.gitignore`.
* `make tmux`: Start a tmux session conveniently with variables from
  `.env`.
* `make env`: Load in `.env` environment variables in a subshell. You
  can also do this for your current session (though it won't affect
  new tmux panes) with `set -a; . .env; set +a`.

### Infrastructure

There are wrappers for `packer` and `terraform` in the repository
`bin` that deal with some niceties automatically. Run `make packer` to
do an AMI build, and use `terraform` commands from any directory.