Write a Dockerfile that can build both entropy and server.

[vitropy]

This commit satisfies the pre-requisite for building container images from the source code herein inside a CI/CD pipeline (or, really, for any other purpose we might need containers for), as described in #423. The long and short of it is that we can use this to have container images built of the current commit, or any commit that we want.

Since the code here depends on access to currently-private GitHub repositories, we need to have set a GitHub Personal Access Token (PAT) that can access the entropyxyz/synedrion and entropyxyz/constraints repositories. This is expected to be part of the entropy-workstation-setup
prerequisites, so do that first if you haven't yet.

The canonical place for such a token on a developer machine is ~/.entropy.auth.sh, so developers can create containers of the entropy binary in a manner like so from their local workstation:

docker buildx build \
    --secret id=credentials,src=$HOME/.entropy.auth.sh \
    --tag entropy .

This assumes a line like the following exists in ~/.entropy.auth.sh:

export GITHUB_TOKEN="ghp_xxxxxxxxxx" # Obviously, put a real PAT here.

You can also build arbitrary packages in the same environment by setting the PACKAGE build argument:

docker buildx build --build-arg PACKAGE=server \
    --secret id=credentials,src=$HOME/.entropy.auth.sh \
    --tag server .

Additional build arguments provide some flexibility for the build environment: RUST_VERSION, DEBIAN_CODENAME, and ALPINE_VERSION.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
entropy-core	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 16, 2023 9:39pm

[HCastano]

Nice work!

A few thoughts/questions:

Can we use this as the canonical way to build release binaries? I worry that the build steps/dependencies will differ from what was introduced in #429 which might make tracking down bugs harder. What if the GHA workflow were to use this to build and publish binaries instead?

What are the tradeoffs between using PATs and SSH? I ask because my SSH setup for GH is already good (and hardware backed) to go and I'd rather not have an extra piece of sensitive data lying around. Would it be possible to use SSH instead of PATs here?

Otherwise everything looks reasonable. I'll approve once I actually build an image locally later today

[HCastano]

Can you add some of the points about the access token assumptions, etc. as comments here?

Will help anybody in the future that runs into issues

[vitropy]

Can we use this as the canonical way to build release binaries? I worry that the build steps/dependencies will differ from what was introduced in #429 which might make tracking down bugs harder. What if the GHA workflow were to use this to build and publish binaries instead?

That is the plan.

What are the tradeoffs between using PATs and SSH? I ask because my SSH setup for GH is already good (and hardware backed) to go and I'd rather not have an extra piece of sensitive data lying around. Would it be possible to use SSH instead of PATs here?

It would be possible to use SSH but then we (and any consumer) has to manage SSH keys. Also, Cargo's implementation of SSH is extremely frustrating; it doesn't always pick up SSH configuration options the way OpenSSH does. There's a fall back possible (net.git-fetch-with-cli, IIRC), but even that flakes sometimes in my testing. SSH deploy keys are used currently in the Vercel builds, for example, as well as the GitHub Actions Workflow, because these already have access to the entropy-core codebase via GitHub/Vercel automations themselves. In contrast, this Dockerfile is intended, as you posited above, to be the canonical method for building our binaries anywhere and for as long as there are dependencies in our binaries that do not point at open source repositories (i.e., synederion and constraints, as of this writing), there are authentication requirements for building the code that need to be addressed. But since this is the canonical way to build the binaries, that means I was writing it with the intention of making it easy for anyone at all to grab the source without needing an SSH private key whose public key was added as a new Deploy Key. Remember also that we manage SSH Deploy Keys as IaC in our Terraform configuration for the GitHub Organization, which means managing another Deploy Key would introduce a dependency on our Terraform GitOps runs, which are themselves not quite ready to run unattended.

Moreover, the PAT here can be what GitHub calls a "fine-grained PAT" instead of a "classic" PAT, which can be scoped not only to permissions of your user account but to specific repositories. This makes the fine-grained PAT option much easier to work with because it is A) symmetrical, B) no more permissive than a deploy key, and C) does not rely on our other infrastructure requirements.

[HCastano]

I wasn't able to get to building this today, so I'll do this tomorrow.

@vitropy Do you need a yolo-approve to unblock you here?

[vitropy]

Do you need a yolo-approve to unblock you here?

@HCastano Yes, pls! 🙏🏻💖

[ameba23]

I had a go at trying this out.

The buildx command above works fine for me with a 'classic' PAT token, but with a fine-grained one i couldn't figure out the right permissions to give it.

Using docker compose up -d i get:
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /home/turnip/software/entropy/entropy-core/.local
What is .local for?

[vitropy]

I had a go at trying this out.

The buildx command above works fine for me with a 'classic' PAT token, but with a fine-grained one i couldn't figure out the right permissions to give it.

It needs Contents permissions for entropy-core and its dependencies, synedrion and constraints. That's all.

Using docker compose up -d i get: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /home/turnip/software/entropy/entropy-core/.local What is .local for?

I think you're looking at the wrong file. You're using a two-year old version of the docker-compose.yml file that Jesse wrote. If you're asking about the "local devnet" I've been working on, it's in my feature branch (vi/docker-compose-for-local-devnet), and the file is docker-compose.yaml.

[vitropy]

@ameba23 Following up from my previous comment, you may also want to check out the newer Dockerfile I made today that can work with any of:

• GitHub PAT (classic)
• GitHub PAT (fine-grained)
• Forwarded SSH agent (i.e., your own SSH credentials, super useful for making things easier on y'all during dev)

Have a look at the 3b453fd commit.