Bare-metal Buildkite agents capable of running Docker and VMs

[emidoots]

Quoting the Distribution roadmap:

Today we cannot release of Sourcegraph, run e2e tests, or perform Docker image tests in an automated fashion because our CI infrastructure does not support running Docker containers (or VMs/Vagrant) in a reliable way. Today we have a side-car DIND container in our CI pipeline but it is flaky, unreliable, and a regular source of issues which has led to us removing automated testing (see Automatic e2e testing).

I would like to see us have some sort of CI infrastructure that is capable of running:

• Docker containers
• Docker Compose
• Vagrant

I propose we create bare-metal Buildkite agents to solve this issue, provisioned through Terraform. This would then allow us to do a number of things:

• Have confidence that our e2e test failures are NOT due to the sidecar dind node, and therefor run e2e tests in CI once again.
• Perform automatic Docker image testing
• Automate the Docker Compose release process (which requires e.g. running docker-compose up -d and doing basic health checks on specifically a Linux host OS)

TODO

• Build some tooling so make the process of building the image repeatable with something simple like make build that can be run locally.

[emidoots]

There appears to be some prior research / context in https://github.com/sourcegraph/sourcegraph/issues/6887 that may be valuable.

[pecigonzalo]

Maybe we could use https://github.com/firecracker-microvm/firecracker in a similar way as RFC-199

[davejrt]

Based on the requirements above, I'm looking at taking some inspiration from the buildkite elastic ci.

[davejrt]

Update: image is now usable in gcp under sourcegraph-dev/buildkite-agent. Still outstanding is to automate this or at least make the image producable in an automated fashion

[marekweb]

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.19 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

[davejrt]

Baremetal buildkite agents are now running in GCP in the sourcegraph-ci project.

Image is built here and deployed with terraform here

sourceraph-deploy-docker uses this infrastructure via sourcegraph/deploy-sourcegraph-docker#141