Always build all binaries and migrations, simplify deployment

[sethvargo]

In preparation for me documenting this, I wanted to do a refactor of our build/deploy/migrate/promote lifecycle. This PR:

• Uses a cache of the /go/pkg directory to make builds beyond the first much faster
• Updates the Terraform to create a bucket to store said cache (and permissions)
• Moves builds to the CI system and copies compiled binaries into the container
• Tags images based on tag-ref instead of user-supplied tag
• Compiles migrations into a container image, so migrations are now tagged and versioned with the deployment
• Gracefully terminates the Cloud SQL Proxy after running migrations! I believe we were slowly leaking connections because the container just kills
• Injects build ID and build tag into the logs
• Injects build ID and build tag into the HTML as a meta tag (it's available as a template global variable, so we can debate putting it somewhere else)
• Codifies a lot of the local toil in scripts where possible
• Makes it easier for someone to build off of a fork (we previously hardcoded a lot of paths)

Fixes GH-411

Release Note

Improve deployment scripts

[google-oss-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sethvargo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sethvargo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sethvargo]

/assign @mikehelmick
/cc @icco

[mikehelmick]

is it possible to also get the commit sha for when a build is done from head

[jeremyfaller]

Does go tool buildid work for your use case???

[sethvargo]

git describe --dirty (which I'm using) does what I think you want, some examples:

v0.1.1
v0.1.1-ab28ced
v0.1.1-ab28ced-dirty

[sethvargo]

@jeremyfaller we explicitly want the Cloud Build Build ID here (it's a UUID).

[icco]

This looks reasonable.

I personally prefer to be more explicit (by just having a Dockerfile per service), at the cost of speed, so that we could just setup cloud build to automatically build every commit, and then just deploy and promote when we want. But given this is optimizing for the local workflow, it seem solid.

I'll let Mike give the final review. Also note you have a merge commit as the e2e service now exists.

[icco]

Is there any way to keep this file? If we have this setup, then we'd at least have continuous builds. Maybe we could simplify it to just call the build script?

[sethvargo]

We can point Cloud Build at the build.yaml file (which was my plan here).

[icco]

Don't you not want to overwrite this?

[sethvargo]

It's actually not possible to overwrite them since we've moved this into the middleware. This is an artifact from whenever they were lazy-added, but now we force-add these attributes very early in the middleware.

[icco]

Remove the version?

[sethvargo]

Eh, two things:

1. I've been bitten by subtle changes in gcloud and would prefer to make deploys as reproducible as possible by reducing variables.

2. Cloud Build is going to cache the container image. So "latest" isn't actually latest anyway.

[icco]

Fair enough. I always prefer latest if possible because folks always forget to update, and the larger the diff on upgrade, the higher the chance of bug.

[sethvargo]

I personally prefer to be more explicit (by just having a Dockerfile per service), at the cost of speed, so that we could just setup cloud build to automatically build every commit, and then just deploy and promote when we want. But given this is optimizing for the local workflow, it seem solid.

@icco I don't think this precludes or eliminates that possibility. In fact, I want to set this up to auto-build on each commit to HEAD, then we just deploy, maybe migrate, and promote.

[icco]

I personally prefer to be more explicit (by just having a Dockerfile per service), at the cost of speed, so that we could just setup cloud build to automatically build every commit, and then just deploy and promote when we want. But given this is optimizing for the local workflow, it seem solid.

@icco I don't think this precludes or eliminates that possibility. In fact, I want to set this up to auto-build on each commit to HEAD, then we just deploy, maybe migrate, and promote.

I talked with Seth and reread and I don't think I truly understood what this is doing. The details that was missing for me is that /workspace is shared between all cloud builds, and that we are going to point cloud build at this build.yaml.

[icco]

/lgtm
/hold - in case Mike or Jeremy have any other comments

[jeremyfaller]

No, I'm all good. If you want a LG, LMK.