Use init containers to preflight-check image pulls

[DrJosh9000]

What

Make image pull failures more obvious:

• fail the job
• with an error message that identifies the image that failed to pull

Why

Previously, if an image failed to pull (within a timeout after the pod starts), the imagePullBackOffWatcher would use the GraphQL API to cancel the job. If the agent container had started, it could detect the mismatch between connected and expected number of containers and try to log something that would warn the user about ImagePullBackOff.

Cancelling the job looks weird, because that's normally an action performed by a human, and the agent's last-gasp log message is both easily ignored and hedges its bets (it could be ImagePullBackOff, but it might not be, ya know?)

The ability to directly acquire and fail a job in BK changes the set of possibilities. If we can detect image pull failures before the agent container runs in the pod, then we can directly fail the job.

How

The scheduler now includes an init container for each distinct image value in the podSpec.
Each new init container is called imagepullcheck-%d and has an entrypoint command that exits immediately with success. (So, in theory, the container should only fail if the image fails to pull.)
imagePullBackOffWatcher is updated to watch init container statuses. If an imagepullcheck-%d container fails to pull, imagePullBackOffWatcher uses the failJob mechanism from #363 to fail the job in Buildkite.

It also asks Kubernetes to evict the pod specifically to save a bit of time. (Maybe we should detect other failure modes and do this as cleanup.) I'm assuming we couldn't do this before, since it would risk evicting the agent before it could upload its last-gasp log and end the other containers.

TestImagePullBackOffOnSidecar is deleted, because it was used to specifically check that imagePullBackOffWatcher didn't touch the sidecar. Now Kubernetes won't even get to thinking about starting the sidecar, because the corresponding init container that pre-checks its image pull will fail.

The job cancellation mechanism is left in because pre-checking for image pull failure is TOCTOU - if the main containers have an image pull policy of "Always", and there is a transient network failure starting after the init containers succeeded, then they could still fail to pull at that moment. Our only termination mechanism after the agent container has started is still to cancel the job.

This also fixes two bugs I just discovered in #363:

• in order to read the agent token from the secret, the controller service account needs secret read permissions. Duh.
• in order to acquire a job, you need the tags when registering the agen (at least the queue, I think). Passed on CI?

[wolfeidau]

👍🏻 looks great