remote-run for toolbox containers

[piax93]

Forgive me PaaSTA gods for the sins I'm about to commit.

Hi, I know this is a bit... hacky, but it's the best way I could think to reuse the same implementation. If I had to describe the change with one image, it'd be something like:

So, to explain a bit, the plan which was discussed internally as result of the CEP is to re-use the same remote-run logic to launch "toolbox container" which would give access to selected CLI tools to developers. We don't want these images to be deployed like mock services, as that would be scope creep of what's supposed to be in soa-configs, and we only require some small set of startup settings anyways. So the plan would be to define those settings as part of the paasta system config, and generate a deployment configuration on the fly from them, which is exactly what this change does.

I tested the generation of a K8s Job spec from this assembled configuration, and it looks reasonable to me. The one thing will lose meaning is the "git_sha" field, as that's strictly related to how paasta actually marks deployments, but I don't think that would create problems.

I had to just make a couple of adjustments to the main deployment logic:

• Added an option to set "privileged" in the pods' security context, as we need that to run sshd with pamd.
• Added an option to override the container registry base URL, as the sandbox images would live outside the PaaSTA registry, since they are not service images (I'm honestly flexible on this, but it seemed a good separation of concerns).

The final notable difference is that, since we set up sshd in these pods, we don't need to kubectl exec into them, but just grab the pod IP as part of the status polling response, and ssh directly into them. This choice was done since it makes it way easier to audit user actions inside the containers.