Always use docker registry server's digest information for pushed spark images

[danielpops]

Problem

Because of containers/podman#14779, docker push has different behavior w.r.t. the local vs remote image digest value when run in privileged vs unprivileged mode. This was causing paasta spark run commands using --build to fail to allocate an executor node in unprivileged container mode because it was referring to an image digest that paasta couldn't find because it only exists on the devbox where paasta spark run is called.

Solution

Parse the message from the remote docker registry when running docker push during paasta spark run --build so that we always use the server version of the image digest as the source of truth.

Testing Done

Ran paasta spark run manually for both --pull and --build with this patch and confirmed the executor can be allocated and the jobs can complete.

[piax93]

nice workaround

[piax93]

I guess there are a couple of unit tests that need to be fixed though

[danielpops]

I guess there are a couple of unit tests that need to be fixed though

ah yeah I had some pre-commit shenanigans that was preventing me from running the tests locally, I will fix these ASAP

[piax93]

🚢

[88manpreet]

lgtm. I assume the plan to push this is after Jan 1st, 2024.

[oholiab]

Amazing <3

Let's get this merged ASAP now that we're in the new year so we can start doing more aggressive rollout to spark-heavy machines