ict 4483 - add job attach to wait for job execution

[davixcky]

Fixes https://github.com/signadot/signadot/issues/4483

[jmsktm]

Looks mostly good to me. But the it seemed connected / attached for a while after the job was completed before disconnecting with the below message:

./signadot --config ~/.signadot/staging.yaml job submit -f control-backend-e2e.yaml --attach
Job control-backend-e2e-ptc0dxq queued on Job Runner Group: control-e2e

Dashboard page: https://app.signadot.com/testing/jobs/control-backend-e2e-ptc0dxq

Waiting for job execution
Cloning signadot repo
Running backend e2e tests
newman

Latest-Operator-Version

→ latest-operator-version (member user)
  GET https://api.staging.signadot.com/api/v1/vars/latest-operator-version
  200 OK ★ 521ms time ★ 1.13kB↑ 306B↓ size ★ 9↑ 7↓ headers ★ 0 cookies
  ┌ ↓ application/json ★ text ★ json ★ utf8 ★ 94B
...
logs
...

&{%!s(*models.JobsCanceledState=<nil>) %!s(*models.JobsCompletedState=&{0 }) %!s(*models.JobsQueuedState=<nil>) %!s(*models.JobsRunningState=<nil>)}

[jmsktm]

I saw the connection persist beyond job completion with the logs command too.

./signadot --config ~/.signadot/staging.yaml logs --job control-backend-e2e-wmmx22i

Even in this case, the connection stayed intact for a while (maybe 20/30 seconds) after the job completed.

Since this seems to be external to the scope of this PR, I will go ahead and approve it.
This could be directly or indirectly related to https://github.com/signadot/signadot/issues/4484 as well. But let's create a separate ticket for the cli so that it doesn't get forgotten.

Dismissing review based on some comments by Daniel.

[daniel-de-vera]

I think the --attach feature should be more robust (at least if we want to use it in the context of CI). Something that in case of errors it retries based on exp backoff (or luby), and only after some time it fails. Otherwise we may end up generating false negatives (claiming that a job failed when it didn't).

For example, imagine the case when we restart the apiserver (e.g. we perform a new control release), all connected clients will error out, but that doesn't mean the Job failed.

[daniel-de-vera]

LGTM