🌱 Improve release staging build speed

[cahillsf]

What this PR does / why we need it:
Last release cut (slack thread for ref) we were blocked pretty early on in the process waiting for the post-cluster-api-push-images job to complete. We cut two patch releases (1.4.6 and 1.5.1) with each taking ~28 minutes to complete.

This PR improves the speed of the release process by:

• splitting the original make command into 3 sub-makes to enable parallelization within the cloudbuild job
• passing -j 8 to the make command to parallelize each sub-make and -O to unify the logging output for each job

The results of my local testing have been promising, with the most recent job with the current changes completing in under 6 minutes: https://a.cl.ly/5zubWldx

Here is some historical testing data from my sandbox:

• builds with the 3 make targets split into 3 separate targets (3 steps defined in the cloudbuild) with varying -j option vals and machine sizes: https://a.cl.ly/6quw9NJX
• local run overview from the original config (the job ran in 26 minutes which is consistent with the results from the last patch release cut): https://a.cl.ly/4guRAqQK

changes based on the conversation and work in this reverted PR: #9392

/area release

[cahillsf]

cc @furkatgofurov7

[killianmuldoon]

I'm happy to get this merged as an experiment - but as with the previous version we should be ready to roll back quickly if needed.

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: killianmuldoon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [killianmuldoon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[CecileRobertMichon]

am I correct to understand that docker-push-all and docker-build-all won't and can't run in parallel (because push depends on build)? if so which parts are parallelized by using the -j option? Is it the docker build commands themselves?

[cahillsf]

am I correct to understand that docker-push-all and docker-build-all won't and can't run in parallel (because push depends on build)?

yes that's true. in one iteration of the previous PR i got around this with the suggestion of mdbooth by defining "docker-build-% as a dependency of docker-push-%". but this also introduced some complexity in the push target that we decided was not worth the tradeoff

if so which parts are parallelized by using the -j option? Is it the docker build commands themselves?

yep exactly, the default is to run one job at a time, by passing -j 8 each sub-make (e.g. docker-build-all, docker-push-all) make will run 8 jobs in parallel

[CecileRobertMichon]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 929d2b0e08e5fb25b720878fe2307ad2ffc12635

[cahillsf]

looks like the first job run with the changes was a success, completed in less time than the average of recent runs but not a huge improvement.

there is still the variable queue time within Cloudbuild that i assume is leading to some variation in the build times, will be interesting to see more data in the subsequent jobs

[cahillsf]

taking a sample of the last 15 runs (as of right now) since the change, the jobs have been completing in ~21 minutes. Looking at the 25 runs prior to that, the average was ~28 mins. decent improvement IMO, going to cherry-pick to the release branches so we have this for the next cycle

/cherry-pick release-1.4

[k8s-infra-cherrypick-robot]

@cahillsf: new pull request created: #9554

In response to this:

taking a sample of the last 15 runs (as of right now) since the change, the jobs have been completing in ~21 minutes. Looking at the 25 runs prior to that, the average was ~28 mins. decent improvement IMO, going to cherry-pick to the release branches so we have this for the next cycle

/cherry-pick release-1.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[cahillsf]

/cherry-pick release-1.5

[k8s-infra-cherrypick-robot]

@cahillsf: new pull request created: #9555

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.