[Umbrella] 1.16 Release Retrospective Action Items

[justaugustus]

Instead of only having the retro AIs in the retro doc, I thought it would be good to have an umbrella issue for these for we can more diligently track them throughout the cycle.

1.15 AIs

• (SIG Scalability / Release Team process improvements #841) Schedule at least 3 2 (ideally 3; once a month) meetings w/ SIG Scalability + SIG Release Chairs + 1.17 RT Leads + CI Signal team throughout the cycle:
  • After the Release Team has been fully staffed (to introduce the new Release Team)
  • Two additional meetings approx. one month following the initial meeting
  • Capture this in the CI Signal and RT Lead handbooks
    section to KEP template
• Clean up SIG Release Testgrid dashboards
  • Move failing jobs to orphaned
    • We also subsequently removed them. See Prune SIG GCP dashboards and orphaned jobs Prune SIG GCP dashboards and orphaned jobs test-infra#14241 (thank you @justaugustus!)
  • Ensure all blocking jobs are alerted on
    • This was put in placed around 1.14 by assign release-blocking job ownership assign release-blocking job ownership test-infra#10870. One of the things to note from that PR is the test TestReleaseBlockingJobsMustHaveTestgridDescriptions (and ones like it) https://github.com/kubernetes/test-infra/blob/master/config/tests/testgrids/config_test.go#L440 which are run as part of pull-test-infra-bazel on all PRs to the master branch of test-infra. That test in particular blocks any job in sig-release-master-informing doesn't have a testgrid-alert-email annotation. (thank you @spiffxp!)
  • ([Tracking] Consolidate and automate creation of sig-release owned testgrid dashboards  test-infra#11977) Ensure relevant blocking/informing jobs are properly forked
  • Move jobs that were consistently flaky from blocking to informing
    • Done by moving flaky gce-cos jobs to -informing dashboards (move flaky gce-cos jobs to informing dashboards test-infra#14572) (thank you @hasheddan!)

1.16 AIs

• (Implement kubernetes job health dashboard test-infra#13879) Investigate dashboard for test-infra health
• (Update Branch Manager handbook after 1.16.0 #803) Is doing the build after 4pm a hard requirement?
• (SIG Scalability / Release Team process improvements #841) Consider a Performance Engineering RT role or SIG Scalability liaison

/assign @justaugustus @claurence @guineveresaenger
/milestone v1.17
/priority important-soon
cc @kubernetes/release-team @kubernetes/release-engineering @kubernetes/sig-release-admins

[justaugustus]

FYI @craiglpeters

[alejandrox1]

Some updates...

For 1.15 "Clean up SIG Release Testgrid dashboards", we have done the following

• "Move failing jobs to orphaned" and we also subsequently removed them. See Prune SIG GCP dashboards and orphaned jobs Prune SIG GCP dashboards and orphaned jobs test-infra#14241 (thank you @justaugustus !)
• "Ensure all blocking jobs are alerted on" this was put in placed around 1.14 by assign release-blocking job ownership assign release-blocking job ownership test-infra#10870 . One of the things to not from that PR is the test TestReleaseBlockingJobsMustHaveTestgridDescriptions (and the ones like it) https://github.com/kubernetes/test-infra/blob/master/config/tests/testgrids/config_test.go#L440 which is run as part of pull-test-infra-bazel on all PRs to the master branch of test-infra. That tests in particular block is any job in sig-release-master-informing dosn't have a testgrid-alert-email annotation. (thank you @spiffxp !)
• "Move jobs that were consistently flaky from blocking to informing" was done by move flaky gce-cos jobs to informing dashboards move flaky gce-cos jobs to informing dashboards test-infra#14572 (thank you @hasheddan !)

[alejandrox1]

quick update on ai from 1.16 "Investigate dashboard for test-infra health" this is being tracked here kubernetes/test-infra#14890

[guineveresaenger]

@justaugustus - do you have updates for us? :)

[justaugustus]

I've linked issues for the AIs that had them, added updates to the sections I had them for, and flagged each of the items that might require an issue for tracking.

Will try to poke at a few more of these before the retro, but in the meantime, let me pass the baton.
/unassign @guineveresaenger @justaugustus
/assign @lachie83
/milestone v1.18

[alejandrox1]

Another update here for 1.16

• "Consider a test-infra freeze, coordinated w/ Code Freeze" https://github.com/kubernetes/kubernetes/issues/86135

[alejandrox1]

/assign @mrbobbytables @onlydole @alejandrox1

[lasomethingsomething]

Hiya, checking in on the status of these items -- any more boxes to check?

[lasomethingsomething]

How might I get editing rights to these umbrella issues so that I can check boxes?

I've also asked in the sig-release channel today if I can consolidate these umbrella retro item issues into a new one (adding a v.17/v.18/v.19/v.etc. note to each item) and close out these old issues. This need was expressed by several 1.19 RT members during prioritisation sessions.

[lasomethingsomething]

This issue's been updated. All unresolved items previously posted here will be collected in a new issue and reviewed with other, more recent, unresolved retro items.