Develop a prow migration plan

[spiffxp]

Break out into issues when we've gotten consensus

Current sketch is:

• decide whether we want a build cluster, and/or how many we we want
• allow prow.k8s.io to schedule jobs on our build cluster(s)
• setup a prow instance (in aaa?)
• figure out how we want to migrate/cut-over github traffic

Some concerns include:

• jobs that run in same cluster as prow.k8s.io to take advantage of the ghproxy instance that lives there
• using a new github bot user/token so as not to conflict with k8s-ci-robot while prow.k8s.io continues to use it
• setting up boskos in our build cluster(s)
• setting up greenhouse in our build cluster(s)
• setting up projects for boskos to manage
• migration of google-owned buckets like kubernetes-jenkins
• on-call

[spiffxp]

/wg k8s-infra
/sig testing
/area cluster-infra
/assign

[spiffxp]

I prototyped a single build cluster here #806

I have it hooked up to my personal prow instance here: https://prow.bashfire.dev

[spiffxp]

Trying to work through what the plan looks like, it's unlikely we'll get to discussing the actual "migrate prow.k8s.io" part in much detail this week.

https://docs.google.com/document/d/1g7RBDhbTAH1VBMl2kJZR71OfzM6cJF277zbysd2AoCQ/edit#

[spiffxp]

Discussed during today's meeting, came away with:

• We'll do two build clusters, each in their own project (trusted, untrusted)
• Untrusted build cluster will use same machine type as used by prow.k8s.io's existing build cluster (n1-highmem-8)
• Will use regional clusters
• Will start with e2e project pool that is not sig/job-specific
  • Adjust when costs get to unknown/unexplained size
  • Consider per-sig project pools
• Naming Is Hard:
  • k8s-infra-prow-build - untrusted build cluster (and project, serviceaccount, etc)
  • k8s-infra-prow-build-trusted - trusted build cluster (and project, serviceaccount, etc)
  • k8s-infra-e2e-{suffix} - e2e projects intended to be pinned to a job
  • k8s-infra-e2e-boskos-{nnn} - e2e projects intended to be managed by boskos
• What I'm aiming for in two weeks:
  • clusters hooked up to prow.k8s.io
  • confirmed migration of:
    • ci-kubernetes-node-kubelet
    • ci-kubernetes-e2e-gci-gce
    • (an image pushing job?)
  • dev/test cycle for migrating jobs

[spiffxp]

#830 implements all of the above except dev/test cycle

[spiffxp]

Opened up some more issues to represent followup work:

• - Migrate release-master-blocking jobs to k8s-infra-prow-build #841 - migrate release-master-blocking jobs to prow-build cluster
• - Setup greenhouse in k8s-infra-prow-build cluster #842 - setup greenhouse
• - Setup ghproxy in k8s-infra-prow-build-trusted #843 - setup ghproxy
• - Develop better ACLs for prow build clusters and e2e projects #844 - better ACLs to empower more people
• - Setup postsubmit jobs to deploy prow build cluster resources #845 - postsubmit jobs to deploy cluster resources
• - Migrate jobs away from gs://kubernetes-release-dev #846 - migrate jobs away from gs://kubernetes-release-dev

[MHBauer]

Learning about sig-node ci, can I help with or shadow along changes to ci-kubernetes-node-kubelet?

[spiffxp]

@MHBauer add yourself to the k8s-infra-prow-viewers@kubernetes.io group in groups/groups.yaml and let me know if you can see https://console.cloud.google.com/monitoring/dashboards?project=k8s-infra-prow-build

[helenfeng737]

I can help with the migration work too if needed. Already added myself in k8s-infra-prow-viewers

[spiffxp]

About ready to enumerate:

• setting up a staging prow instance to experiment with running in community infra
• how to shift traffic from prow.k8s.io

[spiffxp]

/assign @ameukam
since also working on staging prow (ref: #1394)

[spiffxp]

/priority important-longterm
I don't know how much time I'll have to work on completing a plan by end of v1.21

[spiffxp]

The proposal linked in this issue likely needs to be refreshed, but dropping a few thoughts before I forget.

Blockers to migrating all jobs over to a community-run build cluster include:

• setting resource requests/limits on all jobs to properly trigger autoscaling
• using community-owned projects or granting privileges to the community-owned build cluster and boskos to manage google.com projects
  • manually pinned projects (e.g. Remove custom projects from SIG Node test manifests test-infra#23822, the list from Umbrella issue: migrate away from google.com gcp projects #1469)
  • gke projects (Decommission gke-project-pool test-infra#23778)
• more budget than we have capacity for based on our current spend w/ scalability jobs migrated over

Blockers to shutting down prow.k8s.io and then standing it all up over in kubernetes-public include:

• all of the existing kubernetes/test-infra tooling / manifests assume deployment to its own cluster vs. a shared cluster
• allowing a community-owned prow to trigger builds in the google.com-owned "default" build cluster (if we fail to migrate all jobs over
• allowing a community-owned prow to write/update gs://k8s-testgrid
• allowing a community-owned prow to write/update gs://kubernetes-jenkins
• allowing a community-owned prow to write to the google.com-owned k8s-prow project
• prow.k8s.io currently runs jobs for more than the kubernetes community (Migrate non-Kubernetes repos off of prow.k8s.io test-infra#12863)
• (not sure what else I'm missing here...)

[ameukam]

/milestone v1.24

[ameukam]

/milestone v1.30
/lifecycle clear

Rebooted this effort. See: https://docs.google.com/document/d/1erBhuCwY26d0UfPbzt8lEj6bYT2hOUKzc2j36YHVqfM/edit?usp=sharing

[ameukam]

/milestone v1.31

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

We have a timeline for this and an overall plan to do the migration.
Technical requirements need to be discussed.

[ameukam]

We have a migration plan established: https://docs.google.com/document/d/1erBhuCwY26d0UfPbzt8lEj6bYT2hOUKzc2j36YHVqfM/edit

Execution is planned: https://groups.google.com/a/kubernetes.io/g/dev/c/qzNYpcN5la4

cc @BenTheElder @michelle192837

/close

[k8s-ci-robot]

@ameukam: Closing this issue.

In response to this:

We have a migration plan established: https://docs.google.com/document/d/1erBhuCwY26d0UfPbzt8lEj6bYT2hOUKzc2j36YHVqfM/edit

Execution is planned: https://groups.google.com/a/kubernetes.io/g/dev/c/qzNYpcN5la4

cc @BenTheElder @michelle192837

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.