Migration prow.k8s.io to the community infrastructure

[ameukam]

Context:

https://prow.k8s.io is running in the GCP project k8s-prow owned by Google.com and should be migrated the community owned infrastructure.

A document written by @spiffxp give us an overview of the migration plan : https://docs.google.com/document/d/1g7RBDhbTAH1VBMl2kJZR71OfzM6cJF277zbysd2AoCQ/edit#.

Umbrella issue with some indications about the progress of the migration : kubernetes/k8s.io#752.

[ameukam]

I'm working on setup a prow staging instance that will run on aaa cluster : kubernetes/k8s.io#1394

As discussed during sig-testing meeting on 02/23/2021, we agree on run the prow staging instance against the current build clusters using a namespace different than test-prods in the build-clusters.

[BenTheElder]

thanks for working on this!

[ameukam]

Figure out what to do about gs://kubernetes-jenkins (from the migration plan):

In an ideal world, each build cluster would write to its own GCS bucket, and consumption of test results would involve reading from >multiple GCS buckets
It is probable that, today, parts of our test-infra assume everything is in a single bucket (gs://kubernetes-jenkins)
Option A: flip everything over to a CNCF-owned bucket, eg: gs://k8s-infra-notjenkins
Option B: identify everything that assumes only kubernetes-jenkins and update it to support multiple buckets

WDYT ? @alvaroaleman @BenTheElder @cjwagner @spiffxp

[BenTheElder]

I'm not sure what all is trying to read from kubernetes-jenkins, however we have other more problematic buckets to migrate.
(also for context: kubernetes-jenkins is in a google.com project, not kubernetes.io)

see: kubernetes/k8s.io#1526, kubernetes/k8s.io#1569

[alvaroaleman]

Having separate build clusters write to separate GCS buckets is safer in terms of potential blast radius if upload credentials are misused either by a malicious presubmit or accidentally.

Sure, but you can achieve that with different credentials, you do not need different buckets for that.

It is common to use separate build clusters to isolate workloads with different levels of trustworthiness, this is an extension of that to protect the job results in addition to the credentials and environments that the jobs use.

Yeah, clusters can represent trust domains. It is also common to have multiple clusters in the same trust domain to have multiple distinct failure domains (this is the main reason we started to use multiple build clusters for Openshift).

[spiffxp]

Sure, but you can achieve that with different credentials, you do not need different buckets for that.

IAM is way easier to audit if we don't allow per-object ACLs

So "unable delete others' data" becomes "writes to different bucket"

Today we sortof trust that everything writing to kubernetes-jenkins isn't going to misbehave. Even if all of the build clusters have different service accounts, they could still fight it out since they're all pointed at the same bucket, and we haven't done too much with per-object ACLs.

[spiffxp]

Scalability has AFAIK put at least some of what is needed in place to write/read multiple buckets: kubernetes/test-infra#20275

It's not clear to me whether there are any other assumptions lurking out there, but it seems to be working well enough for them?

[spiffxp]

@ameukam I feel like "Option B", if it's possible

• setup a bucket in k8s-infra-prow-build-trusted for that build cluster
• update gcsweb deployment to support displaying that bucket
• setup whatever needs to be setup for kettle to support reading from that bucket (check with @MushuEE)
• setup whatever needs to be setup on prow.k8s.io to support reading from that bucket
• trial some jobs, confirm things still work
  • deck
  • spyglass artifact display
  • spyglass job history
  • spyglass pr history
  • testgrid
• migrate all jobs for that build cluster
• rinse and repeat for k8s-infra-prow-build

But I also agree with @BenTheElder that I'm not sure where "don't write to kubernetes-jenkins" sits in terms of relative priority at the moment. And I'm not sure this is on the critical path to help you setup a staging instance.

[spiffxp]

There is lots more to flesh out here and I hope to be able to work on that with you shortly before or after v1.21 gets released. Thank you for nudging us forward here.