Make critical jobs Guaranteed Pod QOS: ci-kubernetes-e2e-device-plugin-gpu

[spiffxp]

What should be cleaned up or changed:

This is part of #18530

The following jobs should be Guaranteed Pod QOS, meaning they should have CPU and memory resource limits, and matching resource requests:

• ci-kubernetes-e2e-device-plugin-gpu
• ci-kubernetes-e2e-device-plugin-gpu-beta
• ci-kubernetes-e2e-device-plugin-gpu-stable1
• ci-kubernetes-e2e-device-plugin-gpu-stable2
• ci-kubernetes-e2e-device-plugin-gpu-stable3

These jobs run on (google.com only) k8s-prow-build, so @spiffxp has provided the following guess:

• suggest 1 cpu, 3Gi mem (like the other "low effort" e2e jobs in generated.yaml)

General steps to follow:

• update the job definitions in /config/jobs to have matching CPU and memory limits and requests
  • e.g. (ci-kubernetes-e2e-gci-gce has a resources: field with matching entries)
• open a pull request, include a link to this issue and cc @kubernetes/ci-signal in the description
• keep an eye on the jobs for the next few days, and if they start failing more than usual, open followup pull requests to raise resources
  • can look at recent prow.k8s.io runs (e.g. https://prow.k8s.io/?job=ci-kubernetes-e2e-gci-gce)
  • can look at job history (e.g. https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce)
  • can look at testgrid dashboards (e.g. https://testgrid.k8s.io/sig-release-master-blocking#gce-cos-master-default)
• after the job has remained stable for a few days, declare victory

/sig testing
/sig release
/area jobs
/area release-eng

[helenfeng737]

/assign

[spiffxp]

/remove-help
since @ZhiFeng1993 has it

[helenfeng737]

https://prow.k8s.io/?job=ci-kubernetes-e2e-gce-device-plugin-gpu
https://prow.k8s.io/?job=ci-kubernetes-e2e-gce-device-plugin-gpu-beta
https://prow.k8s.io/?job=ci-kubernetes-e2e-gce-device-plugin-gpu-stable1
https://prow.k8s.io/?job=ci-kubernetes-e2e-gce-device-plugin-gpu-stable2
https://prow.k8s.io/?job=ci-kubernetes-e2e-gce-device-plugin-gpu-stable3

https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu-beta
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu-stable1
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu-stable2
https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-device-plugin-gpu-stable3

[spiffxp]

Per #18650 these now run in k8s-infra-prow-build so metrics are visible to k8s-infra-prow-viewers@kubernetes.io

[helenfeng737]

Memory usage dashboard: https://console.cloud.google.com/monitoring/metrics-explorer?project=k8s-infra-prow-build&timeDomain=6h&pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22kubernetes.io%2Fcontainer%2Fmemory%2Fused_bytes%5C%22%20resource.type%3D%5C%22k8s_container%5C%22%20metadata.user_labels.%5C%22prow.k8s.io%2Fjob%5C%22%3Dmonitoring.regex.full_match(%5C%22ci-kubernetes-e2e-gce-device-plugin-gpu.*%5C%22)%22,%22minAlignmentPeriod%22:%223600s%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_MEAN%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22groupByFields%22:%5B%22metadata.user_labels.%5C%22prow.k8s.io%2Fjob%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221w%22%7D%7D

Looks like the memory usages are mostly lower than 2.5Gi. We are using 3Gi here, should be ok.

[helenfeng737]

/close

I think it's safe to close this one. cc. @RobertKielty

[k8s-ci-robot]

@ZhiFeng1993: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

/close

I think it's safe to close this one. cc. @RobertKielty

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

/close
Thanks @ZhiFeng1993

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Thanks @ZhiFeng1993

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.