[FR] Default resource requirement/limits for the KFP UI and system services

[Bobgy]

UPDATE: at the end, we decided to only add resource requirements, see discussion in #5236 (comment)

It's desirable to provide a set of default resource requirement & limits for KFP UI & system services, to make sure their QoS is Guaranteed by default.

https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/
I'm not exactly sure what will be reasonable, because if they are set too low, the services may stop operating when there are workloads reaching a limit.
But setting them to make QoS Guaranteed is also important, because otherwise when there are many other workloads, KFP UI & API services may be evicted because default QoS is BestEffort and BestEffort Pods are the first to be evicted by Kubernetes when it runs out of resources.

[Bobgy]

Got some help from Sid Palas:

A couple of example request settings:
ml-pipeline (api server)
        requests:
          cpu: '2'
          memory: 4Gi
ml-pipeline-ui
        requests:
          cpu: 10m
          memory: 70Mi
workflow-controller (argo)
        requests:
          cpu: 200m
          memory: 3Gi
minio
          requests:
            cpu: 20m
            memory: 25Mi
persistent-agent
          requests:
            cpu: 120m
            memory: 2Gi

see thread https://kubeflow.slack.com/archives/CE10KS9M4/p1613655024114300

[NikeNano]

According to the argo documentation the memory and cpu usage for argo scales linearly with the nbr of workflows, see. So users will probably have to adjust this according if they are running heavier workloads or like to reduce costs.

I would be happy to update this!

/assign

[Bobgy]

thank you @NikeNano

[Bobgy]

/reopen
after #5273, we need to apply default resource requirement to argo pods again

[google-oss-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen
after #5273, we need to apply default resource requirement to argo pods again

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.