[Testing] postsubmit mkp test failure 2021.3.4

[Bobgy]

First failing mkp test: https://oss-prow.knative.dev/view/gs/oss-prow/logs/kubeflow-pipeline-postsubmit-mkp-e2e-test/1365452157549023232
Root cause seems to be: #5158

[Bobgy]

The failure: https://oss-prow.knative.dev/view/gs/oss-prow/logs/kubeflow-pipeline-postsubmit-mkp-e2e-test/1365452157549023232#1:build-log.txt%3A7816

Step #1 - "verify": 37s Warning FailedScheduling pod/ml-pipeline-55bbc45946-m7s66 0/3 nodes are available: 2 Insufficient memory, 3 Insufficient cpu.

[Bobgy]

It seems that the added resource request make it impossible to schedule ml-pipeline pod in the cluster.

[Bobgy]

The default cluster for marketplace has 3 nodes each with 2 CPUs and 3GB memory allocatable.
So the new requirements set in #5158 seems too large, it marked 4GB memory for ml-pipeline server. I assume it was prepared more for large scale production env.

We need to reduce the request to fit into the default cluster.

[Bobgy]

/cc @NikeNano

[Bobgy]

/assign

I'll fix this

[Bobgy]

Another issue to address is that, mkp test should be triggered on presubmit if a PR touches MKP manifest. Let me add the auto trigger.

[Bobgy]

Some reading on requests & limits: https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-resource-requests-and-limits

[Bobgy]

Some investigation into other OSS projects, argo doesn't provide default for requests/limits for the most part: https://github.com/argoproj/argo-workflows/tree/master/manifests.
There's a doc page to introduce how to improve cost optimization: https://argoproj.github.io/argo-workflows/cost-optimisation/.

[Bobgy]

I think we'll need an operator manual documentation to tell people how to adjust their resource requests, but maybe as a next step.
Adding some requests as default only make it better than before -- without setting it, it's assumed to be 0. Also, when requested number is reached, the Pod is not killed like a limit.

[NikeNano]

I think we'll need an operator manual documentation to tell people how to adjust their resource requests, but maybe as a next step.

I think this sounds good, also if we could provide some ball park figures. I guess however if we set them to high we will request more resources than actually used for most people.

[Bobgy]

Caused by #5148