Add E2E tests

[terrytangyuan]

Part of #217.

• E2E tests should be run on presubmit, postsubmit, and periodically
• Tests should cover deployment (kustomize packages) as well as the application functionality

[NikeNano]

I am looking into this, but may not be super fast. Need to catch up on prow and how the test infrastructure for Kubeflow works.

Do I understand it correct that prow is not used atm, since the prow config is empty @terrytangyuan ?

[NikeNano]

/assign

[terrytangyuan]

Thanks! Yes that’s correct.

[NikeNano]

I have played around with the test infra a bite. Do we have any thought about how we like the e2e test to look like? Should we build it based upon one of the examples? If so I suggest that we use the v1alpha2/tensorflow-benchmark.

[terrytangyuan]

Yes that would be a good example to use for end2end test.

[terrytangyuan]

Actually v1 controller is the one we want to graduate so use this example instead: https://github.com/kubeflow/mpi-operator/blob/master/examples/v1/tensorflow-benchmarks.yaml

[naveensrinivasan]

@terrytangyuan How about doing e2e using something like k3d or kind? This will give confidence for someone to run locally. Similar to this kubeflow/kubeflow#5400

[terrytangyuan]

See #296 for an alternative approach to leverage AWS's test infra.

[terrytangyuan]

cc @kubeflow/wg-training-leads @PatrickXYS

[PatrickXYS]

FYI, you can check migration doc here https://github.com/kubeflow/testing/blob/master/aws/Doc/onboarding.md

[naveensrinivasan]

I agree that the end goal is to use the real infrastructure.

As a contributor I would like to run functional test prior submitting the PR. Also I would like to run partial set of tests based on the changes.

Can you explain what is the disadvantages of using something in memory and also have an option to use the actual infrastructure based on a flag?

This gives significant advantages of having a short dev/test cycle.

[naveensrinivasan]

FYI, you can check migration doc here https://github.com/kubeflow/testing/blob/master/aws/Doc/onboarding.md

@PatrickXYS How do I get access to the repo aws-kf-ci-bot?

I would like to start working on the e2e tests.

[PatrickXYS]

@PatrickXYS How do I get access to the repo aws-kf-ci-bot?

You won't, you should follow the doc to proceed

[alculquicondor]

I started to look into this (along with integration tests). Did anyone make any progress?
I'm thinking of using this package https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest

[alculquicondor]

With #399 and #403 merged, I think we should be good to close this.

@terrytangyuan wdyt?

[terrytangyuan]

Sure. Thank you!