Implement v2 controller that sets up SSH for communication

[alculquicondor]

Implementation for https://github.com/kubeflow/mpi-operator/blob/master/proposals/scalable-robust-operator.md

• Core Features:
  • Replace kubectl and support roles with ssh and secrets Do inter-pod communication through SSH #375
  • Add support for running as non-root Allow running MPI applications as non-root #383
  • Ensure retries in case of startup delays Allow launcher to start after workers are ready #386
  • Ensure all container image examples support the SSH setup
  • Support Intel MPI Add support for Intel MPI #389
• Production readiness
  • Upgrade k8s dependencies Upgrade v2 dependencies to k8s 1.19 #370
  • Add integration tests Do inter-pod communication through SSH #375
  • Add E2E tests E2E test for v2 controller #399 Add E2E tests for failure, root and Intel #403
  • Add API defaulting and validation Consolidate validation and defaulting logic #376
  • Update API to apiextensions.k8s.io/v1 for support in k8s 1.22 Upgrade to apiextensions.k8s.io/v1 #379
  • Use fully-qualified label names Use fully-qualified label names from common #409
  • Developer documentation Preparing release of v0.3.0 #414
  • End-user documentation
  • Graduate API from v2beta1 to v2 once matured.
• Controller simplifications: delegating pod management to core k8s APIs
  • Replace the plain pod workers with StatefulSet or Indexed Job
  • Replace the plain pod launcher with a Job

[gaocegege]

https://www.kubeflow.org/docs/about/contributing/#joining-the-kubeflow-github-org

Hi, could you please join the kubeflow org? Then we do not need to trigger the CICD for your PR manually.

[alculquicondor]

Sent PR kubeflow/internal-acls#473

Thanks for the suggestion

[alculquicondor]

I verified that images docker.io/kubeflow/mpi-horovod-mnist and docker.io/mpioperator/tensorflow-benchmarks just work with the new controller. Marking that as done.

[Jeffwan]

@alculquicondor Has community discussed tradeoffs about job vs pod for launcher, statefulsets vs plain pods for workers?

[alculquicondor]

Yes for launcher. See the discussion here #386

For workers, it's still open for discussion. We could do Statefulsets, but I think plain pods might be fine for now. We might migrate to Indexed Jobs at some point, but since it's only available in k8s 1.22, it's kind of early to discuss.

[alculquicondor]

I think this is pretty much ready. The last things I would like to do are:

• update the labels to use fully-qualified names. For that, I'm waiting for a release of the common library Cut release-0.4 branch and cut 0.4.0 tag common#153
• Add documentation (is there a website, or should I just do it on readmes)?

[terrytangyuan]

* Add documentation (is there a website, or should I just do it on readmes)?

There's this page https://www.kubeflow.org/docs/components/training/mpi/

[tenzen-y]

Maybe we can introduce Indexed Job to mpi-operator v2 once kubernetes/enhancements#3715 is graduated to beta.

[tenzen-y]

Consider introducing JobSet instead of managing raw pods for the workers: https://github.com/kubernetes-sigs/jobset