Add MPI example and tests

[TommyLike]

For #42

[k82cn]

minAvailable should be 3 to run whole job.

[k82cn]

let us this email: volcano-sh@googlegroups.com

[jeefy]

That's what I call timing: kubernetes-retired/kube-batch#688

I'm worried my review is going to be biased since I just did this PR for Kube-Batch, so keep that in mind. :)

• This assumes that the job will use your container. It might be better to support people bringing their own images, so long as they have MPI support.
  • I feel like each pod having/using SSH is inefficient. My approach was to use kubectl exec as a transport layer, again, assuming that a user just had to bring a container with MPI.
    https://github.com/kubernetes-sigs/kube-batch/pull/688/files#diff-7da03fcb2b6bbf8383b563b9abbb7a30R120
    https://github.com/kubernetes-sigs/kube-batch/pull/688/files#diff-dd2739b2b61ebae852d4626a0d53bedcR33
• While I actually like not requiring a CRD, I feel like this approach leads to poor UX
  • This requires a user to break out the executor from the workers manually. The job definition becomes bloated and redundant because of this.
  • For longer running MPI jobs, it makes sense to wrap them in RBAC and, eventually, NetworkPolicies. This should be done with a CRD and operator logic, and cleanup will be easy since we can set OwnerReferences to that new CRD object
• This code is so much cleaner than mine 😂
• I love that you used kind

I'm the new guy looking at this, and I'm not sure the direction or end-state y'all are looking to achieve, so take these comments as uninformed. :)

[k82cn]

This assumes that the job will use your container.

Nop, user can build container on demand, with open-mpi & ssh support. For kubectl, it try to avoid unnecessary access to api-server; and that'll be great if you can build a 'kubectl' plugin for mpi job to use.

[k82cn]

btw, let's discuss the detail at #42 ; I'd like to get this merged firstly for next step :)