add support for mpi4py

[basnijholt]

No description provided.

[dalcinl]

        ex.bootup() # wait until all workers are up and running
        return executor._pool.size  # not public API!

[basnijholt]

That's better, does ex._pool.size work before all the workers are up and running? Because Adaptive can handle scaling of the pool size.

[jbweston]

does this "just work"? Isn't there some extra bits needed for launching workers? We should probably document this somehow...

[basnijholt]

@jbweston I'll add some more details to the docs later.

In a nutshell, it works when calling your Python script like:

mpiexec -n 16 python -m mpi4py.futures run_learner.py

or in a SLURM job

srun -n $SLURM_NTASKS --mpi=pmi2 python -m mpi4py.futures run_learner.py

[dalcinl]

In a nutshell, it works when calling your Python script like:

mpiexec -n 16 python -m mpi4py.futures run_learner.py

In your desktop or laptop, it can also work like this:

export MPI4PY_MAX_WORKERS=15
mpiexec -n 1 python run_learner.py

Or you can pass max_workers=15 programmatically when creating the executor instance.

In this case, the 15 workers will be MPI-spawned at runtime. I consider this the preferred way of using mpi4py.futures, unfortunately it is not always supported by batch systems or vendor MPI implementations in supercomputers. If your code uses no more than one executor instance at a time, then both methods are practically equivalent. The difference is when you create a second executor, in the first form, all executors share all the workers, in the second form (spawn), each executor has its own set of workers. BTW, this is explained in the docs! Folks, for once in my life that I care to write docs, and you do not RTFM? Come on! 😉

[dalcinl]

does this "just work"?

You are hurting my feelings 😉

[basnijholt]

@dalcinl thanks for the comments!

I've updated the explanation for the docs.

@akhmerov or @jbweston merge if you are happy with it.

[jbweston]

LGTM