Proper Locking

[JBlaschke]

Following our discussion in the HPC call, there is a need for Julia to manage locks when calling into MPI. The thinking so far is:

• Currently MPI.jl defaults to THREAD_SERIALIZE, but we expect the user to guarantee serial access into MPI functions:
  • https://stackoverflow.com/a/6987529 -- calls locks "extremely necessary"
  • https://www.osti.gov/servlets/purl/1346519
• The current solution is problematic because we're using c-locks to lock out Julia threads. The better solution is to manage locks for Julia threads from Julia. This will make things easier for things like: Cooperative_waitall / cooperative_waitany

Open questions:

• Look at the MPI standard: Should lock be on communicator level? Or for MPI in its entirety?

Tagging: @vchuravy @eschnett @simonbyrne