updated multi-threading branch

[stevengj]

WIP to rebase @smartalecH's multithreading branch to fix #1627 and fix #228.

Includes some additional thread-safety fixes (to parallelize the DFT updates, and to avoid calling back to Python or other user callbacks from multiple threads).

[stevengj]

The tests don't currently exercise the openmp code (we need to configure with --enable-openmp and set OMP_NUM_THREADS to enable this functionality), but at least they are green, so this hopefully isn't harming anything.

It would be good to do a quick performance benchmark without openmp on this branch, to make sure that @smartalecH's loop rewriting didn't hurt the serial performance.

[stevengj]

We probably want to parallelize the LOOP_OVER_IVECS in dft_chunk::update_dft as well, since that is often a significant fraction of the runtime.

[stevengj]

@smartalecH, should this file be included in make check?

[smartalecH]

No, this was just for my personal debugging. Didn't realize it was added.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.46%. Comparing base (21c94c0) to head (79ffca8).
Report is 396 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1628   +/-   ##
=======================================
  Coverage   73.46%   73.46%           
=======================================
  Files          13       13           
  Lines        4557     4557           
=======================================
  Hits         3348     3348           
  Misses       1209     1209           

[oskooi]

This branch should be rebased after #1737 is merged first.

[smartalecH]

All of the LOOP macros that involve python callbacks do not currently use the multithreading implementation.

There's a nifty swig feature, -threads, that allows us to temporarily release the python GIL, call whatever python function we need (in parallel), and then reenable the GIL. Many of the callbacks aren't required to be sequential -- rather they are just evaluating some user-defined function (e.g. a permittivity distribution). So this would be a perfect use case (and consequently, speed up the simulation init step quite a bit).

[oskooi]

After this PR was merged earlier today, I noticed that if building Meep from the master branch (commit: 74cb6dd) using --with-openmp to enable multithreading from this branch, if you run e.g. any of the unit tests in python/tests and the environment variable OMP_NUM_THREADS is not set, the timestepping rate slows down by nearly two orders of magnitude. It's only when this environment variable is set (e.g., $ OMP_NUM_THREADS=2 mpirun -np 2 python test_force.py) that the timestepping rate is what it should be. Perhaps we need to check if OMP_NUM_THREADS has not been set and if so set it to a default value of 1?

[oskooi]

According to this page:

If you forget to set OMP_NUM_THREADS to any value, the default value of your cluster environment will be used. In many cases, the default is 1, so that your program is executed serially. If this envvar is not set at all the OpenMP run time may also deciede to use up all cores of your computer which must not always be the expected outcome, so it is a good idea always to set a meaningful value.

[stevengj]

Maybe we should call omp_set_num_threads(1) somewhere if the OMP_NUM_THREADS environment variable is not set.