Parallel runs

[algebravic]

I would like to run a large number of experiments. Since I have a number of CPUs and GPUs available, I'd like to run them in parallel. I didn't see anything in the docs to address this. Are there race conditions in observers that I should be worried about?

[daliasen]

I was wondering about the same thing and eventually settled down for the SqlObserver.

I use the SqlObserver with a PostgreSQL database. The fix in pull request #813 solved the issues with race conditions I was having when running multiple experiments in parallel.

I would be cautious using SqlObserver with an SQLite database though. I'm not sure if simultaneous writers would be handled correctly.

My understanding is that the TinyDbObserver is inherently not usable if you want to be able to view the progress of your experiments while they are running but perhaps it's fine otherwise.

It would be useful to have some information about how each observer handles concurrency in the docs.

[thequilo]

I don't know of any race conditions that are not handled. The FileStorageObserver has code to handle simultaneous creation of run dirs from different threads/processes and each run uses a different directory after that. I think that MogoDb is itself already thread-safe.

I agree that this information would be helpful in the docs!