Prevent file locking for HDF5 files #1688
Closed
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation or Problem
There appears to be an issue with saving HDF5 files, which we now use to save the filter tensors (thanks to @mjohnson541 for discovering this). On remote servers, it is possible to get the following error on current master:
I found this issue on the h5py Github, which suggests that this is a problem with the code trying to lock the file when writing, which causes problems for NFS mounted file systems (i.e. most network mounted drives on servers). The only suggested fix is to export and environment variable with
export HDF5_USE_FILE_LOCKING=FALSE
, though this fix does not appear to work for everyone.Description of Changes
Use
os.environ
to set this environment variable wheneverrmgpy.rmg.main
is imported, which contains the code that saves the filter tensors as HDF5 files. This way, we do not have to tell people to add this on to their job submission scripts.Testing
@mjohnson541 has confirmed that this fix removed this issue for him