Update visualization

[yalaudah]

I think moving forward, we'll use smaller PRs. But here are the changes in this one:

Fixes issue #236 that involves rewriting a big portion of train.py such that:

• All the tensorboard event handlers are organized in tensorboard_handlers.py and only called in train.py to log training and validation results in Tensorboard
• The code logs the same results for training and validation. Also, it adds the class IoU score as well.
• All single-use functions (e.g. _select_max, _tensor_to_numpy, _select_pred_and_mask) are lambda functions now
• The code is organized into more meaningful "chunks".. e.g. all the optimizer-related code should be together if possible, same thing for logging, configuration, loaders, tensorboard, ..etc.

In addition:

• Fixed a visualization bug where the seismic images where not normalized correctly. This solves Issue seismic images not displayed correctly in Tensorboard #217.
• Fixed a visualization bug where the predictions where not masked where the input image was padded. This improves the ability to visually inspect and evaluate the results. This solves Issue interpret the padding and validation set mIoU vs predictoins - currently the quality of output predictions doesn't match the mIoU metric #230.
• Fixes a potential issue where Tensorboard can crash when a large training batchsize is used. Now the number of images visualized in Tensorboard from every batch has an upper limit.
• Completely removed OpenCV as a dependency from the DeepSeismic Repo. It was only used in a small part of the code where it wasn't really necessary, and OpenCV is a huge library.
• Fixes Issue Tensorboard should reflect the correct validation epoch number when visualizing validation images  #218 where the epoch number for the images in Tensorboard was always logged as 1 (therefore, not allowing use to see the epoch number of the different results in Tensorboard.
• Removes the HorovodLRScheduler class since its no longer used
• Removes toolz.take from Debug mode, and uses PyTorch's native Subset() dataset class
• Changes default patch size for the HRNet model to 256
• In addition to several other minor changes

[review-notebook-app]

Check out this pull request on 

You'll be able to see Jupyter notebook diff and discuss changes. Powered by ReviewNB.

[yalaudah]

Closing again. Being replaced by #264