transform documentation enhancements

[Spenhouet]

Is your feature request related to a problem? Please describe.

Until now we are using MONAI for our training pipeline and torchio for data augmentations (offline preprocessed).
The data augmentations torchio provides are very medically relevant.
Their documentation, naming of the augmentations and the visual examples are very helpful.

Original	Random blur
Random flip	Random noise
Random affine transformation	Random elastic transformation
Random bias field artifact	Random motion artifact
Random spike artifact	Random ghosting artifact
Random Anisotropy	Random Swap

We struggled finding equivalent methods in MONAI and even when we did, we were not certain that these are actually the same methods.
We therefore never bothered using MONAI for data augmentations.
We would like to replace torchio with MONAI to further reduce our dependencies.

Describe the solution you'd like

1. We wish that MONAI would provide the same transformations as torchio does.
2. We also wish that MONAI provides a equally good documentation and uses medical terms for naming for these transforms.
3. MONAI should add visual examples for these transforms (in the form of GIFs if possible)

While this issue is meant as a feature request, I see that it has a wide scope. It is also a question in that I might just not see the equivalents.
Let's see were MONAI currently is. I will to try find the equivalent transforms of both packages.

torchio	MONAI
RandomAnisotropy	Downsampling + Upsampling with Spacing
RandomBlur	RandGaussianSmooth
RandomFlip	RandFlip
RandomNoise	RandGaussianNoise
RandomAffine	RandAffine
RandomElasticDeformation	Rand3DElastic
RandomBiasField	RandBiasField
RandomMotion	?
RandomSpike	RandKSpaceSpikeNoise
RandomGhosting	?
RandomGamma	RandAdjustContrast
RandomSwap	?

Having visual examples in the documentation of the transforms would really help.

Other open issues on data augmentations I could find:

• Monai equivalent to tio.OneOf transform? (ETA 8/13) #1847
• GridDistortion data augmentation #2186
• Augmentation Transform | InCut & OutCut Dropping (ETA 08/27/21) #2707
• Augmentation Transform | Patch Shuffling (ETA 8/27) #2701
• histogram equalisation transform? #2691
• RandFlip operates on all axes or none #2245
• random plane sampling transform #1993

I would like to collect all task which are needed for MONAI to match torchio on its documentation and transforms.

The following tasks will be necessary to match torchio:

• Implement RandAnisotropy similar to RandomAnisotropy. This should be implemented with a sequential down and up sampling via Spacing.
• Implement a RandMotion similar to RandomMotion. See also MRI k-Space Motion Artefact Augmentation: Model Robustness and Task-Specific Uncertainty.
• Implement a RandPatchSwap similar to RandomSwap. Coordinate with Augmentation Transform | Patch Shuffling (ETA 8/27) #2701 where a RandPatchShuffle is proposed.
• Implement a RandGhosting similar to . See also Ghost Artifacts - Why do motion artifacts often form into discrete ghosts?
• Add example images to documentation of transforms (especially the augmentation transforms). If possible create GIFs like it is done in torchio (see above).

[Nic-Ma]

Hi @Spenhouet ,

Thanks for your great feedback!
I think that are very good points and welcome contributions.
Maybe we can work together to achieve them.

Thanks.

[Spenhouet]

Hi @Nic-Ma,

this is definitely not something I will have time for any time soon but I did hope I could get this going.
The first thing needed would be a precise list of steps / small TODOs so that the community can open up pull request for individual tasks.

From the table I included, is there something where I missed the MONAI equivalent? Is the monai.RandKSpaceSpikeNoise the same as the torchio.RandomSpike?

What would be preferred here:

1. Have this issue as a big issue which will track all tasks and pull request?
2. or have this issue to identify all tasks and then create individual issues for every task?

[Nic-Ma]

Hi @Spenhouet ,

1. I think RandKSpaceSpikeNoise is the correct mapping.
2. RandomBlur may map to MONAI RandGaussianSmooth transform.
3. RandomAnisotropy may be achieved by MONAI Spacing or Resize transforms.
4. RandomMotion may map to MONAI RandBiasField transform? I didn't check the details carefully.
5. RandomGamma maps to MONAI RandAdjustContrast transform.
6. RandomSwap is an interesting feature, I think @finalelement is working on it in ticket: Augmentation Transform | Patch Shuffling (ETA 8/27) #2701.

So only RandomGhosting of your list is not existing.

I think you can make a clear list in this ticket and then submit sub-tickets to track every separate concrete task?
Every PR can close a sub-ticket in theory. @wyli @ericspod @rijobro do you guys have other opinions?

Thanks in advance.

[Spenhouet]

Hi @Nic-Ma,

• We should consider renaming RandGaussianSmooth to RandGaussianBlur since that seems much more common. https://en.wikipedia.org/wiki/Gaussian_blur
• Yes, RandomAnisotropy is a downsampling + upsampling with Spacing. Nonetheless, we should implement it as a method since it fulfills a very important purpose - simulating lower resolution input during training. This helps with 2D vs 3D MR acquisition.
• I don't see how RandBiasField relates to RandomMotion. These are very different operations. So I assume a RandomMotion is also missing. See MRI k-Space Motion Artefact Augmentation: Model Robustness and Task-Specific Uncertainty
• RandomSwap is different to what is proposed in Augmentation Transform | Patch Shuffling (ETA 8/27) #2701. One is about taking blocks and putting them at different locations and one is about mixing pixels within random blocks.

I'm not sure how sub tickets in GitHub work but I will create and maintain a task list in my original post. In the comments we can discuss necessary and potential steps. I will add everything to the task list which seems decided to me. Everything else should be discussed first.

[Spenhouet]

• We should also think about renaming Rand2DElastic and Rand3DElastic to Rand2DElasticDeformation and Rand3DElasticDeformation since this is a much more precise and intuitive name
• One thing I like about torchio is that they explain everything in medical relevant terms. I believe the MONAI documentation could take a page from their book on that and improve all the existing doc strings. This might seem like unnecessary work but I strongly believe that it is important work.
• Another thing is that they always make sure to cite relevant work in the individual transform topics. Doing the same would make MONAI more reputable.

[Nic-Ma]

Hi @Spenhouet ,

Thanks for your great suggestions! Very valuable feedback for MONAI project.
And I think maybe we can start from non-breaking feature requests to add the new transforms you listed?
@wyli @ericspod @rijobro What do you guys think about the GIF demo of medical transforms?

Thanks.

[Nic-Ma]

About RandomSwap, I just confirmed it's different from MONAI ticket #2701 , the latter is to shuffle the content of patches.
So this RandomSwap transform is also not in our radar so far.

Thanks.

[diazandr3s]

This is a great point.

CC @fepegar

[fepegar]

Hi all! Thanks for pinging, @diazandr3s. I have a couple of comments:

1. Feel free to ping me in any discussions or questions related to TorchIO, I'm more than happy to help and collaborate. See e.g. Monai equivalent to tio.OneOf transform? (ETA 8/13) #1847, Random Patch Dataset #606, How to integrate transforms from BatchGenerator, TorchIO and Rising #493
2. I'm not sure it's necessary to spend resources in implementing more and more transforms. TorchIO and MONAI can be mixed more or less easily, as shown in
   a. Integrate 3rd party transforms into a MONAI program
   b. TorchIO + MONAI + PyTorch Lightning

[aylward]

Two thoughts on these visualizations:

1. would be great if the generated examples/images/documentation could also be part of the MONAI testing framework.

2. perhaps the visualizations (and documentation) could be generated in jupyter notebooks. A nice way of embedding interactive 2D and 3D visualizations of numpy, tensor, and other 2D and 3D structures in jupyter notebooks is via itkWidgets:
   https://github.com/InsightSoftwareConsortium/itkwidgets
   and
   https://blog.kitware.com/monai-and-itkwidgets-getting-started/

• Also, there is a nice way of capturing jupyter notebook visualizations generated by itkWidgets and saving them as pngs/gifs, etc.

[diazandr3s]

@aylward do you know whether itkwidgets work in Google Colab as well?
Another option for visualization is to have a Slicer module to try/test image transformations in Slicer. Similar to what is being created for TorchIO https://github.com/fepegar/SlicerTorchIO

[aylward]

@diazandr3s - great point...we are in the process of adding Colab support in collaboration with the NIH/NCI. It does work with Binder and other cloud-based jupyter notebooks, but google changed the jupyter notebook just enough such that itkWidgets and some (many?) other jupyter notebook / jupyter lab widgets don't quite work (they don't crash, but they simply don't appear in a cell).

The nice part about these widgets is that they are embedded in the notebooks, and you can use automated cell capturing to generate images and movies. So, it might be possible to automate the image generation for docs and for testing using the same notebook.

To capture a view do...

from itkwidgets import view
import ipywebrtc as webrtc
... read your image / numpy array / tensor / ...
... generate or read your label image ...
view1 = view(image=<your_image>, label_image=<your_label_image>, cmap=itkwidgets.cm.viridis, gradient_opacity=0.9, background=[0.5,0.5,0.5], slicing_planes=True)
view1

image_recorder1 = webrtc.ImageRecorder(stream=view1, filename='<your_filename.png>', autosave=True)
image_recorder1.recording=True

For all of the view options, use

view?

Enjoy.

[Spenhouet]

Hi @fepegar, first of I would like to thank you for your great library. As I already wrote above I really value the documentation and that it is highly medically relevant.
Sorry that I did not tag you on this issue. No bad intention.

I'm not sure it's necessary to spend resources in implementing more and more transforms. TorchIO and MONAI can be mixed more or less easily

Having MONAI transform wrappers around torch.io transforms would be an option but this would add torch.io as dependency to MONAI and that is a decision I have no say in but doubt that it is likely.
For us, as a medtech software company, reducing dependencies is crucial. For each SOUP (software of unknown provenance) we have to perform validations, risk assessments, .... and then again on updates. This is a lot of work for a Startup.

I agree that this seemingly binds "unnecessary" resources but having a single framework instead of multiple is way better for us (and probably many more). Currently there are way to many individual projects on the same topics and combining resources would be the best. But it is how it is. We did see the most potential for our use case in MONAI and therefore went with it. But MONAI doesn't yet provide the same data augmentations as torch.io does why we are still dependent on torch.io. Therefore my wish to close this gap.

[fepegar]

@Spenhouet Thanks for your detailed explanation and kind words.

As I said, I'm happy to help!

[rijobro]

Hi @Spenhouet, I think your issue boils down to two main points:

1. improved documentation/visualisation of transforms
2. add a few transforms that exist in TorchIO but not MONAI.

For the latter, I would recommend creating a feature request issue per transform. These can be grouped together in a Github Project if that helps keep things organised.

As a small side note, you mentioned renaming some transforms e.g., Rand2DElastic -> Rand2DElasticDeformation. Personally I think we should avoid this where possible for backwards compatibility.

[Spenhouet]

I would recommend creating a feature request issue per transform.

Makes sense. We can also do that. Should we keep this issue open until all individual issues are created? And maybe a project?

you mentioned renaming some transforms [...] we should avoid this where possible for backwards compatibility.

MONAI isn't even version 1.0 yet and your are already stopping change for backwards compatibility? I feel like this library needs to undergo a lot of change until it reaches v1. Personally I would prioritize refactoring to a good initial v1 instead of directly starting with a legacy product.

Naming is hard but integral.

For backward compatibility we can always do something like this:

Rand2DElastic = Rand2DElasticDeformation

What do you think?

[aylward]

MONAI isn't even version 1.0 yet and your are already stopping change for backwards compatibility? I feel like this library needs to undergo a lot of change until it reaches v1. Personally I would prioritize refactoring to a good initial v1 instead of directly starting with a legacy product.

Agreed. Maybe now is a good time to start introducing @deprecated usage and policies. This was a huge point of discussion with ITK. You can see opinions from various folks here:
https://itk.org/Wiki/ITK/Backward_Compatibility_Open_Discussion

Your comments on the importance of naming (to reduce learning curve) and suggestion for maintaining backward compatibility is perfect (if deprecation notifications are also provided).

[wyli]

Visualisations are included, I'm closing this ticket. Please feel free to file additional feature requests, e.g. Project-MONAI/tutorials#436.

[Nic-Ma]

Thanks for the suggestions about visualization ideas.
I tried to use itkwidget in jupyter notebook:
view(image=data["image"], label=data["label"])
No image or chart shows, just got a log print:

Viewer(geometries=[], gradient_opacity=0.22, point_sets=[], rendered_image=<itk.itkImagePython.itkImageF3; pro…

@ericspod is this the same issue you mentioned before?

And I also checked the itkwidget tutorial, seems it also only shows the log:
https://github.com/KitwareMedical/MONAI-GettingStarted/blob/main/MONAI-ITKWidgets.ipynb

So I will not show itkwidget usage in PR Project-MONAI/tutorials#448 so far, will try to add it when we figured out how to use it later.

Thanks.

[aylward]

Hi, It isn't working in Google CoLab, but should be working otherwise. Can you please show the type of the data["image"] and data["label"] variables and their intensity range? Thanks! s

…

On Mon, Nov 22, 2021 at 12:31 AM Nic Ma ***@***.***> wrote: Thanks for the suggestion about visualization ideas. I tried to use itkwidget in jupyter notebook: view(image=data["image"], label=data["label"]) No image or chart shows, just got a log print: Viewer(geometries=[], gradient_opacity=0.22, point_sets=[], rendered_image=<itk.itkImagePython.itkImageF3; pro… @ericspod <https://github.com/ericspod> is this the same issue you mentioned before? Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2721 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACEJLZJQ2ZCIUK3Z6PUS73UNHITNANCNFSM5BYI4GWA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Stephen R. Aylward, Ph.D. Senior Director of Strategic Initiatives --- Kitware: *Delivering innovative, open source, scientific software.*

[Nic-Ma]

Hi @aylward ，

They are PyTorch Tensors and the intensity range is (0.0, 1.0) in my case.
Maybe someone from ITK team can help update the ITKWidget tutorial first as a reference?
https://github.com/KitwareMedical/MONAI-GettingStarted/blob/main/MONAI-ITKWidgets.ipynb

Thanks in advance.

[aylward]

Hi,

It looks like you're running this in Jupyter Lab 3, CoLab, Sagemaker, or something other than Jupyter Notebooks, correct?

Jupyter Lab 2 works, but compatibility broke in Jupyter Lab 3. Also, pyWidgets in general do not work without modification in CoLab and Sagemaker - Google and Amazon created their own, incompatible kernel communication protocols. We hope to have updated itkWidgets (which derive from pyWidgets) with support for CoLab within 1-2 weeks.

The good news is that all seems to be working as expected with Jupyter Notebooks....to generate the image shown below, I did the following:

1. updated the script you provided to work with MONAI 0.7, e.g., post_transform -> postprocessing in the call to 'SupervisedTrainer()'.
2. added the argument gradient_opacity=0.4 to the view command, so that no slider adjustments need to be made.

Here is the resulting image:

I'd like to update this script to use MONAI and UNets more succinctly - so that it is a more up-to-date representation of how to use MONAI. I'll create a pull request later in the week with these changes.

Hope this helps. Let me know if you have questions/suggestions. Hopefully a 1-2 week for CoLab support is acceptable.

@thewtex is the creator of itkWidgets and he is updating them to have CoLab support.

Stephen

[ericspod]

Thanks for the suggestions about visualization ideas. I tried to use itkwidget in jupyter notebook: view(image=data["image"], label=data["label"]) No image or chart shows, just got a log print:

Viewer(geometries=[], gradient_opacity=0.22, point_sets=[], rendered_image=<itk.itkImagePython.itkImageF3; pro…

@ericspod is this the same issue you mentioned before?

And I also checked the itkwidget tutorial, seems it also only shows the log: https://github.com/KitwareMedical/MONAI-GettingStarted/blob/main/MONAI-ITKWidgets.ipynb

So I will not show itkwidget usage in PR Project-MONAI/tutorials#448 so far, will try to add it when we figured out how to use it later.

Thanks.

I've seen that behaviour in places that aren't compatible as @aylward discussed, I think also in places where it should work but where I'm missing the necessary packages.

[Nic-Ma]

Hi @aylward ,

Thanks for your detailed sharing.
I am using jupyter notebook instead of jupyter lab, I will try again later when your tutorial notebook is ready.

Thanks for your support!

[Nic-Ma]

Hi @aylward and @ericspod ,

Thanks for your help and examples, I submitted PR to add itkwidgets example in the MONAI visualization notebook:
Project-MONAI/tutorials#454

Thanks.