Proposal: Component benchmarking for registration

[YipengHu]

1. to separate registration-related and registration-dedicated components in to a few major benchmark-able components (tickets/issues);
2. in each issue, to identify the units in each part that need to be benckmarked;
3. to suggest a number of external, trusted libraries (or other sources) for benchmark candidates for each unit;
4. to discuss feasibility for individual candidates and best strategy for benckmarking.

Relevant tickets will be linked here.
Details above issues will be discussed here.

@wyli @tvercaut @kate-sann5100 @mathpluscode

[YipengHu]

That's my first attempt to define these components:

• image loss
• registration-related label loss*
• registration-dedicated label loss
• deformation loss
• encoder only networks (rigid, affine)
• encoder-decoder networks (ddf, dvf...)
• resampling, spatial transformation and image warping
• metrics on corresponding ROIs, e.g. tre, dice etc

*Separate registration-related with -dedicated label losses as the former may not be benchmarked separately.

[wyli]

For reference, the list of the highly relevant components that are currently in MONAI:

• losses https://github.com/Project-MONAI/MONAI/tree/dev/monai/losses
  • deform.py
  • image_dissimilarity.py
  • multi_scale.py
• network/blocks https://github.com/Project-MONAI/MONAI/tree/dev/monai/networks/blocks
  • regunet_block.py
  • localnet_block.py
  • warp.py
  • regunet.py

[kate-sann5100]

Speaking of registration-related components, I am curious have there been any attempt of benchmarking other components in MONAI? @wyli

[wyli]

yes, in terms of segmentation quality we did some comparisons:
https://github.com/Project-MONAI/tutorials/tree/master/modules/dynunet_pipeline#examples

in terms of speed/gpu utilisation, @yiheng-wang-nv is currently looking at DLProf for MONAI https://developer.nvidia.com/blog/profiling-and-optimizing-deep-neural-networks-with-dlprof-and-pyprof/
previously we did some basic analysis: https://github.com/Project-MONAI/tutorials/tree/master/acceleration

[kate-sann5100]

For segmentation quality, if I understand the tutorial correct, is trying to reimplement a SOTA paper. I am wondering is that the only way to benchmark networks cause that maybe a rabbit hole to go down. Maybe we could benchmark the loss and warp components before reimplementation.

For the speed/gpu utilisation, I will look at the tutorial first.

Thanks for the useful information!

[kate-sann5100]

I have created two issues respectively for detail discussions on BendingEnergyLoss and Image loss components.

[YipengHu]

I have done a search, but could not found any implemented python wrapper for deformation loss (bending energy or L2), e.g. in ANTs or elastix.

With c++ implemntations, we have NiftyReg: https://github.com/KCL-BMEIS/niftyreg/blob/master/reg-lib/cpu/_reg_localTrans_regul.cpp#L78

@tvercaut are you aware anything?

Alternatives, we have voxelmorph and deepreg for reference, although the former is not tested code among others.

Image losses seem available in almost all. @kate-sann5100 I think we can start with these.

[tvercaut]

Note that elastix provides bending energy as well but I don't know if they expose it in python/simpleelastix:
https://elastix.lumc.nl/doxygen/classelastix_1_1TransformBendingEnergyPenalty.html

I looked at AIRLAB as well but they don't seem to have bending energy (L2 is tere through):
https://github.com/airlab-unibas/airlab/tree/master/airlab/regulariser

[YipengHu]

Yup - it looks like they only exist in the cpp:
https://github.com/SuperElastix/elastix/tree/develop/Components/Metrics/BendingEnergyPenalty
and tested here: https://github.com/SuperElastix/elastix/blob/develop/Testing/CMakeLists.txt#L812

[kate-sann5100]

Thank you both for the valuable informations. I am starting to write the benchmarking of image loss.

[wyli]

notes from our meeting today:

main approach:

• to embed the expected results as CSV files in MONAI,
• (might host a separate repo to document how to generate those CSVs using the external packages)

assumptions:

• assuming autodiff. of pytorch is good

two ways to handle the potential numerical differences:

• ignoring the outcome at the image borders
• measuring the proportion of matched pixel-level results