Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redesign whole slide image reading #4107

Merged
merged 25 commits into from
Apr 19, 2022
Merged

Redesign whole slide image reading #4107

merged 25 commits into from
Apr 19, 2022

Conversation

drbeh
Copy link
Member

@drbeh drbeh commented Apr 12, 2022 •
edited
Loading

Fixes #4071
Fixes #4098

Description

This PR is a re-design and re-implementation of WSIReader while keeping the API the same as before and provide full backward compatibility (support for all the backends will come in the next PRs). This PR implements BaseWSIReader, WSIReader, and CuCIMWSIReader.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

drbeh added 4 commits April 12, 2022 01:18
@drbeh
Redesign BaseWSIReader, WSIReader, CuCIMWSIReader
2baa344 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Add unittests for WSIReader
e46fea0 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Add image mode for output validation
f946a63 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Merge branch 'dev' into new-wsireader
9b41019
@drbeh drbeh changed the base branch from dev to pathology April 12, 2022 20:44
drbeh added 8 commits April 13, 2022 13:08
@drbeh
Update docs
9c15ea3 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Update references to new WSIReader
e004538 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Remove legacy WSIReader
df0a61e 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Update unittests
b64c087 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Update docs
7f09e73 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
sort imports
d95e1c2 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Clean up imports
025376b 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Update docstrings
3e6fb27 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh drbeh marked this pull request as ready for review April 13, 2022 20:32
@drbeh drbeh requested review from Nic-Ma and wyli April 13, 2022 20:33
drbeh and others added 5 commits April 13, 2022 20:36
@drbeh
Update docs and docstrings
b8a2444 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
0268a03 
for more information, see https://pre-commit.ci
@drbeh
Fix a typo
f2b14aa 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Remove redundant checking
aed0ac5 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Merge branch 'new-wsireader' of github.com:drbeh/MONAI into new-wsire…
272d11f 
…ader
Nic-Ma
Nic-Ma reviewed Apr 14, 2022
View reviewed changes
monai/data/wsi_reader.py Outdated Show resolved Hide resolved
monai/data/wsi_reader.py Show resolved Hide resolved
drbeh added 2 commits April 14, 2022 17:34
@drbeh
Update read and other methods
ae4ff17 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Merge branch 'pathology' of github.com:Project-MONAI/MONAI into new-w…
e0bad98 
…sireader
Nic-Ma
Nic-Ma reviewed Apr 15, 2022
View reviewed changes
monai/data/wsi_reader.py Outdated Show resolved Hide resolved
monai/data/wsi_reader.py Show resolved Hide resolved
monai/data/wsi_reader.py Show resolved Hide resolved
drbeh added 3 commits April 18, 2022 14:43
@drbeh
Update wsireader to support multi image and update docstrings
e5a7a18 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Make workaround for CuImage objects
da6b675 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Add unittests for multi image reading
1ed8154 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Copy link
Member Author

drbeh commented Apr 18, 2022

Hi @Nic-Ma, do you have any other comments here? Thanks

@drbeh
Update a note about cucim
77f8747 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
Nic-Ma
Nic-Ma reviewed Apr 19, 2022
View reviewed changes
Copy link
Contributor

@Nic-Ma Nic-Ma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @drbeh ,

Thanks for your quick update!
I put some minor comments inline, others look good to me.

Thanks.

tests/test_wsireader.py Show resolved Hide resolved
monai/data/wsi_reader.py Outdated Show resolved Hide resolved
monai/data/wsi_reader.py Show resolved Hide resolved
monai/data/wsi_reader.py Outdated Show resolved Hide resolved
monai/data/wsi_reader.py Outdated Show resolved Hide resolved
monai/data/wsi_reader.py Show resolved Hide resolved
monai/data/wsi_reader.py Outdated Show resolved Hide resolved
drbeh added 2 commits April 19, 2022 13:52
@drbeh
Update type hints and docstrings
36e5b17 
Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
@drbeh
Merge branch 'pathology' of github.com:Project-MONAI/MONAI into new-w…
0328fe6 
…sireader
@drbeh
Copy link
Member Author

drbeh commented Apr 19, 2022

Hi @Nic-Ma, thanks for your comments. I have addressed them all and will merge this PR but please let me know if you have any comment and I will include it in the following PRs.

@drbeh drbeh merged commit 78340a9 into Project-MONAI:pathology Apr 19, 2022
drbeh added a commit that referenced this pull request Apr 19, 2022
@drbeh
Revert "Redesign whole slide image reading (#4107)"
36c77bb 
This reverts commit 78340a9.
@Nic-Ma
Copy link
Contributor

Nic-Ma commented Apr 19, 2022

Cool, thanks for your quick update.
Looks good to me.

@drbeh drbeh mentioned this pull request Apr 19, 2022
Merged
7 tasks
drbeh added a commit that referenced this pull request Apr 23, 2022
@drbeh
Split transform (#4153)
0cdec83 
* Redesign whole slide image reading (#4107)

* Redesign BaseWSIReader,  WSIReader, CuCIMWSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add image mode for output validation

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update references to new WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove legacy WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update unittests

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* sort imports

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Clean up imports

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs and docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix a typo

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove redundant checking

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update read and other methods

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update wsireader to support multi image and update docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Make workaround for CuImage objects

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for multi image reading

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update a note about cucim

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update type hints and docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Implement Split transform

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update formatting

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Implement SplitDict

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for SplitDict

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove images from docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Address all comments

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add example and size check

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Revert references to new wsireader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add missing comma

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
wyli added a commit that referenced this pull request Apr 26, 2022
@wyli @yiheng-wang-nv
@pre-commit-ci @Nic-Ma @dongyang0122 @drbeh @pxLi @ericspod
sync metatensor branch with dev (#4184)
724f73f 
* 4095 Add bundle download (#4114)

* draft download

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* update bundle download

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* add url and load

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* rename args and remove a few places

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix flake8 issue

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* enhance with reviews

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* add instantiate for load

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* fix black error

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* add unittest

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* add load to docs

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* add skip

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* add schemaerror

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* fix partial places

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* download zip bundle

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* [DLMED] restore Exception for test

Signed-off-by: Nic Ma <nma@nvidia.com>

* update ts features

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* add config_files test case

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* enhance docstring example for args_file

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Nic Ma <nma@nvidia.com>

* Disable pylint error and fix CI tests of new tifffile (#4162)

* workaround

Signed-off-by: Nic Ma <nma@nvidia.com>

* [DLMED] fix tifffile issue

Signed-off-by: Nic Ma <nma@nvidia.com>

* Fixed an error in DiNTS model implementation and enabled act and norm layer options (#4157)

* fixed a bug

Signed-off-by: dongy <dongy@nvidia.com>

* autofix

Signed-off-by: dongy <dongy@nvidia.com>

* update test case

Signed-off-by: dongy <dongy@nvidia.com>

Co-authored-by: dongy <dongy@nvidia.com>

* Split transform (#4153)

* Redesign whole slide image reading (#4107)

* Redesign BaseWSIReader,  WSIReader, CuCIMWSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add image mode for output validation

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update references to new WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove legacy WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update unittests

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* sort imports

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Clean up imports

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs and docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix a typo

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove redundant checking

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update read and other methods

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update wsireader to support multi image and update docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Make workaround for CuImage objects

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for multi image reading

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update a note about cucim

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update type hints and docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Implement Split transform

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update formatting

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Implement SplitDict

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for SplitDict

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove images from docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Address all comments

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add example and size check

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Revert references to new wsireader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add missing comma

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* fix bundle download test issue (#4169)

Signed-off-by: Yiheng Wang <vennw@nvidia.com>

* 4094 Enhance `ckpt_export` to save config files (#4159)

* [DLMED] enhance checkpoint export

Signed-off-by: Nic Ma <nma@nvidia.com>

* [DLMED] update according to comments

Signed-off-by: Nic Ma <nma@nvidia.com>

* Move RGB/RGBA checks to base class (#4171)

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

Co-authored-by: Nic Ma <nma@nvidia.com>

* [CICD] To support temp dgx runner (#4175)

* Support new temp dgx runner

Signed-off-by: Peixin Li <pxli@nyu.edu>

* atol 1e-5

Signed-off-by: Wenqi Li <wenqil@nvidia.com>

Co-authored-by: Wenqi Li <wenqil@nvidia.com>

* Test fix for AMP kwargs (#4178)

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>

Co-authored-by: Yiheng Wang <68361391+yiheng-wang-nv@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Nic Ma <nma@nvidia.com>
Co-authored-by: dongyang0122 <don.yang.mech@gmail.com>
Co-authored-by: dongy <dongy@nvidia.com>
Co-authored-by: Behrooz <3968947+drbeh@users.noreply.github.com>
Co-authored-by: Peixin <pxli@nyu.edu>
Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
Can-Zhao pushed a commit to Can-Zhao/MONAI that referenced this pull request May 10, 2022
@drbeh @Can-Zhao
Split transform (Project-MONAI#4153)
3ea72e6 
* Redesign whole slide image reading (Project-MONAI#4107)

* Redesign BaseWSIReader,  WSIReader, CuCIMWSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add image mode for output validation

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update references to new WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove legacy WSIReader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update unittests

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* sort imports

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Clean up imports

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs and docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix a typo

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove redundant checking

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update read and other methods

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update wsireader to support multi image and update docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Make workaround for CuImage objects

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for multi image reading

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update a note about cucim

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update type hints and docstrings

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Implement Split transform

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update formatting

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Implement SplitDict

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add unittests for SplitDict

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Remove images from docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Address all comments

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add example and size check

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Update docs

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Revert references to new wsireader

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>

* Add missing comma

Signed-off-by: Behrooz <3968947+drbeh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Nic-Ma Nic-Ma Nic-Ma left review comments

@wyli wyli Awaiting requested review from wyli

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

WSIReader to return patch information Redesign of WSIReader
2 participants
@drbeh @Nic-Ma