Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8185 - Refactor test #8231

Merged
merged 104 commits into from
Feb 12, 2025
Merged

8185 - Refactor test #8231

merged 104 commits into from
Feb 12, 2025

Conversation

garciadias
Copy link
Contributor

@garciadias garciadias commented Nov 21, 2024

Fixes #8185

Description

Reorganize tests

I have looked at the imports in each test file and the test title to identify which files were being tested. I mirrored the file structure of MONAI on the tests folder and moved the files accordingly. I used some helper scripts, but the process required substantial manual intervention. When uncertain, I moved the tests to the integration folder since the confusion always involved many imports, and I could not find clarity from the test name.

Please review the integration folder carefully, which is the one that I feel the least confident about.


### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not applicable items -->
- [x] Non-breaking change (fix or new feature that would not break existing functionality).
- [x] Quick tests passed locally by running `./runtests.sh --quick --unittests  --disttests`.

Performance Before:
94.81s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_default_value_1_model
20.95s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_default_value_0_
15.26s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_export_2_model
14.86s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_default_value_2_model
14.55s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_export_1_model
14.28s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_export_0_

Performance after:

1.62s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_ckpt_export_2_model
1.25s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_ckpt_export_default_2_model
0.64s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_ckpt_export_0_
0.57s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_ckpt_export_1_model
0.57s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_ckpt_export_default_1_model
0.55s call     tests/test_bundle_ckpt_export.py::TestCKPTExport::test_ckpt_export_default_0_
0.01s setup    tests/test_bundle_ckpt_export.py::TestCKPTExport::test_ckpt_export_0_
@garciadias
Copy link
Contributor Author

Hi @ericspod,

Just for context, last week, I started working in Jorge Cardoso's team at KCL, so I am still getting access to the cluster and waiting for a new laptop with a dedicated GPU. I am currently working on a laptop with integrated graphics, which is not very powerful. So, it has been hard for me to tackle the slow tests since they would run SUPER slow for me. This is why I moved to reorganise the tests.

Since this obscures any further changes in these files on the tracking records, I suggest you review this part first, and I create a new pull request for the other points in your issue.

Please let me know what you think.

Many thanks

@ericspod
Copy link
Member

ericspod commented Dec 2, 2024

Hi @ericspod,

Just for context, last week, I started working in Jorge Cardoso's team at KCL, so I am still getting access to the cluster and waiting for a new laptop with a dedicated GPU. I am currently working on a laptop with integrated graphics, which is not very powerful. So, it has been hard for me to tackle the slow tests since they would run SUPER slow for me. This is why I moved to reorganise the tests.

Since this obscures any further changes in these files on the tracking records, I suggest you review this part first, and I create a new pull request for the other points in your issue.

Please let me know what you think.

Many thanks

Hi @garciadias I see that the tests have been moved which is good, but there's a few files that seem to have gone missing and some conflicts that need to be resolved. I'm sure you've kept what you were working on earlier so we can come back to that when you're ready. These changes are going to represent a significant change from what's in the tests folder so we'll have to discuss with other developers more. We discussed it briefly at the Core meeting but we should again, what I can suggest now is that the directory structure doesn't have to be so deep as you have it, so perhaps the array/dictionary directories for the transforms should have their contents moved to the parent, or some directories with one file should also have that file moved to the parent. It looks much better with more structure, I just don't want too deep a structure that is cumbersome itself. This is definitely in the right direction!

Good to hear you've come on board with us, we should meet in person and discuss things further. Thanks!

@garciadias garciadias closed this Dec 2, 2024
@garciadias garciadias reopened this Dec 2, 2024
@garciadias
Copy link
Contributor Author

Hi @ericspod,
Just for context, last week, I started working in Jorge Cardoso's team at KCL, so I am still getting access to the cluster and waiting for a new laptop with a dedicated GPU. I am currently working on a laptop with integrated graphics, which is not very powerful. So, it has been hard for me to tackle the slow tests since they would run SUPER slow for me. This is why I moved to reorganise the tests.
Since this obscures any further changes in these files on the tracking records, I suggest you review this part first, and I create a new pull request for the other points in your issue.
Please let me know what you think.
Many thanks

Hi @garciadias I see that the tests have been moved which is good, but there's a few files that seem to have gone missing and some conflicts that need to be resolved. I'm sure you've kept what you were working on earlier so we can come back to that when you're ready. These changes are going to represent a significant change from what's in the tests folder so we'll have to discuss with other developers more. We discussed it briefly at the Core meeting but we should again, what I can suggest now is that the directory structure doesn't have to be so deep as you have it, so perhaps the array/dictionary directories for the transforms should have their contents moved to the parent, or some directories with one file should also have that file moved to the parent. It looks much better with more structure, I just don't want too deep a structure that is cumbersome itself. This is definitely in the right direction!

Good to hear you've come on board with us, we should meet in person and discuss things further. Thanks!

Hi @ericspod, thank you very much for reviewing this and for the welcoming regards.

Missing files:

When you say there are some missing files, are you referring to these file files?

  • tests/test_bundle_ckpt_export.py
  • tests/test_fl_monai_algo_dist.py
  • tests/test_handler_metrics_saver_dist.py
  • tests/test_integration_classification_2d.py
  • tests/test_integration_segmentation_3d.py

If these are all missing files you are referring to, I can confirm that git marked them as deleted, but they are present at:

  • tests/bundle/test_bundle_ckpt_export.py
  • tests/fl/client/monai_algo/test_fl_monai_algo_dist.py
  • tests/handlers/test_handler_metrics_saver_dist.py
  • tests/integration/test_integration_classification_2d.py
  • tests/integration/test_integration_segmentation_3d.py

I don't understand why these were flagged as deleted.

Conflicts:

I will do my best to merge the current dev branch to this and keep solving the conflicts while we wait for approval of this change.

Folder depth:

Fair enough, I will move them. I am happy to keep adjusting until we are satisfied with the result.

Please keep me posted on the conversation with the core group.
If possible, I would love to be included in any of the meetings you have.

Many thanks, Eric.

@ericspod
Copy link
Member

ericspod commented Dec 2, 2024

Hi @ericspod,
Just for context, last week, I started working in Jorge Cardoso's team at KCL, so I am still getting access to the cluster and waiting for a new laptop with a dedicated GPU. I am currently working on a laptop with integrated graphics, which is not very powerful. So, it has been hard for me to tackle the slow tests since they would run SUPER slow for me. This is why I moved to reorganise the tests.
Since this obscures any further changes in these files on the tracking records, I suggest you review this part first, and I create a new pull request for the other points in your issue.
Please let me know what you think.
Many thanks

Hi @garciadias I see that the tests have been moved which is good, but there's a few files that seem to have gone missing and some conflicts that need to be resolved. I'm sure you've kept what you were working on earlier so we can come back to that when you're ready. These changes are going to represent a significant change from what's in the tests folder so we'll have to discuss with other developers more. We discussed it briefly at the Core meeting but we should again, what I can suggest now is that the directory structure doesn't have to be so deep as you have it, so perhaps the array/dictionary directories for the transforms should have their contents moved to the parent, or some directories with one file should also have that file moved to the parent. It looks much better with more structure, I just don't want too deep a structure that is cumbersome itself. This is definitely in the right direction!
Good to hear you've come on board with us, we should meet in person and discuss things further. Thanks!

Hi @ericspod, thank you very much for reviewing this and for the welcoming regards.

Missing files:

When you say there are some missing files, are you referring to these file files?

* tests/test_bundle_ckpt_export.py

* tests/test_fl_monai_algo_dist.py

* tests/test_handler_metrics_saver_dist.py

* tests/test_integration_classification_2d.py

* tests/test_integration_segmentation_3d.py

If these are all missing files you are referring to, I can confirm that git marked them as deleted, but they are present at:

* tests/bundle/test_bundle_ckpt_export.py

* tests/fl/client/monai_algo/test_fl_monai_algo_dist.py

* tests/handlers/test_handler_metrics_saver_dist.py

* tests/integration/test_integration_classification_2d.py

* tests/integration/test_integration_segmentation_3d.py

I don't understand why these were flagged as deleted.

Conflicts:

I will do my best to merge the current dev branch to this and keep solving the conflicts while we wait for approval of this change.

Folder depth:

Fair enough, I will move them. I am happy to keep adjusting until we are satisfied with the result.

Please keep me posted on the conversation with the core group. If possible, I would love to be included in any of the meetings you have.

Many thanks, Eric.

Hi @garciadias There were 2 files mentioned as deleted in the "Files changed" section here, the second of these test_clip_intensity_percentilesd.py appears missing entirely. The conflicts appear resolved but there are other errors now to resolve. This looks like it's related to how we were excluding files in the past that assumed a flat structure that isn't there anymore. We can fix that for now but then think about improving that process later. Thanks!

@garciadias
Copy link
Contributor Author

garciadias commented Dec 3, 2024

Hi @garciadias There were 2 files mentioned as deleted in the "Files changed" section here, the second of these test_clip_intensity_percentilesd.py appears missing entirely. The conflicts appear resolved but there are other errors now to resolve. This looks like it's related to how we were excluding files in the past that assumed a flat structure that isn't there anymore. We can fix that for now but then think about improving that process later. Thanks!

Thank you, @ericspod. I have now restored the test_clip_intensity_percentilesd.py file, and the other one 'missing', is at tests/bundle/test_bundle_ckpt_export.py.

I will be working on solving the other issues.

Am I pushing too often? I see the whole pipeline is triggered. Should I accumulate some changes before pushing?

Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
@KumoLiu
Copy link
Contributor

KumoLiu commented Feb 10, 2025

/build

Copy link
Contributor

@KumoLiu KumoLiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[2025-02-10T16:54:55.127Z] 
[2025-02-10T16:54:55.127Z] ----------------------------------------------------------------------
[2025-02-10T16:54:55.127Z] Ran 0 tests in 0.000s
[2025-02-10T16:54:55.127Z] 
[2025-02-10T16:54:55.127Z] OK
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_init_bundle.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_dints_cell.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_dints_mixop.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_regunet.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_localnet.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_get_data.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_verify_metadata.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_verify_net.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_reference_resolver.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_ckpt_export.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_push_to_hf_hub.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_config_parser.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_config_item.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_download.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_workflow.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_component_locator.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_utils.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_trt_export.py
[2025-02-10T16:54:55.127Z] []
[2025-02-10T16:54:55.127Z] time to discover tests: 0.04873630404472351s, total cases: 0.
[2025-02-10T16:54:55.383Z] coverage
[2025-02-10T16:54:55.639Z] Combined data file .coverage/.coverage.pt250-cuda126-jenkins-monai-premerge-3411-vcdbg-6rzzn.3784.XYkZCFzx
[2025-02-10T16:54:55.639Z] Combined data file .coverage/.coverage.pt250-cuda126-jenkins-monai-premerge-3411-vcdbg-6rzzn.946.XIWTFPdx
[2025-02-10T16:54:56.202Z] /usr/local/lib/python3.10/dist-packages/coverage/report_core.py:116: CoverageWarning: Couldn't parse '/home/jenkins/agent/workspace/MONAI-premerge/monai/config-3.10.py': No source for code: '/home/jenkins/agent/workspace/MONAI-premerge/monai/config-3.10.py'. (couldnt-parse)
[2025-02-10T16:54:56.202Z]   coverage._warn(msg, slug="couldnt-parse")
[2025-02-10T16:54:56.202Z] /usr/local/lib/python3.10/dist-packages/coverage/report_core.py:116: CoverageWarning: Couldn't parse '/home/jenkins/agent/workspace/MONAI-premerge/monai/config.py': No source for code: '/home/jenkins/agent/workspace/MONAI-premerge/monai/config.py'. (couldnt-parse)
[2025-02-10T16:54:56.202Z]   coverage._warn(msg, slug="couldnt-parse")

I would suggest create a small pr only for "test_utils" name change to previous block daily ci.

@garciadias
Copy link
Contributor Author

[2025-02-10T16:54:55.127Z] 
[2025-02-10T16:54:55.127Z] ----------------------------------------------------------------------
[2025-02-10T16:54:55.127Z] Ran 0 tests in 0.000s
[2025-02-10T16:54:55.127Z] 
[2025-02-10T16:54:55.127Z] OK
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_init_bundle.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_dints_cell.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_dints_mixop.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_regunet.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_localnet.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_get_data.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_verify_metadata.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_verify_net.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_reference_resolver.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_ckpt_export.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_push_to_hf_hub.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_config_parser.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_config_item.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_download.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_workflow.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_component_locator.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_utils.py
[2025-02-10T16:54:55.127Z] monai test runner: excluding test_bundle_trt_export.py
[2025-02-10T16:54:55.127Z] []
[2025-02-10T16:54:55.127Z] time to discover tests: 0.04873630404472351s, total cases: 0.
[2025-02-10T16:54:55.383Z] coverage
[2025-02-10T16:54:55.639Z] Combined data file .coverage/.coverage.pt250-cuda126-jenkins-monai-premerge-3411-vcdbg-6rzzn.3784.XYkZCFzx
[2025-02-10T16:54:55.639Z] Combined data file .coverage/.coverage.pt250-cuda126-jenkins-monai-premerge-3411-vcdbg-6rzzn.946.XIWTFPdx
[2025-02-10T16:54:56.202Z] /usr/local/lib/python3.10/dist-packages/coverage/report_core.py:116: CoverageWarning: Couldn't parse '/home/jenkins/agent/workspace/MONAI-premerge/monai/config-3.10.py': No source for code: '/home/jenkins/agent/workspace/MONAI-premerge/monai/config-3.10.py'. (couldnt-parse)
[2025-02-10T16:54:56.202Z]   coverage._warn(msg, slug="couldnt-parse")
[2025-02-10T16:54:56.202Z] /usr/local/lib/python3.10/dist-packages/coverage/report_core.py:116: CoverageWarning: Couldn't parse '/home/jenkins/agent/workspace/MONAI-premerge/monai/config.py': No source for code: '/home/jenkins/agent/workspace/MONAI-premerge/monai/config.py'. (couldnt-parse)
[2025-02-10T16:54:56.202Z]   coverage._warn(msg, slug="couldnt-parse")

I would suggest create a small pr only for "test_utils" name change to previous block daily ci.

But then it would be called util, as it was before? Otherwise, it will conflict with the utils folder.

@KumoLiu
Copy link
Contributor

KumoLiu commented Feb 11, 2025

But then it would be called util, as it was before? Otherwise, it will conflict with the utils folder.

No, what I meant was to move the changes from test.utils to test.test_utils, as updated in this PR, to another PR and merge that first. I have already modified the Blossom CI based on this, and merging it as is will break the daily CI.

I guess you may need some additional time to debug the refactor; based on what I pasted, it shows that 0 tests have been run.
running command: ./runtests.sh --build --quick --unittests

@ericspod ericspod mentioned this pull request Feb 11, 2025
7 tasks
@ericspod
Copy link
Member

But then it would be called util, as it was before? Otherwise, it will conflict with the utils folder.

No, what I meant was to move the changes from test.utils to test.test_utils, as updated in this PR, to another PR and merge that first. I have already modified the Blossom CI based on this, and merging it as is will break the daily CI.

I guess you may need some additional time to debug the refactor; based on what I pasted, it shows that 0 tests have been run. running command: ./runtests.sh --build --quick --unittests

I have #8335 to address this now. Let's merge that, then we can update this PR.

@garciadias
Copy link
Contributor Author

But then it would be called util, as it was before? Otherwise, it will conflict with the utils folder.

No, what I meant was to move the changes from test.utils to test.test_utils, as updated in this PR, to another PR and merge that first. I have already modified the Blossom CI based on this, and merging it as is will break the daily CI.

I guess you may need some additional time to debug the refactor; based on what I pasted, it shows that 0 tests have been run. running command: ./runtests.sh --build --quick --unittests

Sorry about that. I think I have now fixed this.

ericspod added a commit that referenced this pull request Feb 11, 2025
Related to #8185.

### Description

This changes the name of `tests/utils.py` to `tests/test_utils.py` to
conform to the changes introduced with the daily CICD tests. This is
done in #8231 but the change is being pre-merged now to get tests
working again while issues are sorted out there. This also includes
changes this PR made to that file, and changes anywhere else to
correctly import the module.

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>
@garciadias garciadias requested a review from KumoLiu February 12, 2025 09:03
@KumoLiu
Copy link
Contributor

KumoLiu commented Feb 12, 2025

/build

Signed-off-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
@KumoLiu
Copy link
Contributor

KumoLiu commented Feb 12, 2025

/build

Copy link
Contributor

@KumoLiu KumoLiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, LGTM now!

@KumoLiu KumoLiu merged commit af9e8f9 into Project-MONAI:dev Feb 12, 2025
28 checks passed
Can-Zhao pushed a commit to Can-Zhao/MONAI that referenced this pull request Mar 10, 2025
Related to Project-MONAI#8185.

### Description

This changes the name of `tests/utils.py` to `tests/test_utils.py` to
conform to the changes introduced with the daily CICD tests. This is
done in Project-MONAI#8231 but the change is being pre-merged now to get tests
working again while issues are sorted out there. This also includes
changes this PR made to that file, and changes anywhere else to
correctly import the module.

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Can-Zhao pushed a commit to Can-Zhao/MONAI that referenced this pull request Mar 10, 2025
Fixes Project-MONAI#8185

### Description

## Reorganize tests

I have looked at the imports in each test file and the test title to
identify which files were being tested. I mirrored the file structure of
MONAI on the `tests` folder and moved the files accordingly. I used some
helper scripts, but the process required substantial manual
intervention. When uncertain, I moved the tests to the `integration`
folder since the confusion always involved many imports, and I could not
find clarity from the test name.

Please review the integration folder carefully, which is the one that I
feel the least confident about.

```

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not applicable items -->
- [x] Non-breaking change (fix or new feature that would not break existing functionality).
- [x] Quick tests passed locally by running `./runtests.sh --quick --unittests  --disttests`.

---------

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com>
Signed-off-by: Rafael Garcia-Dias <rafaelagd@gmail.com>
Signed-off-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Rafael Garcia-Dias <rd24@lihe055-pc.isd.kcl.ac.uk>
Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test Refactor
4 participants