Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Ignore the 'CUDA_VISIBLE_DEVICE' setting after importing Monai in the latest Pytorch #7575

Closed
KumoLiu opened this issue Mar 25, 2024 · 0 comments
Closed

Ignore the 'CUDA_VISIBLE_DEVICE' setting after importing Monai in the latest Pytorch #7575

KumoLiu opened this issue Mar 25, 2024 · 0 comments

Comments

@KumoLiu
Copy link
Contributor

KumoLiu commented Mar 25, 2024 

FAIL: test_visible_devices (tests.test_set_visible_devices.TestVisibleDevices)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/MONAI/tests/test_set_visible_devices.py", line 37, in test_visible_devices
    self.assertEqual(num_gpus_before, num_gpus_after)
AssertionError: 0 != 2

The root cause could be issues listed here:
https://github.com/pytorch/pytorch/issues/122085
KumoLiu added a commit to KumoLiu/MONAI that referenced this issue Mar 25, 2024
@KumoLiu
workaround for Project-MONAI#7575
b87ae5e 
Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
@KumoLiu KumoLiu mentioned this issue Mar 25, 2024
Merged
7 tasks
KumoLiu added a commit to KumoLiu/MONAI that referenced this issue Mar 25, 2024
@KumoLiu
workaround for Project-MONAI#7575
9319a3c 
Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
KumoLiu added a commit to KumoLiu/MONAI that referenced this issue Mar 26, 2024
@KumoLiu
workaround for Project-MONAI#7575
30131db 
Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
@KumoLiu KumoLiu mentioned this issue Mar 26, 2024
Merged
7 tasks
KumoLiu added a commit that referenced this issue Mar 27, 2024
@KumoLiu @ericspod
Remove device count cache when import monai (#7581)
2716b6a 
workaround for #7575


### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
KumoLiu added a commit that referenced this issue Apr 1, 2024
@KumoLiu
Utilizing subprocess for nnUNet training. (#7576)
15d2abf 
Workaround for #7575 

### Description
- Due to the impact of #7575, the operation to set the device within
nnUNetV2Runner will become ineffective. This PR is intended to resolve
this issue.
- Add a version check for #7575, will revisit after the update from
pytorch team.

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Yu0610 pushed a commit to Yu0610/MONAI that referenced this issue Apr 11, 2024
@KumoLiu @ericspod @Yu0610
Remove device count cache when import monai (Project-MONAI#7581)
fc651fa 
workaround for Project-MONAI#7575

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>
Signed-off-by: Yu0610 <612410030@alum.ccu.edu.tw>
@KumoLiu KumoLiu closed this as completed Apr 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@KumoLiu