-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reloading of corrupted cached files for PersistentDataset #5723
Labels
Comments
sure please go ahead, and I think would be great to create a warning message if that error handling logic is triggered |
@wyli here is a proposed try-except block to handle this:
|
Hi @ishandutta0098, thanks for the proposal, would you like to create a PR? |
Sure @KumoLiu please assign the issue to me, I will start working on it |
7 tasks
KumoLiu
added a commit
that referenced
this issue
Nov 20, 2023
Fixes #5723 ### Description Corrupted cached files in the PersistentDataset cause the exception: `RuntimeError: Invalid magic number; corrupt file?` With this PR we handle that case in the try-except block and continue the usual functionality if the cached file was absent. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [x] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [x] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Ishan Dutta <ishandutta0098@gmail.com> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com>
marksgraham
pushed a commit
to marksgraham/MONAI
that referenced
this issue
Jan 30, 2024
…t-MONAI#7244) Fixes Project-MONAI#5723 ### Description Corrupted cached files in the PersistentDataset cause the exception: `RuntimeError: Invalid magic number; corrupt file?` With this PR we handle that case in the try-except block and continue the usual functionality if the cached file was absent. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [x] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [x] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Ishan Dutta <ishandutta0098@gmail.com> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Mark Graham <markgraham539@gmail.com>
juampatronics
pushed a commit
to juampatronics/MONAI
that referenced
this issue
Mar 25, 2024
…t-MONAI#7244) Fixes Project-MONAI#5723 ### Description Corrupted cached files in the PersistentDataset cause the exception: `RuntimeError: Invalid magic number; corrupt file?` With this PR we handle that case in the try-except block and continue the usual functionality if the cached file was absent. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [x] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [x] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Ishan Dutta <ishandutta0098@gmail.com> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Juan Pablo de la Cruz Gutiérrez <juampatronics@gmail.com>
Yu0610
pushed a commit
to Yu0610/MONAI
that referenced
this issue
Apr 11, 2024
…t-MONAI#7244) Fixes Project-MONAI#5723 ### Description Corrupted cached files in the PersistentDataset cause the exception: `RuntimeError: Invalid magic number; corrupt file?` With this PR we handle that case in the try-except block and continue the usual functionality if the cached file was absent. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [x] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [x] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Ishan Dutta <ishandutta0098@gmail.com> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Yu0610 <612410030@alum.ccu.edu.tw>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I use PersistentDataset in my train loop. I needed to abort train script and when I run it again, one of cached files was corrupted because of abortion. PersistentDataset did not finish saving this cache file.
It threw an exception
RuntimeError: Invalid magic number; corrupt file?
I would like to add a possibility to handle corrupt files. If cached file is corrupted, PersistenDataset will continue like there was no cached file and replace it with a good one.
I see, that if I add "except" block here, I can handle such behaviour.
May I take this ticket and make a PR?
The text was updated successfully, but these errors were encountered: