Add rectified flow noise scheduler for accelerated diffusion model #8374
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KumoLiu
approved these changes
Mar 10, 2025
guopengf
reviewed
Mar 10, 2025
Related to Project-MONAI#8274 , this PR is used to check potential issues. When I used the same environment as the nightly test, the error was not reproduced. Therefore, I hope the new change can show more information about the error. ### Description A few sentences describing the changes proposed in this pull request. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Yiheng Wang <vennw@nvidia.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes # . ### Description A few sentences describing the changes proposed in this pull request. ### Types of changes - [x] Non-breaking change (fix or new feature that would not break existing functionality). Fixing issue Project-MONAI#8284 In this format there are no duplicates: ``` t = [ (0, 0, 0), (1, 0, 0), (0, 1, 0), (0, 0, 1), (1, 0, 1), (1, 1, 0), (0, 1, 1), (1, 1, 1), ] print(set(t)) # {(1, 0, 1), (1, 1, 0), (0, 1, 0), (0, 0, 0), (1, 0, 0), (0, 0, 1), (1, 1, 1), (0, 1, 1)} ``` --------- Signed-off-by: pooya-mohammadi <pooyamohammadikazaj@gmail.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8274 . ### Description The new test has already tested with the same 24.08 + A100 env. I did some tests but cannot reproduce the original test case error (there are NaN values or significant small/large data). Since only 24.08 base image has the issue (24.10 does not have), I decided to use a different test case for 24.08 and prepared this PR ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Yiheng Wang <vennw@nvidia.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8298 ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8267 . ### Description Fix channel-wise intensity normalization for integer type inputs. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [ ] Non-breaking change (fix or new feature that would not break existing functionality). - [x] Breaking change (fix or new feature that would cause existing functionality to change). - [x] New tests added to cover the changes. - [x] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [x] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [x] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: advcu987 <adrianvoicu.tm@gmail.com> Signed-off-by: advcu <65158236+advcu987@users.noreply.github.com> Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8306 This previous api has been deprecated, update based on: https://docs.ngc.nvidia.com/api/?urls.primaryName=Private%20Artifacts%20(Models)%20API#/artifact-file-controller/downloadAllArtifactFiles ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8298 ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Related to Project-MONAI#8241 . ### Description A few sentences describing the changes proposed in this pull request. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Yiheng Wang <vennw@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
) Fixes Project-MONAI#8298. ### Description This includes the tests for the `compressor` argument when testing with Zarr before version 3.0 when this argument was deprecated. A fix to upgrade the version of `pycln` used is also included. The version of PyTorch is also fixed to below 2.6 to avoid issues with misuse of `torch.load` which must be addressed later. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Signed-off-by: Can-Zhao <canz@nvidia.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Related to Project-MONAI#8185. ### Description This changes the name of `tests/utils.py` to `tests/test_utils.py` to conform to the changes introduced with the daily CICD tests. This is done in Project-MONAI#8231 but the change is being pre-merged now to get tests working again while issues are sorted out there. This also includes changes this PR made to that file, and changes anywhere else to correctly import the module. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8185 ### Description ## Reorganize tests I have looked at the imports in each test file and the test title to identify which files were being tested. I mirrored the file structure of MONAI on the `tests` folder and moved the files accordingly. I used some helper scripts, but the process required substantial manual intervention. When uncertain, I moved the tests to the `integration` folder since the confusion always involved many imports, and I could not find clarity from the test name. Please review the integration folder carefully, which is the one that I feel the least confident about. ``` ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [x] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. --------- Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com> Signed-off-by: Rafael Garcia-Dias <rafaelagd@gmail.com> Signed-off-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Rafael Garcia-Dias <rd24@lihe055-pc.isd.kcl.ac.uk> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8186. ### Description A few sentences describing the changes proposed in this pull request. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: Ben Murray <ben.murray@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
…val (Project-MONAI#8296) ### Description This bumps the minimum required `torch` version from 1.9.0 to 1.13.1. See GHSA-47fc-vmwq-366v for more details such as the highest severity scoring of "Critical". - https://nvd.nist.gov/vuln/detail/CVE-2022-45907 - https://security.snyk.io/vuln/SNYK-PYTHON-TORCH-3149871 Maintainers will need to update the required status checks for the [`dev`](https://github.com/Project-MONAI/MONAI/tree/dev) branch to: - Remove min-dep-pytorch (1.10.2) - Remove min-dep-pytorch (1.11.0) - Remove min-dep-pytorch (1.12.1) - Remove min-dep-pytorch (1.13) - Add min-dep-pytorch (1.13.1) cc: @KumoLiu ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Breaking change (fix or new feature that would cause existing functionality to change). (drop of older `torch` versions) - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. --------- Signed-off-by: James Butler <james.butler@revvity.com> Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
…djustment in latent diffusion inferers (Project-MONAI#8343) Fixes Project-MONAI#8334 ### Description There was an if save_intermediates missing in the code that was trying to run crop of the latent spaces on the sample function of the Latent Diffusion Inferers (normal one and ControlNet one) even when intermediates aren't created. ### Types of changes - [x] Non-breaking change (fix or new feature that would not break existing functionality). --------- Signed-off-by: Virginia Fernandez <virginia.fernandez@kcl.ac.uk> Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: Virginia Fernandez <virginia.fernandez@kcl.ac.uk> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8349 ### Description The current behaviour is that `pkging, has_ver = optional_import("packaging.Version")` always returns `has_ver=False` because the import always fails (the `Version` class is exposed by the `packaging.version` submodule). This issue previously didn't surface, because when the import fails, it would just continue to use the fallback logic. However, there seem to be more hidden and more severe implications, which ultimately led me to discovering this particular bug: Function like `floor_divide()` in `monai.transforms` that check the module version using this logic are called many times in common ML dataloading workflows. The failed imports somehow can lead to OOM errors and the main process being killed (see Project-MONAI#8348). Maybe when `optional_import` fails to import a module, the lazy exceptions somehow stack up in memory when this function is called many times in a short time period? ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. Signed-off-by: Can-Zhao <volcanofly@gmail.com>
…ject-MONAI#8359) Fixes Project-MONAI#8348 ### Description Support for `torch` versions prior to `1.13` has been dropped, so those `1.8` version checks are not required anymore. Furthermore, as reported in the issue description, those checks led to unstable behaviour when using certain transforms in data pipelines. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Nicolas Kaenzig <nkaenzig@gmail.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
### Description `CommonKeys()` docstring mentions `INFO` which doesn't exist. Instead there is a `METADATA` field, so the docstring was updated accordingly. ### Types of changes - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. Signed-off-by: Bartosz Grabowski <58475557+bartosz-grabowski@users.noreply.github.com> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8085. ### Description Average Precision is very similar to ROCAUC, so I was very much inspired by the ROCAUC implementation. More precisely, I created: - `AveragePrecisionMetric` and `compute_average_precision` in `monai.metrics`, - a handler called `AveragePrecision` in `monai.handlers`, - three unittest modules: `test_compute_average_precision.py`, `test_handler_average_precision.py` and `test_handler_average_precision_dist.py`. I also modified the docs to mention Average Precision. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [x] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [x] In-line docstrings updated. - [x] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: thibaultdvx <thibault.devarax@icm-institute.org> Signed-off-by: Thibault de Varax <154365476+thibaultdvx@users.noreply.github.com> Signed-off-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Fixes Project-MONAI#8354 ### Description Fixes path on test that is only run on special conditions. ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [x] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: R. Garcia-Dias <rafaelagd@gmail.com> Co-authored-by: YunLiu <55491388+KumoLiu@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
When there is no need to perform tensor parallel, skip all the unnecessary splitting steps Signed-off-by: Can-Zhao <volcanofly@gmail.com>
for more information, see https://pre-commit.ci Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
…github.com> I, Can Zhao <69829124+Can-Zhao@users.noreply.github.com>, hereby add my Signed-off-by to this commit: efdc623 I, Can Zhao <69829124+Can-Zhao@users.noreply.github.com>, hereby add my Signed-off-by to this commit: 131f7e6 Signed-off-by: Can Zhao <69829124+Can-Zhao@users.noreply.github.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
for more information, see https://pre-commit.ci Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Signed-off-by: monai-bot <monai.miccai2019@gmail.com> Signed-off-by: monai-bot <monai.miccai2019@gmail.com> Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Partially addresses Project-MONAI#8303. This changes the maximum Numpy version to be below 3.0 for testing with 2.x compatibility. This appears to be resolved with newer versions of dependencies. This will also include fixes for Pytorch 2.6 mostly relating to `torch.load` and `autocast` usage. <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. --------- Signed-off-by: Eric Kerfoot <eric.kerfoot@kcl.ac.uk> Signed-off-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
There was a problem hiding this comment.
I am not very familiar with this type of method. From the code that has been modified, I assume this scheduler is not meant to be used to train diffusion models, but merely sampling from them (such as PNDM)?
Even in this case, they could be used with latent diffusion models and controlnets, right? Could these inferers be tested with this scheduler as well?
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
for more information, see https://pre-commit.ci
Signed-off-by: Can-Zhao <volcanofly@gmail.com>
Thank you! I resolved the conflict |
|
/build |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes # .
Description
A few sentences describing the changes proposed in this pull request.
Types of changes
./runtests.sh -f -u --net --coverage../runtests.sh --quick --unittests --disttests.make htmlcommand in thedocs/folder.