Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude echo entity from optimally combined derivatives #3166

Merged
merged 9 commits into from
Dec 6, 2023

Conversation

tsalo
Copy link
Collaborator

@tsalo tsalo commented Dec 1, 2023

Closes #3165.

Changes proposed in this pull request

  • Ensure that the echo entity is removed from CIFTI optimally combined derivatives.
  • Ensure that the echo entity is retained in derivatives when --echo-idx is used to process multi-echo data like single-echo.
    • I initially ran the ds210 test with --echo-idx 1 to determine if echo was being retained in the filenames. It wasn't, but the dismiss_echo function fixed that.
  • Add function, fmriprep.utils.bids.dismiss_echo, to include echo in dismiss_entities parameters only when applicable, based on the config.

Documentation that should be reviewed

Copy link

codecov bot commented Dec 1, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (3c836e8) 72.28% compared to head (1253d28) 72.37%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3166      +/-   ##
==========================================
+ Coverage   72.28%   72.37%   +0.08%     
==========================================
  Files          55       55              
  Lines        4085     4098      +13     
==========================================
+ Hits         2953     2966      +13     
  Misses       1132     1132              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tsalo
Copy link
Collaborator Author

tsalo commented Dec 1, 2023

The test fails because of the following:

< sub-02/anat/sub-02_desc-brain_mask.json
< sub-02/anat/sub-02_desc-brain_mask.nii.gz
< sub-02/anat/sub-02_desc-preproc_T1w.json
< sub-02/anat/sub-02_desc-preproc_T1w.nii.gz
< sub-02/anat/sub-02_dseg.nii.gz
< sub-02/anat/sub-02_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5
< sub-02/anat/sub-02_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5
< sub-02/anat/sub-02_label-CSF_probseg.nii.gz
< sub-02/anat/sub-02_label-GM_probseg.nii.gz
< sub-02/anat/sub-02_label-WM_probseg.nii.gz
22a13
> sub-02/anat/sub-02_space-MNI152NLin2009cAsym_desc-preproc_dseg.nii.gz
25,28c16,18
< sub-02/anat/sub-02_space-MNI152NLin2009cAsym_dseg.nii.gz
< sub-02/anat/sub-02_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz
< sub-02/anat/sub-02_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz
< sub-02/anat/sub-02_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz
---
> sub-02/anat/sub-02_space-MNI152NLin2009cAsym_label-CSF_desc-preproc_probseg.nii.gz
> sub-02/anat/sub-02_space-MNI152NLin2009cAsym_label-GM_desc-preproc_probseg.nii.gz
> sub-02/anat/sub-02_space-MNI152NLin2009cAsym_label-WM_desc-preproc_probseg.nii.gz
36,37d25
< sub-02/func/sub-02_task-cuedSGT_run-01_desc-brain_mask.json
< sub-02/func/sub-02_task-cuedSGT_run-01_desc-brain_mask.nii.gz
44,49d31
< sub-02/func/sub-02_task-cuedSGT_run-01_echo-1_desc-preproc_bold.json
< sub-02/func/sub-02_task-cuedSGT_run-01_echo-1_desc-preproc_bold.nii.gz
< sub-02/func/sub-02_task-cuedSGT_run-01_echo-2_desc-preproc_bold.json
< sub-02/func/sub-02_task-cuedSGT_run-01_echo-2_desc-preproc_bold.nii.gz
< sub-02/func/sub-02_task-cuedSGT_run-01_echo-3_desc-preproc_bold.json
< sub-02/func/sub-02_task-cuedSGT_run-01_echo-3_desc-preproc_bold.nii.gz
60,61d41
< sub-02/func/sub-02_task-cuedSGT_run-01_space-MNI152NLin2009cAsym_T2starmap.json
< sub-02/func/sub-02_task-cuedSGT_run-01_space-MNI152NLin2009cAsym_T2starmap.nii.gz

From what I can tell, the T2starmap and echo-wise outputs are not generated, which is what I would expect. However, the other functional outputs don't have the echo entity, which means that the outputs are not correctly named when using --echo-idx.

I'm not sure why there are differences in the anatomical outputs.

EDIT: Ah, I think I understand why native-space outputs weren't generated when --echo-idx was used. It must be because of --me-output-echos!

Still, why did the MNI152NLin2009cAsym-space dseg and probseg files have desc-preproc when --echo-idx was used, but no desc field when it wasn't?

@tsalo tsalo requested a review from effigies December 4, 2023 17:24
@tsalo tsalo marked this pull request as ready for review December 4, 2023 17:24
@tsalo tsalo requested a review from madisoth December 4, 2023 17:24
Copy link
Member

@effigies effigies left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not in love with the name. The intuitive meaning of dismiss_echo(["part"]) is not obviously ["part", "echo"] or ["part"], but I also haven't come up with a better name for the function. We can change the name if it ever becomes necessary.

fmriprep/utils/bids.py Outdated Show resolved Hide resolved
@effigies effigies merged commit d24f585 into nipreps:master Dec 6, 2023
8 of 9 checks passed
@effigies effigies added this to the 23.2.0 milestone Dec 6, 2023
@madisoth
Copy link
Collaborator

madisoth commented Dec 7, 2023

Just tested this and can confirm it ran without issues.

@tsalo
Copy link
Collaborator Author

tsalo commented Dec 7, 2023

Thanks @madisoth!

@tsalo tsalo deleted the cifti-optcom-fname branch December 7, 2023 20:00
effigies added a commit that referenced this pull request Jan 10, 2024
23.2.0 (January 10, 2024)

New feature release in the 23.2.x series.

This release wraps up a significant refactor of fMRIPrep. The main new features
can be used with the ``--level`` and ``--derivatives`` flags.

The ``--level`` flag can take the arguments ``minimal``, ``resampling`` or
``full``. The default is ``full``, which should produce nearly the same results
as previous versions. ``minimal`` will produce only the minimum necessary to
deterministically generate the remaining derivatives. ``resampling`` will produce
some additional derivatives, intended to simplify resampling with other tools.

The ``--derivatives`` flag takes arguments of the form ``name=/path/to/dir``,
for example ``--derivatives anat=$SMRIPREP_DIR``.  If provided, fMRIPrep will
read the specified directories for pre-computed derivatives. If a derivative is
found, it will be used instead of computing it from scratch. If a derivative is
not found, fMRIPrep will compute it and proceed as usual.

Taken together, these features can allow a dataset provider to run a minimal
fMRIPrep run, targeting many output spaces, while a user can then run a
``--derivatives`` run to generate additional derivatives in only the output
spaces they need. Another use case is to provide an precomputed derivative
to override the default fMRIPrep behavior, enabling easier workarounds for
bugs or experimentation with alternatives.

Additionally, this release includes a number of bug fixes and improvements.
This release adds support for MSM-Sulc, improving the alignment of subject
surfaces to the fsLR template. This process is enabled by default, but may
be disabled with the ``--no-msm`` flag.

This release resolves a number of issues with fieldmaps inducing distortions
during correction. Phase difference and direct fieldmaps are now masked correctly,
preventing the overestimation of distortions outside the brain. Additionally,
we now implement Jacobian weighting during unwarping, which corrects for compression
and expansion effects on signal intensity. To disable Jacobian weighting, use
``--ignore fmap-jacobian``.

Finally, a new resampling method has been added, to better account for
susceptibility distortion and motion in a single shot resampling to a volumetric
target space. We anticipate extending this to surface targets in the future.

* FIX: Restore --ignore sbref functionality (#3180)
* FIX: Retrieve atlas ROIs at requested density (#3179)
* FIX: Keep minctracc executable in FreeSurfer installation (#3175)
* FIX: Exclude echo entity from optimally combined derivatives (#3166)
* FIX: Disable boldref-space outputs unless requested (#3159)
* FIX: Tag memory estimates in resamplers (#3150)
* FIX: Final revisions for next branch (#3134)
* FIX: Minor fixes to work with MSMSulc-enabled smriprep-next (#3098)
* FIX: Connect EPI-to-fieldmap transform (#3099)
* FIX: Use Py2-compatible version file template for fmriprep-docker (#3101)
* FIX: Update connections to unwarp_wf, convert ITK transforms to text (#3077)
* ENH: Allow --ignore fmap-jacobian to disable Jacobian determinant modulation during fieldmap correction (#3186)
* ENH: Exclude non-steady-state volumes from confound correlation plot (#3171)
* ENH: Pass FLAIR images to anatomical workflow builder to include in boilerplate (#3146)
* ENH: Restore carpetplot and other final adjustments (#3131)
* ENH: Restore CIFTI-2 generation (#3129)
* ENH: Restore resampling to surface GIFTIs (#3126)
* ENH: Restore confound generation (#3120)
* ENH: Restore resampling BOLD to volumetric templates (#3121)
* ENH: Restore resampling to T1w target (#3116)
* ENH: Add MSMSulc (#3085)
* ENH: Add reporting workflow for BOLD fit (#3082)
* ENH: Generate anatomical derivatives useful for resampling (#3081)
* RF: Load reportlets interfaces from nireports rather than niworkflows (#3176, #3184)
* RF: Separate goodvoxels mask creation from fsLR resampling (#3170)
* RF: Write out anatomical template derivatives (#3136)
* RF: Update primary bold workflow to incorporate single shot resampling (#3114)
* RF: Update derivative cache spec, calculate per-BOLD, reuse boldref2fmap (#3078)
* RF: Split fMRIPrep into fit and derivatives workflows (#2913)
* RPT: Rename CSF/WM confounds in fMRIPlot (#3172)
* TST: Add smoke tests for full workflow and most branching flags (#3155)
* TST: Add smoke-tests for bold_fit_wf (#3152)
* DOC: Fix documentation and description for init_bold_grayords_wf (#3051)
* DOC: Minor updates in outputs.rst (#3148)
* STY: Apply a couple refurb suggestions (#3151)
* STY: Fix flake8 warnings (#3044)
* STY: Apply pyupgrade suggestions (#3043)
* MNT: Restore mritotal subcommands to Dockerfile (#3149)
* MNT: Update smriprep to 0.13.1 (#3153)
* MNT: optimise size of PNG files (#3145)
* MNT: update vendored docs script ``github_link.py`` (#3144)
* MNT: Update tedana pin, test on Python 3.12 (#3141)
* MNT: Bump environment (#3132)
* MNT: Bump version requirements (#3107)
* MNT: http:// → https:// (#3097)
* MNT: Remove mritotal and dependencies from FreeSurfer ignore file (#3090)
* MNT: Update environment (#3073)
* MNT: Depend on newer sphinx (#3067)
* MNT: Install ANTs from conda-forge (#3061)
* MNT: Drop Python 3.8 and numpy 1.21 support (NEP29) (#3052)
* MNT: update update_zenodo.py script (#3042)
* MNT: Fix welcome message formatting and instructions (#3039)
* MNT: Python 3.11 should be supported (#3038)
* CI: Bump actions/setup-python from 4 to 5 (#3181)
* CI: Stop testing legacy layout (#3079)
* CI: Improve tag detection for docker builds (#3066)
* CI: Clean up pre-release builds (#3040)
NingAnMe added a commit to NingAnMe/fmriprep that referenced this pull request Jan 11, 2024
23.2.0 (January 10, 2024)

New feature release in the 23.2.x series.

This release wraps up a significant refactor of fMRIPrep. The main new features
can be used with the ``--level`` and ``--derivatives`` flags.

The ``--level`` flag can take the arguments ``minimal``, ``resampling`` or
``full``. The default is ``full``, which should produce nearly the same results
as previous versions. ``minimal`` will produce only the minimum necessary to
deterministically generate the remaining derivatives. ``resampling`` will produce
some additional derivatives, intended to simplify resampling with other tools.

The ``--derivatives`` flag takes arguments of the form ``name=/path/to/dir``,
for example ``--derivatives anat=$SMRIPREP_DIR``.  If provided, fMRIPrep will
read the specified directories for pre-computed derivatives. If a derivative is
found, it will be used instead of computing it from scratch. If a derivative is
not found, fMRIPrep will compute it and proceed as usual.

Taken together, these features can allow a dataset provider to run a minimal
fMRIPrep run, targeting many output spaces, while a user can then run a
``--derivatives`` run to generate additional derivatives in only the output
spaces they need. Another use case is to provide an precomputed derivative
to override the default fMRIPrep behavior, enabling easier workarounds for
bugs or experimentation with alternatives.

Additionally, this release includes a number of bug fixes and improvements.
This release adds support for MSM-Sulc, improving the alignment of subject
surfaces to the fsLR template. This process is enabled by default, but may
be disabled with the ``--no-msm`` flag.

This release resolves a number of issues with fieldmaps inducing distortions
during correction. Phase difference and direct fieldmaps are now masked correctly,
preventing the overestimation of distortions outside the brain. Additionally,
we now implement Jacobian weighting during unwarping, which corrects for compression
and expansion effects on signal intensity. To disable Jacobian weighting, use
``--ignore fmap-jacobian``.

Finally, a new resampling method has been added, to better account for
susceptibility distortion and motion in a single shot resampling to a volumetric
target space. We anticipate extending this to surface targets in the future.

* FIX: Restore --ignore sbref functionality (nipreps#3180)
* FIX: Retrieve atlas ROIs at requested density (nipreps#3179)
* FIX: Keep minctracc executable in FreeSurfer installation (nipreps#3175)
* FIX: Exclude echo entity from optimally combined derivatives (nipreps#3166)
* FIX: Disable boldref-space outputs unless requested (nipreps#3159)
* FIX: Tag memory estimates in resamplers (nipreps#3150)
* FIX: Final revisions for next branch (nipreps#3134)
* FIX: Minor fixes to work with MSMSulc-enabled smriprep-next (nipreps#3098)
* FIX: Connect EPI-to-fieldmap transform (nipreps#3099)
* FIX: Use Py2-compatible version file template for fmriprep-docker (nipreps#3101)
* FIX: Update connections to unwarp_wf, convert ITK transforms to text (nipreps#3077)
* ENH: Allow --ignore fmap-jacobian to disable Jacobian determinant modulation during fieldmap correction (nipreps#3186)
* ENH: Exclude non-steady-state volumes from confound correlation plot (nipreps#3171)
* ENH: Pass FLAIR images to anatomical workflow builder to include in boilerplate (nipreps#3146)
* ENH: Restore carpetplot and other final adjustments (nipreps#3131)
* ENH: Restore CIFTI-2 generation (nipreps#3129)
* ENH: Restore resampling to surface GIFTIs (nipreps#3126)
* ENH: Restore confound generation (nipreps#3120)
* ENH: Restore resampling BOLD to volumetric templates (nipreps#3121)
* ENH: Restore resampling to T1w target (nipreps#3116)
* ENH: Add MSMSulc (nipreps#3085)
* ENH: Add reporting workflow for BOLD fit (nipreps#3082)
* ENH: Generate anatomical derivatives useful for resampling (nipreps#3081)
* RF: Load reportlets interfaces from nireports rather than niworkflows (nipreps#3176, nipreps#3184)
* RF: Separate goodvoxels mask creation from fsLR resampling (nipreps#3170)
* RF: Write out anatomical template derivatives (nipreps#3136)
* RF: Update primary bold workflow to incorporate single shot resampling (nipreps#3114)
* RF: Update derivative cache spec, calculate per-BOLD, reuse boldref2fmap (nipreps#3078)
* RF: Split fMRIPrep into fit and derivatives workflows (nipreps#2913)
* RPT: Rename CSF/WM confounds in fMRIPlot (nipreps#3172)
* TST: Add smoke tests for full workflow and most branching flags (nipreps#3155)
* TST: Add smoke-tests for bold_fit_wf (nipreps#3152)
* DOC: Fix documentation and description for init_bold_grayords_wf (nipreps#3051)
* DOC: Minor updates in outputs.rst (nipreps#3148)
* STY: Apply a couple refurb suggestions (nipreps#3151)
* STY: Fix flake8 warnings (nipreps#3044)
* STY: Apply pyupgrade suggestions (nipreps#3043)
* MNT: Restore mritotal subcommands to Dockerfile (nipreps#3149)
* MNT: Update smriprep to 0.13.1 (nipreps#3153)
* MNT: optimise size of PNG files (nipreps#3145)
* MNT: update vendored docs script ``github_link.py`` (nipreps#3144)
* MNT: Update tedana pin, test on Python 3.12 (nipreps#3141)
* MNT: Bump environment (nipreps#3132)
* MNT: Bump version requirements (nipreps#3107)
* MNT: http:// → https:// (nipreps#3097)
* MNT: Remove mritotal and dependencies from FreeSurfer ignore file (nipreps#3090)
* MNT: Update environment (nipreps#3073)
* MNT: Depend on newer sphinx (nipreps#3067)
* MNT: Install ANTs from conda-forge (nipreps#3061)
* MNT: Drop Python 3.8 and numpy 1.21 support (NEP29) (nipreps#3052)
* MNT: update update_zenodo.py script (nipreps#3042)
* MNT: Fix welcome message formatting and instructions (nipreps#3039)
* MNT: Python 3.11 should be supported (nipreps#3038)
* CI: Bump actions/setup-python from 4 to 5 (nipreps#3181)
* CI: Stop testing legacy layout (nipreps#3079)
* CI: Improve tag detection for docker builds (nipreps#3066)
* CI: Clean up pre-release builds (nipreps#3040)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimally combined ME-EPI CIFTI derivatives incorrectly have "echo" entity in filename
3 participants