Skip to content
This repository has been archived by the owner on Jul 29, 2023. It is now read-only.

Flatfield correction and metadata #150

Merged
merged 81 commits into from
May 31, 2022
Merged

Flatfield correction and metadata #150

merged 81 commits into from
May 31, 2022

Conversation

jennyfolkesson
Copy link
Contributor

@jennyfolkesson jennyfolkesson commented May 2, 2022

Here's another PR for you where:

  • intensity values are computed after flatfield correction in preprocessing
  • metadata is generated in preprocessing if no meta csv file is found in the input directory

Now that @JohannaRahm 's PR is merged with master I will rebase to master, then get the master_tests branch merged, then it's time for this one.

After this, I will start working on adding support for zarr input data.

@codecov
Copy link

codecov bot commented May 16, 2022

Codecov Report

Merging #150 (99bcd1d) into master (d321807) will decrease coverage by 4.19%.
The diff coverage is 67.56%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #150      +/-   ##
==========================================
- Coverage   80.59%   76.39%   -4.20%     
==========================================
  Files          50       53       +3     
  Lines        3756     4554     +798     
==========================================
+ Hits         3027     3479     +452     
- Misses        729     1075     +346     
Flag Coverage Δ
build 76.39% <67.56%> (-4.20%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
micro_dl/cli/dataset_pooling.py 0.00% <0.00%> (ø)
micro_dl/cli/train_script.py 0.00% <0.00%> (ø)
micro_dl/preprocessing/tile_3d.py 30.00% <ø> (ø)
micro_dl/preprocessing/tile_nonuniform_images.py 81.57% <ø> (ø)
micro_dl/train/trainer.py 0.00% <0.00%> (ø)
micro_dl/utils/train_utils.py 11.11% <0.00%> (ø)
micro_dl/train/losses.py 56.25% <12.50%> (-6.25%) ⬇️
micro_dl/networks/vqnet.py 25.33% <25.33%> (ø)
micro_dl/inference/stitch_predictions.py 63.33% <25.97%> (-26.84%) ⬇️
micro_dl/cli/inference_script.py 75.67% <50.00%> (-13.22%) ⬇️
... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b9d6045...99bcd1d. Read the comment docs.

@jennyfolkesson
Copy link
Contributor Author

@mattersoflight @smguo @JohannaRahm This PR is now ready for review. Thanks!

Copy link
Collaborator

@smguo smguo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @jennyfolkesson! The PR looks great. Please see my questions below.

np.save(file_name, np.squeeze(im_pred), allow_pickle=True)
else:
raise ValueError(
'Unsupported file extension: {}'.format(self.image_ext),
)
if self.save_figs:
if self.save_figs and self.image_ext != '.npy':
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't figures be saved if the file extension is .npy?

Copy link
Contributor Author

@jennyfolkesson jennyfolkesson May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. It's not the file format I was trying to catch but 3D images which are usually the ones saved as numpy format but that's totally unclear. I'll update to check for 3d images.

flat_field_dir,
'flat-field_channel-{}.npy'.format(channel_idx)
)
mp_fn_args.append((im_path, ff_path, block_size, meta_row))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks to me that if channel_ids is only a subset of channels in frames_metadata, then for channels not in channel_ids, their ff_path won't get updated. Is this the behavior you expect or I missed something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No that's not the behavior I want, thanks for catching that but. I'm now setting ff_path to None inside the for loop so no channels outside channel_ids flatfields should be messing up the calculations.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting ff_path to None inside the for loop makes sense to me. One more comment: It looks like flatfield correction share the parameter channel_ids as other preprocessing steps, but in most use cases we might apply flatfield correction to fluorescence channels but usually not to label-free channels. Should we make a separate channel_ids for flatfield correction? It can be done in the next PR if it requires more code changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! Let's create an issue and I'll take care of it in a separate PR.

Copy link
Collaborator

@smguo smguo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me! I have one more comment but it can be addressed in the future PR. Feel free to merge the PR.

@jennyfolkesson jennyfolkesson merged commit c5a3a8b into master May 31, 2022
@jennyfolkesson jennyfolkesson deleted the normalization branch May 31, 2022 19:56
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants