Flatfield correction and metadata (#150)

* fixed generate meta tests

* fixed inference script tests

* fixed metrics tests

* fixed preprocessing (masks, tile, mp aux)

* fixed evaluation metrics tests

* lots of issues with inference. fixed 2d so far

* fixed 2.5d inference, still working on 3d

* 3 remaining errors in 3d inference

* debugging 3d inference

* debugging 3d inference

* debugging 3d inference, fixed overlap shape

* fixed 5d tiling bug

* fixed 3d inference!

* fixed stitch tests

* fixed dataset tests

* fixed dataset w mask tests

* fixed inference dataset tests

* fixed plot utils tests

* added flatfield tests

* fixed gen mask tests

* working on tiling tests

* fixed tile nonuni tests

* fixed uniform tile tests

* fixed aux utils tests

* debugging image utils

* updated pandas version to avoid attribute error in pandas

* fixed image utils tests

* fixed mask utils tests

* fixed mp utils tests

* debugging tile utils

* fixed tile utils tests

* newer mpl version

* updated skimage, debugging flatfield

* sort output of os.listdir

* initial work on moving intensity computations

* added metadata check in preprocessing script

* adding tests for intensity meta

* simplified ints_meta_generator, added test

* debugging preprocessing tests

* fixed preprocess tests

* added mp utils test

* adding meta utils tests

* adding meta tests

* debugging zscore test

* fixed zscore test

* added preprocessing tests for meta gen and norm

* f
ixed plot utils test

* Master tests (#149)

* fixed generate meta tests

* fixed inference script tests

* fixed metrics tests

* fixed preprocessing (masks, tile, mp aux)

* fixed evaluation metrics tests

* lots of issues with inference. fixed 2d so far

* fixed 2.5d inference, still working on 3d

* 3 remaining errors in 3d inference

* debugging 3d inference

* debugging 3d inference

* debugging 3d inference, fixed overlap shape

* fixed 5d tiling bug

* fixed 3d inference!

* fixed stitch tests

* fixed dataset tests

* fixed dataset w mask tests

* fixed inference dataset tests

* fixed plot utils tests

* added flatfield tests

* fixed gen mask tests

* working on tiling tests

* fixed tile nonuni tests

* fixed uniform tile tests

* fixed aux utils tests

* debugging image utils

* updated pandas version to avoid attribute error in pandas

* fixed image utils tests

* fixed mask utils tests

* fixed mp utils tests

* debugging tile utils

* fixed tile utils tests

* newer mpl version

* updated skimage, debugging flatfield

* sort output of os.listdir

* Inference features (progressbar, define prediction folder name, metrics in figures) (#147)

* minor fix for preprocessing_script

* changed the layout of sub panels in predicted figures

* removed redundant slice margin adjustment

* disable the check for modelcheckpoint monitor

* Added flags to save predicted images in image directory or model directory

* Added data normalization options to preprocessing

* updated image_inference script with data normalization

* bug fix

* updated tests

* updated tests and 3d preprocessing

* added dataset otsu mask type

* added script for pooling multiple datasets

* created meta_utils; added multi-processing option to meta_generator; estimate dataset z-score parameters from foreground images only

* added function to sample values at block centers

* added blocks_meta.csv

* bug fix

* updated functions to sample pixels and compute zscore parameters

* Fixed the bug caused by mixed numpy datatypes; make data normalization backward compatible

* updated metrics_script to new normalization options
added get_pp_config function to preprocess_utils

* unzscore prediction before computing SSIM;
added multi-threading to tiling

* unzscore predition for 3D inference

* bug fix

* turned off normalization for reading 3D target images for computing metrics

* change output dtype to float32

* bug fix

* made metrics_script backward compatible

* Rename "workers" to "num_workers" in the training config

* fixed tests

* fixed tests

* added pool config file

* made Maskgenerator backward compatible

* Add large 2D inference

* bug fix

* computed metrics stats for single FOV

* update inference_script.py

* bug fix

* generate mask meta for user supplied masks; add watershed

* update config files

* update notebook

* update conda env yaml

* edit config

* edit notebook

* Add README for the notebook

* edit comment

* update comment blocks

* update comment blocks

* update README.md

* update notebook

* update README.md

* fix plots not displayed issue

* update README.md

* adding a shell script for setup

* shell script for setup

* update setup script

* update README.md

* update notebook

* edited the image translation exercies for clarity, added TOC, and added jupyterlab to conda environment config.

* update README

* update the paths to data to avoid conflict with 04_image_translation repo

* update instructions

* move README to course repo, clean up the notebook

* fix typos

* update paths for backup tiles

* bux fix

* update plotting

* add config for 2.5D model; add warning for too small min_fraction

* fix margin issue with 2.5D inference

* fix indexing issues in uniform tiling

* bug fix

* bug fix

* bug fix

* bug fix

* bug fix

* fix plotting bug

* fix tiling z indexing bug

* fix inference 2.5D model bug

* fix inference 2.5D model bug, cleaned

* inference: add progressbar, colorbar for figure-target, decrease margin in figure

* fix figure metric assignment

* fix inference input middle slice selection

* plot multiple inputs in figures

* add tqdm to requirements.txt file

* inference refactoring

* changed pd version, convert nan to none when reading meta

* fixed generate meta tests

* fixed inference script tests

* fixed metrics tests

* fixed preprocessing (masks, tile, mp aux)

* fixed evaluation metrics tests

* lots of issues with inference. fixed 2d so far

* fixed 2.5d inference, still working on 3d

* 3 remaining errors in 3d inference

* debugging 3d inference

* debugging 3d inference

* debugging 3d inference, fixed overlap shape

* fixed 5d tiling bug

* fixed 3d inference!

* fixed stitch tests

* fixed dataset tests

* fixed dataset w mask tests

* fixed inference dataset tests

* fixed plot utils tests

* added flatfield tests

* fixed gen mask tests

* working on tiling tests

* fixed tile nonuni tests

* fixed uniform tile tests

* fixed aux utils tests

* debugging image utils

* updated pandas version to avoid attribute error in pandas

* fixed image utils tests

* fixed mask utils tests

* fixed mp utils tests

* debugging tile utils

* fixed tile utils tests

* newer mpl version

* updated skimage, debugging flatfield

* sort output of os.listdir

* changed pd version, convert nan to none when reading meta

* making tests compatible with progress bar changes, still debugging 3d

* removed requirement to run xy metrics, fixed tests for 3d inference

* fixed plot utils test

* sorted glob output

Co-authored-by: Johanna Rahm <48733135+JohannaRahm@users.noreply.github.com>

* fixed inference script tests

* fixed metrics tests

* fixed preprocessing (masks, tile, mp aux)

* fixed evaluation metrics tests

* lots of issues with inference. fixed 2d so far

* fixed 2.5d inference, still working on 3d

* 3 remaining errors in 3d inference

* debugging 3d inference

* debugging 3d inference

* debugging 3d inference, fixed overlap shape

* fixed 5d tiling bug

* fixed 3d inference!

* working on tiling tests

* fixed aux utils tests

* updated skimage, debugging flatfield

* sort output of os.listdir

* initial work on moving intensity computations

* added metadata check in preprocessing script

* adding tests for intensity meta

* simplified ints_meta_generator, added test

* debugging preprocessing tests

* fixed preprocess tests

* added mp utils test

* adding meta utils tests

* adding meta tests

* debugging zscore test

* fixed zscore test

* added preprocessing tests for meta gen and norm

* updated docstring

* switched from None to NaN due to efficiency in pandas

* smguo feedback

* trying to fix protocol error

Co-authored-by: Johanna Rahm <48733135+JohannaRahm@users.noreply.github.com>

micro_dl/cli/dataset_pooling.py

@@ -38,7 +38,7 @@ def pool_dataset(config):
     num_workers = pool_config['num_workers']
     pool_mode = pool_config['pool_mode']
     frames_meta_dst_path = os.path.join(dst_dir, 'frames_meta.csv')
-    ints_meta_dst_path = os.path.join(dst_dir, 'ints_meta.csv')
+    ints_meta_dst_path = os.path.join(dst_dir, 'intensity_meta.csv')
     pos_idx_cur = 0
     os.makedirs(dst_dir, exist_ok=True)
     if os.path.exists(frames_meta_dst_path) and pool_mode == 'add':

micro_dl/cli/generate_meta.py

@@ -36,6 +36,12 @@ def parse_args():
         default=4,
         help="number of workers for multiprocessing",
     )
+    parser.add_argument(
+        '--block_size',
+        type=int,
+        default=256,
+        help="Pixel block size for intensity sampling",
+    )
     parser.add_argument(
         '--normalize_im',
         type=str,
@@ -46,19 +52,40 @@ def parse_args():
 
 
 def main(parsed_args):
-    meta_utils.frames_meta_generator(parsed_args.input,
-                                     parsed_args.order,
-                                     parsed_args.name_parser,
-                                     )
+    """
+    Generate metadata for each file by interpreting the file name.
+    Writes found data in frames_metadata.csv in input directory.
+    Assumed default file naming convention is:
+    dir_name
+    |
+    |- im_c***_z***_t***_p***.png
+    |- im_c***_z***_t***_p***.png
+
+    c is channel
+    z is slice in stack (z)
+    t is time
+    p is position (FOV)
+
+    Other naming convention is:
+    img_channelname_t***_p***_z***.tif for parse_sms_name
+
+    :param argparse parsed_args: Input arguments
+    """
+    # Collect metadata for all image files
+    meta_utils.frames_meta_generator(
+        input_dir=parsed_args.input,
+        order=parsed_args.order,
+        name_parser=parsed_args.name_parser,
+    )
+    # Compute intensity stats for all images
     if parsed_args.normalize_im in ['dataset', 'volume', 'slice']:
-        meta_utils.ints_meta_generator(parsed_args.input,
-                                       parsed_args.order,
-                                       parsed_args.name_parser,
-                                       parsed_args.num_workers,
-                                       )
+        meta_utils.ints_meta_generator(
+            input_dir=parsed_args.input,
+            num_workers=parsed_args.num_workers,
+            block_size=parsed_args.block_size,
+        )
 
 
 if __name__ == '__main__':
     parsed_args = parse_args()
     main(parsed_args)
-