Update readme with directory structure

README.md

@@ -17,6 +17,7 @@ This repo is currently in beta testing. None of the code has been published yet,
   - [Setting up the virtual environment](#setting-up-the-virtual-environment)
   - [Using the repo](#using-the-repo)
   - [Updating the repo](#updating-the-repo)
+- [Directory structure](#directory-structure)
 - [Panel format](#panel-format)
 - [Median Pulse Height](#median-pulse-height)
 - [Questions?](#questions)
@@ -25,7 +26,7 @@ This repo is currently in beta testing. None of the code has been published yet,
 The repo has four main parts, with associated code and jupyter notebooks for each.
 
 ### 1. Using toffy for the first time
-The first time you use toffy on one of the commercial instruments, you'll need to perform some basic tasks to ensure everything is working properly. The [set up](./templates/1_set_up_toffy.ipynb) jupyter notebook will guide you through this process
+The first time you use toffy on one of the commercial instruments, you'll need to perform some basic tasks to ensure everything is working properly. The [set up](./templates/1_set_up_toffy.ipynb) jupyter notebook will guide you through this process and the resulting directory structure is explained below ([directory structure](#directory-structure)).
 
 ### 2. Setting up a MIBI run 
 For large MIBI runs, it is often convenient to automatically generate the JSON file containing the individual FOVs. There are two notebooks for this task, one for large tiled regions, the second for TMAs. If you will be tiling multiple adjacent FOVs together into a single image, the [tiling](./templates/2_create_tiled_mibi_run.ipynb) notebook can automate this process. You provide the location of the top corner of the tiled region, along with the number of fovs along the rows and columns, and it will automatically create the appropriate JSON file. 
@@ -146,6 +147,35 @@ conda remove --name toffy_env --all
 conda env create -f environment.yml
 ```
 
+## Directory structure
+Data from each run on the MIBI will be stored in the default base directory `D:\\Data`, in a subdirectory labeled with the run name.
+The [set up](./templates/1_set_up_toffy.ipynb) jupyter notebook creates the following folders that will be used throughout toffy.
+<br> <br>
+Four new folders are created on the D drive:
+- `Extracted_Images`: stores the raw images extracted from the bin files in either the [MIBI monitoring notebook](./templates/3a_monitor_MIBI_run.ipynb) or the [extraction notebook](./templates/3b_extract_images_from_bin.ipynb).
+- `Rosetta_Compensated_Images`: contains the images after being processed with the Rosetta algorithm in the [compensate image data notebook](./templates/4a_compensate_image_data.ipynb)
+- `Normalized_Images`: contains the images after accounting for intensity normalization in the [normalization notebook](./templates/4b_normalize_image_data.ipynb)
+- `Cohorts`: final location of your completely processed image data after renaming the folders and files in the [renaming and reorganizing notebook](./templates/5_rename_and_reorganize.ipynb)
+
+![alt text](templates/img/D_dirs.png)
+<br><br>
+Within `C:\\Users\\Customer.ION\\Documents` are directories that store necessary files used to set up and monitor a MIBI run.
+- `tiled_image_jsons`: stores all files used to set up a tiled run in the [tiling notebook](./templates/2_create_tiled_mibi_run.ipynb)
+- `autolabeled_tma_jsons`: stores all files used to set up a tma run in the [tma notebook](./templates/2_create_tma_mibi_run.ipynb)
+- `panel_files`: directory containing the run panel file, needed for notebooks 3a, 3b, 4a, and 4b.
+- `run_metrics`: contains the data files produced by the [QC](./templates/3c_generate_qc_metrics.ipynb) and [MPH notebooks](./templates/3d_compute_median_pulse_height.ipynb)
+- `watcher_logs`: contains the log file of FOVs which have been processed in the [monitoring notebook](./templates/3a_monitor_MIBI_run.ipynb)
+- `normalization_curve`: directory which stores the normalization curve file for the machine that was produced by the [set up notebook](./templates/1_set_up_toffy.ipynb) and necessary for [notebook 4b](./templates/4b_normalize_image_data.ipynb)
+
+![alt text](templates/img/C_dirs.gif)
+<br><br>
+You can see below how to pin a folder to Quick Access, which can then be easily located in the section of the same name on the left side of File Explorer. 
+
+We suggest pinning the following folders: `tiled_image_jsons`, `autolabeled_jsons`, `run_metrics`.
+
+![alt text](templates/img/quick_access.gif)
+
+
 ## Panel format
 Many of the scripts in `toffy` require a panel file. This file identifies which targets have been put on which masses. For an example of what the format should be, you can look at the [example panel file](https://github.com/angelolab/toffy/blob/main/files/example_panel_file.csv). Some panels will not have targets on every mass; in this case, it's important that you just leave the placeholder row in the panel, and not delete it, in order to ensure that all the notebooks work as expected. Similarly, if you have multiple targets on the same mass, don't add a unique row for each, just give them a consolidated name. 
 

templates/1_set_up_toffy.ipynb

@@ -52,7 +52,7 @@
     "**If this is the first time toffy has been installed on your CAC, run the following cell to generate the folders necessary.**\n",
     "\n",
     "\n",
-    "Read more about the overall directory structure, and the purpose of each one here (**LINK**)."
+    "More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
    ]
   },
   {
@@ -269,7 +269,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.8.13 ('toffy_env')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },

templates/2_create_tiled_mibi_run.ipynb

@@ -51,7 +51,7 @@
    "source": [
     "Define the following input and output paths:\n",
     "\n",
-    "* `json_tiling_dir`: the directory containing the information to read and write the FOV info\n",
+    "* `json_tiling_dir`: the directory created in notebook one which contains the information to read and write the FOV info <br> (see the [README](https://github.com/angelolab/toffy#directory-structure) for more details)\n",
     "* `region_corners_path`: the JSON file containing the FOVs defining the top-left corner of each region\n",
     "* `tiled_region_fovs_path`: where to store JSON defining the FOVs for each tiled region\n",
     "* `moly_path`: the path to the Moly point, needed if you want to insert between FOVs"
@@ -166,9 +166,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "toffy_env",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "toffy_env"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {

templates/2_create_tma_mibi_run.ipynb

@@ -86,7 +86,7 @@
     "* `manual run file`: this file, which contains the manually selected FOVs from your TMA, should be named `tma_name_manual.json`.\n",
     "* `optical image file`: this file, which contains the image of your slide, should be named either `tma_name.bmp` or `tma_name.jpg`. **This file can be found on the CAC at:** `D:\\\\Data\\\\optical-image\\\\`\n",
     "\n",
-    "Each of these files should be copied to `C:\\\\Users\\\\Customer.ION\\\\Documents\\\\autolabeled_tma_jsons`"
+    "Each of these files should be copied to `C:\\\\Users\\\\Customer.ION\\\\Documents\\\\autolabeled_tma_jsons`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
    ]
   },
   {
@@ -338,7 +338,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.12"
+   "version": "3.8.13"
   }
  },
  "nbformat": 4,

templates/3a_monitor_MIBI_run.ipynb

@@ -58,20 +58,29 @@
     "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Everything necessary for and subsequently outputted from this notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
+    "# load panel\n",
+    "panel = pd.read_csv(panel_path)\n",
+    "\n",
     "# these are set automatically\n",
     "base_dir = os.path.join('D:\\\\Data', run_name)\n",
+    "extraction_dir = os.path.join('D:\\\\Extracted_Images', run_name)\n",
+    "metrics_dir = os.path.join('C:\\\\Users\\\\Customer.ION\\\\Documents\\\\run_metrics', run_name)\n",
     "\n",
     "# path to log folder\n",
-    "log_path = os.path.join('C:\\\\Users\\\\Customer.ION\\\\Documents\\\\watcher_logs', run_name)\n",
-    "\n",
-    "# load panel\n",
-    "panel = pd.read_csv(panel_path)"
+    "log_path = os.path.join('C:\\\\Users\\\\Customer.ION\\\\Documents\\\\watcher_logs', run_name)"
    ]
   },
   {
@@ -101,9 +110,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "extraction_dir = os.path.join('D:\\\\Extracted_Images', run_name)\n",
-    "metrics_dir = os.path.join('C:\\\\Users\\\\Customer.ION\\\\Documents\\\\run_metrics', run_name)\n",
-    "\n",
     "fov_callback, run_callback = build_callbacks(\n",
     "    run_callbacks = ['plot_qc_metrics', 'plot_mph_metrics', 'image_stitching'],\n",
     "    fov_callbacks = ['extract_tiffs'],\n",
@@ -127,9 +133,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "toffy_env",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "toffy_env"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {

templates/3b_extract_images_from_bin.ipynb

@@ -60,6 +60,14 @@
     "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "4491bec6",
+   "metadata": {},
+   "source": [
+    "Everything necessary for and subsequently outputted from this notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -101,9 +109,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "toffy_env",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "toffy_env"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {

templates/3c_generate_qc_metrics.ipynb

@@ -77,8 +77,24 @@
     "run_name = 'YYYY-MM-DD_run_name'\n",
     "\n",
     "# path to user panel\n",
-    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'\n",
-    "\n",
+    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2572f3e4",
+   "metadata": {},
+   "source": [
+    "Everything necessary for and subsequently outputted from this notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a8ced90",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "# path where plots will get saved\n",
     "bin_file_path = os.path.join('D:\\\\Data', run_name)\n",
     "extracted_imgs_path = os.path.join('D:\\\\Extracted_Images', run_name)\n",

templates/3d_compute_median_pulse_height.ipynb

@@ -45,9 +45,7 @@
    "source": [
     "### Define file parameters\n",
     "You will need to define the `run_name` argument below, which will automatically assign the correct bin file and saving directories. The rest of the cells in this notebook can be run just as they are.\n",
-    " * run_name: should contain the exact name of the MIBI run to locate the mph data from\n",
-    " * bin_file_path: the directory containing your bin files \n",
-    " * mph_out_dir: the directory to save the MPH visualizations to (can be left as defafult)"
+    " * `run_name`: should contain the exact name of the MIBI run to locate the mph data from"
    ]
   },
   {
@@ -58,8 +56,24 @@
    "outputs": [],
    "source": [
     "# set up directories for current run\n",
-    "run_name = 'YYYY-MM-DD_run_name'\n",
-    "\n",
+    "run_name = 'YYYY-MM-DD_run_name'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1e5b85f6",
+   "metadata": {},
+   "source": [
+    "Everything necessary for and subsequently outputted from this notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "3277f8f3",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "bin_file_dir = os.path.join('D:\\\\Data', run_name)\n",
     "mph_out_dir = os.path.join('C:\\\\Users\\\\Customer.ION\\\\Documents\\\\run_metrics', run_name)\n",
     "extracted_imgs_path = os.path.join('D:\\\\Extracted_Images', run_name)\n",
@@ -104,7 +118,7 @@
     "### Visualize MPH Plots\n",
     "Now, we can combine the metrics into a single `mph_pulse_combined.csv` file and then visualize the plots. The second visualization code cell will additionally produce a linear regession line. \n",
     "\n",
-    "Note that the plot produced last will get saved as a png file in `C:\\\\Users\\\\Customer.ION\\\\Documents\\\\run_metrics\\\\run_name` for later reference. If you would like to ensure which version of your plot you are saving, run the corresponding cell again before closing the notebook."
+    "Note that the plot produced last will get saved as a png file in `C:\\\\Users\\\\Customer.ION\\\\Documents\\\\run_metrics\\\\run_name` for later reference. **If you would like to ensure which version of your plot you are saving, run the corresponding cell again before closing the notebook.**"
    ]
   },
   {
@@ -170,9 +184,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "toffy_env",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "toffy_env"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {

templates/3e_stitch_images.ipynb

@@ -62,6 +62,14 @@
     "channel_list = None"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "5c060eba",
+   "metadata": {},
+   "source": [
+    "Everything necessary for and subsequently outputted from this notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -98,9 +106,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "toffy_env",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "toffy_env"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {

templates/4a_compensate_image_data.ipynb

@@ -223,7 +223,9 @@
    "id": "bd4f51e0-8bcc-4b5b-b2d3-6bc00140e0ca",
    "metadata": {},
    "source": [
-    "Once you're satisfied that the Rosetta is working appropriately, you can use it to process your run. First select the run you want to process, and define the relevant top-level folders"
+    "Once you're satisfied that the Rosetta is working appropriately, you can use it to process your run. First select the run you want to process, and define the relevant top-level folders. <br>\n",
+    "\n",
+    "Everything necessary for and subsequently outputted from this section of the notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
    ]
   },
   {
@@ -300,7 +302,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3.8.13 ('toffy_env')",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },

templates/4b_normalize_image_data.ipynb

@@ -38,7 +38,10 @@
    "id": "dbe49e0b-34c2-4f86-b786-403c24b2f678",
    "metadata": {},
    "source": [
-    "### You'll first need to specify the location of the relevant files to enable image normalization"
+    "### You'll first need to specify the location of the relevant files to enable image normalization\n",
+    "\n",
+    " - `run_name` should contain the exact name of the MIBI run that you'll be normalizing\n",
+    " - `panel_path` should point to a panel csv specifying the targets on your panel. For details on panel formatting, see the [panel description](https://github.com/angelolab/toffy#panel-format)"
    ]
   },
   {
@@ -48,11 +51,29 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# First specify the name of the run that you'll be normalizing\n",
+    "# The name of the run\n",
     "run_name = '20220101_run_to_be_processed'\n",
     "\n",
-    "# Then provide the path to your panel\n",
-    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'\n",
+    "# Path to user panel\n",
+    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "bc348f78",
+   "metadata": {},
+   "source": [
+    "Everything necessary for and subsequently outputted from this notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "20ad0c47",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load the panel\n",
     "panel = pd.read_csv(panel_path)\n",
     "\n",
     "# These paths should point to the folders containing each step of the processing pipeline\n",
@@ -157,7 +178,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.12"
+   "version": "3.8.13"
   }
  },
  "nbformat": 4,