templates/4b_normalize_image_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8e012798-3632-4107-8d5a-5f47e7671a9d",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Image Normalization\n",
    "\n",
    "This notebook will walk you through the process of normalizing your image data.\n",
    "\n",
    "Changes in detector sensitivity over a run can result in different image intensities, even when there are no actual difference in biological signal. To correct for this, we use the median pulse height (MPH) to measure detector sensitivity. We then combine this estimate of sensitivity with the tuning curve generated by `1_set_up_toffy.ipynb` to determine the normalization coefficient for each FOV.\n",
    "\n",
    "Before running through the notebook, make sure you've completed section 3 of `1_set_up_toffy.ipynb`, which is used to create the necessary normalization curve. In addition, you should have already compensated your data with rosetta using `4_compensate_image_data.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c9751084-7855-4005-b152-55895ff40823",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../')\n",
    "\n",
    "import os\n",
    "\n",
    "from toffy import normalize\n",
    "from toffy.panel_utils import load_panel\n",
    "from ark.utils.io_utils import list_files, list_folders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbe49e0b-34c2-4f86-b786-403c24b2f678",
   "metadata": {},
   "source": [
    "### You'll first need to specify the location of the relevant files to enable image normalization\n",
    " - `run_name` should contain the exact name of the MIBI run to extract from\n",
    " - `panel_path` should point to a panel csv specifying the targets on your panel (see [panel format](https://github.com/angelolab/toffy#panel-format) for more information)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f48127b8-84da-4763-9c44-01a5bde4c4ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The name of the run\n",
    "run_name = '20220101_run_to_be_processed'\n",
    "\n",
    "# Path to user panel\n",
    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc348f78",
   "metadata": {},
   "source": [
    "Everything necessary for and subsequently outputted from this notebook is stored in the automatic directories established in `1_set_up_toffy.ipynb`. More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51f96a31-79cb-46d7-a720-aa013bc0116b",
   "metadata": {},
   "outputs": [],
   "source": [
    "panel = load_panel(panel_path)\n",
    "\n",
    "# These paths should point to the folders containing each step of the processing pipeline\n",
    "bin_base_dir = 'D:\\\\Data'\n",
    "rosetta_base_dir = 'D:\\\\Rosetta_Compensated_Images'\n",
    "normalized_base_dir = 'D:\\\\Normalized_Images'\n",
    "mph_run_dir = os.path.join('C:\\\\Users\\\\Customer.ION\\\\Documents\\\\run_metrics', run_name, 'fov_data')\n",
    "\n",
    "# check the run for changes in detector voltage\n",
    "normalize.check_detector_voltage(os.path.join(bin_base_dir, run_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62142cdc-fb78-4c21-8cb2-77b3fb60d399",
   "metadata": {},
   "source": [
    "### Within the defined directories, we'll specify the relevant run_dir based on the run_name provided above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e323824-38a1-4c67-8088-a1725ecbc910",
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify sub-folder for rosetta images\n",
    "img_sub_folder = 'rescaled'\n",
    "\n",
    "# create directory to hold normalized images\n",
    "normalized_run_dir = os.path.join(normalized_base_dir, run_name)\n",
    "if not os.path.exists(normalized_run_dir):\n",
    "    os.makedirs(normalized_run_dir)\n",
    "    \n",
    "# create directory to hold associated processing files\n",
    "if not os.path.exists(mph_run_dir):\n",
    "    os.makedirs(mph_run_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "748a2e7c-ecb0-421a-8192-d4bfaf889f37",
   "metadata": {},
   "source": [
    "### Then, we'll loop over each FOV, generating the necessary normalization files if they weren't already created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfe67691-1676-4c3a-93fc-491e3abd6a24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all FOVs\n",
    "fovs = list_folders(os.path.join(rosetta_base_dir, run_name), substrs='fov-')\n",
    "\n",
    "# loop over each FOV\n",
    "for fov in fovs:\n",
    "    # generate mph values\n",
    "    mph_file_path = os.path.join(mph_run_dir, fov + '_pulse_heights.csv')\n",
    "    if not os.path.exists(mph_file_path):\n",
    "        normalize.write_mph_per_mass(base_dir=os.path.join(bin_base_dir, run_name), output_dir=mph_run_dir, \n",
    "                                     fov=fov, masses=panel['Mass'].values, start_offset=0.3, stop_offset=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e60fc2eb-9072-4ffe-a302-7ddb175bf4e7",
   "metadata": {},
   "source": [
    "###  Finally, we'll normalize the images, and save them to the output folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59f49779-5e77-4017-8e68-50afce126883",
   "metadata": {},
   "outputs": [],
   "source": [
    "normalize.normalize_image_data(img_dir=os.path.join(rosetta_base_dir, run_name), norm_dir=normalized_run_dir, pulse_height_dir=mph_run_dir,\n",
    "                               panel_info=panel, img_sub_folder=img_sub_folder)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}