templates/1_set_up_toffy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ec35cb18-c771-48e6-859f-e10e074545d6",
   "metadata": {},
   "source": [
    "# Setting up toffy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e36293c5-aa89-4029-a3fa-e8ea841bb8b5",
   "metadata": {},
   "source": [
    "There are three parts to this notebook.\n",
    "1. The first part creates the necessary folders that toffy is expecting, and **only needs to be run the first time you install it on a new CAC**. \n",
    "2. The second part updates the co-registration parameters between the slide image (optical image) and the stage coordinates. This **needs to be run when Ionpath changes the co-registration, which usually happens after a column realignment**\n",
    "3. The third part generates a tuning curve to correct for shifts in instrument sensitivity, and **only needs to be run once per instrument**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c65c5bf-c7c1-43a7-882e-4134d9de0007",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "import sys\n",
    "sys.path.append('../')\n",
    "\n",
    "from datetime import datetime\n",
    "import json\n",
    "import numpy as np\n",
    "import os\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "from toffy import tiling_utils, normalize\n",
    "from ark.utils import io_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb3f2573-2c06-4212-8abe-7fb41b83c878",
   "metadata": {},
   "source": [
    "## 1. Create necessary folders\n",
    "\n",
    "**If this is the first time toffy has been installed on your CAC, run the following cell to generate the folders necessary.**\n",
    "\n",
    "\n",
    "More information on the uses and locations of the directories in toffy can be found in the [README](https://github.com/angelolab/toffy#directory-structure)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e62bf338-873f-4c3f-a58c-6f908e6bdb24",
   "metadata": {},
   "outputs": [],
   "source": [
    "folders = ['D:\\\\Extracted_Images', 'D:\\\\Rosetta_Compensated_Images', 'D:\\\\Normalized_Images', 'D:\\\\Cohorts',\n",
    "           'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\run_metrics', 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\watcher_logs',\n",
    "           'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\tiled_run_jsons', \n",
    "           'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\autolabeled_tma_jsons', \n",
    "           'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files', 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\normalization_curve', \n",
    "           'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\rosetta_testing',\n",
    "           'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\rosetta_matrices']\n",
    "\n",
    "for folder in folders:\n",
    "    if not os.path.exists(folder):\n",
    "        os.makedirs(folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edb13b4a-9589-4c2d-814c-0cae835cc255",
   "metadata": {},
   "source": [
    "## 2. Update coregistration parameters\n",
    "In order to display the location of FOVs on the slide image, we need to map the location of pixels in the image to stage coordinates on the sldie. This notebook allows you to generate this information, which can change over time for a number of reasons. We use the ficudial marks, the dots on each side of the slide, to accomplish this. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44bf4948-eebe-43a9-8e0e-a993dc8e726b",
   "metadata": {},
   "source": [
    "### Set stage and optical coordinates of the fiducials"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d3cc736-faba-41e7-a406-ff900074d7fe",
   "metadata": {},
   "source": [
    "The code below will automatically prompt you to specify the coordinates of the top 3 fiducial pairs in both stage coordinate and corresponding optical coordinate values.\n",
    "\n",
    "For each fiducial:\n",
    "\n",
    "1: Use the pop-out window to open the slide image on the commercial instrument viewer.\n",
    "\n",
    "2: Open the developer console. Right click anywhere on the slide image, this menu should open:\n",
    "\n",
    "<div>\n",
    "    <img src=\"img/inspect_menu.png\" width=\"150\" height=\"137\"/>\n",
    "</div>\n",
    "\n",
    "\n",
    "Click `Inspect`. The developer console should appear on the right side:\n",
    "\n",
    "<div>\n",
    "    <img src=\"img/developer_console.png\" width=\"254\" height=\"240\"/>\n",
    "</div>\n",
    "\n",
    "3: In the console (above the tabs `What's New` and `Console`), type `logger.level=4` and hit `Enter`\n",
    "\n",
    "<div>\n",
    "    <img src=\"img/logger_level.png\" width=\"253\" height=\"76\"/>\n",
    "</div>\n",
    "\n",
    "\n",
    "4: Click on one of the slide image fiducials. Retrieve the stage coordinates on top of the image next to `Target Point`:\n",
    "\n",
    "<div>\n",
    "    <img src=\"img/stage_coords.png\" width=\"164\" height=\"21\"/>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "Retrieve the corresponding optical coordinates from the console:\n",
    "\n",
    "<div>\n",
    "    <img src=\"img/optical_coords.png\" width=\"254\" height=\"51\"/>\n",
    "</div>\n",
    "\n",
    "\n",
    "\n",
    "5: For each fiducial, we prompt you for the stage x-coordinate, stage y-coordinate, optical x-coordinate, and optical y-coordinate values separately. Enter the correct value for the coordinate and hit `Enter` to continue.\n",
    "\n",
    "6: The co-registration parameter settings will be saved to the `toffy` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64d6d2f5-ac4c-4917-a874-b8310f75f5e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "fiducial_info = tiling_utils.read_fiducial_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce74c91b-86ef-4569-87ac-b8553ecb23bc",
   "metadata": {},
   "source": [
    "### Generate new co-registration parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3db446dd-2575-420b-a00e-874802b7a3ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "coreg_params = tiling_utils.generate_coreg_params(fiducial_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df88849-a7ba-400c-92bd-88e165a5b515",
   "metadata": {},
   "source": [
    "### Save co-registration settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f126bc6a-4695-4722-9d73-b9dc66bb7195",
   "metadata": {},
   "outputs": [],
   "source": [
    "tiling_utils.save_coreg_params(coreg_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14c82566-d6f5-4096-a249-92fae371ab39",
   "metadata": {},
   "source": [
    "## 3. Generate sensitivity calibration curve\n",
    "The sensitivity of the detector will slowly decay over time, decreasing the counts recorded for the same amount of true signal. The Median Pulse Height (MPH) can be used as metric to evaluate the sensitivity of the detector.\n",
    "\n",
    "We create a calibration curve to relate MPH to the percentage of max signal to understand their relationship.\n",
    "\n",
    "Your curve should look like the image below. It's okay if your values are a bit different, but the shape of the curve should be qualitatively the same. Please notice the limited range of MPH values used for our fit. The curve will be saved in the `sweep_path` folder you define below.\n",
    "<figure>\n",
    "    <img src=\"img/tuning_curve.png\" width=\"500\" style=\"display=block; margin:auto\"/>\n",
    "    <figcaption> The x axis is the Detector Gain, y axis is the % of Max Signal </figcaption>\n",
    "</figure>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4170c03-1619-479c-99f0-ea03ac13d76b",
   "metadata": {},
   "source": [
    "### Identify detector sweep\n",
    "The first step is selecting a detector sweep. The goal is for this sweep to cover the range of values most often seen during image acqusition. Therefore, it's best to pick a sweep where the suggested change in voltage following the sweep was less than 50V.\n",
    "If you're operating the commercial MIBIscope, simply use the data from the second of two consecutive detector sweeps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "525de367-672a-416c-9c1a-1cb20a397cda",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pick a name for the sweep, such as the date it was run\n",
    "sweep_name = '2022-10-09_molybdenum'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e661bba8-6657-4f9f-ba8b-ebb9eab874bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new folder with the sweep name\n",
    "normalization_dir = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\normalization_curve'\n",
    "sweep_path = os.path.join(normalization_dir, sweep_name)\n",
    "if not os.path.exists(sweep_path):\n",
    "    os.makedirs(sweep_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cab012d-9092-4136-b81b-9c8b3a969e15",
   "metadata": {},
   "source": [
    "Now, copy all of the FOVs from the sweep into the newly created folder, which can be found in *C:\\\\Users\\\\Customer.ION\\\\Documents\\\\normalization_curve*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86941169-09a1-43fe-9299-f9f1ca8766b3",
   "metadata": {},
   "source": [
    "### Create calibration curve\n",
    "We will now use the detector sweep data in order to create a signal intensity calbration curve. **Sometimes FOV channel counts are too low, when the detector is missing detections (false negative = Type II error) due to sub-threshold detector gain values. Sometimes FOV channel counts are too high, when the detector is stochastically adding counts (false positive = Type I error) due to very high detector gain values. Only valid counts from a detector running within its optimal detector gain range should be used for the creation of the calibration curve: You should therefore adjust the default maximum peak height (MPH) range below and run the code cell multiple times until your calibration curve looks like the example curve.** Two curves will be produced, one including all of the data and one excluding values outside of the maximum peak height range. \n",
    "\n",
    "A barplot will be produced below so you can determine where the channel counts seem to level off; this will indicate what the upper threshold for the MPH range should be. If you would like to simply use the curve with all values included, set `mph_limits=(0, float('inf'))` and run the cell again. The default range is restored with `mph_limits(3_000, 10_000)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7d0aa94-a481-4628-b41a-fa2b74d489d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define masses to use\n",
    "normalize.create_tuning_function(sweep_path=sweep_path, mph_limits=(3_000, 10_000))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "b6fb10fd04ae96ae15bad5607f85b9bdcc66c9c776108a7102620b79ac3bec79"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}