Norm workflow

src/toffy/normalize.py

@@ -382,7 +382,7 @@ def show_multiple_plots(rows, cols, image_paths, image_size=(17, 12)):
 def create_tuning_function(
     sweep_path,
     moly_masses=[92, 94, 95, 96, 97, 98, 100],
-    save_path=os.path.join("..", "toffy", "norm_func.json"),
+    save_path=os.path.join("..", "tuning_curves", "sample_tuning_curve.json"),
     count_range=(0, 3000000),
 ):
     """Creates a tuning curve for an instrument based on the provided moly sweep
@@ -777,7 +777,7 @@ def normalize_image_data(
     img_sub_folder="",
     mass_obj_func="poly_2",
     extreme_vals=(0.4, 1.1),
-    norm_func_path=os.path.join("..", "toffy", "norm_func.json"),
+    norm_func_path=os.path.join("..", "tuning_curves", "avg_norm_func_2600.json"),
 ):
     """Normalizes image data based on median pulse height from the run and a tuning curve
 

templates/1_set_up_toffy.ipynb

@@ -13,10 +13,9 @@
    "id": "e36293c5-aa89-4029-a3fa-e8ea841bb8b5",
    "metadata": {},
    "source": [
-    "There are three parts to this notebook.\n",
+    "There are two parts to this notebook.\n",
     "1. The first part creates the necessary folders that toffy is expecting, and **only needs to be run the first time you install it on a new CAC**. \n",
-    "2. The second part updates the co-registration parameters between the slide image (optical image) and the stage coordinates. This **needs to be run when Ionpath changes the co-registration, which usually happens after a column realignment**\n",
-    "3. The third part generates a tuning curve to correct for shifts in instrument sensitivity, and **only needs to be run once per instrument**"
+    "2. The second part updates the co-registration parameters between the slide image (optical image) and the stage coordinates. This **needs to be run when Ionpath changes the co-registration, which usually happens after a column realignment**"
    ]
   },
   {
@@ -178,92 +177,11 @@
    "source": [
     "tiling_utils.save_coreg_params(coreg_params)"
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "14c82566-d6f5-4096-a249-92fae371ab39",
-   "metadata": {},
-   "source": [
-    "## 3. Generate sensitivity tuning curve\n",
-    "The sensitivity of the detector will slowly decay over time, decreasing the counts recorded for the same amount of true signal. The Median Pulse Height (MPH) can be used as metric to evaluate the sensitivity of the detector.\n",
-    "\n",
-    "We create a tuning curve to relate MPH to the percentage of max signal to understand their relationship.\n",
-    "\n",
-    "Your curve should look like the image below. It's okay if your values are a bit different, but the shape of the curve should be qualitatively the same. The curve will be saved in the `sweep_path` folder you define below.\n",
-    "<figure>\n",
-    "    <img src=\"img/tuning_curve.png\" width=\"500\" style=\"display=block; margin:auto\"/>\n",
-    "    <figcaption> The x axis is the Detector Gain, y axis is the % of Max Signal </figcaption>\n",
-    "</figure>"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b4170c03-1619-479c-99f0-ea03ac13d76b",
-   "metadata": {},
-   "source": [
-    "### Identify detector sweep\n",
-    "The first step is selecting a detector sweep. The goal is for this sweep to cover the range of values most often seen during image acqusition. Therefore, it's best to pick a sweep where the suggested change in voltage following the sweep was less than 50V."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "525de367-672a-416c-9c1a-1cb20a397cda",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# pick a name for the sweep, such as the date it was run\n",
-    "sweep_name = '20220417_pmma'"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e661bba8-6657-4f9f-ba8b-ebb9eab874bc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# create a new folder with the sweep name\n",
-    "normalization_dir = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\normalization_curve'\n",
-    "sweep_path = os.path.join(normalization_dir, sweep_name)\n",
-    "if not os.path.exists(sweep_path):\n",
-    "    os.makedirs(sweep_path)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9cab012d-9092-4136-b81b-9c8b3a969e15",
-   "metadata": {},
-   "source": [
-    "Now, copy all of the FOVs from the sweep into the newly created folder, which can be found in *C:\\\\Users\\\\Customer.ION\\\\Documents\\\\normalization_curve*"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "86941169-09a1-43fe-9299-f9f1ca8766b3",
-   "metadata": {},
-   "source": [
-    "### Create tuning curve\n",
-    "We'll then use these FOVs in order to create the curve. **Sometimes FOV channel counts which are too low or too high can interfere with the curve generation, so you can adjust the default range below and run the cell multiple times.** Two tuning curves will be produced, one including all of the data and one excluding values outside of the count range. \n",
-    "\n",
-    "A barplot will be produced below so you can determine where the channel counts seem to level off; this will indicate what the upper threshold for the count range should be. If you would like to simply use the curve with no values exlcuded, set `count_range=None` and run the cell again."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e7d0aa94-a481-4628-b41a-fa2b74d489d9",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# define masses to use\n",
-    "normalize.create_tuning_function(sweep_path=sweep_path, count_range=(0, 3000000))"
-   ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "toffy38",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -277,7 +195,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.15 (default, Nov 11 2022, 16:55:28) \n[Clang 14.0.0 (clang-1400.0.29.202)]"
+   "version": "3.8.15"
   },
   "vscode": {
    "interpreter": {

templates/4b_normalize_image_data.ipynb

@@ -37,7 +37,10 @@
    "source": [
     "### You'll first need to specify the location of the relevant files to enable image normalization\n",
     " - `run_name` should contain the exact name of the MIBI run to extract from\n",
-    " - `panel_path` should point to a panel csv specifying the targets on your panel (see [panel format](https://github.com/angelolab/toffy#panel-format) for more information)"
+    " - `panel_path` should point to a panel csv specifying the targets on your panel (see [panel format](https://github.com/angelolab/toffy#panel-format) for more information)\n",
+    " - `tuning_curve_file` should point to a tuning curve contained in `toffy/tuning_curves` (`avg_norm_func_450.json`, `avg_norm_func_1300.json`, or `avg_norm_func_2600.json`).\n",
+    "\n",
+    "`avg_norm_func_2600.json` is the curve that should be used when running the MIBI with Ionpath's default settings. `avg_norm_func_450.json` and `avg_norm_func_1300.json` are curves generated for Angelo Lab-specific parameters. Use the power supply settings to determine which curve is most applicable."
    ]
   },
   {
@@ -51,7 +54,10 @@
     "run_name = '20220101_run_to_be_processed'\n",
     "\n",
     "# Path to user panel\n",
-    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'"
+    "panel_path = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\panel_files\\\\my_cool_panel.csv'\n",
+    "\n",
+    "# Name of tuning curve file\n",
+    "tuning_curve_file = 'avg_norm_func_2600.json'"
    ]
   },
   {
@@ -152,13 +158,14 @@
    "outputs": [],
    "source": [
     "normalize.normalize_image_data(img_dir=os.path.join(rosetta_base_dir, run_name), norm_dir=normalized_run_dir, pulse_height_dir=mph_run_dir,\n",
-    "                               panel_info=panel, img_sub_folder=img_sub_folder)"
+    "                               panel_info=panel, img_sub_folder=img_sub_folder,\n",
+    "                               norm_func_path=os.path.join('..', 'tuning_curves', tuning_curve_file))"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -172,7 +179,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.0 (main, Nov 15 2022, 09:24:41) [Clang 14.0.0 (clang-1400.0.29.202)]"
+   "version": "3.9.16"
   },
   "vscode": {
    "interpreter": {

templates/generate_tuning_curve.ipynb

@@ -0,0 +1,141 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "161300a0-eb1a-4ccd-8938-1f67cda4c930",
+   "metadata": {
+    "tags": []
+   },
+   "source": [
+    "# Generating sensitivity tuning curve\n",
+    "\n",
+    "**NOTE: standardized tuning curves have been generated which are cross-instrument compatible. These can be found in the `tuning_curves` folder. Unless you have a specific reason to generate a new curve, please bypass this notebook and select from the `tuning_curves` folder instead.**\n",
+    "\n",
+    "The sensitivity of the detector will slowly decay over time, decreasing the counts recorded for the same amount of true signal. The Median Pulse Height (MPH) can be used as metric to evaluate the sensitivity of the detector.\n",
+    "\n",
+    "We create a tuning curve to relate MPH to the percentage of max signal to understand their relationship.\n",
+    "\n",
+    "Your curve should look like the image below. It's okay if your values are a bit different, but the shape of the curve should be qualitatively the same. The curve will be saved in the `sweep_path` folder you define below.\n",
+    "<figure>\n",
+    "    <img src=\"img/tuning_curve.png\" width=\"500\" style=\"display=block; margin:auto\"/>\n",
+    "    <figcaption> The x axis is the Detector Gain, y axis is the % of Max Signal </figcaption>\n",
+    "</figure>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7aba2d6a-e8f2-4429-984b-413c150b2dca",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# imports\n",
+    "import os, warnings\n",
+    "from toffy import normalize"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "5ddc9f0c-8d10-42b8-9426-af1efb1c8933",
+   "metadata": {},
+   "source": [
+    "### Identify detector sweep\n",
+    "The first step is selecting a detector sweep. The goal is for this sweep to cover the range of values most often seen during image acqusition. Therefore, it's best to pick a sweep where the suggested change in voltage following the sweep was less than 50V."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6e54837d-3cdf-4732-94ee-c1817a614a2d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# pick a name for the sweep, such as the date it was run\n",
+    "sweep_name = '20220417_pmma'"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b78d600c-036e-4162-a972-cae96879eb60",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# create a new folder with the sweep name\n",
+    "normalization_dir = 'C:\\\\Users\\\\Customer.ION\\\\Documents\\\\normalization_curve'\n",
+    "sweep_path = os.path.join(normalization_dir, sweep_name)\n",
+    "if not os.path.exists(sweep_path):\n",
+    "    os.makedirs(sweep_path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "82cf2530-42fa-4594-836f-6fa020ab9fbb",
+   "metadata": {},
+   "source": [
+    "Now, copy all of the FOVs from the sweep into the newly created folder, which can be found in *C:\\\\\\Users\\\\\\Customer.ION\\\\\\Documents\\\\\\normalization_curve*"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "851f43b3-e8dc-4fe0-8084-fde28112402d",
+   "metadata": {},
+   "source": [
+    "### Create tuning curve\n",
+    "We'll then use these FOVs in order to create the curve. **Sometimes FOV channel counts which are too low or too high can interfere with the curve generation, so you can adjust the default range below and run the cell multiple times.** Two tuning curves will be produced, one including all of the data and one excluding values outside of the count range. \n",
+    "\n",
+    "A barplot will be produced below so you can determine where the channel counts seem to level off; this will indicate what the upper threshold for the count range should be. If you would like to simply use the curve with no values exlcuded, set `count_range=None` and run the cell again."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "43009bc9-39bb-402d-b212-383767508582",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# define the name of the tuning file\n",
+    "tuning_curve_file = 'default_tuning_curve.json'\n",
+    "\n",
+    "if os.path.exists(os.path.join('..', 'tuning_curves', tuning_curve_file)):\n",
+    "    warnings.warn('Tuning curve file %s already exists, are you sure you want to overwrite?' % tuning_curve_file)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a7715f21-8c8e-4c1f-af67-fe38809d58cb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# define masses to use\n",
+    "normalize.create_tuning_function(\n",
+    "    sweep_path=sweep_path,\n",
+    "    save_path=os.path.join('..', 'tuning_curves', tuning_curve_file),\n",
+    "    count_range=(0, 3000000)\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

tuning_curves/avg_norm_func_1300.json

@@ -0,0 +1 @@
+{"name": "exp", "weights": [119.8713651649559, -14.206805861739168, 0.5620928991242043, -336.7403258779316]}

tuning_curves/avg_norm_func_2600.json

@@ -0,0 +1 @@
+{"name": "exp", "weights": [89.11213813383279, -9.530756091348142, 0.3403483027593935, -277.2129746816949]}

tuning_curves/avg_norm_func_450.json

@@ -0,0 +1 @@
+{"name": "exp", "weights": [92.08712376, -12.34594787, 0.55308083, -228.5327305]}