standard_workflow_besca25.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Standard-BESCA-workflow\" data-toc-modified-id=\"Standard-BESCA-workflow-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Standard BESCA workflow</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Import-section\" data-toc-modified-id=\"Import-section-1.0.1\"><span class=\"toc-item-num\">1.0.1&nbsp;&nbsp;</span>Import section</a></span></li></ul></li><li><span><a href=\"#Setup-Standard-Wokflow\" data-toc-modified-id=\"Setup-Standard-Wokflow-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Setup Standard Wokflow</a></span><ul class=\"toc-item\"><li><span><a href=\"#Parameters-to-be-set---on-the-command-line-or-here\" data-toc-modified-id=\"Parameters-to-be-set---on-the-command-line-or-here-1.1.1\"><span class=\"toc-item-num\">1.1.1&nbsp;&nbsp;</span>Parameters to be set - on the command line or here</a></span></li><li><span><a href=\"#Define-Input\" data-toc-modified-id=\"Define-Input-1.1.2\"><span class=\"toc-item-num\">1.1.2&nbsp;&nbsp;</span>Define Input</a></span></li><li><span><a href=\"#Standard-parameters---these-should-be-kept-as-stable-as-possible\" data-toc-modified-id=\"Standard-parameters---these-should-be-kept-as-stable-as-possible-1.1.3\"><span class=\"toc-item-num\">1.1.3&nbsp;&nbsp;</span>Standard parameters - these should be kept as stable as possible</a></span></li></ul></li><li><span><a href=\"#Standard-Pipeline\" data-toc-modified-id=\"Standard-Pipeline-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Standard Pipeline</a></span><ul class=\"toc-item\"><li><span><a href=\"#Visualization-of-quality-control-plots-and-selected-filtering-parameters\" data-toc-modified-id=\"Visualization-of-quality-control-plots-and-selected-filtering-parameters-1.2.1\"><span class=\"toc-item-num\">1.2.1&nbsp;&nbsp;</span>Visualization of quality control plots and selected filtering parameters</a></span><ul class=\"toc-item\"><li><span><a href=\"#Count-occurrence\" data-toc-modified-id=\"Count-occurrence-1.2.1.1\"><span class=\"toc-item-num\">1.2.1.1&nbsp;&nbsp;</span>Count occurrence</a></span></li><li><span><a href=\"#Transcript-capture-efficiency\" data-toc-modified-id=\"Transcript-capture-efficiency-1.2.1.2\"><span class=\"toc-item-num\">1.2.1.2&nbsp;&nbsp;</span>Transcript capture efficiency</a></span></li><li><span><a href=\"#Library-size-distribution\" data-toc-modified-id=\"Library-size-distribution-1.2.1.3\"><span class=\"toc-item-num\">1.2.1.3&nbsp;&nbsp;</span>Library size distribution</a></span></li><li><span><a href=\"#The-effect-of-filtering-parameters\" data-toc-modified-id=\"The-effect-of-filtering-parameters-1.2.1.4\"><span class=\"toc-item-num\">1.2.1.4&nbsp;&nbsp;</span>The effect of filtering parameters</a></span></li><li><span><a href=\"#Scanpy-plots-of-genes,-counts,-and-mitochondria-gene-counts\" data-toc-modified-id=\"Scanpy-plots-of-genes,-counts,-and-mitochondria-gene-counts-1.2.1.5\"><span class=\"toc-item-num\">1.2.1.5&nbsp;&nbsp;</span>Scanpy plots of genes, counts, and mitochondria gene counts</a></span></li><li><span><a href=\"#Mitochondrial-genes,-genes,-and-counts-by-samples-grouped-by-the-split-condition\" data-toc-modified-id=\"Mitochondrial-genes,-genes,-and-counts-by-samples-grouped-by-the-split-condition-1.2.1.6\"><span class=\"toc-item-num\">1.2.1.6&nbsp;&nbsp;</span>Mitochondrial genes, genes, and counts by samples grouped by the split condition</a></span></li></ul></li><li><span><a href=\"#Filtering\" data-toc-modified-id=\"Filtering-1.2.2\"><span class=\"toc-item-num\">1.2.2&nbsp;&nbsp;</span>Filtering</a></span><ul class=\"toc-item\"><li><span><a href=\"#Filtering-with-thresholds-of-gene-and-cell-counts\" data-toc-modified-id=\"Filtering-with-thresholds-of-gene-and-cell-counts-1.2.2.1\"><span class=\"toc-item-num\">1.2.2.1&nbsp;&nbsp;</span>Filtering with thresholds of gene and cell counts</a></span></li><li><span><a href=\"#Filtering-with-thresholds-of-proportion-of-mitochondrial-genes-and-the-upper-limit-of-gene-counts\" data-toc-modified-id=\"Filtering-with-thresholds-of-proportion-of-mitochondrial-genes-and-the-upper-limit-of-gene-counts-1.2.2.2\"><span class=\"toc-item-num\">1.2.2.2&nbsp;&nbsp;</span>Filtering with thresholds of proportion of mitochondrial genes and the upper limit of gene counts</a></span></li><li><span><a href=\"#Visualising-QC-metrics-of-the-filtered-dataset\" data-toc-modified-id=\"Visualising-QC-metrics-of-the-filtered-dataset-1.2.2.3\"><span class=\"toc-item-num\">1.2.2.3&nbsp;&nbsp;</span>Visualising QC metrics of the filtered dataset</a></span></li></ul></li><li><span><a href=\"#Per-cell-normalization-and-output-of-the-normalized-data\" data-toc-modified-id=\"Per-cell-normalization-and-output-of-the-normalized-data-1.2.3\"><span class=\"toc-item-num\">1.2.3&nbsp;&nbsp;</span>Per-cell normalization and output of the normalized data</a></span></li><li><span><a href=\"#Feature-selection-(highly-variable-genes)-for-clustering\" data-toc-modified-id=\"Feature-selection-(highly-variable-genes)-for-clustering-1.2.4\"><span class=\"toc-item-num\">1.2.4&nbsp;&nbsp;</span>Feature selection (highly variable genes) for clustering</a></span></li><li><span><a href=\"#Regression-steps,-and-output-of-regressed-data\" data-toc-modified-id=\"Regression-steps,-and-output-of-regressed-data-1.2.5\"><span class=\"toc-item-num\">1.2.5&nbsp;&nbsp;</span>Regression steps, and output of regressed data</a></span></li><li><span><a href=\"#PCA-based-neighborhood-analysis-and-UMAP-with-optional-batch-correction\" data-toc-modified-id=\"PCA-based-neighborhood-analysis-and-UMAP-with-optional-batch-correction-1.2.6\"><span class=\"toc-item-num\">1.2.6&nbsp;&nbsp;</span>PCA-based neighborhood analysis and UMAP with optional batch correction</a></span></li><li><span><a href=\"#Clustering\" data-toc-modified-id=\"Clustering-1.2.7\"><span class=\"toc-item-num\">1.2.7&nbsp;&nbsp;</span>Clustering</a></span></li><li><span><a href=\"#Additional-Labeling\" data-toc-modified-id=\"Additional-Labeling-1.2.8\"><span class=\"toc-item-num\">1.2.8&nbsp;&nbsp;</span>Additional Labeling</a></span></li><li><span><a href=\"#CiteSeq-Standard-Wokflow-(only-executed-if-applicable)\" data-toc-modified-id=\"CiteSeq-Standard-Wokflow-(only-executed-if-applicable)-1.2.9\"><span class=\"toc-item-num\">1.2.9&nbsp;&nbsp;</span>CiteSeq Standard Wokflow (only executed if applicable)</a></span></li><li><span><a href=\"#Complete-the-log-file\" data-toc-modified-id=\"Complete-the-log-file-1.2.10\"><span class=\"toc-item-num\">1.2.10&nbsp;&nbsp;</span>Complete the log-file</a></span></li><li><span><a href=\"#Write-QC-Report\" data-toc-modified-id=\"Write-QC-Report-1.2.11\"><span class=\"toc-item-num\">1.2.11&nbsp;&nbsp;</span>Write QC Report</a></span></li><li><span><a href=\"#Session-info\" data-toc-modified-id=\"Session-info-1.2.12\"><span class=\"toc-item-num\">1.2.12&nbsp;&nbsp;</span>Session info</a></span></li></ul></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Standard BESCA workflow\n",
    "\n",
    "\n",
    "### Import section ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scanpy as sc\n",
    "import besca as bc \n",
    "\n",
    "vers = bc.print_software_versions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl\n",
    "from scipy import sparse, io\n",
    "import seaborn as sns\n",
    "import os\n",
    "import re\n",
    "import time\n",
    "import IPython\n",
    "from random import sample\n",
    "\n",
    "import logging\n",
    "import seaborn as sns\n",
    "from sinfo import sinfo\n",
    "import warnings\n",
    "\n",
    "# Seed is fixed for reproducible analysis.\n",
    "import random \n",
    "random.seed(1)\n",
    "\n",
    "# for standard processing, set verbosity to minimum\n",
    "sc.settings.verbosity = 0  # verbosity: errors (0), warnings (1), info (2), hints (3)\n",
    "sc.settings.set_figure_params(dpi=80)\n",
    "sns.set_context(\"paper\", font_scale=0.9) #changes font sizes so better readable\n",
    "plt.rcParams['svg.fonttype'] = 'none' #ensures that any saved svgs have editable fonts\n",
    "version = '3.0' # This is the third version of the standard workflow, introduced with besca v2.5\n",
    "start0 = time.time()\n",
    "warnings.simplefilter(action=\"always\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Standard Wokflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters to be set - on the command line or here ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# decisions to be made\n",
    "use_example_dataset = False # If True, the pipeline will run using the example dataset shipped with BESCA\n",
    "\n",
    "species = 'human'\n",
    "batch_to_correct = 'None' # must be \"None\" or any one of the labels in \"metadata.tsv\", ID, SPECIES, TISSUE, DONOR, TREATMENT; typically \"ID\" or \"DONOR\"\n",
    "analysis_name = 'sw_besca2' # analysis name (will appear in db and viz), only alphanumeric and _ allowed (not - .)\n",
    "split_condition='sample_id' #'readout_id' or 'experiment' is generally a good default\n",
    "citeseq = False #specify if the dataset contains citeseq data: True or False \n",
    "adt_norm = 'CLR' # only CLR currently, DSB deprecated (required R packages)\n",
    "dynrange=['B2m','Actb','Pgk1','Ctcf'] #genes for which to plot dynamic range\n",
    "example_markers = ['PTPRC', 'CD14'] # Will be shown in the first umaps\n",
    "\n",
    "if species=='human': dynrange=[x.upper() for x in dynrange]\n",
    "if not re.match(r'^\\w+$', analysis_name):  raise Warning('Analysis name is NOT compliant with MongoDB, only alphanumeric and _ allowed (not - .)')\n",
    "    \n",
    "#additional labeling\n",
    "labeling_to_use = 'None' # must be \"None\" or any one of the labels in \"metadata.tsv\", ID, SPECIES, TISSUE, DONOR, TREATMENT; typically \"ID\" or \"DONOR\"\n",
    "labeling_name = 'MyAnno' # define name under which the labeling should be exported\n",
    "labeling_description = 'celltype annotation' #define description which should be saved to labeling_info file\n",
    "labeling_author = 'MyAnnotAuthor' #define author which should be saved to labeling info file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Input ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define filepath (this is the folder that contains \"raw\" and \"analyzed\")\n",
    "root_path = os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard parameters - these should be kept as stable as possible ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the standard parameter section\n",
    "standard_min_genes = 600\n",
    "standard_min_cells = 20\n",
    "standard_min_counts = 1000\n",
    "standard_n_genes = 6000 # this is the most tricky one to set\n",
    "standard_percent_mito = 0.15\n",
    "standard_max_counts = 40000 #might be redundant with n_genes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard Pipeline\n",
    "(note nothing below this point should be modified!!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define standardized filepaths based on above input\n",
    "results_folder = os.path.join(root_path, 'analyzed', analysis_name)\n",
    "results_file = os.path.join(results_folder, analysis_name + '.h5ad') # specify a .h5ad file for storing the results\n",
    "results_file_raw = os.path.join(results_folder, analysis_name + '.raw.h5ad') # specify a .h5ad file for storing the raw counts after filtering (might be needed for other analyses/tools)\n",
    "log_file = os.path.join(results_folder, analysis_name + '.standard.log') # specify a log file for keeping a short summary and overview\n",
    "sc.settings.figdir = os.path.join(results_folder, 'figures')\n",
    "\n",
    "#setup standard workflow (generates output directories and setsup logging file)\n",
    "bc.st.setup(results_folder, \n",
    "              analysis_name, \n",
    "              labeling_name, \n",
    "              labeling_to_use, \n",
    "              log_file, \n",
    "              version,\n",
    "              root_path, \n",
    "              species, \n",
    "              batch_to_correct, \n",
    "              standard_min_genes,\n",
    "              standard_min_cells,\n",
    "              standard_min_counts,\n",
    "              standard_n_genes,\n",
    "              standard_percent_mito,\n",
    "              standard_max_counts)\n",
    "\n",
    "\n",
    "# read input data\n",
    "if use_example_dataset:\n",
    "    print('utilizing example data')\n",
    "    # subset the cells for speed purpose\n",
    "    adata = bc.datasets.Kotliarov2020_raw()\n",
    "    subset = sample([i for i in range(adata.shape[0])], 10000) \n",
    "    adata = adata[subset,:]\n",
    "    if citeseq:\n",
    "        adata_prot = bc.datasets.Kotliarov2020_citeSeq()[subset,:]\n",
    "        adata_prot.obs.index = adata.obs.index\n",
    "        results_folder_citeseq = os.path.join(results_folder, 'citeseq' )\n",
    "        results_folder_merged= os.path.join(results_folder_citeseq, 'citeseq_merged')\n",
    "        results_file_citeseq = os.path.join(results_folder_citeseq, analysis_name + '.h5ad')\n",
    "        results_file_merged = os.path.join(results_folder_merged, analysis_name + '.h5ad')\n",
    "        bc.st.setup_citeseq(results_folder_citeseq,\n",
    "                            labeling_name, \n",
    "                            labeling_to_use)\n",
    "        print('utilizing citeseq data')\n",
    "        \n",
    "elif citeseq:\n",
    "    #generate file strucutre for citeseq data\n",
    "    results_folder_citeseq = os.path.join(results_folder, 'citeseq' )\n",
    "    results_folder_merged= os.path.join(results_folder_citeseq, 'citeseq_merged')\n",
    "    results_file_citeseq = os.path.join(results_folder_citeseq, analysis_name + '.h5ad')\n",
    "    results_file_merged = os.path.join(results_folder_merged, analysis_name + '.h5ad')\n",
    "    bc.st.setup_citeseq(results_folder_citeseq,\n",
    "                        labeling_name, \n",
    "                        labeling_to_use)\n",
    "    \n",
    "    print('utilizing citeseq data')\n",
    "    #read citeseq data and normal data\n",
    "    logging.info('Reading Gene Expression Data...')\n",
    "    adata = bc.st.read_matrix(root_path, citeseq = 'gex_only')\n",
    "    logging.info('Reading Citeseq Data...')\n",
    "    adata_prot = bc.st.read_matrix(root_path, citeseq = 'citeseq_only')\n",
    "else:\n",
    "    #read data\n",
    "    adata = bc.st.read_matrix(root_path, citeseq = None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#calculate mitochondrial gene content\n",
    "bc.pp.fraction_counts(adata=adata, species=species)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true,
    "toc-nb-collapsed": true
   },
   "source": [
    "### Visualization of quality control plots and selected filtering parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Count occurrence\n",
    "\n",
    "This plot shows cell counts per sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "temp=bc.tl.count_occurrence(adata,split_condition)\n",
    "sns.set(rc={'figure.figsize':(11.7,8.27)})\n",
    "sns.set_style(\"whitegrid\")\n",
    "sns.barplot(y=temp.index,x=temp.Counts,color='gray',orient='h')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transcript capture efficiency\n",
    "\n",
    "This plot gives you an idea about the sequencing depth and if the sequencing has reached saturation or not.\n",
    "- Each dot represents a gene. \n",
    "- X-axis: Total gene count across all cells\n",
    "- Y-axis: Proportion of cells the gene is detected in\n",
    "- There are non-linear ranges where the genecounts are too low to get a reasonable detection\n",
    "probability (this is basically below the sensitivity threshold) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1)\n",
    "fig.set_figwidth(8)\n",
    "fig.set_figheight(5)\n",
    "fig.tight_layout()\n",
    "\n",
    "#bc.pl.transcript_capture_efficiency(adata,ax=ax)\n",
    "#fig.savefig(os.path.join(results_folder, 'figures/transcriptcaptureefficiency.png'), format='png', bbox_inches = 'tight') #save figure for QC report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Library size distribution\n",
    "\n",
    "This plot gives you an idea about the library size distribution in your dataset before processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = bc.pl.librarysize_overview(adata, bins=100)\n",
    "fig.savefig(os.path.join(results_folder, 'figures/librarysize.png'), format='png', bbox_inches = 'tight') #save figure for QC report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata_unfiltered = adata.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The effect of filtering parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ((ax1, ax2, ax3), (ax4, ax5, ax6)) = plt.subplots(ncols=3, nrows=2)\n",
    "fig.set_figwidth(17)\n",
    "fig.set_figheight(9)\n",
    "fig.tight_layout(pad=4.5)\n",
    "\n",
    "bc.pl.kp_genes(adata, min_genes=standard_min_genes, ax = ax1)\n",
    "bc.pl.kp_counts(adata, min_counts=standard_min_counts, ax = ax2)\n",
    "bc.pl.kp_cells(adata, min_cells=standard_min_cells, ax = ax3)\n",
    "bc.pl.max_genes(adata, max_genes=standard_n_genes, ax = ax4)\n",
    "bc.pl.max_mito(adata, max_mito=standard_percent_mito, annotation_type='SYMBOL', species=species, ax = ax5)\n",
    "bc.pl.max_counts(adata, max_counts=standard_max_counts, ax=ax6)\n",
    "fig.savefig(os.path.join(results_folder, 'figures/filtering_thresholds.png'), format='png', bbox_inches = 'tight') #save figure for QC report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Scanpy plots of genes, counts, and mitochondria gene counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.pl.violin(adata, ['n_genes', 'n_counts', 'percent_mito'], jitter=0.2, multi_panel=True, save = '.before_filtering.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mitochondrial genes, genes, and counts by samples grouped by the split condition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for myvals in ['percent_mito','n_genes', 'n_counts']:\n",
    "    sc.pl.violin(adata, myvals, groupby=split_condition,jitter=0.1,rotation=90, save = myvals+'.before_filtering.split.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering\n",
    "\n",
    "#### Filtering with thresholds of gene and cell counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture filtering1\n",
    "adata = bc.st.filtering_cells_genes_min(adata, standard_min_cells, standard_min_genes, standard_min_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtering1.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filtering with thresholds of proportion of mitochondrial genes and the upper limit of gene counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture filtering2\n",
    "adata = bc.st.filtering_mito_genes_max(adata, standard_percent_mito, standard_n_genes, standard_max_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtering2.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualising QC metrics of the filtered dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for myvals in ['percent_mito','n_genes', 'n_counts']:\n",
    "    sc.pl.violin(adata, ['n_genes', 'n_counts', 'percent_mito'], jitter=0.2, multi_panel=True, save = myvals+'.after_filtering.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata.var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check mitochondrial reads per sample \n",
    "for myvals in ['percent_mito','n_genes', 'n_counts']:\n",
    "    sc.pl.violin(adata, myvals, groupby=split_condition,jitter=0.1,rotation=90, save = myvals+'.after_filtering.split.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cell counts per sample\n",
    "temp=bc.tl.count_occurrence(adata,split_condition)\n",
    "sns.barplot(y=temp.index,x=temp.Counts,color='gray',orient='h')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#display the top 25 genes in the dataset\n",
    "#fig, ax = plt.subplots(ncols=1, nrows=1, figsize = (8, 6))\n",
    "#bc.pl.top_genes_counts(adata=adata, top_n=25, ax = ax )\n",
    "#fig.savefig(os.path.join(results_folder, 'figures/top_genes.png'), format='png', bbox_inches = 'tight') #save figure for QC report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata.var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export adata object with raw counts and filtered cells/genes (might be needed by other analyses)\n",
    "adata.write(results_file_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Per-cell normalization and output of the normalized data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata = bc.st.per_cell_normalize(adata, results_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We perform an additional QC, which checks the dynamic range of ubiquitously expressed marker genes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature selection (highly variable genes) for clustering ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata = bc.st.highly_variable_genes(adata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data scaling ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.pp.scale(adata, max_value=10, copy=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PCA-based neighborhood analysis and UMAP with optional batch correction\n",
    "\n",
    "We use the Batch Balanced K-Nearest Neighbourhood (bbknn, [Teichlab/bbknn](https://github.com/Teichlab/bbknn)) method as the batch correction method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (batch_to_correct != 'None'):\n",
    "    #save a copy of uncorrected in case we need it for something later\n",
    "    adata_uncorrected = adata.copy()\n",
    "    adata.obs['batch'] = adata.obs[batch_to_correct]\n",
    "    adata = bc.st.pca_neighbors_umap(adata,results_folder, method='bbknn')\n",
    "else:\n",
    "    adata = bc.st.pca_neighbors_umap(adata, results_folder)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# leiden clustering is the default \n",
    "adata = bc.st.clustering(adata, results_folder, myres=1.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata.var"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.settings.set_figure_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.pl.umap(adata, color=['treatment_id','readout_id','individual_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# everything that was done so far goes to the .h5ad file for later use\n",
    "adata.write(results_file)\n",
    "print(results_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "marker = [x for x in example_markers if x in adata.raw.var_names ] \n",
    "if len(marker) >0 :\n",
    "    sc.pl.umap( adata, color = marker , color_map = 'viridis')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.pl.umap( adata, color = ['batch','treatment_id','compound_name','individual_id', 'readout_id'] , ncols=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.pl.umap( adata, color = ['CD3D','CD8A','S1PR1','S1PR5','PDCD1',\n",
    "                            'FOXP3', 'MKI67', 'IFITM1','TOX',\n",
    "                            'GZMA', 'GZMB','GZMH','GZMK', 'IL7R', 'TCF7',\n",
    "                            'CXCR3','CXCR5'] , color_map = 'viridis')\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#goi=[MCP-1, MIP-1a,  MIP-1b,  IL-8, IP-10, IL-1b, TNF-a, IFNg, IL-6, IL-2 and IL-10]\n",
    "goi=['CCL2','CCL3','CCL4','CXCL8', 'CXCL10', 'IL1B','TNF', 'IFNG','IL6','IL2','IL10']\n",
    "\n",
    "sc.pl.umap( adata, color = goi , color_map = 'viridis')\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.pl.umap( adata, color = ['n_genes', 'n_counts', 'percent_mito'] , color_map = 'viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional Labeling ###\n",
    "\n",
    "If `labeling_to_use` is specified, additional labels are taken from annotations in \"metadata.tsv\", and the data associated with additional labelling will be exported to files. And the fract_pos.gct and average.gct files are generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (labeling_to_use != 'None'):\n",
    "    adata = bc.st.additional_labeling(adata, labeling_to_use, labeling_name, labeling_description, labeling_author, results_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complete the log-file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info('Entire workflow completed.')\n",
    "logging.info('\\tTime for entire workflow: '+str(round(time.time()-start0, 3))+'s')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write QC Report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc.st.write_qc(adata_unfiltered = adata_unfiltered, \n",
    "                 adata_filtered = adata,\n",
    "                 version = version, \n",
    "                 analysis_name = analysis_name, \n",
    "                 standard_min_genes = standard_min_genes, \n",
    "                 standard_min_cells = standard_min_cells, \n",
    "                 standard_min_counts = standard_min_counts, \n",
    "                 standard_percent_mito = standard_percent_mito, \n",
    "                 standard_max_counts = standard_max_counts,\n",
    "                 standard_n_genes = standard_n_genes,\n",
    "                 filtering_output1 = filtering1,\n",
    "                 filtering_output2 = filtering2,\n",
    "                 results_folder = results_folder,\n",
    "                 css_path = os.path.join(os.path.dirname(bc.__file__),'st', 'style.css'))\n",
    "logging.info('QC Report generated and saved as .html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Session info\n",
    "\n",
    "Finally, we report the session info with the package [sinfo](https://gitlab.com/joelostblom/sinfo)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture session_info\n",
    "sinfo()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logging.info(session_info)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert to html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bc.convert_notebook_to_HTML()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adata.obs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "besca_py39_dev",
   "language": "python",
   "name": "besca_py39_dev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "vscode": {
   "interpreter": {
    "hash": "b9dea2d003d87676325b87da227df116e4cd201f5bea5c1857be972557bccd45"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}