reproducing_hQTL_SNP_effect_prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c9fd58c6",
   "metadata": {},
   "source": [
    "# Reproducing the hQTL SNP effect predicition analysis\n",
    "\n",
    "This workbook explains the process of reproducing the hQTL SNP effect predicition analysis\n",
    "on the hQTL datasets as seen in our publication. The steps covered are:\n",
    "\n",
    "  * downloading the hQTL datasets\n",
    "  * preprocessing hQTL datasets\n",
    "  * precomputing reference and alternative DNA embeddings for SNPs\n",
    "  * running SNP effect predictions with checkpoints"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a3864be",
   "metadata": {},
   "source": [
    "## 1. Downloading the hQTL datasets\n",
    "\n",
    "The hQTL (histone mark quantative trait loci) datasets used in our publications come from BLUEPRINT phase 2 data from [Kundu et al](https://www.biorxiv.org/content/10.1101/2020.01.15.907436v1.full). This data contains both H3K4me1 and H3K27ac for three blood immune cell types: Neutrophils, Monocytes and T-Cells for a total of 6 hQTL datasets. The phase 2 data needs an access request to be downloaded but the phase 1 data from [chen et al., 2016](https://cancerdiscovery.aacrjournals.org/content/6/8/827) is readily available. See the function `qtl()` in `./bin/download_data_parallel.py` for an easy method to download them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4ffb3d9",
   "metadata": {},
   "source": [
    "## 2. Preprocessing the hQTL datasets\n",
    "\n",
    "There are two preprocessing steps to be run for the hQTL datasets:\n",
    "  1. standardise and quality control the BLUEPRINT hQTL datasets. This is done with [MungeSumstats](https://academic.oup.com/bioinformatics/article/37/23/4593/6380562). The R script `./bin/munge.R` is set up for this.\n",
    "  2. Process the hQTL datasets to be compatible with SLDP requirements. The python script `./bin/process_qtl_as_sumstats.py` is set up to run this. It also filters the SNPs to just those in HapMap3. See `./bin/get_sldp_dependencies.py` for where this file is downloaded."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41491f9f",
   "metadata": {},
   "source": [
    "## 3. Precomputing reference and alternative DNA embeddings for SNPs\n",
    "\n",
    "Predictions of SNP effects will be made for every SNP in any of the six hQTL datasets. This equates to >850K SNPs to be predicted on. For each SNP, two predictions are necessary to measure the effect - one for the reference and one for the alternative allele. And for each of these predictions we must predict three 3 times sliding the input window to capture the effect across the full input 200k base-pairs (see `Predict the effect of a genetic variant` in [using_enformer_celltyping](https://github.com/neurogenomics/EnformerCelltyping/blob/master/using_enformer_celltyping.ipynb) for further details on this). And finally, we need to repeat this for every cell type we want to test. This equates to `867,568*2*3*X` where X is the number of cell types of interest predictions.\n",
    "\n",
    "Suffice to say, this is a lot of predicitions. To help alleviate some of the time consumption, we can precompute the DNA embeddings from enformer for every SNP position so these don't have to be recomputed for every new cell type of interest. This is a similar idea to precomputing DNA embeddings when making genome-wide predictions of histone marks in a cell type - see `Predict histone marks for a cell type of interest` in [using_enformer_celltyping](https://github.com/neurogenomics/EnformerCelltyping/blob/master/using_enformer_celltyping.ipynb). \n",
    "\n",
    "We can do this by running `./bin/mke_sldp_train_data_arr.py`. Note this is set up to be used in parallel runs (like array jobs on a HPC). See the txt file `./metadata/create_sldp_train_dat.txt` for start and end numbers to run through all 867,568 unique SNPs. The downside to this approach is that it requires a massive amount of disk space. For every SNP (reference and alternative) it takes up **~110 TB** so this is worth bearing in mind."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc7e0785",
   "metadata": {},
   "source": [
    "## 4. Running SNP effect predictions with checkpoints \n",
    "\n",
    "The section `Predict the effect of a genetic variant` in [using_enformer_celltyping](https://github.com/neurogenomics/EnformerCelltyping/blob/master/using_enformer_celltyping.ipynb) gives details of the function `predict_snp_effect_sldp()` which makes predicting the effect of a genetic variant across Enformer Celltyping's full 200k base-pair input relatively easy. However, this function is not set up to make use of the precomputed DNA embeddings. To use these we have made the function `predict_snp_effect_sldp_checkpoint()` which works in the same way but will use the precomputed data, if it is available and otherwise it will create it.\n",
    "\n",
    "So now we have a method to compute genetic variants aggregated effect which will run more efficiently since it uses precomputed DNA embeddings. However, running this across all the variants in all the hQTL datasets for all cell types of interest like we did for the manuscript is still massively time consumming. Since genetic variants are shared across the different hQTL datasets, these can be computed just once for each cell type, we have two methods we used to take advantage of this and run the predictions in parallel.\n",
    "\n",
    "The first, is to save the computed genetic effect predictions so that they can be shared across different hQTL runs for the same cell type. For the manuscript, we predicted across the 6 BLUEPRINT 2 hQTL datasets for 6 cell types. You can do this using `pred_snp_effect_checkpoint_arr.py` and the inputs from `./metadata/with snp_eff_combns.txt`.\n",
    "\n",
    "The second option is to combine all six hQTL summary statistics and save the predicted effect of the genetic variants for the unique set of genetic variants (>850k genetic variants) for each cell type. You can do this again using the precomputed DNA embeddings for the 6 hQTL summary statistics and the 6 cell types we used in the manuscript using `pred_snp_effect_all_qtl_arr.py` and the inputs from `./metadata/snp_eff_all_qtls.txt`. This approach is compatable with the first so can be used in combination however, it is actually much quicker to just run this second approach with all the CPU machines you have available. To compute the 40k SNPs for each job in `./metadata/snp_eff_all_qtls.txt` will take roughly 35-40 hours (on a 40 CPU machine).\n",
    "\n",
    "One note on running this, it is better to get all parallel runs started than waiting for GPUs to free up, the time saving of solely using GPUs isn't worth it if you can get access to a lot more CPU machines to run this.\n",
    "\n",
    "Finally, once you have run all the SNP predictions you can run SLDP on the data. This can be done in parallel for each QTL dataset, each histone mark predicited effect in each cell type. This accounts to 432 separate runs for the 6 QTL datasets and 6 cell types we tested for the manuscript. You can run this using `./bin/run_sldp.py` and the inputs in `./metadata/sldp_runs.txt`, using the `sldp` conda environment. First just install the conda environment:\n",
    "\n",
    "```\n",
    "make sldpenv\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "EnformerCelltyping",
   "language": "python",
   "name": "enformercelltyping"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}