notebooks/pyllelic_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example Pyllelic Use-Case Notebook\n",
    "\n",
    "<img src=\"https://github.com/Paradoxdruid/pyllelic/blob/master/assets/pyllelic_logo.png?raw=true\" alt=\"pyllelic logo\" width=\"100\"/>\n",
    "\n",
    "## Background\n",
    "\n",
    "This notebook illustrates the import and use of `pyllelic` in a jupyter environment.\n",
    "\n",
    "Source code: <https://github.com/Paradoxdruid/pyllelic>\n",
    "\n",
    "Documentation: <https://paradoxdruid.github.io/pyllelic/>\n",
    "\n",
    "## Pre-setup / File preparation\n",
    "\n",
    "### Obtaining fastq data\n",
    "\n",
    "We can download rrbs (reduced representation bisulfite sequencing) data from the Encode project:\n",
    "<http://hgdownload.cse.ucsc.edu/goldenPath/hg19/encodeDCC/wgEncodeHaibMethylRrbs/>\n",
    "\n",
    "Those files are in unaligned fastq format.  We will need to align these to a reference human genome.\n",
    "\n",
    "### Retrieve promoter sequence\n",
    "\n",
    "We will download the genomic region of interest from the UCSC Genome browser and save it in a text file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Allow import of module from above notebooks directory\n",
    "import sys\n",
    "sys.path.append('..')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "from pyllelic import process\n",
    "# Retrieve promoter genomic sequence\n",
    "process.retrieve_seq(\"{prom_filename}.txt\", chrom=\"chr5\", start=1293200, end=1296000)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing bisultife-converted genome\n",
    "\n",
    "To prepare the genome and align reads, we'll use [bismark](https://github.com/FelixKrueger/Bismark), bowtie2, and samtools (through its pysam wrapper).\n",
    "\n",
    "First, we need to download a genomic index sequence: <http://hgdownload.soe.ucsc.edu/goldenPath/hg19>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Processing imports\n",
    "from pathlib import Path\n",
    "\n",
    "# Set up file paths\n",
    "genome = Path(\"/{your_directory}/{genome_file_directory}\")\n",
    "fastq = Path(\"/{your_directory}/{your_fastq_file.fastq.gz}\")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**WARNING:** The next command is processor, RAM, and time intensive, and only needs to be run once!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Prepare genome via bismark\n",
    "process.prepare_genome(genome) # can optionally give path to bowtie2 if not in PATH\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aligning reads\n",
    "\n",
    "**WARNING:** The next command is processor, RAM, and time intensive, and only needs to be run once!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Convert fastq to bismark-aligned bam\n",
    "from pyllelic import process\n",
    "process.bismark(genome, fastq)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notes:\n",
    "\n",
    "* We recommend renaming your `.bam` file to encode cell-line and tissue.  Our convention is: `fh_CELLLINE_TISSUE.REGION.bam`\n",
    "\n",
    "Next, we need to sort and index the bam file using samtools functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Sort the bamfile\n",
    "bamfile = Path(\"/{your_directory}/{bam_filename}.bam\")\n",
    "process.sort_bam(bamfile)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Create an index of the sorted bamfile\n",
    "sorted_bam = Path(\"/{your_directory}/{bam_filename}_sorted.bam\")\n",
    "process.index_bam(sorted_bam)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Organize directory contents\n",
    "\n",
    "* Place sorted bam files and index files (again, rename to capture cell-line and tissue info) in the `test` folder for analysis by pyllelic.\n",
    "* Place the promoter sequence in your main directory.\n",
    "\n",
    "## Set-up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyllelic import pyllelic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Set up your disk location:  ```base_path``` should be the directory we'll do our work in\n",
    "2. Make a sub-directory under ```base_path``` with a folder named ```test``` and put the ```.bam``` and ```.bai``` files in ```test```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = pyllelic.configure(\n",
    "    base_path=\"/home/jovyan/assets/\",\n",
    "    prom_file=\"tert_genome.txt\",\n",
    "    prom_start=1293200,\n",
    "    prom_end=1296000,\n",
    "    chrom=\"5\",\n",
    "    offset=1293000,\n",
    "    # viz_backend=\"plotly\",\n",
    "    # fname_pattern=r\"^[a-zA-Z]+_([a-zA-Z0-9]+)_.+bam$\",\n",
    "    # test_dir=\"test\",\n",
    "    # results_dir=\"results\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main Parsing Functions\n",
    "\n",
    "### Find files to analyze"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "files_set = pyllelic.make_list_of_bam_files(config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# files_set # uncomment for debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Perform full methylation analysis and generate data object\n",
    "\n",
    "**Warning**: This step is processor and time intensive and will perform all processing and analysis of the bam file data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pyllelic.pyllelic(config=config, files_set=files_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check main data outcomes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# \", \".join(data.positions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.means.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.modes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.diffs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data.diffs.loc[:,data.diffs.any()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# data.quma_results[data.means.index[0]].values.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.individual_data.loc[data.means.index[0], data.means.columns[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Write Output\n",
    "\n",
    "### Save entire object as pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data.save_pickle(\"test_data.pickle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Quma Results to excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data.save(\"output.xlsx\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save analysis files (means, modes, diffs) to excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data.write_means_modes_diffs(\"Full_\") # Sets the filename stub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reopen saved object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data = pyllelic.GenomicPositionData.from_pickle(\"test_data2.pickle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Analysis\n",
    "\n",
    "### View raw data of methylation percentage per read"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = data.individual_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical Tests for Normality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summ = data.summarize_allelic_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summ.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ad_big_diffs = summ.pivot(index=\"position\", columns=\"cellLine\", values=\"ad_stat\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ad_big_diffs = ad_big_diffs.dropna(axis=1, how=\"all\").count(axis=1).to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ad_big_diffs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Raw quma results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.quma_results[data.means.index[0]].values.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.individual_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Data\n",
    "\n",
    "### Histograms of reads at a cell line and genomic position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "CELL = data.means.index[1]\n",
    "POS = data.means.columns[5]\n",
    "data.histogram(CELL, POS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Heatmap of mean methylation values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.heatmap(min_values=1, width=600, height=800, data_type=\"means\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bar chart of significant methylation differences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.sig_methylation_differences()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bar graphs of individual reads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.reads_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find values with a methylation difference above a threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import plotly.express as px"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DIFF_THRESHOLD = 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "large_diffs = data.diffs[(data.diffs >= DIFF_THRESHOLD).any(1)].dropna(\n",
    "    how=\"all\", axis=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "large_diffs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interesting = {}\n",
    "for index, row in large_diffs.iterrows():\n",
    "    if row.loc[(row != 0)].dropna().any():\n",
    "        interesting[index] = row.loc[(row != 0)].dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "big_diffs = pd.DataFrame.from_dict(interesting)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "big_diffs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "big_diffs_counts = big_diffs.dropna(axis=1, how=\"all\").count(axis=1).to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = px.bar(big_diffs_counts, template=\"seaborn\")\n",
    "fig.update_layout(xaxis_type=\"linear\", showlegend=False, barmode=\"stack\")\n",
    "fig.update_xaxes(\n",
    "    tickformat=\"r\", tickangle=45, nticks=40, title=\"Position\", range=[1292981, 1295979]\n",
    ")\n",
    "fig.update_yaxes(title=\"# of large differences\")\n",
    "fig.update_traces(width=50)\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting read methylation distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = data.individual_data.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_1D(series):\n",
    "    \"\"\"From https://towardsdatascience.com/dealing-with-list-values-in-pandas-dataframes-a177e534f173\"\"\"\n",
    "    if isinstance(series, pd.Series):\n",
    "        series = series.dropna()\n",
    "        return pd.Series([x for _list in series for x in _list])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "POS = data.means.columns[5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_1D(df.loc[:,POS]).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_1D(df.loc[:,POS])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "to_1D(df.loc[:,POS]).mode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Distribution at one cell line, position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "px.bar(to_1D(df.loc[:,POS]).value_counts(normalize=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Distribution across a cell line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CELL = data.means.index[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = df.loc[CELL]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2.explode().value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "px.violin(df2.explode())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Distribution across all cell lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_values = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for each in df.index:\n",
    "    temp = pd.Series(df.loc[each].explode())\n",
    "    #     print(temp)\n",
    "    all_values.append(temp)\n",
    "#     print(all_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flat_list = [item for sublist in all_values for item in sublist]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flat_series = pd.Series(flat_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flat_series.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "px.violin(flat_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify large differences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_big_diffs(df, min_diff: float):\n",
    "    out = {}\n",
    "    for each in df.columns:\n",
    "        mean = to_1D(df[each].dropna()).mean()\n",
    "        mode = to_1D(df[each].dropna()).mode()\n",
    "        if not mode.empty:\n",
    "            diff = abs(mean - mode.values[0])\n",
    "            if diff > min_diff:\n",
    "                out[each] = diff\n",
    "\n",
    "    if out:\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "big_ones = find_big_diffs(df, 0.1)\n",
    "big_ones"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}