ipynb/tutorials/Phylogenetic-IBD-Tutorial-QIIME2-CLI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Phylogenetic Compositional Tensor Factorization (phylo-CTF) Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial builds upon the CTF tutorial and explains how you can also perform CTF weighted by phylogenetic information. If you would like to better understand CTF please first complete that tutorial. In this tutorial we will use Phylogenetic Compositional Tensor Factorization (phylo-CTF) which we provide in the software package [gemelli](https://github.com/biocore/gemelli). Phylo-CTF can account for repeated measures, compositionality, and sparsity in microbiome data.\n",
    "\n",
    "In this tutorial we use _gemelli_ to perform phylo-CTF on a time series dataset comparing Crohn's and control subjects over a period of 25 weeks published in [Vázquez-Baeza et al](https://gut.bmj.com/content/67/9/1743). First we will download the processed data originally from [here](https://qiita.ucsd.edu/study/description/2538#). This data can be downloaded with the following links:\n",
    "\n",
    "* **Table** (table.qza) | [download](https://github.com/biocore/gemelli/tree/master/ipynb/tutorials/IBD-2538/data/table.qza)\n",
    "* **Rarefied Table** (rarefied-table.qza) | [download](https://github.com/biocore/gemelli/tree/master/ipynb/tutorials/IBD-2538/data/rarefied-table.qza)\n",
    "* **Sample Metadata** (metadata.tsv) | [download](https://github.com/biocore/gemelli/tree/master/ipynb/tutorials/IBD-2538/data/metadata.tsv)\n",
    "* **Feature Metadata** (taxonomy.qza) | [download](https://github.com/biocore/gemelli/tree/master/ipynb/tutorials/IBD-2538/data/taxonomy.qza)\n",
    "* **Tree** (sepp-insertion-tree.qza) | [download](https://github.com/biocore/gemelli/tree/master/ipynb/tutorials/IBD-2538/data/sepp-insertion-tree.qza)\n",
    "\n",
    "**Note**: This tutorial assumes you have installed [QIIME2](https://qiime2.org/) using one of the procedures in the [install documents](https://docs.qiime2.org/2020.2/install/). This tutorial also assumed you have installed, [Qurro](https://github.com/biocore/qurro), [DEICODE](https://github.com/biocore/DEICODE), and [gemelli](https://github.com/biocore/gemelli).\n",
    "\n",
    "First, we will make a tutorial directory and download the data above and move the files to the `IBD-2538/data` directory:\n",
    "\n",
    "```bash\n",
    "mkdir IBD-2538\n",
    "```\n",
    "```bash\n",
    "# move downloaded data here\n",
    "mkdir IBD-2538/data\n",
    "```\n",
    "\n",
    "Next we will import our data with the QIIME2 Python API. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: IBD-2538/core-metric-output: File exists\r\n"
     ]
    }
   ],
   "source": [
    "!mkdir IBD-2538\n",
    "!mkdir IBD-2538/core-metric-output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to account for the correlation among samples from the same subject we will employ phylogenetic compositional tensor factorization (CTF). CTF builds on the ability to account for compositionality and sparsity using the robust center log-ratio transform covered in the RPCA tutorial (found [here](https://forum.qiime2.org/t/robust-aitchison-pca-beta-diversity-with-deicode)) but restructures and factors the data as a tensor. Here we will run CTF through _gemelli_ and explore/interpret the different results. Additionally, phylogenetic CTF incorporates internal nodes and branch lengths of a tree into the factorization. \n",
    "\n",
    "There are two forms of phylogenetic CTF bieng (1) with taxonomy information and (2) without. Here we will run phylogenetic-ctf-with-taxonomy but it can also be run without taxonomy called just phylogenetic-ctf.\n",
    "\n",
    "To run phylo-CTF we only need to run one command (gemelli phylogenetic-ctf-with-taxonomy). The required input requirements are:\n",
    "\n",
    "1. table\n",
    "    - The table is of type `FeatureTable[Frequency]` which is a table where the rows are features (e.g. ASVs/microbes), the columns are samples, and the entries are the number of sequences for each sample-feature pair.\n",
    "2. tree\n",
    "    - This is a phylogeny of type `Phylogeny[Rooted]` where all the features in the `table` are represented in the tree. \n",
    "3. feature-metadata-file\n",
    "    - This is a metadata file (e.g. tsv, or `FeatureTable[Taxonomy]` .qza) where the rows are matched to the table features and the columns are feature metadata such as taxonomy, gene pathway, etc... \n",
    "4. sample-metadata\n",
    "    - This is a QIIME2 formatted [metadata](https://docs.qiime2.org/2020.2/tutorials/metadata/) (e.g. tsv format) where the rows are samples matched to the (1) table and the columns are different sample data (e.g. time point). \n",
    "5. individual-id-column\n",
    "    - This is the name of the column in the (2) metadata that indicates the individual subject/site (e.g. subject ID) that was sampled repeatedly.\n",
    "6. state-column\n",
    "    - This is the name of the column in the (2) metadata that indicates the numeric repeated measure (e.g., Time in months/days) or non-numeric category (i.e. decade/body-site). \n",
    "7. output-dir\n",
    "    - The desired location of the output. We will cover each output independently below.  \n",
    "\n",
    "\n",
    "In this tutorial our individual-id-column is `host_subject_id` and our state-column is different time points denoted as `timepoint` in the sample metadata. Now we are ready to run phylo-CTF:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved FeatureTable[Frequency] to: IBD-2538/data/matched-table/filtered_table.qza\u001b[0m\n",
      "\u001b[32mSaved FeatureTable[Frequency] to: IBD-2538/data/matched-table/removed_table.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults % Properties('biplot') to: IBD-2538/ctf-results/subject_biplot.qza\u001b[0m\n",
      "\u001b[32mSaved PCoAResults % Properties('biplot') to: IBD-2538/ctf-results/state_biplot.qza\u001b[0m\n",
      "\u001b[32mSaved DistanceMatrix to: IBD-2538/ctf-results/distance_matrix.qza\u001b[0m\n",
      "\u001b[32mSaved SampleData[SampleTrajectory] to: IBD-2538/ctf-results/state_subject_ordination.qza\u001b[0m\n",
      "\u001b[32mSaved FeatureData[FeatureTrajectory] to: IBD-2538/ctf-results/state_feature_ordination.qza\u001b[0m\n",
      "\u001b[32mSaved Phylogeny[Rooted] to: IBD-2538/ctf-results/counts_by_node_tree.qza\u001b[0m\n",
      "\u001b[32mSaved FeatureTable[Frequency] to: IBD-2538/ctf-results/counts_by_node.qza\u001b[0m\n",
      "\u001b[32mSaved FeatureData[Taxonomy] to: IBD-2538/ctf-results/t2t_taxonomy.qza\u001b[0m\n",
      "\u001b[32mSaved FeatureTable[Frequency] to: IBD-2538/ctf-results/subject_table.qza\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime fragment-insertion filter-features\\\n",
    "    --i-table IBD-2538/data/table.qza  \\\n",
    "    --i-tree IBD-2538/data/sepp-insertion-tree.qza  \\\n",
    "    --output-dir IBD-2538/data/matched-table\n",
    "\n",
    "!qiime gemelli phylogenetic-ctf-with-taxonomy\\\n",
    "    --i-table  IBD-2538/data/matched-table/filtered_table.qza\\\n",
    "    --i-phylogeny  IBD-2538/data/sepp-insertion-tree.qza\\\n",
    "    --m-sample-metadata-file IBD-2538/data/metadata.tsv\\\n",
    "    --m-taxonomy-file IBD-2538/data/taxonomy.qza\\\n",
    "    --p-state-column timepoint\\\n",
    "    --p-individual-id-column host_subject_id\\\n",
    "    --p-min-feature-frequency 10\\\n",
    "    --output-dir IBD-2538/ctf-results\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We will now cover the output files:\n",
    "* subject_biplot\n",
    "* state_biplot\n",
    "* distance_matrix\n",
    "* state_subject_ordination\n",
    "* state_feature_ordination\n",
    "* counts_by_node_tree\n",
    "* counts_by_node\n",
    "* t2t_taxonomy\n",
    "* subject_table\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will explore the `subject_biplot` which is an ordination where dots represent _subjects_ not _samples_ and arrows represent features (e.g. ASVs). First, we will need to aggregate the metadata by subject (i.e. collapsing the metadata of all samples from a given subject). This can be done by hand or using DataFrames in python (with pandas) or R like so:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ibd</th>\n",
       "      <th>active_disease</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>#SampleID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>s1000100</th>\n",
       "      <td>Control</td>\n",
       "      <td>quiescent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>s1000200</th>\n",
       "      <td>Control</td>\n",
       "      <td>quiescent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>s1000300</th>\n",
       "      <td>Control</td>\n",
       "      <td>quiescent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>s1000500</th>\n",
       "      <td>Control</td>\n",
       "      <td>quiescent</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>s1000600</th>\n",
       "      <td>Control</td>\n",
       "      <td>quiescent</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               ibd active_disease\n",
       "#SampleID                        \n",
       "s1000100   Control      quiescent\n",
       "s1000200   Control      quiescent\n",
       "s1000300   Control      quiescent\n",
       "s1000500   Control      quiescent\n",
       "s1000600   Control      quiescent"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from qiime2 import Metadata\n",
    "import numpy as np\n",
    "from biom import Table\n",
    "from skbio import OrdinationResults\n",
    "\n",
    "# first we import the metdata into pandas\n",
    "mf = pd.read_csv('IBD-2538/data/metadata.tsv', sep='\\t',index_col=0)\n",
    "# next we aggregate by subjects (i.e. 'host_subject_id') \n",
    "# and keep the first instance of 'diagnosis_full' by subject.\n",
    "mf = mf.groupby('host_subject_id').agg({'ibd':'first','active_disease':'first'})\n",
    "# now we save the metadata in QIIME2 format.\n",
    "mf.index.name = '#SampleID'\n",
    "mf.to_csv('IBD-2538/data/subject-metadata.tsv', sep='\\t')\n",
    "mf.head(5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike conventional CTF, the arrows in the phylo-CTF biplot can be both internal nodes and features of the table. In order to label the taxonomy we use the lowest common ancestor of that internal node. To do this we feed the `t2t_taxonomy` output from phylo-CTF into the biplot. Additionally, we can use Empress to generate plots combining both the tree and biplot view at one time. This view can help us understand what phylogenetic partitions separate our samples along a PC axis. \n",
    "\n",
    "Phylo-CTF output contains a table that allows us to use the internal nodes of the tree as possible log-ratios (i.e. `counts_by_node`). In this table the internal nodes contain the sum of all the counts up to that node. `Note,` it is important not to take the log-ratio of two internal nodes in the tree so we will visualize the loadings on the tree using [Empress](https://github.com/biocore/empress).\n",
    "\n",
    "The log-ratio can then be taken with [Qurro](https://github.com/biocore/qurro) to explore the feature loading partitions (arrows) in this biplot as a log-ratio of the original table counts. This allows us to relate these low-dimensional representations back to our original data. Additionally, log-ratios provide a nice set of data points for additional analysis such as LME models. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: IBD-2538/ctf-results/empress.qzv\u001b[0m\n",
      "\u001b[32mSaved Visualization to: IBD-2538/ctf-results/qurro.qza.qzv\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!qiime empress community-plot\\\n",
    "    --i-tree IBD-2538/ctf-results/counts_by_node_tree.qza\\\n",
    "    --i-feature-table IBD-2538/ctf-results/subject_table.qza\\\n",
    "    --i-pcoa IBD-2538/ctf-results/subject_biplot.qza\\\n",
    "    --m-sample-metadata-file IBD-2538/data/subject-metadata.tsv\\\n",
    "    --m-feature-metadata-file IBD-2538/ctf-results/t2t_taxonomy.qza\\\n",
    "    --p-filter-missing-features\\\n",
    "    --p-number-of-features 50\\\n",
    "    --o-visualization IBD-2538/ctf-results/empress.qzv\n",
    "\n",
    "!qiime qurro loading-plot\\\n",
    "    --i-ranks IBD-2538/ctf-results/subject_biplot.qza\\\n",
    "    --i-table IBD-2538/ctf-results/counts_by_node.qza\\\n",
    "    --m-sample-metadata-file IBD-2538/data/metadata.tsv\\\n",
    "    --m-feature-metadata-file  IBD-2538/ctf-results/t2t_taxonomy.qza\\\n",
    "    --o-visualization IBD-2538/ctf-results/qurro.qza\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this visualization we can see that the Crohn's subjects clearly separate from the healthy controls in the ordination on the right. We can also observe, by adding a barplot of the PC2 loadings (see the [empress tutorial](https://github.com/biocore/empress) for more info on how to make these plots). \n",
    "\n",
    "![image.png](etc/ctf_empress_plot_one2.png)\n",
    "\n",
    "One additional benefit to having the phylogeny is the ability to explore ratios between phylogenetic paritions. For example the nodes n58 and n52 within the Bacteroidetes phylum which represent Bacteroides and Prevotella genera respectively. \n",
    "\n",
    "\n",
    "![image.png](etc/ctf_empress_plot_two2.png)\n",
    "\n",
    "![image.png](etc/ctf_empress_plot_three2.png)\n",
    "\n",
    "We can then obtain the log-ratio based between these nodes from the qurro visualization.\n",
    "\n",
    "\n",
    "![image.png](etc/ctf_phylo_qurro.png)\n",
    "\n",
    "\n",
    "We can then export the log-ratio output and plot the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>timepoint</th>\n",
       "      <th>host_subject_id</th>\n",
       "      <th>ibd</th>\n",
       "      <th>Current_Natural_Log_Ratio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>#SampleID</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2538.1000102</th>\n",
       "      <td>25</td>\n",
       "      <td>s1000100</td>\n",
       "      <td>Control</td>\n",
       "      <td>4.663439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2538.1000104000004</th>\n",
       "      <td>6</td>\n",
       "      <td>s1000100</td>\n",
       "      <td>Control</td>\n",
       "      <td>0.470004</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    timepoint host_subject_id      ibd  \\\n",
       "#SampleID                                                \n",
       "2538.1000102               25        s1000100  Control   \n",
       "2538.1000104000004          6        s1000100  Control   \n",
       "\n",
       "                    Current_Natural_Log_Ratio  \n",
       "#SampleID                                      \n",
       "2538.1000102                         4.663439  \n",
       "2538.1000104000004                   0.470004  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# import log-ratio data\n",
    "metadata_one = pd.read_csv('IBD-2538/data/metadata.tsv',\n",
    "                           sep='\\t', index_col=0)\n",
    "# import rest of the metadata\n",
    "metadata_two = pd.read_csv('IBD-2538/ctf-results/sample_plot_data.tsv',\n",
    "                           sep='\\t', index_col=0)[['Current_Natural_Log_Ratio']]\n",
    "# merge the data\n",
    "log_ratio_metdata = pd.concat([metadata_two, metadata_one], axis=1)\n",
    "# ensure no duplicate columns\n",
    "log_ratio_metdata = log_ratio_metdata.dropna(subset=['Current_Natural_Log_Ratio'])\n",
    "log_ratio_metdata.index = log_ratio_metdata.index.astype(str)\n",
    "# export in QIIME2 format\n",
    "log_ratio_metdata = log_ratio_metdata[['timepoint','host_subject_id',\n",
    "                                       'ibd','Current_Natural_Log_Ratio']]\n",
    "\n",
    "log_ratio_metdata.index.name = '#SampleID'\n",
    "log_ratio_metdata.to_csv('IBD-2538/ctf-results/merged_sample_plot_data.tsv', sep='\\t')\n",
    "log_ratio_metdata.head(2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "As you can see above the metadata now has the added column of `Current_Natural_Log_Ratio` from Qurro. So now we will continue to explore this log-ratio by first plotting it explicitly over time with q2-longitudinal.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: IBD-2538/ctf-results/log_ratio_plot.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime longitudinal volatility \\\n",
    "    --m-metadata-file IBD-2538/ctf-results/merged_sample_plot_data.tsv\\\n",
    "    --p-state-column timepoint \\\n",
    "    --p-individual-id-column host_subject_id \\\n",
    "    --p-default-group-column ibd \\\n",
    "    --p-default-metric Current_Natural_Log_Ratio \\\n",
    "    --o-visualization IBD-2538/ctf-results/log_ratio_plot.qzv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "This demonstrates that we can recreate the separation by IBD that we saw in both the `subject_biplot` & `state_subject_ordination`, allowing us to associate specific taxa (in the numerator or denominator) with a particular phenotype.\n",
    "\n",
    "![image.png](etc/ctf_phylo-lr-vol.png)\n",
    "\n",
    "We can test the statistical power of this log-ratio to differentiate samples by IBD status using a linear mixed effects (LME) through q2-longitudinal. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32mSaved Visualization to: IBD-2538/ctf-results/lme_log_ratio.qzv\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "!qiime longitudinal linear-mixed-effects\\\n",
    "    --m-metadata-file IBD-2538/ctf-results/merged_sample_plot_data.tsv\\\n",
    "    --p-state-column timepoint \\\n",
    "    --p-individual-id-column host_subject_id \\\n",
    "    --p-group-columns ibd \\\n",
    "    --p-metric Current_Natural_Log_Ratio \\\n",
    "    --o-visualization IBD-2538/ctf-results/lme_log_ratio.qzv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this LME model we can see that indeed the IBD grouping is significant across time. \n",
    "\n",
    "![image.png](etc/ctf_lme-logratio-phylo.png)\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}