minapharm_ml_2022.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Minapharm ML Engineer Technical Assesment\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Candidate Name** : WALEED ELBADRY\n",
    "\n",
    "**Date** : 14/04/2022\n",
    "\n",
    "**Email** : wbadry@live.com\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "---\n",
    "\n",
    "A data repository is composed of _5000 articles_ is provided to be utilized to extract the most important topics and assigning it to each article with a probability score.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Libraries and dependancies\n",
    "\n",
    "---\n",
    "\n",
    "The developed notebook ran on Ubuntu _Linux 20.04_\n",
    "\n",
    "![Linux OS](images/os.png)\n",
    "\n",
    "Since it is odd to have dataset in **rar format**, an extra step is needed to extract it.\n",
    "\n",
    "It is recommended to install `unrar` application using terminal command\n",
    "\n",
    "```bash\n",
    "sudo apt install unrar\n",
    "```\n",
    "\n",
    "The application will be used as a tool to unrar the `minapharm downloaded dataset`.\n",
    "\n",
    "You may run the following command in the terminal to install all of the used libraries\n",
    "\n",
    "```bash\n",
    "python -m pip install -U pandas numpy scikit-learn gensim urlib progressbar rarfile nltk spacy pprint matplotlib pyldavis \n",
    "\n",
    "```\n",
    "\n",
    "Last , but not least, installing the dictionary needed for tokenization\n",
    "\n",
    "```bash\n",
    "python -m spacy download en_core_web_sm\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Libraries:'\n",
      "'progressbar : 2.5'\n",
      "'pandas : 1.4.2'\n",
      "'numpy : 1.21.5'\n",
      "'scikit-learn : 1.0.2'\n",
      "'gensim : 4.1.2'\n"
     ]
    }
   ],
   "source": [
    "# imports\n",
    "# Run in terminal or command prompt\n",
    "# python3 -m spacy download en\n",
    "import urllib\n",
    "import progressbar\n",
    "from rarfile import RarFile\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re, nltk, spacy, gensim\n",
    "\n",
    "# Sklearn\n",
    "import sklearn\n",
    "from sklearn.decomposition import LatentDirichletAllocation, TruncatedSVD\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from pprint import pprint\n",
    "\n",
    "# Plotting tools\n",
    "import pyLDAvis\n",
    "import pyLDAvis.sklearn\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "pd.set_option('mode.chained_assignment', None)\n",
    "pd.set_option('display.max_columns', 100)\n",
    "\n",
    "# Libraries versions\n",
    "pprint('Libraries:')\n",
    "pprint('progressbar : {:}'.format(progressbar.__version__))\n",
    "pprint('pandas : {:}'.format(pd.__version__))\n",
    "pprint('numpy : {:}'.format(np.__version__))\n",
    "pprint('scikit-learn : {:}'.format(sklearn.__version__))\n",
    "pprint('gensim : {:}'.format(gensim.__version__))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class MyProgressBar():\n",
    "    \"\"\"Class to display the progress bar during the download of the dataset\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        \"\"\"Initialize the progress bar count\n",
    "        \"\"\"\n",
    "        self.pbar = None\n",
    "\n",
    "    def __call__(self, block_num, block_size, total_size):\n",
    "        \"\"\"Progress bar visualization\n",
    "\n",
    "        Args:\n",
    "            block_num (byte): The current block number to be downloaded\n",
    "            block_size (int): Block size\n",
    "            total_size (_type_): Total number of blocks to be downloaded\n",
    "        \"\"\"\n",
    "        if not self.pbar:\n",
    "            self.pbar = progressbar.ProgressBar(maxval=total_size)\n",
    "            self.pbar.start()\n",
    "\n",
    "        downloaded = block_num * block_size\n",
    "        if downloaded < total_size:\n",
    "            self.pbar.update(downloaded)\n",
    "        else:\n",
    "            self.pbar.finish()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading the compressed file, please wait ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% |########################################################################|\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Download is completed\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the dataset from minapharm provided link\n",
    "url = 'https://www.minapharm.com/gShare/Pubmed5k.rar'\n",
    "print('Downloading the compressed file, please wait ...')\n",
    "status = urllib.request.urlretrieve(url, 'data/Pubmed5k.rar', MyProgressBar())\n",
    "print('Download is completed')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the excel file from the rar file\n",
    "RarFile.UNRAR_TOOL = 'unrar'\n",
    "rar_file_path = 'data/Pubmed5k.rar'\n",
    "with RarFile(rar_file_path) as file:\n",
    "    file.extract(file.namelist()[0], path='data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read the excel file into dataframe\n",
    "excel_file_path = 'data/Pubmed5k.xlsx'\n",
    "df = pd.read_excel(excel_file_path, sheet_name='random 5k', header=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Data wrangling\n",
    "\n",
    "---\n",
    "\n",
    "The first step after importing the data is to check some insights regarding the **data structure** .\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset has:\n",
      " 4999 records\n",
      "    3 columns\n",
      "\n",
      "The number of empty records in any column:\n",
      "ArticleID    0\n",
      "Title        0\n",
      "Abstract     0\n",
      "dtype: int64\n",
      "\n",
      "Number of unique records:\n",
      "ArticleID    4999\n",
      "Title        4999\n",
      "Abstract     4989\n",
      "dtype: int64\n",
      "\n",
      "Exploring columns datatypes:\n",
      "ArticleID     int64\n",
      "Title        object\n",
      "Abstract     object\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Some quick insights\n",
    "print('The dataset has:')\n",
    "print('{0:5d} records'.format(df.shape[0]))\n",
    "print('{0:5d} columns'.format(df.shape[1]))\n",
    "print()\n",
    "\n",
    "\n",
    "print('The number of empty records in any column:')\n",
    "print(df.isnull().sum())\n",
    "print()\n",
    "\n",
    "print('Number of unique records:')\n",
    "print(df.nunique())\n",
    "print()\n",
    "\n",
    "print('Exploring columns datatypes:')\n",
    "print(df.dtypes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the _previous insights_, there are several duplicated records, mainly in the `Abstract` section\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Cleaning articles with missing abstracts\n",
    "\n",
    "---\n",
    "\n",
    "After wrangling the data, each article is stored into 3 columns:\n",
    "\n",
    "- `Article ID` : serving as the unique ID for storing the article.\n",
    "- `Title` : The article published title or main header.\n",
    "- `Abstract` : Summary of the article.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 10 duplicated records in the abstract\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ArticleID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Abstract</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2590</th>\n",
       "      <td>34669440</td>\n",
       "      <td>Peptide-based urinary monitoring of fibrotic nonalcoholic steatohepatitis by mass-barcoded activity-based sensors.</td>\n",
       "      <td>[Figure: see text].</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2591</th>\n",
       "      <td>34669441</td>\n",
       "      <td>A rapid assay provides on-site quantification of tetrahydrocannabinol in oral fluid.</td>\n",
       "      <td>[Figure: see text].</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2592</th>\n",
       "      <td>34669442</td>\n",
       "      <td>Fatal enhanced respiratory syncytial virus disease in toddlers.</td>\n",
       "      <td>[Figure: see text].</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2593</th>\n",
       "      <td>34669443</td>\n",
       "      <td>Macrophage migration inhibitory factor drives pathology in a mouse model of spondyloarthritis and is associated with human disease.</td>\n",
       "      <td>[Figure: see text].</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2594</th>\n",
       "      <td>34669444</td>\n",
       "      <td>Development of ICT01, a first-in-class, anti-BTN3A antibody for activating V?9Vd2 T cell-mediated antitumor immune response.</td>\n",
       "      <td>[Figure: see text].</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3872</th>\n",
       "      <td>34258891</td>\n",
       "      <td>Too much of a good thing in ischemic mitral: lessons for surgeons and cardiologists.</td>\n",
       "      <td>No abstract present.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3873</th>\n",
       "      <td>34258892</td>\n",
       "      <td>COVID-19 infection and cardiometabolic complications: short- and long-term treatment and management considerations.</td>\n",
       "      <td>No abstract present.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3874</th>\n",
       "      <td>34258893</td>\n",
       "      <td>Comments on Cardiovascular effects of waterpipe smoking: a systematic review and meta-analysis.</td>\n",
       "      <td>No abstract present.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3875</th>\n",
       "      <td>34258894</td>\n",
       "      <td>A case of COVID-19 infection quickly relieved after nasal instillations and gargles with povidone iodine.</td>\n",
       "      <td>No abstract present.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4757</th>\n",
       "      <td>34425679</td>\n",
       "      <td>Study of anabolic activity of dry extracts of leaves and rhizomes of Iris hungarica on a model of hydrocortisone-induced protein catabolism.</td>\n",
       "      <td>This article presents the results of the study of the anabolic effect of dry extracts of Iris hungarica leaves and rhizomes on the model of hydrocortisoneinduced protein catabolism. Previous studies have established the presence of anabolic activity of dry extracts of Iris hungarica leaves and rhizomes in intact animals. Therefore, it was reasonable to study the effect of the experimental extracts on the state of protein metabolism, which is regulated by glucocorticoids. The model of hydrocortisoneinduced protein catabolism was used to determine anabolic activity for dry extracts of Iris hungarica leaves and rhizomes at a  dose of 150  mg/kg by monitoring the recovery of body weight and the increase in the total protein in the cardiac muscle of rats and in muscle tissue homogenate, which is aimed to promote myofibrillar hypertrophy. Dry extract of Iris hungarica rhizomes reduced urea excretion, normalized metabolism, restored nitrogen balance, and inhibited protein catabolism. The ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      ArticleID  \\\n",
       "2590   34669440   \n",
       "2591   34669441   \n",
       "2592   34669442   \n",
       "2593   34669443   \n",
       "2594   34669444   \n",
       "3872   34258891   \n",
       "3873   34258892   \n",
       "3874   34258893   \n",
       "3875   34258894   \n",
       "4757   34425679   \n",
       "\n",
       "                                                                                                                                             Title  \\\n",
       "2590                            Peptide-based urinary monitoring of fibrotic nonalcoholic steatohepatitis by mass-barcoded activity-based sensors.   \n",
       "2591                                                          A rapid assay provides on-site quantification of tetrahydrocannabinol in oral fluid.   \n",
       "2592                                                                               Fatal enhanced respiratory syncytial virus disease in toddlers.   \n",
       "2593           Macrophage migration inhibitory factor drives pathology in a mouse model of spondyloarthritis and is associated with human disease.   \n",
       "2594                  Development of ICT01, a first-in-class, anti-BTN3A antibody for activating V?9Vd2 T cell-mediated antitumor immune response.   \n",
       "3872                                                          Too much of a good thing in ischemic mitral: lessons for surgeons and cardiologists.   \n",
       "3873                           COVID-19 infection and cardiometabolic complications: short- and long-term treatment and management considerations.   \n",
       "3874                                               Comments on Cardiovascular effects of waterpipe smoking: a systematic review and meta-analysis.   \n",
       "3875                                     A case of COVID-19 infection quickly relieved after nasal instillations and gargles with povidone iodine.   \n",
       "4757  Study of anabolic activity of dry extracts of leaves and rhizomes of Iris hungarica on a model of hydrocortisone-induced protein catabolism.   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     Abstract  \n",
       "2590                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      [Figure: see text].  \n",
       "2591                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      [Figure: see text].  \n",
       "2592                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      [Figure: see text].  \n",
       "2593                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      [Figure: see text].  \n",
       "2594                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      [Figure: see text].  \n",
       "3872                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     No abstract present.  \n",
       "3873                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     No abstract present.  \n",
       "3874                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     No abstract present.  \n",
       "3875                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     No abstract present.  \n",
       "4757  This article presents the results of the study of the anabolic effect of dry extracts of Iris hungarica leaves and rhizomes on the model of hydrocortisoneinduced protein catabolism. Previous studies have established the presence of anabolic activity of dry extracts of Iris hungarica leaves and rhizomes in intact animals. Therefore, it was reasonable to study the effect of the experimental extracts on the state of protein metabolism, which is regulated by glucocorticoids. The model of hydrocortisoneinduced protein catabolism was used to determine anabolic activity for dry extracts of Iris hungarica leaves and rhizomes at a  dose of 150  mg/kg by monitoring the recovery of body weight and the increase in the total protein in the cardiac muscle of rats and in muscle tissue homogenate, which is aimed to promote myofibrillar hypertrophy. Dry extract of Iris hungarica rhizomes reduced urea excretion, normalized metabolism, restored nitrogen balance, and inhibited protein catabolism. The ...  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the dupplicated records\n",
    "count = df.Abstract.duplicated().sum()\n",
    "print(\"There are {:} duplicated records in the abstract\".format(count))\n",
    "\n",
    "df_dup = df[df.Abstract.duplicated()]\n",
    "df_dup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the removal of these records won't affect the **topic modeling task** since the information loss is expected to be:\n",
    "\n",
    "$$\n",
    "Information \\ Loss (\\%) = {10 \\over 4999} \\times 100 = 0.2 \\%\n",
    "$$\n",
    "\n",
    "The loss was assessed by the candidate to be acceptable. Therefore, _all of the duplicated records_ would be removed.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cleaned dataset has:\n",
      " 4989 records\n",
      "    3 columns\n"
     ]
    }
   ],
   "source": [
    "# Remove duplicated records\n",
    "df_clean = df[~df.Abstract.duplicated()]\n",
    "\n",
    "# Show number of records after cleaning it\n",
    "print('The cleaned dataset has:')\n",
    "print('{0:5d} records'.format(df_clean.shape[0]))\n",
    "print('{0:5d} columns'.format(df_clean.shape[1]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Apparently**, the two columns, namely `Title` and `Abstract` are string but was read as object. We may convert it into string for easy tokenization later.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Verifying columns datatypes:\n",
      "ArticleID     int64\n",
      "Title        string\n",
      "Abstract     string\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Convert the Title and Abstract to string\n",
    "df_clean['Title'] = df_clean['Title'].astype(\"string\")\n",
    "df_clean['Abstract'] = df_clean['Abstract'].astype(\"string\")\n",
    "\n",
    "# Verify the conversion\n",
    "print('Verifying columns datatypes:')\n",
    "print(df_clean.dtypes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Topic modeling\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the **Natural Language Processing (NLP)** , `topic modeling` is to select which topic a given text or article is about.\n",
    "\n",
    "In other words, extracting descriptive hidden topics from large volumes of text aka _corpus_. This approach can be achieved by searching for a statistical model by means of :\n",
    "\n",
    "#### 3.1. _Dimensionality reduction_ :\n",
    "\n",
    "If we consider text ${T}$ in the language vocabulary ${V}$ , we may use features encoding to map words ${Word_i}$ found in that text into topics ${Topic_i}$ with weights or probability ${Weight_i}$. Mathematically can be reformed as:\n",
    "\n",
    "$${Word_i,T} \\in {V} \\rightarrow {Topic_i,Weight_i} \\in {Topics}$$\n",
    "\n",
    "#### 3.2. _Unsupervised learning_ :\n",
    "\n",
    "As of clustering algorithms where objects are mapped to each cluster centroid by its Euclidean distance, text or words are mapped to topics with probability or weight score.\n",
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"images/topic_modeling.jpeg\" alt=\"Image\"/>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Algorithms for topic modeling\n",
    "\n",
    "---\n",
    "\n",
    "Several algorithms has been proposed over years for topic modeling. The three common algorithms are:\n",
    "\n",
    "- Latent Dirichlet Allocation (LDA)\n",
    "- Non Negative Matrix Factorization (NMF)\n",
    "- Latent Semantic Analysis (LSA)\n",
    "- Parallel Latent Dirichlet Allocation (PLDA)\n",
    "- Pachinko Allocation Model (PAM)\n",
    "\n",
    "Many resources including Kaggle and medium posts suggested using LDA, therefore, we may commence with it.\n",
    "\n",
    "### 5. Latent Dirichlet Allocation (LDA)\n",
    "\n",
    "---\n",
    "\n",
    "Latent Dirichlet Allocation is a statistical and graphical model which are used to obtain relationships between multiple documents in a corpus. It is developed using Variational Exception Maximization (VEM) algorithm for obtaining the maximum likelihood estimate from the whole corpus of text.\n",
    "\n",
    "Mathematically :\n",
    "\n",
    "$${P(topic \\ T | document \\ d)} \\ * {P(word \\ d | topic \\ T)}$$\n",
    "\n",
    "where:\n",
    "\n",
    "${P(topic \\ T | document \\ d)}$ : Proportion of words in document d that are assigned to topic t.\n",
    "\n",
    "${P(word \\ d | topic \\ T)}$ :Proportion of assignments to topic t across all documents from words that come from w\n",
    "\n",
    "Just like the clustering algorithm, it is an iterative process until reaching the same conditional probability of topic assignment with no changes after successive iteration.\n",
    "\n",
    "The expectation would be generating an output topics with mixtures such as in pharmaceuticals :\n",
    "\n",
    "- Topic 1 : 60% Analgesics , 40% Antacids\n",
    "- Topic 2 : 80% Antiarrhythmics, 6% Antacids, 4% Antibacterials\n",
    "\n",
    "Thereafter, each article would be assigned to each topic with a probability :\n",
    "\n",
    "<div align=\"center\">\n",
    "\n",
    "| Article ID |       Topic 1       |       Topic 2       |    Topic n    |\n",
    "| :--------: | :-----------------: | :-----------------: | :-----------: |\n",
    "|  5654562   |         0.6         |         0.3         |      ...      |\n",
    "|  4564562   |        0.95         |        0.01         |      ...      |\n",
    "|     M      | ${P_{max}(t\\|d)_1}$ | ${P_{max}(t\\|d)_2}$ | ${P(t\\|d)_n}$ |\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.1. Abstract cleaning and lemmatization\n",
    "\n",
    "Since the abstract would cover a summary of the article, we need to preprocess it first :\n",
    "\n",
    "- Converting the `Abstract` column into list of lists\n",
    "- Using [regular expressions](https://www.dataquest.io/blog/regex-cheatsheet/) , we may :\n",
    "  - remove `author emails`, `new line` breaks, `single quotes` and `characters less than 3`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Coordination variability ()  commonly analyzed  understand dynamical '\n",
      " 'qualities  human locomotion.  purpose  this study   develop guidelines   '\n",
      " 'number  trials required  inform  calculation   stable mean lower limb  '\n",
      " 'during overground locomotion. Three-dimensional lower limb kinematics were '\n",
      " 'captured   recreational runners performing  trials each  preferred  fixed '\n",
      " 'speed walking  running. Stance phase   calculated   segment  joint couplings '\n",
      " 'using  modified vector coding technique.  number  trials required  achieve   '\n",
      " 'mean within %   strides average  determined  each coupling  individual.  '\n",
      " 'statistical outputs  mode (walking  running)  speed (preferred  fixed) were '\n",
      " 'compared when informed  differing numbers  trials.  minimum   trials were '\n",
      " 'required  stable mean stance phase . With fewer than  trials,   '\n",
      " 'underestimated     oversight  significant differences between mode  speed. '\n",
      " 'Future overground locomotion  research  healthy populations using  vector '\n",
      " 'coding approach should   trials   standard minimum. Researchers should  '\n",
      " 'aware   notable consequences   insufficient number  trials  overall study '\n",
      " 'findings.']\n"
     ]
    }
   ],
   "source": [
    "# Convert to list\n",
    "data = df.Abstract.values.tolist()\n",
    "# Remove Emails\n",
    "data = [re.sub(r'\\S*@\\S*\\s?', '', sent) for sent in data]\n",
    "# Remove new line characters\n",
    "data = [re.sub(r'\\s+', ' ', sent) for sent in data]\n",
    "# Remove distracting single quotes\n",
    "data = [re.sub(r\"\\'\", \"\", sent) for sent in data]\n",
    "# Remove characters less than 3\n",
    "data = [re.sub(r'\\b\\w{1,3}\\b',\"\", sent) for sent in data]\n",
    "\n",
    "# Example of cleaned abstract (still having some punctuation)\n",
    "pprint(data[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step , is to convert the sentences into words using [gensim library](https://radimrehurek.com/gensim/) to be ready for [tokenization](https://www.analyticsvidhya.com/blog/2019/07/how-get-started-nlp-6-unique-ways-perform-tokenization/) by splitting the `sentences` into `words`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['coordination', 'variability', 'commonly', 'analyzed', 'understand', 'dynamical', 'qualities', 'human', 'locomotion', 'purpose', 'this', 'study', 'develop', 'guidelines', 'number', 'trials', 'required', 'inform', 'calculation', 'stable', 'mean', 'lower', 'limb', 'during', 'overground', 'locomotion', 'three', 'dimensional', 'lower', 'limb', 'kinematics', 'were', 'captured', 'recreational', 'runners', 'performing', 'trials', 'each', 'preferred', 'fixed', 'speed', 'walking', 'running', 'stance', 'phase', 'calculated', 'segment', 'joint', 'couplings', 'using', 'modified', 'vector', 'coding', 'technique', 'number', 'trials', 'required', 'achieve', 'mean', 'within', 'strides', 'average', 'determined', 'each', 'coupling', 'individual', 'statistical', 'outputs', 'mode', 'walking', 'running', 'speed', 'preferred', 'fixed', 'were', 'compared', 'when', 'informed', 'differing', 'numbers', 'trials', 'minimum', 'trials', 'were', 'required', 'stable', 'mean', 'stance', 'phase', 'with', 'fewer', 'than', 'trials', 'underestimated', 'oversight', 'significant', 'differences', 'between', 'mode', 'speed', 'future', 'overground', 'locomotion', 'research', 'healthy', 'populations', 'using', 'vector', 'coding', 'approach', 'should', 'trials', 'standard', 'minimum', 'researchers', 'should', 'aware', 'notable', 'consequences', 'insufficient', 'number', 'trials', 'overall', 'study', 'findings']]\n"
     ]
    }
   ],
   "source": [
    "def sent_to_words(sentences):\n",
    "    for sentence in sentences:\n",
    "        # Remove punctuation\n",
    "        yield(gensim.utils.simple_preprocess(str(sentence), deacc=True))\n",
    "data_words = list(sent_to_words(data))\n",
    "\n",
    "# Tokenization of the first abstract in the articles dataset\n",
    "print(data_words[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2. Lemmatization\n",
    "\n",
    "By using [`SpaCy` library](https://spacy.io/), the words are lammetized. In other words, using a dictionary , every verb is converted to its root such as `has` and `had` becomes `have`. We are only interested in nouns and verbs for articles topic modeling  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['analyze understand quality locomotion purpose study develop guideline number trial require inform calculation mean limb locomotion limb kinematic capture runner perform trial prefer fix speed walk run stance phase calculate segment coupling use modify vector code technique number trial require achieve average determine couple output mode walk run speed prefer fix compare inform differ number trial trial require stance phase trial underestimate oversight difference mode speed locomotion research population use vector code approach trial researcher aware consequence number trial study finding', 'scenario knee valgus alteration knee lead risk injury weakness musculature abduction extension hext rotation contribute increase landing task focus question decrease strength associate increase landing task athlete summary finding study include randomize control trial cohort study case control study find decrease strength contribute increase landing task study find extensor contribute control adduction factor mechanism injury study recommend strengthen hext decrease reduce risk injury knee bottom line hext contribute increase college athlete strengthen lead decrease reduce risk injury knee strength recommendation article grade level evidence give grade strength recommendation hext associate increase athlete']\n"
     ]
    }
   ],
   "source": [
    "def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']): #'NOUN', 'ADJ', 'VERB', 'ADV'\n",
    "    \"\"\"Lemmatization of abstract words\n",
    "\n",
    "    Args:\n",
    "        texts (string): list of words\n",
    "        allowed_postags (list, optional): word type. Defaults to ['NOUN', 'ADJ', 'VERB', 'ADV'].\n",
    "\n",
    "    Returns:\n",
    "        lemma of the word\n",
    "    \"\"\"\n",
    "    texts_out = []\n",
    "    for sent in texts:\n",
    "        doc = nlp(\" \".join(sent)) \n",
    "        texts_out.append(\" \".join([token.lemma_ if token.lemma_ not in ['-PRON-'] else '' for token in doc if token.pos_ in allowed_postags]))\n",
    "    return texts_out\n",
    "\n",
    "# Initialize spacy ‘en’ model, keeping only tagger component (for efficiency)\n",
    "nlp = spacy.load(\"en_core_web_sm\", disable=[\"parser\", \"ner\"])\n",
    "# Do lemmatization keeping only Noun, Adj, Verb, Adverb\n",
    "data_lemmatized = lemmatization(data_words, allowed_postags=['NOUN', 'VERB']) #select noun and verb\n",
    "# Example of the words after lemmatization\n",
    "print(data_lemmatized[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.3. Vectorization\n",
    "\n",
    "[Text Vectorization](geeksforgeeks.org/using-countvectorizer-to-extracting-features-from-text/) is the process of counting the frequency of word occurance across the document to perform encoded word matrix. This matrix can be used to compute the conditional probability of the word with respect to a topic or document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x2930 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 47 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(analyzer='word',\n",
    "                             min_df=10,\n",
    "# minimum reqd occurences of a word\n",
    "                             stop_words='english',\n",
    "# remove stop words\n",
    "                             lowercase=True,\n",
    "# convert all words to lowercase\n",
    "                             token_pattern='[a-zA-Z0-9]{3,}',\n",
    "# num chars > 3\n",
    "                             max_features=50000,\n",
    "# max number of uniq words    \n",
    ")\n",
    "# Vectorization\n",
    "data_vectorized = vectorizer.fit_transform(data_lemmatized)\n",
    "\n",
    "data_vectorized[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Hyperparameter Tuning\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By investigating [the documentation](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html) of scikit-learn, there are many parameters to tune. However, the most effective hyperparameters:\n",
    "\n",
    "- `n_componentsint` : Number of topics.\n",
    "- `learning_decay` : It is a parameter that control learning rate in the online learning method. The value should be set between (0.5, 1.0] to guarantee asymptotic convergence.\n",
    "\n",
    "These parameters could be tuned using `Grid Search` which creates a grid of possible parameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 15 candidates, totalling 75 fits\n",
      "[CV] END ................learning_decay=0.5, n_components=10; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.5, n_components=10; total time=   6.3s\n",
      "[CV] END ................learning_decay=0.5, n_components=10; total time=   6.3s\n",
      "[CV] END ................learning_decay=0.5, n_components=10; total time=   6.3s\n",
      "[CV] END ................learning_decay=0.5, n_components=10; total time=   6.2s\n",
      "[CV] END ................learning_decay=0.5, n_components=15; total time=   6.1s\n",
      "[CV] END ................learning_decay=0.5, n_components=15; total time=   6.2s\n",
      "[CV] END ................learning_decay=0.5, n_components=15; total time=   6.2s\n",
      "[CV] END ................learning_decay=0.5, n_components=15; total time=   6.3s\n",
      "[CV] END ................learning_decay=0.5, n_components=15; total time=   6.2s\n",
      "[CV] END ................learning_decay=0.5, n_components=20; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.5, n_components=20; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.5, n_components=20; total time=   6.5s\n",
      "[CV] END ................learning_decay=0.5, n_components=20; total time=   6.5s\n",
      "[CV] END ................learning_decay=0.5, n_components=20; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.5, n_components=25; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.5, n_components=25; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.5, n_components=25; total time=   6.5s\n",
      "[CV] END ................learning_decay=0.5, n_components=25; total time=   6.6s\n",
      "[CV] END ................learning_decay=0.5, n_components=25; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.5, n_components=30; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.5, n_components=30; total time=   6.6s\n",
      "[CV] END ................learning_decay=0.5, n_components=30; total time=   6.6s\n",
      "[CV] END ................learning_decay=0.5, n_components=30; total time=   6.9s\n",
      "[CV] END ................learning_decay=0.5, n_components=30; total time=   6.6s\n",
      "[CV] END ................learning_decay=0.7, n_components=10; total time=   6.6s\n",
      "[CV] END ................learning_decay=0.7, n_components=10; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.7, n_components=10; total time=   6.8s\n",
      "[CV] END ................learning_decay=0.7, n_components=10; total time=   6.9s\n",
      "[CV] END ................learning_decay=0.7, n_components=10; total time=   6.8s\n",
      "[CV] END ................learning_decay=0.7, n_components=15; total time=   6.5s\n",
      "[CV] END ................learning_decay=0.7, n_components=15; total time=   6.8s\n",
      "[CV] END ................learning_decay=0.7, n_components=15; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.7, n_components=15; total time=   6.9s\n",
      "[CV] END ................learning_decay=0.7, n_components=15; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.7, n_components=20; total time=   6.3s\n",
      "[CV] END ................learning_decay=0.7, n_components=20; total time=   7.1s\n",
      "[CV] END ................learning_decay=0.7, n_components=20; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.7, n_components=20; total time=   7.1s\n",
      "[CV] END ................learning_decay=0.7, n_components=20; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.7, n_components=25; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.7, n_components=25; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.7, n_components=25; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.7, n_components=25; total time=   6.9s\n",
      "[CV] END ................learning_decay=0.7, n_components=25; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.7, n_components=30; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.7, n_components=30; total time=   7.1s\n",
      "[CV] END ................learning_decay=0.7, n_components=30; total time=   7.2s\n",
      "[CV] END ................learning_decay=0.7, n_components=30; total time=   7.3s\n",
      "[CV] END ................learning_decay=0.7, n_components=30; total time=   7.1s\n",
      "[CV] END ................learning_decay=0.9, n_components=10; total time=   6.2s\n",
      "[CV] END ................learning_decay=0.9, n_components=10; total time=   6.3s\n",
      "[CV] END ................learning_decay=0.9, n_components=10; total time=   6.6s\n",
      "[CV] END ................learning_decay=0.9, n_components=10; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.9, n_components=10; total time=   6.8s\n",
      "[CV] END ................learning_decay=0.9, n_components=15; total time=   5.7s\n",
      "[CV] END ................learning_decay=0.9, n_components=15; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.9, n_components=15; total time=   6.8s\n",
      "[CV] END ................learning_decay=0.9, n_components=15; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.9, n_components=15; total time=   6.7s\n",
      "[CV] END ................learning_decay=0.9, n_components=20; total time=   5.5s\n",
      "[CV] END ................learning_decay=0.9, n_components=20; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.9, n_components=20; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.9, n_components=20; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.9, n_components=20; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.9, n_components=25; total time=   7.1s\n",
      "[CV] END ................learning_decay=0.9, n_components=25; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.9, n_components=25; total time=   6.4s\n",
      "[CV] END ................learning_decay=0.9, n_components=25; total time=   6.5s\n",
      "[CV] END ................learning_decay=0.9, n_components=25; total time=   6.3s\n",
      "[CV] END ................learning_decay=0.9, n_components=30; total time=   6.5s\n",
      "[CV] END ................learning_decay=0.9, n_components=30; total time=   6.9s\n",
      "[CV] END ................learning_decay=0.9, n_components=30; total time=   7.0s\n",
      "[CV] END ................learning_decay=0.9, n_components=30; total time=   7.1s\n",
      "[CV] END ................learning_decay=0.9, n_components=30; total time=   7.1s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(estimator=LatentDirichletAllocation(learning_method='online',\n",
       "                                                 learning_offset=50.0,\n",
       "                                                 max_iter=5, random_state=0),\n",
       "             param_grid={'learning_decay': [0.5, 0.7, 0.9],\n",
       "                         'n_components': [10, 15, 20, 25, 30]},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Define Search Param\n",
    "search_params = {'n_components': [10, 15, 20, 25, 30], 'learning_decay': [.5, .7, .9]}\n",
    "# Init the Model\n",
    "lda = LatentDirichletAllocation(max_iter=5, learning_method='online', learning_offset=50.,random_state=0)\n",
    "# Init Grid Search Class\n",
    "model = GridSearchCV(lda, param_grid=search_params, verbose=2)\n",
    "# Do the Grid Search\n",
    "model.fit(data_vectorized)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.1. Model selection\n",
    "\n",
    "A model with `higher log-likelihood` and `lower perplexity` (exp(-1. * log-likelihood per word)) is considered to be `good`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Model's Params:  {'learning_decay': 0.5, 'n_components': 10}\n",
      "Best Log Likelihood Score:  -539653.5520135149\n",
      "Model Perplexity:  907.480004528889\n"
     ]
    }
   ],
   "source": [
    "# Best Model\n",
    "best_lda_model = model.best_estimator_\n",
    "# Model Parameters\n",
    "print(\"Best Model's Params: \", model.best_params_)\n",
    "# Log Likelihood Score\n",
    "print(\"Best Log Likelihood Score: \", model.best_score_)\n",
    "# Perplexity\n",
    "print(\"Model Perplexity: \", best_lda_model.perplexity(data_vectorized))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the `Grid search` , we may find:\n",
    "- The best `learning rate` is `0.5`\n",
    "- The articles dataset is best to be described by `10 topics`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Evaluation of the topic model\n",
    "\n",
    "--------------------------------------------\n",
    "\n",
    "As `requested` , we may then select the `highest 3 topics` in terms of probabilities and display the probability of the top `10 topics` as it is the best model based on LDA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_26886/3409709740.py:12: FutureWarning: Support for multi-dimensional indexing (e.g. `obj[:, None]`) is deprecated and will be removed in a future version.  Convert to a numpy array before indexing instead.\n",
      "  dominant_topic = pd.DataFrame(df_document_topic.columns[np.argsort(-np.array(df_document_topic.values) ,axis=1)[:, :3]], index =docnames,columns = ['1st Max','2nd Max','3rd Max'])\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ArticleID</th>\n",
       "      <th>Topic0</th>\n",
       "      <th>Topic1</th>\n",
       "      <th>Topic2</th>\n",
       "      <th>Topic3</th>\n",
       "      <th>Topic4</th>\n",
       "      <th>Topic5</th>\n",
       "      <th>Topic6</th>\n",
       "      <th>Topic7</th>\n",
       "      <th>Topic8</th>\n",
       "      <th>Topic9</th>\n",
       "      <th>1st Max</th>\n",
       "      <th>2nd Max</th>\n",
       "      <th>3rd Max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>34153941</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.46</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.24</td>\n",
       "      <td>Topic5</td>\n",
       "      <td>Topic9</td>\n",
       "      <td>Topic4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>34153942</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.29</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.34</td>\n",
       "      <td>Topic9</td>\n",
       "      <td>Topic2</td>\n",
       "      <td>Topic4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>34153964</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.38</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.27</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.08</td>\n",
       "      <td>Topic2</td>\n",
       "      <td>Topic5</td>\n",
       "      <td>Topic3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34153968</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.21</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Topic2</td>\n",
       "      <td>Topic3</td>\n",
       "      <td>Topic8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>34153978</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.27</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Topic2</td>\n",
       "      <td>Topic8</td>\n",
       "      <td>Topic0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>34153979</td>\n",
       "      <td>0.38</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.29</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.32</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Topic0</td>\n",
       "      <td>Topic8</td>\n",
       "      <td>Topic2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>34153980</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.38</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.13</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Topic1</td>\n",
       "      <td>Topic3</td>\n",
       "      <td>Topic5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>34153982</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.36</td>\n",
       "      <td>0.49</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.09</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Topic3</td>\n",
       "      <td>Topic2</td>\n",
       "      <td>Topic8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>34153983</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.62</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.09</td>\n",
       "      <td>Topic3</td>\n",
       "      <td>Topic5</td>\n",
       "      <td>Topic9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>34153984</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.24</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.41</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.06</td>\n",
       "      <td>Topic3</td>\n",
       "      <td>Topic1</td>\n",
       "      <td>Topic5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>34153985</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.32</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.08</td>\n",
       "      <td>Topic3</td>\n",
       "      <td>Topic5</td>\n",
       "      <td>Topic1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>34153986</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.41</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.46</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Topic3</td>\n",
       "      <td>Topic1</td>\n",
       "      <td>Topic4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>34153989</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.08</td>\n",
       "      <td>0.06</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0.35</td>\n",
       "      <td>Topic9</td>\n",
       "      <td>Topic8</td>\n",
       "      <td>Topic2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>34154038</td>\n",
       "      <td>0.29</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.22</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Topic0</td>\n",
       "      <td>Topic4</td>\n",
       "      <td>Topic8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>34159821</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.98</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Topic3</td>\n",
       "      <td>Topic0</td>\n",
       "      <td>Topic1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   ArticleID  Topic0  Topic1  Topic2  Topic3  Topic4  Topic5  Topic6  Topic7  \\\n",
       "0   34153941    0.00    0.00    0.00    0.00    0.18    0.46    0.03    0.03   \n",
       "1   34153942    0.09    0.00    0.29    0.00    0.14    0.14    0.00    0.00   \n",
       "2   34153964    0.00    0.00    0.38    0.13    0.00    0.27    0.00    0.00   \n",
       "3   34153968    0.00    0.00    0.70    0.21    0.00    0.00    0.00    0.00   \n",
       "4   34153978    0.15    0.00    0.43    0.00    0.00    0.00    0.15    0.00   \n",
       "5   34153979    0.38    0.00    0.29    0.00    0.00    0.00    0.00    0.00   \n",
       "6   34153980    0.00    0.38    0.00    0.30    0.00    0.19    0.00    0.00   \n",
       "7   34153982    0.00    0.00    0.36    0.49    0.05    0.00    0.00    0.00   \n",
       "8   34153983    0.03    0.00    0.00    0.62    0.00    0.24    0.00    0.00   \n",
       "9   34153984    0.00    0.24    0.00    0.41    0.00    0.18    0.11    0.00   \n",
       "10  34153985    0.00    0.18    0.15    0.32    0.00    0.26    0.00    0.00   \n",
       "11  34153986    0.00    0.41    0.04    0.46    0.08    0.00    0.00    0.00   \n",
       "12  34153989    0.00    0.00    0.12    0.10    0.00    0.08    0.06    0.00   \n",
       "13  34154038    0.29    0.00    0.15    0.04    0.28    0.00    0.00    0.00   \n",
       "14  34159821    0.00    0.00    0.00    0.98    0.00    0.00    0.00    0.00   \n",
       "\n",
       "    Topic8  Topic9 1st Max 2nd Max 3rd Max  \n",
       "0     0.04    0.24  Topic5  Topic9  Topic4  \n",
       "1     0.00    0.34  Topic9  Topic2  Topic4  \n",
       "2     0.13    0.08  Topic2  Topic5  Topic3  \n",
       "3     0.08    0.00  Topic2  Topic3  Topic8  \n",
       "4     0.27    0.00  Topic2  Topic8  Topic0  \n",
       "5     0.32    0.00  Topic0  Topic8  Topic2  \n",
       "6     0.13    0.00  Topic1  Topic3  Topic5  \n",
       "7     0.09    0.00  Topic3  Topic2  Topic8  \n",
       "8     0.00    0.09  Topic3  Topic5  Topic9  \n",
       "9     0.00    0.06  Topic3  Topic1  Topic5  \n",
       "10    0.00    0.08  Topic3  Topic5  Topic1  \n",
       "11    0.00    0.00  Topic3  Topic1  Topic4  \n",
       "12    0.28    0.35  Topic9  Topic8  Topic2  \n",
       "13    0.22    0.00  Topic0  Topic4  Topic8  \n",
       "14    0.00    0.00  Topic3  Topic0  Topic1  "
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create Document — Topic Matrix\n",
    "lda_output = best_lda_model.transform(data_vectorized)\n",
    "# column names\n",
    "topicnames = ['Topic' + str(i) for i in range(best_lda_model.n_components)]\n",
    "# index names\n",
    "# docnames = [“Doc” + str(i) for i in range(len(data))]\n",
    "docnames = [str(ID) for ID in df.ArticleID]\n",
    "# Make the pandas dataframe\n",
    "df_document_topic = pd.DataFrame(np.round(lda_output, 2), columns=topicnames, index=docnames)\n",
    "# Get dominant topic for each document\n",
    "# dominant_topic = np.argmax(df_document_topic.values, axis=1)\n",
    "dominant_topic = pd.DataFrame(df_document_topic.columns[np.argsort(-np.array(df_document_topic.values) ,axis=1)[:, :3]], index =docnames,columns = ['1st Max','2nd Max','3rd Max'])\n",
    "df_document_topic = pd.concat([df_document_topic, dominant_topic], axis=1)\n",
    "#df_document_topic['dominant_topic'] =  dominant_topic\n",
    "# change article ID to be column\n",
    "df_document_topic.reset_index(inplace=True)\n",
    "df_document_topic = df_document_topic.rename(columns = {'index':'ArticleID'})\n",
    "df_document_topic_15 = df_document_topic.head(15)\n",
    "df_document_topic_15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the result to article_topic.csv\n",
    "df_document_topic.to_excel('data/article_topic.xlsx',sheet_name='topics for articles')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Topics insights\n",
    "\n",
    "----------------------------------------------------\n",
    "\n",
    "We may check how many words were representing each topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Each topic has:'\n",
      "'  2930 words'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/wbadry/.local/lib/python3.8/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aberration</th>\n",
       "      <th>ability</th>\n",
       "      <th>ablation</th>\n",
       "      <th>abnormality</th>\n",
       "      <th>absence</th>\n",
       "      <th>absorb</th>\n",
       "      <th>absorption</th>\n",
       "      <th>abstract</th>\n",
       "      <th>abundance</th>\n",
       "      <th>abuse</th>\n",
       "      <th>accelerate</th>\n",
       "      <th>acceleration</th>\n",
       "      <th>accept</th>\n",
       "      <th>acceptability</th>\n",
       "      <th>acceptance</th>\n",
       "      <th>access</th>\n",
       "      <th>accessibility</th>\n",
       "      <th>accessory</th>\n",
       "      <th>accident</th>\n",
       "      <th>accompany</th>\n",
       "      <th>accord</th>\n",
       "      <th>accordance</th>\n",
       "      <th>account</th>\n",
       "      <th>accounting</th>\n",
       "      <th>accumulate</th>\n",
       "      <th>accumulation</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>ace</th>\n",
       "      <th>achieve</th>\n",
       "      <th>achievement</th>\n",
       "      <th>acid</th>\n",
       "      <th>acknowledge</th>\n",
       "      <th>acquire</th>\n",
       "      <th>acquisition</th>\n",
       "      <th>act</th>\n",
       "      <th>action</th>\n",
       "      <th>activate</th>\n",
       "      <th>activation</th>\n",
       "      <th>activity</th>\n",
       "      <th>actor</th>\n",
       "      <th>acuity</th>\n",
       "      <th>acute</th>\n",
       "      <th>adapt</th>\n",
       "      <th>adaptability</th>\n",
       "      <th>adaptation</th>\n",
       "      <th>add</th>\n",
       "      <th>addiction</th>\n",
       "      <th>addition</th>\n",
       "      <th>additive</th>\n",
       "      <th>address</th>\n",
       "      <th>...</th>\n",
       "      <th>ward</th>\n",
       "      <th>warning</th>\n",
       "      <th>warrant</th>\n",
       "      <th>waste</th>\n",
       "      <th>wastewater</th>\n",
       "      <th>water</th>\n",
       "      <th>wave</th>\n",
       "      <th>wavelength</th>\n",
       "      <th>way</th>\n",
       "      <th>weaken</th>\n",
       "      <th>weakness</th>\n",
       "      <th>wealth</th>\n",
       "      <th>wear</th>\n",
       "      <th>weather</th>\n",
       "      <th>website</th>\n",
       "      <th>week</th>\n",
       "      <th>weight</th>\n",
       "      <th>welfare</th>\n",
       "      <th>wellbee</th>\n",
       "      <th>wheat</th>\n",
       "      <th>width</th>\n",
       "      <th>wildlife</th>\n",
       "      <th>willingness</th>\n",
       "      <th>window</th>\n",
       "      <th>winter</th>\n",
       "      <th>wish</th>\n",
       "      <th>withdraw</th>\n",
       "      <th>withdrawal</th>\n",
       "      <th>woman</th>\n",
       "      <th>women</th>\n",
       "      <th>word</th>\n",
       "      <th>work</th>\n",
       "      <th>worker</th>\n",
       "      <th>workflow</th>\n",
       "      <th>workforce</th>\n",
       "      <th>working</th>\n",
       "      <th>workload</th>\n",
       "      <th>workplace</th>\n",
       "      <th>workshop</th>\n",
       "      <th>world</th>\n",
       "      <th>worsen</th>\n",
       "      <th>wound</th>\n",
       "      <th>write</th>\n",
       "      <th>xenograft</th>\n",
       "      <th>year</th>\n",
       "      <th>yeast</th>\n",
       "      <th>yield</th>\n",
       "      <th>youth</th>\n",
       "      <th>zinc</th>\n",
       "      <th>zone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Topic0</th>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.107395</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100020</td>\n",
       "      <td>0.100042</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>21.068396</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>33.776803</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100101</td>\n",
       "      <td>0.100130</td>\n",
       "      <td>0.102737</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100288</td>\n",
       "      <td>0.100323</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100032</td>\n",
       "      <td>0.100067</td>\n",
       "      <td>21.454473</td>\n",
       "      <td>13.791648</td>\n",
       "      <td>48.927458</td>\n",
       "      <td>11.199270</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100142</td>\n",
       "      <td>0.100352</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100044</td>\n",
       "      <td>12.296457</td>\n",
       "      <td>0.100046</td>\n",
       "      <td>0.310672</td>\n",
       "      <td>0.100052</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>270.689381</td>\n",
       "      <td>2.520677</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100360</td>\n",
       "      <td>19.146317</td>\n",
       "      <td>2.166852</td>\n",
       "      <td>0.100061</td>\n",
       "      <td>11.417815</td>\n",
       "      <td>22.677746</td>\n",
       "      <td>28.007477</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>43.988327</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100057</td>\n",
       "      <td>9.875940</td>\n",
       "      <td>0.100148</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.109528</td>\n",
       "      <td>0.101326</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100038</td>\n",
       "      <td>0.100103</td>\n",
       "      <td>0.100034</td>\n",
       "      <td>31.895891</td>\n",
       "      <td>0.100043</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>29.553382</td>\n",
       "      <td>0.374262</td>\n",
       "      <td>4.901540</td>\n",
       "      <td>0.100143</td>\n",
       "      <td>30.067521</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100022</td>\n",
       "      <td>0.100050</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100154</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>442.695719</td>\n",
       "      <td>39.751305</td>\n",
       "      <td>0.100103</td>\n",
       "      <td>106.965331</td>\n",
       "      <td>18.234956</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>12.869422</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>9.311776</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>28.988408</td>\n",
       "      <td>20.591144</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>0.702614</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>303.274724</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>4.502490</td>\n",
       "      <td>57.075486</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>0.101914</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic1</th>\n",
       "      <td>20.198554</td>\n",
       "      <td>67.496501</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>22.778556</td>\n",
       "      <td>5.426876</td>\n",
       "      <td>0.102874</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>46.015356</td>\n",
       "      <td>0.105511</td>\n",
       "      <td>20.915125</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100051</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100459</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>17.477084</td>\n",
       "      <td>0.101733</td>\n",
       "      <td>0.100053</td>\n",
       "      <td>9.107230</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>34.355251</td>\n",
       "      <td>75.043939</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>24.759219</td>\n",
       "      <td>0.119445</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>208.441880</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>13.511978</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>36.611142</td>\n",
       "      <td>70.095557</td>\n",
       "      <td>117.592907</td>\n",
       "      <td>158.481859</td>\n",
       "      <td>420.188303</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100059</td>\n",
       "      <td>0.100170</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>11.947367</td>\n",
       "      <td>0.100294</td>\n",
       "      <td>122.782462</td>\n",
       "      <td>0.100039</td>\n",
       "      <td>9.904374</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>3.721624</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100056</td>\n",
       "      <td>0.100139</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>3.455308</td>\n",
       "      <td>0.100020</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100025</td>\n",
       "      <td>45.544659</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>35.005460</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>21.888960</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100110</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100043</td>\n",
       "      <td>18.121381</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100051</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100029</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>32.434335</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>16.148006</td>\n",
       "      <td>0.100032</td>\n",
       "      <td>59.090164</td>\n",
       "      <td>10.709311</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>6.282828</td>\n",
       "      <td>0.100016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic2</th>\n",
       "      <td>0.100025</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>9.700153</td>\n",
       "      <td>25.380010</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100857</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>0.100053</td>\n",
       "      <td>0.100065</td>\n",
       "      <td>5.597971</td>\n",
       "      <td>0.100017</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100022</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>25.616259</td>\n",
       "      <td>0.100032</td>\n",
       "      <td>65.523623</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>34.474274</td>\n",
       "      <td>2.987933</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100164</td>\n",
       "      <td>0.100062</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>6.889040</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>162.464531</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>4.743919</td>\n",
       "      <td>12.962936</td>\n",
       "      <td>0.100123</td>\n",
       "      <td>0.100039</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100053</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>19.514503</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100031</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>13.554809</td>\n",
       "      <td>10.536692</td>\n",
       "      <td>0.100397</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100042</td>\n",
       "      <td>55.472075</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100202</td>\n",
       "      <td>2.138227</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>6.759169</td>\n",
       "      <td>17.897085</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>304.692156</td>\n",
       "      <td>175.347712</td>\n",
       "      <td>0.100106</td>\n",
       "      <td>0.100020</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100017</td>\n",
       "      <td>0.100081</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>6.351425</td>\n",
       "      <td>0.100073</td>\n",
       "      <td>114.813399</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>0.100022</td>\n",
       "      <td>28.837509</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>6.204825</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100029</td>\n",
       "      <td>4.749965</td>\n",
       "      <td>13.353788</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>750.244116</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic3</th>\n",
       "      <td>0.100071</td>\n",
       "      <td>0.100064</td>\n",
       "      <td>25.055031</td>\n",
       "      <td>60.393058</td>\n",
       "      <td>19.029974</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100041</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.100050</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>17.174452</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>13.610915</td>\n",
       "      <td>7.585698</td>\n",
       "      <td>15.064317</td>\n",
       "      <td>74.638834</td>\n",
       "      <td>0.100060</td>\n",
       "      <td>16.831334</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100032</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>52.137059</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>28.813705</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100178</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100122</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>28.593208</td>\n",
       "      <td>0.100020</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>19.636451</td>\n",
       "      <td>0.100025</td>\n",
       "      <td>46.966385</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>...</td>\n",
       "      <td>5.170173</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>18.310894</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100057</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100017</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>3.603899</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100020</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>26.204043</td>\n",
       "      <td>9.476335</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>1.281008</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>2.594475</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100046</td>\n",
       "      <td>6.627129</td>\n",
       "      <td>185.589669</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.100022</td>\n",
       "      <td>0.100546</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>6.702462</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>45.780213</td>\n",
       "      <td>27.495483</td>\n",
       "      <td>80.186915</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>1.584407</td>\n",
       "      <td>632.047058</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>7.477801</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>6.641749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic4</th>\n",
       "      <td>0.100086</td>\n",
       "      <td>30.144463</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100025</td>\n",
       "      <td>40.972990</td>\n",
       "      <td>5.490550</td>\n",
       "      <td>0.100082</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>3.546516</td>\n",
       "      <td>12.431171</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100042</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>8.509455</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100123</td>\n",
       "      <td>0.100031</td>\n",
       "      <td>31.797824</td>\n",
       "      <td>0.107859</td>\n",
       "      <td>48.525995</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>9.760327</td>\n",
       "      <td>58.276992</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>40.177603</td>\n",
       "      <td>0.100062</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100056</td>\n",
       "      <td>26.173632</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100017</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>39.282746</td>\n",
       "      <td>0.100038</td>\n",
       "      <td>13.465467</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>15.957531</td>\n",
       "      <td>0.100054</td>\n",
       "      <td>39.612514</td>\n",
       "      <td>2.724872</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>5.568338</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.234742</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100038</td>\n",
       "      <td>0.239765</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.116527</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.100147</td>\n",
       "      <td>0.101624</td>\n",
       "      <td>0.100068</td>\n",
       "      <td>0.100156</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>12.845155</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100031</td>\n",
       "      <td>12.758868</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>42.756608</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100089</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100119</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100043</td>\n",
       "      <td>44.730340</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>8.298018</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.100469</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.100224</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100061</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100044</td>\n",
       "      <td>61.277265</td>\n",
       "      <td>0.100056</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.103039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic5</th>\n",
       "      <td>0.100026</td>\n",
       "      <td>14.931481</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>0.100038</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100315</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>0.255394</td>\n",
       "      <td>17.140379</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100033</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.487866</td>\n",
       "      <td>0.100891</td>\n",
       "      <td>71.778714</td>\n",
       "      <td>14.921450</td>\n",
       "      <td>0.100050</td>\n",
       "      <td>0.100033</td>\n",
       "      <td>0.100032</td>\n",
       "      <td>0.100045</td>\n",
       "      <td>253.068252</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>54.908340</td>\n",
       "      <td>0.100030</td>\n",
       "      <td>3.474495</td>\n",
       "      <td>0.100047</td>\n",
       "      <td>5.457576</td>\n",
       "      <td>0.100032</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>2.929346</td>\n",
       "      <td>0.110073</td>\n",
       "      <td>4.733668</td>\n",
       "      <td>0.100049</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>6.380554</td>\n",
       "      <td>0.100048</td>\n",
       "      <td>0.100294</td>\n",
       "      <td>14.486820</td>\n",
       "      <td>13.976595</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>18.327891</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>1.167011</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100066</td>\n",
       "      <td>0.316329</td>\n",
       "      <td>0.100033</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100038</td>\n",
       "      <td>0.100029</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100175</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100057</td>\n",
       "      <td>0.100045</td>\n",
       "      <td>0.100044</td>\n",
       "      <td>89.607387</td>\n",
       "      <td>39.229147</td>\n",
       "      <td>0.100039</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.460407</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100054</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>4.097326</td>\n",
       "      <td>9.472477</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>0.150763</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>0.100044</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100029</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.177934</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>3.859796</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>15.080189</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic6</th>\n",
       "      <td>0.100066</td>\n",
       "      <td>35.061954</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.100218</td>\n",
       "      <td>57.016749</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>13.632296</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100049</td>\n",
       "      <td>0.100152</td>\n",
       "      <td>0.100030</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100020</td>\n",
       "      <td>0.100081</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100022</td>\n",
       "      <td>0.100073</td>\n",
       "      <td>0.100079</td>\n",
       "      <td>0.100034</td>\n",
       "      <td>0.100022</td>\n",
       "      <td>28.141853</td>\n",
       "      <td>0.100044</td>\n",
       "      <td>5.461470</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>105.026931</td>\n",
       "      <td>3.861633</td>\n",
       "      <td>170.796525</td>\n",
       "      <td>0.100054</td>\n",
       "      <td>0.100025</td>\n",
       "      <td>2.405345</td>\n",
       "      <td>0.100033</td>\n",
       "      <td>0.100030</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100039</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100030</td>\n",
       "      <td>0.100194</td>\n",
       "      <td>0.100201</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>22.401617</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>88.245722</td>\n",
       "      <td>15.118781</td>\n",
       "      <td>13.159733</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>52.058296</td>\n",
       "      <td>60.646644</td>\n",
       "      <td>471.068532</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>16.741859</td>\n",
       "      <td>0.100025</td>\n",
       "      <td>8.583673</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>3.647817</td>\n",
       "      <td>0.100115</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100049</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100053</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100038</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100080</td>\n",
       "      <td>0.100056</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.992914</td>\n",
       "      <td>103.443687</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100072</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100063</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.126923</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100208</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.101127</td>\n",
       "      <td>0.100092</td>\n",
       "      <td>95.144298</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>26.688902</td>\n",
       "      <td>0.100031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic7</th>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>32.093823</td>\n",
       "      <td>0.100052</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>1.654593</td>\n",
       "      <td>41.723982</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>1.349595</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>0.100017</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100030</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100044</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100041</td>\n",
       "      <td>21.521805</td>\n",
       "      <td>0.100052</td>\n",
       "      <td>0.100046</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>0.100034</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100371</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.120210</td>\n",
       "      <td>7.981186</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100025</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100020</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>0.100128</td>\n",
       "      <td>21.444545</td>\n",
       "      <td>0.100051</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>30.466070</td>\n",
       "      <td>0.100073</td>\n",
       "      <td>0.174031</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>0.100068</td>\n",
       "      <td>0.100774</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>93.606937</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>0.100080</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100047</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>27.164715</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>14.521201</td>\n",
       "      <td>37.912137</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100041</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>1.280437</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100141</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100066</td>\n",
       "      <td>35.905690</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>55.524840</td>\n",
       "      <td>0.100023</td>\n",
       "      <td>24.404672</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>93.396939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic8</th>\n",
       "      <td>0.100000</td>\n",
       "      <td>40.850804</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100057</td>\n",
       "      <td>0.100043</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>7.728950</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.753593</td>\n",
       "      <td>16.628437</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100187</td>\n",
       "      <td>36.960055</td>\n",
       "      <td>44.641845</td>\n",
       "      <td>251.573111</td>\n",
       "      <td>14.657784</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100104</td>\n",
       "      <td>8.638034</td>\n",
       "      <td>17.413369</td>\n",
       "      <td>0.100037</td>\n",
       "      <td>0.100053</td>\n",
       "      <td>0.100045</td>\n",
       "      <td>0.100052</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>52.063892</td>\n",
       "      <td>24.867351</td>\n",
       "      <td>0.100002</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>0.100033</td>\n",
       "      <td>1.595047</td>\n",
       "      <td>0.100041</td>\n",
       "      <td>46.520105</td>\n",
       "      <td>0.100029</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>14.774789</td>\n",
       "      <td>9.480564</td>\n",
       "      <td>0.100079</td>\n",
       "      <td>0.100229</td>\n",
       "      <td>33.762113</td>\n",
       "      <td>8.565183</td>\n",
       "      <td>0.100045</td>\n",
       "      <td>8.743048</td>\n",
       "      <td>0.100027</td>\n",
       "      <td>56.269091</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>191.463851</td>\n",
       "      <td>...</td>\n",
       "      <td>15.459584</td>\n",
       "      <td>0.100173</td>\n",
       "      <td>0.192755</td>\n",
       "      <td>0.100094</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100008</td>\n",
       "      <td>0.100251</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>10.330470</td>\n",
       "      <td>0.100143</td>\n",
       "      <td>0.100047</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.114577</td>\n",
       "      <td>0.100038</td>\n",
       "      <td>0.100036</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>34.218304</td>\n",
       "      <td>0.100073</td>\n",
       "      <td>0.100007</td>\n",
       "      <td>0.100012</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>39.301219</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100025</td>\n",
       "      <td>7.517493</td>\n",
       "      <td>0.100200</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>2.935329</td>\n",
       "      <td>8.799969</td>\n",
       "      <td>5.149436</td>\n",
       "      <td>203.587470</td>\n",
       "      <td>231.622139</td>\n",
       "      <td>0.100020</td>\n",
       "      <td>18.835271</td>\n",
       "      <td>0.100153</td>\n",
       "      <td>3.003370</td>\n",
       "      <td>8.852491</td>\n",
       "      <td>21.718439</td>\n",
       "      <td>34.353675</td>\n",
       "      <td>0.145267</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>47.443062</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>124.043528</td>\n",
       "      <td>0.100004</td>\n",
       "      <td>0.100043</td>\n",
       "      <td>25.242914</td>\n",
       "      <td>0.100001</td>\n",
       "      <td>0.100009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Topic9</th>\n",
       "      <td>0.100030</td>\n",
       "      <td>47.141969</td>\n",
       "      <td>0.100005</td>\n",
       "      <td>0.100032</td>\n",
       "      <td>0.100034</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100018</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>2.118168</td>\n",
       "      <td>0.100131</td>\n",
       "      <td>0.100135</td>\n",
       "      <td>0.107568</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>12.932976</td>\n",
       "      <td>112.315175</td>\n",
       "      <td>0.100068</td>\n",
       "      <td>0.100168</td>\n",
       "      <td>10.497145</td>\n",
       "      <td>0.100040</td>\n",
       "      <td>0.100051</td>\n",
       "      <td>0.100064</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100061</td>\n",
       "      <td>0.100016</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100006</td>\n",
       "      <td>38.847641</td>\n",
       "      <td>0.100041</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>6.480777</td>\n",
       "      <td>42.054799</td>\n",
       "      <td>0.100046</td>\n",
       "      <td>21.506813</td>\n",
       "      <td>21.682854</td>\n",
       "      <td>0.100061</td>\n",
       "      <td>22.414747</td>\n",
       "      <td>140.687553</td>\n",
       "      <td>0.100032</td>\n",
       "      <td>0.100013</td>\n",
       "      <td>0.100065</td>\n",
       "      <td>51.895171</td>\n",
       "      <td>0.100145</td>\n",
       "      <td>0.100036</td>\n",
       "      <td>10.222037</td>\n",
       "      <td>0.100011</td>\n",
       "      <td>85.034086</td>\n",
       "      <td>0.100044</td>\n",
       "      <td>78.505097</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100031</td>\n",
       "      <td>0.100330</td>\n",
       "      <td>10.219934</td>\n",
       "      <td>0.100054</td>\n",
       "      <td>0.100009</td>\n",
       "      <td>0.100022</td>\n",
       "      <td>79.306650</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>30.576518</td>\n",
       "      <td>0.102733</td>\n",
       "      <td>9.583115</td>\n",
       "      <td>0.100029</td>\n",
       "      <td>41.857209</td>\n",
       "      <td>0.100139</td>\n",
       "      <td>0.100034</td>\n",
       "      <td>0.101334</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>30.123782</td>\n",
       "      <td>0.100029</td>\n",
       "      <td>0.100028</td>\n",
       "      <td>0.100019</td>\n",
       "      <td>0.100010</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>0.100095</td>\n",
       "      <td>0.100025</td>\n",
       "      <td>0.100285</td>\n",
       "      <td>0.100075</td>\n",
       "      <td>3.293459</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100038</td>\n",
       "      <td>31.549948</td>\n",
       "      <td>195.621075</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>39.538741</td>\n",
       "      <td>0.100015</td>\n",
       "      <td>0.100109</td>\n",
       "      <td>5.811612</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>0.100014</td>\n",
       "      <td>55.419418</td>\n",
       "      <td>0.100024</td>\n",
       "      <td>0.100035</td>\n",
       "      <td>0.100026</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>15.942244</td>\n",
       "      <td>0.100021</td>\n",
       "      <td>5.303020</td>\n",
       "      <td>3.285782</td>\n",
       "      <td>0.100003</td>\n",
       "      <td>6.725997</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 2930 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        aberration    ability   ablation  abnormality    absence    absorb  \\\n",
       "Topic0    0.100000   0.107395   0.100006     0.100020   0.100042  0.100013   \n",
       "Topic1   20.198554  67.496501   0.100010     0.100028  22.778556  5.426876   \n",
       "Topic2    0.100025   0.100023   0.100001     9.700153  25.380010  0.100035   \n",
       "Topic3    0.100071   0.100064  25.055031    60.393058  19.029974  0.100000   \n",
       "Topic4    0.100086  30.144463   0.100010     0.100025  40.972990  5.490550   \n",
       "Topic5    0.100026  14.931481   0.100009     0.100024   0.100038  0.100010   \n",
       "Topic6    0.100066  35.061954   0.100005     0.100018   0.100028  0.100218   \n",
       "Topic7    0.100018   0.100015   0.100000     0.100027  32.093823  0.100052   \n",
       "Topic8    0.100000  40.850804   0.100002     0.100007   0.100057  0.100043   \n",
       "Topic9    0.100030  47.141969   0.100005     0.100032   0.100034  0.100009   \n",
       "\n",
       "        absorption   abstract  abundance      abuse  accelerate  acceleration  \\\n",
       "Topic0    0.100011  21.068396   0.100009  33.776803    0.100019      0.100101   \n",
       "Topic1    0.102874   0.100011  46.015356   0.105511   20.915125      0.100019   \n",
       "Topic2    0.100013   0.100007   0.100857   0.100024    0.100053      0.100065   \n",
       "Topic3    0.100006   0.100041   0.100003   0.100002    0.100050      0.100002   \n",
       "Topic4    0.100082   0.100007   0.100008   0.100015    3.546516     12.431171   \n",
       "Topic5    0.100315   0.100023   0.100012   0.100010    0.100040      0.255394   \n",
       "Topic6   57.016749   0.100011   0.100028   0.100003   13.632296      0.100027   \n",
       "Topic7    0.100004   1.654593  41.723982   0.100001    1.349595      0.100003   \n",
       "Topic8    0.100011   7.728950   0.100001   0.753593   16.628437      0.100005   \n",
       "Topic9    0.100019   0.100018   0.100015   0.100011    2.118168      0.100131   \n",
       "\n",
       "           accept  acceptability  acceptance      access  accessibility  \\\n",
       "Topic0   0.100130       0.102737    0.100018    0.100288       0.100323   \n",
       "Topic1   0.100051       0.100027    0.100010    0.100021       0.100007   \n",
       "Topic2   5.597971       0.100017    0.100010    0.100022       0.100021   \n",
       "Topic3  17.174452       0.100009    0.100016    0.100027       0.100002   \n",
       "Topic4   0.100016       0.100042    0.100018    8.509455       0.100024   \n",
       "Topic5  17.140379       0.100028    0.100012    0.100033       0.100018   \n",
       "Topic6   0.100049       0.100152    0.100030    0.100018       0.100020   \n",
       "Topic7   0.100040       0.100017    0.100004    0.100030       0.100004   \n",
       "Topic8   0.100187      36.960055   44.641845  251.573111      14.657784   \n",
       "Topic9   0.100135       0.107568    0.100009   12.932976     112.315175   \n",
       "\n",
       "        accessory   accident  accompany     accord  accordance    account  \\\n",
       "Topic0   0.100004   0.100032   0.100067  21.454473   13.791648  48.927458   \n",
       "Topic1   0.100459   0.100014  17.477084   0.101733    0.100053   9.107230   \n",
       "Topic2   0.100000  25.616259   0.100032  65.523623    0.100012  34.474274   \n",
       "Topic3  13.610915   7.585698  15.064317  74.638834    0.100060  16.831334   \n",
       "Topic4   0.100018   0.100123   0.100031  31.797824    0.107859  48.525995   \n",
       "Topic5   0.100001   0.487866   0.100891  71.778714   14.921450   0.100050   \n",
       "Topic6   0.100081   0.100007   0.100022   0.100073    0.100079   0.100034   \n",
       "Topic7   0.100044   0.100026   0.100041  21.521805    0.100052   0.100046   \n",
       "Topic8   0.100004   0.100104   8.638034  17.413369    0.100037   0.100053   \n",
       "Topic9   0.100068   0.100168  10.497145   0.100040    0.100051   0.100064   \n",
       "\n",
       "        accounting  accumulate  accumulation    accuracy        ace  \\\n",
       "Topic0   11.199270    0.100026      0.100004    0.100008   0.100018   \n",
       "Topic1    0.100040   34.355251     75.043939    0.100004  24.759219   \n",
       "Topic2    2.987933    0.100040      0.100008    0.100012   0.100014   \n",
       "Topic3    0.100027    0.100009      0.100032    0.100014   0.100006   \n",
       "Topic4    0.100005    0.100028      9.760327   58.276992   0.100002   \n",
       "Topic5    0.100033    0.100032      0.100045  253.068252   0.100003   \n",
       "Topic6    0.100022   28.141853      0.100044    5.461470   0.100005   \n",
       "Topic7    0.100023    0.100034      0.100040    0.100008   0.100009   \n",
       "Topic8    0.100045    0.100052      0.100002    0.100011   0.100007   \n",
       "Topic9    0.100026    0.100061      0.100016    0.100015   0.100006   \n",
       "\n",
       "           achieve  achievement        acid  acknowledge    acquire  \\\n",
       "Topic0    0.100027     0.100142    0.100352     0.100027   0.100044   \n",
       "Topic1    0.119445     0.100012  208.441880     0.100014  13.511978   \n",
       "Topic2    0.100164     0.100062    0.100023     6.889040   0.100028   \n",
       "Topic3   52.137059     0.100021    0.100026     0.100016  28.813705   \n",
       "Topic4   40.177603     0.100062    0.100012     0.100015   0.100056   \n",
       "Topic5   54.908340     0.100030    3.474495     0.100047   5.457576   \n",
       "Topic6  105.026931     3.861633  170.796525     0.100054   0.100025   \n",
       "Topic7    0.100019     0.100018    0.100371     0.100006   0.120210   \n",
       "Topic8   52.063892    24.867351    0.100002     0.100035   0.100033   \n",
       "Topic9   38.847641     0.100041    0.100011     6.480777  42.054799   \n",
       "\n",
       "        acquisition        act     action    activate  activation    activity  \\\n",
       "Topic0    12.296457   0.100046   0.310672    0.100052    0.100028  270.689381   \n",
       "Topic1     0.100014  36.611142  70.095557  117.592907  158.481859  420.188303   \n",
       "Topic2     0.100021   0.100026   0.100027    0.100006    0.100021  162.464531   \n",
       "Topic3     0.100007   0.100178   0.100011    0.100002    0.100014    0.100122   \n",
       "Topic4    26.173632   0.100012   0.100017    0.100010    0.100040   39.282746   \n",
       "Topic5     0.100032   0.100035   0.100021    2.929346    0.110073    4.733668   \n",
       "Topic6     2.405345   0.100033   0.100030    0.100021    0.100016    0.100039   \n",
       "Topic7     7.981186   0.100024   0.100019    0.100005    0.100006    0.100025   \n",
       "Topic8     1.595047   0.100041  46.520105    0.100029    0.100013   14.774789   \n",
       "Topic9     0.100046  21.506813  21.682854    0.100061   22.414747  140.687553   \n",
       "\n",
       "           actor     acuity      acute      adapt  adaptability  adaptation  \\\n",
       "Topic0  2.520677   0.100013   0.100360  19.146317      2.166852    0.100061   \n",
       "Topic1  0.100006   0.100008   0.100015   0.100059      0.100170    0.100024   \n",
       "Topic2  0.100003   4.743919  12.962936   0.100123      0.100039    0.100015   \n",
       "Topic3  0.100012   0.100023  28.593208   0.100020      0.100005    0.100011   \n",
       "Topic4  0.100038  13.465467   0.100011  15.957531      0.100054   39.612514   \n",
       "Topic5  0.100049   0.100006   6.380554   0.100048      0.100294   14.486820   \n",
       "Topic6  0.100000   0.100007   0.100030   0.100194      0.100201    0.100011   \n",
       "Topic7  0.100001   0.100001   0.100020   0.100027      0.100128   21.444545   \n",
       "Topic8  9.480564   0.100079   0.100229  33.762113      8.565183    0.100045   \n",
       "Topic9  0.100032   0.100013   0.100065  51.895171      0.100145    0.100036   \n",
       "\n",
       "              add  addiction    addition   additive     address  ...  \\\n",
       "Topic0  11.417815  22.677746   28.007477   0.100018   43.988327  ...   \n",
       "Topic1  11.947367   0.100294  122.782462   0.100039    9.904374  ...   \n",
       "Topic2   0.100053   0.100024   19.514503   0.100012    0.100031  ...   \n",
       "Topic3  19.636451   0.100025   46.966385   0.100007    0.100023  ...   \n",
       "Topic4   2.724872   0.100035    5.568338   0.100008    0.234742  ...   \n",
       "Topic5  13.976595   0.100005   18.327891   0.100015    1.167011  ...   \n",
       "Topic6  22.401617   0.100001   88.245722  15.118781   13.159733  ...   \n",
       "Topic7   0.100051   0.100000   30.466070   0.100073    0.174031  ...   \n",
       "Topic8   8.743048   0.100027   56.269091   0.100004  191.463851  ...   \n",
       "Topic9  10.222037   0.100011   85.034086   0.100044   78.505097  ...   \n",
       "\n",
       "             ward    warning    warrant      waste  wastewater       water  \\\n",
       "Topic0   0.100018   0.100057   9.875940   0.100148    0.100002    0.109528   \n",
       "Topic1   0.100023   0.100001   3.721624   0.100016    0.100056    0.100139   \n",
       "Topic2   0.100027  13.554809  10.536692   0.100397    0.100009    0.100042   \n",
       "Topic3   5.170173   0.100021  18.310894   0.100015    0.100004    0.100004   \n",
       "Topic4   0.100014   0.100008   0.100038   0.239765    0.100005    0.116527   \n",
       "Topic5   0.100012   0.100066   0.316329   0.100033    0.100013    0.100013   \n",
       "Topic6   0.100015   0.100006   0.100013  52.058296   60.646644  471.068532   \n",
       "Topic7   0.100006   0.100023   0.100068   0.100774    0.100019   93.606937   \n",
       "Topic8  15.459584   0.100173   0.192755   0.100094    0.100005    0.100008   \n",
       "Topic9   0.100031   0.100330  10.219934   0.100054    0.100009    0.100022   \n",
       "\n",
       "             wave  wavelength        way    weaken  weakness     wealth  \\\n",
       "Topic0   0.101326    0.100011   0.100038  0.100103  0.100034  31.895891   \n",
       "Topic1   0.100011    0.100005   0.100018  3.455308  0.100020   0.100000   \n",
       "Topic2  55.472075    0.100003   0.100008  0.100202  2.138227   0.100026   \n",
       "Topic3   0.100057    0.100010   0.100017  0.100014  3.603899   0.100014   \n",
       "Topic4   0.100028    0.100147   0.101624  0.100068  0.100156   0.100007   \n",
       "Topic5   0.100038    0.100029   0.100019  0.100175  0.100012   0.100015   \n",
       "Topic6   0.100027   16.741859   0.100025  8.583673  0.100008   0.100002   \n",
       "Topic7   0.100015    0.100002   0.100012  0.100040  0.100024   0.100006   \n",
       "Topic8   0.100251    0.100001  10.330470  0.100143  0.100047   0.100012   \n",
       "Topic9  79.306650    0.100019  30.576518  0.102733  9.583115   0.100029   \n",
       "\n",
       "             wear    weather    website        week      weight    welfare  \\\n",
       "Topic0   0.100043   0.100008  29.553382    0.374262    4.901540   0.100143   \n",
       "Topic1   0.100001   0.100001   0.100000    0.100025   45.544659   0.100005   \n",
       "Topic2   6.759169  17.897085   0.100005  304.692156  175.347712   0.100106   \n",
       "Topic3   0.100020   0.100008   0.100005   26.204043    9.476335   0.100006   \n",
       "Topic4  12.845155   0.100014   0.100008    0.100031   12.758868   0.100013   \n",
       "Topic5   0.100057   0.100045   0.100044   89.607387   39.229147   0.100039   \n",
       "Topic6   3.647817   0.100115   0.100006    0.100013    0.100049   0.100015   \n",
       "Topic7   0.100006   0.100080   0.100016    0.100013    0.100028   0.100026   \n",
       "Topic8   0.114577   0.100038   0.100036    0.100028    0.100014  34.218304   \n",
       "Topic9  41.857209   0.100139   0.100034    0.101334    0.100021  30.123782   \n",
       "\n",
       "          wellbee      wheat      width   wildlife  willingness     window  \\\n",
       "Topic0  30.067521   0.100006   0.100008   0.100001     0.100011   0.100022   \n",
       "Topic1   0.100000  35.005460   0.100009   0.100007     0.100002  21.888960   \n",
       "Topic2   0.100020   0.100019   0.100026   0.100005     0.100017   0.100081   \n",
       "Topic3   0.100001   0.100006   1.281008   0.100003     0.100015   2.594475   \n",
       "Topic4   0.100016   0.100012  42.756608   0.100004     0.100007   0.100089   \n",
       "Topic5   0.100008   0.100011   0.460407   0.100008     0.100005   0.100054   \n",
       "Topic6   0.100001   0.100053   0.100018   0.100008     0.100001   0.100038   \n",
       "Topic7   0.100013   0.100047   0.100010  27.164715     0.100014  14.521201   \n",
       "Topic8   0.100073   0.100007   0.100012   0.100006    39.301219   0.100016   \n",
       "Topic9   0.100029   0.100028   0.100019   0.100010     0.100024   0.100095   \n",
       "\n",
       "           winter      wish  withdraw  withdrawal       woman      women  \\\n",
       "Topic0   0.100050  0.100013  0.100154    0.100019  442.695719  39.751305   \n",
       "Topic1   0.100009  0.100001  0.100006    0.100110    0.100014   0.100001   \n",
       "Topic2   0.100035  0.100005  6.351425    0.100073  114.813399   0.100021   \n",
       "Topic3   0.100014  0.100007  0.100046    6.627129  185.589669   0.100003   \n",
       "Topic4   0.100005  0.100001  0.100119    0.100035    0.100010   0.100011   \n",
       "Topic5   0.100003  0.100008  4.097326    9.472477    0.100018   0.100027   \n",
       "Topic6   0.100004  0.100004  0.100080    0.100056    0.100007   0.100011   \n",
       "Topic7  37.912137  0.100000  0.100041    0.100016    0.100005   0.100015   \n",
       "Topic8   0.100025  7.517493  0.100200    0.100004    2.935329   8.799969   \n",
       "Topic9   0.100025  0.100285  0.100075    3.293459    0.100015   0.100038   \n",
       "\n",
       "             word        work      worker   workflow  workforce    working  \\\n",
       "Topic0   0.100103  106.965331   18.234956   0.100006   0.100015  12.869422   \n",
       "Topic1   0.100043   18.121381    0.100006   0.100051   0.100000   0.100013   \n",
       "Topic2   0.100022   28.837509    0.100018   0.100027   0.100007   6.204825   \n",
       "Topic3   0.100022    0.100546    0.100006   0.100006   0.100002   6.702462   \n",
       "Topic4   0.100043   44.730340    0.100004   8.298018   0.100002   0.100469   \n",
       "Topic5   0.100040    0.150763    0.100023   0.100044   0.100001   0.100040   \n",
       "Topic6   0.992914  103.443687    0.100005   0.100072   0.100001   0.100063   \n",
       "Topic7   0.100028    1.280437    0.100008   0.100141   0.100002   0.100008   \n",
       "Topic8   5.149436  203.587470  231.622139   0.100020  18.835271   0.100153   \n",
       "Topic9  31.549948  195.621075    0.100015  39.538741   0.100015   0.100109   \n",
       "\n",
       "        workload  workplace   workshop      world     worsen      wound  \\\n",
       "Topic0  0.100012   9.311776   0.100010  28.988408  20.591144   0.100023   \n",
       "Topic1  0.100000   0.100001   0.100000   0.100029   0.100012  32.434335   \n",
       "Topic2  0.100010   0.100007   0.100029   4.749965  13.353788   0.100008   \n",
       "Topic3  0.100008   0.100001   0.100011  45.780213  27.495483  80.186915   \n",
       "Topic4  0.100019   0.100007   0.100003   0.100224   0.100011   0.100061   \n",
       "Topic5  0.100015   0.100018   0.100001   0.100029   0.100007   0.177934   \n",
       "Topic6  0.100026   0.100005   0.100000   0.126923   0.100004   0.100208   \n",
       "Topic7  0.100001   0.100001   0.100066  35.905690   0.100021   0.100004   \n",
       "Topic8  3.003370   8.852491  21.718439  34.353675   0.145267   0.100021   \n",
       "Topic9  5.811612   0.100014   0.100014  55.419418   0.100024   0.100035   \n",
       "\n",
       "            write  xenograft        year      yeast      yield      youth  \\\n",
       "Topic0   0.702614   0.100002  303.274724   0.100035   4.502490  57.075486   \n",
       "Topic1   0.100001  16.148006    0.100032  59.090164  10.709311   0.100004   \n",
       "Topic2   0.100013   0.100026  750.244116   0.100007   0.100024   0.100012   \n",
       "Topic3   0.100010   1.584407  632.047058   0.100001   7.477801   0.100006   \n",
       "Topic4   0.100010   0.100044   61.277265   0.100056   0.100035   0.100015   \n",
       "Topic5   0.100015   0.100003    3.859796   0.100009  15.080189   0.100011   \n",
       "Topic6   0.100003   0.100028    0.101127   0.100092  95.144298   0.100006   \n",
       "Topic7   0.100010   0.100003   55.524840   0.100023  24.404672   0.100001   \n",
       "Topic8  47.443062   0.100000  124.043528   0.100004   0.100043  25.242914   \n",
       "Topic9   0.100026   0.100003   15.942244   0.100021   5.303020   3.285782   \n",
       "\n",
       "             zinc       zone  \n",
       "Topic0   0.100021   0.101914  \n",
       "Topic1   6.282828   0.100016  \n",
       "Topic2   0.100004   0.100019  \n",
       "Topic3   0.100003   6.641749  \n",
       "Topic4   0.100003   0.103039  \n",
       "Topic5   0.100018   0.100039  \n",
       "Topic6  26.688902   0.100031  \n",
       "Topic7   0.100027  93.396939  \n",
       "Topic8   0.100001   0.100009  \n",
       "Topic9   0.100003   6.725997  \n",
       "\n",
       "[10 rows x 2930 columns]"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Topic-Keyword Matrix\n",
    "df_topic_keywords = pd.DataFrame(best_lda_model.components_)\n",
    "# Assign Column and Index\n",
    "df_topic_keywords.columns = vectorizer.get_feature_names()\n",
    "df_topic_keywords.index = topicnames\n",
    "\n",
    "pprint(\"Each topic has:\")\n",
    "pprint( \"  {:} words\".format(df_topic_keywords.shape[1]))\n",
    "# View\n",
    "df_topic_keywords.head(model.best_params_['n_components'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that each `topic` is composed of `2930` words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the result to topic_words.xlsx\n",
    "df_topic_keywords.to_excel('data/topic_words.xlsx',sheet_name='topics for articles')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Communication and findings\n",
    "\n",
    "---------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- After `wrangling the data`, the dataset is composed of nearly 5000 articles composed of:\n",
    "  - ArticleID : the article ID.\n",
    "  - Title : The article title.\n",
    "  - Abstract : The summary of the article.\n",
    "- `0.2 % of the articles` had `no summary`, that's why it was removed from the dataset.\n",
    "- `The abstract` was the selected field for `topic modeling` since it would contain rich set of words summarizing the article.\n",
    "- The procedure of topic modeling is commenced by `slicing` the `sentences` into `words`. Then after tokenization, the words are `lammetized` to its roots and cleaned from punctuations and common 3 letters words.\n",
    "- The `Latent Dirichlet Allocation (LDA) algorithm` is forming `topic modeling` based on the `likelihood` of words `frequency`.\n",
    "- There are many parameters affecting the performance of topic modeling, two major `hyperparameters` are the `learning rate` and the `number of topics`.\n",
    "- `Grid search` is one of the methods that can be used for selection of best parameters. The pitfall is its intensive computation time.\n",
    "- For the `current dataset`, a total of `10 topics` was found to be enough for modeling the whole articles with around `2900 words per topic`.\n",
    "- The number of words can be `drastically reduced` if further optimization is carried out to find the words with higher frequency and omit the words that is unlikely to be repeated. \n",
    "- `Each article` in the dataset was `mapped` to all of the `10 topics` with `probability` of each topic and `the highest 3 topics` and saved to excel file.\n",
    "- The `bag of words` for each topic was `listed` and also saved in excel file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10 References\n",
    "\n",
    "--------------------------------------------------------------------\n",
    "1. [How to generate an LDA Topic Model for Text Analysis](https://yanlinc.medium.com/how-to-build-a-lda-topic-model-using-from-text-601cdcbfd3a6)\n",
    "2. [Topic Modeling and Latent Dirichlet Allocation (LDA) in Python](https://towardsdatascience.com/topic-modeling-and-latent-dirichlet-allocation-in-python-9bf156893c24)\n",
    "3. [David M. Blei et al, Latent Dirichlet Allocation,Journal of Machine Learning Research 3 (2003) 993-1022,2003](https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf)\n",
    "4. [Topic Modeling with Gensim](https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/)\n",
    "5. [A Deeper Meaning: Topic Modeling in Python](https://www.toptal.com/python/topic-modeling-python)\n",
    "6. [Latent Dirichlet Allocation: Intuition, math, implementation and visualisation with pyLDAvis](https://towardsdatascience.com/latent-dirichlet-allocation-intuition-math-implementation-and-visualisation-63ccb616e094)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "2db524e06e9f5f4ffedc911c917cb75e12dbc923643829bf417064a77eb14d37"
  },
  "kernelspec": {
   "display_name": "Python 3.8.7 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}