Lecture04_TopicModels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XLMx-_ei-Fe1"
   },
   "source": [
    "# Lecture 4 - Topic Models\n",
    "\n",
    "In this notebook we will learn how to cluster text into topics using different embeddings and the K-means clustering algorithm. \n",
    "\n",
    "Below is the overview of this notebook.\n",
    "\n",
    "0. Install required packages (only need to do this the first time we run the notebook)\n",
    "\n",
    "1. Load corpus of tweets\n",
    "\n",
    "2. Make word clouds of the tweets\n",
    "\n",
    "3. Create tf and tf-idf embeddings of the tweets\n",
    "\n",
    "4. Create LDA topic model embeddings of the tweets\n",
    "\n",
    "5. Create low dimensional embeddings of the tweets using UMAP\n",
    "\n",
    "6. Cluster the tweets using K-means clustering\n",
    "\n",
    "7. Analyze clusters using word clouds and screen name histograms\n",
    "\n",
    "This notebook can be opened in Colab \n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/zlisto/social_media_analytics/blob/main/Lecture04_TopicModels.ipynb)\n",
    "\n",
    "Before starting, select \"Runtime->Factory reset runtime\" to start with your directories and environment in the base state.\n",
    "\n",
    "If you want to save changes to the notebook, select \"File->Save a copy in Drive\" from the top menu in Colab.  This will save the notebook in your Google Drive.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rFXIL5Xh-yPz"
   },
   "source": [
    "# Clone GitHub Repository\n",
    "This will clone the repository to your machine.  This includes the code and data files.  Then change into the directory of the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "G8VCy2Cd-ifl"
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/zlisto/social_media_analytics\n",
    "\n",
    "import os\n",
    "os.chdir(\"social_media_analytics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kU1he2BH-1EM"
   },
   "source": [
    "## Install Requirements \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "lQR9ab2a-38-"
   },
   "outputs": [],
   "source": [
    "!pip install -r requirements.txt --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sNr1_aR5_LJ_"
   },
   "source": [
    "## Import packages\n",
    "\n",
    "We import the packages we are going to use.  A package contains several useful functions that make our life easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YQE1kWWE_FyT"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "\n",
    "#import umap\n",
    "import gensim.downloader as api\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "from sklearn.decomposition import NMF, LatentDirichletAllocation\n",
    "\n",
    "import sklearn.cluster as cluster\n",
    "from sklearn import metrics\n",
    "from scipy import stats\n",
    "\n",
    "from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator\n",
    "\n",
    "import pyLDAvis\n",
    "import pyLDAvis.lda_model\n",
    "pyLDAvis.enable_notebook()\n",
    "\n",
    "import scripts.TextAnalysis as ta\n",
    "from scripts.api import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HIfy_RVh_4DN"
   },
   "source": [
    "# Data Cleaning\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "izI_8uY9_-bw"
   },
   "source": [
    "\n",
    "### Load data\n",
    "\n",
    "We will load csv files containing tweets from several users into a dataframe **df**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cK_Oun_ZACuV"
   },
   "outputs": [],
   "source": [
    "fname_db = \"data/lecture_04\"\n",
    "df = DB.fetch(table_name = 'user_tweets', path = fname_db)\n",
    "n = len(df)\n",
    "print(f\"{n} tweets in dataframe\")\n",
    "df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h7-L9EWdAATF"
   },
   "source": [
    "###  Remove Superfluous Columns \n",
    "\n",
    "We don't need all the columns.  We can remove them from this dataframe using the column selection operation.  We just choose which columns we want to keep and put them in a list.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "370OFx6RATdS"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qH5Zu2DvAOBm"
   },
   "outputs": [],
   "source": [
    "df = df[ ['screen_name', 'text', 'retweet_count']]\n",
    "df.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tU90isveAcUn"
   },
   "source": [
    "### Plot Tweets per User\n",
    "\n",
    "A count plot shows us how many tweets each user has in the dataset.  If we choose `y` to be `\"screen_name\"` the plot will be vertical.\n",
    "\n",
    "We can choose the `palette` for the plot from this list here: https://seaborn.pydata.org/tutorial/color_palettes.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5fPL923_AemY"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fNuGH-leAOD1"
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(8,8))\n",
    "sns.countplot(data=df,y='screen_name',  palette = \"Set2\")\n",
    "plt.ylabel(\"Screen name\", fontsize = 14)\n",
    "plt.xlabel(\"Tweet count\", fontsize = 14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rrXhiI52BM3r"
   },
   "source": [
    "### Cleaning Text Data\n",
    "Next we will clean the tweet text.  We use the `clean_tweet` function in the TextAnalytics module.  This function removes punctuation and hyperlinks, and also makes all the text lower case.  We remove any cleaned tweets which have zero length, as these won't be useful for clustering.  We add a column to `df` called `\"text_clean\"` with the cleaned tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HmA-JBgJBSGh"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2TFlFikVAOGG"
   },
   "outputs": [],
   "source": [
    "df['text_clean'] = df.text.apply(ta.clean_tweet)  #clean the tweets\n",
    "df = df[df.text_clean.str.len() >0]  #remove cleaned tweets of lenght 0\n",
    "\n",
    "nclean = len(df)\n",
    "\n",
    "print(f\"{n} tweets, {nclean} clean tweets\")\n",
    "\n",
    "df.sample(n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Copy of Dataframe\n",
    "\n",
    "Sometimes you want to work on a slice of a dataframe.  For example, maybe you want to work with a slice that contains tweets from a single screen name.  If you want to add a column to the slice, you will get a warning, because the slice is tied to the original dataframe.  To avoid this, use the `copy` function when creating the slice.  This makes the slice an independent copy and now you can add colummns without any error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Adding a column to a slice of a dataframe (not a copy) gives a warning\")\n",
    "\n",
    "df_aoc = df[df.screen_name=='AOC']\n",
    "\n",
    "df_aoc['test'] = df.retweet_count\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Adding a column to a copy of a slice of a dataframe is ok\")\n",
    "\n",
    "df_aoc = df[df.screen_name=='AOC'].copy()\n",
    "\n",
    "df_aoc['test'] = df.retweet_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jZGPOfQJLfrq"
   },
   "source": [
    "# Word Cloud\n",
    "\n",
    "We can make a word cloud of the tweets using the `WordCloud` function which takes as input a list of stopwords and many other parameters. \n",
    "\n",
    "The list of stopwords is `STOPWORDS`.  We apply the `set` function so we can remove duplicates and easily add new words.  To add a new stopword to the set, we use the `add` function.\n",
    "\n",
    "We convert the `text` column of our dataframe into a single giant string called `text` using the `tolist` and `join` functions.  We then apply the `generate` function to `text` to make the word cloud.  The `imshow` function allows us to visualize the word cloud.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wFeHW7VfLlG8"
   },
   "outputs": [],
   "source": [
    "stopwords = set(STOPWORDS)\n",
    "stopwords.add(\"de\")\n",
    "stopwords.add(\"que\")\n",
    "\n",
    "text=' '.join(df.text_clean.tolist()).lower()\n",
    "wordcloud = WordCloud(stopwords=stopwords,max_font_size=150, \n",
    "                      max_words=100, \n",
    "                      background_color=\"black\",\n",
    "                      width=1000, \n",
    "                      height=600)\n",
    "\n",
    "wordcloud.generate(text)\n",
    "\n",
    "#visualize word cloud\n",
    "fig = plt.figure(figsize = (10,8))\n",
    "plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "plt.axis(\"off\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CbIQglU1Btzc"
   },
   "source": [
    "# Create Text Embeddings\n",
    "\n",
    "To cluster the tweets, we need to create vector embeddings for them.  We can do this using vectorizers.  We have two simple options here.  One is as a term frequency (tf) vectorizer called *CountVectorizer*.  The other is a term-frequency inverse document-frequency (tf-idf) vectorizer called *TfidfVectorizer*.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WfKOwIp9Bxmo"
   },
   "source": [
    "### Term Frequency (TF) Embedding\n",
    "\n",
    "We initialize the `CountVectorizer` and tell it to remove English stopwords with the `stop_words` parameter set to `\"english\"`.  We also tell it to remove any word that occur in less than 5 documents with the `min_df` parameter.  Then we use the `fit_transform` method applied to the `text_clean` column of `df` to create the document vectors, which we call `tf_embedding`.  We store the words for each element of the vector in `tf_feature_names`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WurebVnSBvx-"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TPk5HOSVBv0W"
   },
   "outputs": [],
   "source": [
    "tf_vectorizer = CountVectorizer(min_df=5, stop_words='english')\n",
    "tf_embedding = tf_vectorizer.fit_transform(df.text_clean)\n",
    "tf_feature_names = tf_vectorizer.get_feature_names_out()\n",
    "\n",
    "nvocab = len(tf_feature_names)\n",
    "ntweets = len(df.text_clean)\n",
    "print(f\"{ntweets} tweets, {nvocab} words in vocabulary\")\n",
    "print(f\"TF embedding shape is {tf_embedding.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bnW_EVBqB4ZQ"
   },
   "source": [
    "### Term Frequency-Inverse Document Frequency (TF-IDF) Embedding\n",
    "\n",
    "We initialize the `TfidfVectorizer` as we did the `CountVectorizer`.  Then we use the `fit_transform` method applied to the `text_clean` column of `df` to create the document vectors, which we call `tfidf_embedding`.  We store the words for each element of the vector in `tfidf_feature_names`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dyi8GblJBv2n"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZBFg4vgjBv49"
   },
   "outputs": [],
   "source": [
    "tfidf_vectorizer = TfidfVectorizer(min_df=5, stop_words='english')\n",
    "tfidf_embedding = tfidf_vectorizer.fit_transform(df.text_clean)\n",
    "tfidf_feature_names = tfidf_vectorizer.get_feature_names_out()\n",
    "\n",
    "nvocab = len(tfidf_feature_names)\n",
    "print(f\"{ntweets} tweets, {nvocab} words in vocabulary\")\n",
    "print(f\"TF-IDF embedding shape is {tfidf_embedding.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J0rZXZYIW-E-"
   },
   "source": [
    "### Latent Dirichlet Allocation (LDA) Embedding\n",
    "\n",
    "We will fit an LDA topic model on the tf embedding of the tweets. Much of this section pulls code from this blog:\n",
    "\n",
    "https://medium.com/mlreview/topic-modeling-with-scikit-learn-e80d33668730\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7rKthY1LXtQj"
   },
   "source": [
    "#### Fitting LDA Model\n",
    "\n",
    "To fit an LDA model we need to specify the number of topics.  There are sophisticated ways to do this, but because it takes some time to fit the model, we will cheat here.  We set `num_topics` equal to the number of unique users in the dataset.  Hopefully we find one topic for each user.  To fit the model we use the `LatentDirichletAllocation` function.  We first initialize this object with the number of topics, and then use the `fit` function to fit the model to `tf_embedding` (we can't use `tfidf_embedding` because LDA data must be word counts (integers)).  The fit model object is called `lda`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "C1mWy0VRXd3D"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "num_topics = len(df.screen_name.unique())\n",
    "lda = LatentDirichletAllocation(n_components=num_topics, max_iter=5, \n",
    "                                learning_method='online', learning_offset=50.,\n",
    "                                random_state=0).fit(tf_embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AnIDK1rfYN6g"
   },
   "source": [
    "#### Convert Tweets into Topic Embedding Vectors Using LDA Model\n",
    "\n",
    "Next we convert each tweet into a topic embedding vector.  This vector length is the number of topics in the LDA model.  The value of each element tells us the probability the tweet contains this topic.  The conversion is done using the `transform` function of `lda`.  The resulting topic vectors are called `lda_embedding`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UT4xqBZGY6BP"
   },
   "outputs": [],
   "source": [
    "lda_embedding = lda.transform(tf_embedding)\n",
    "print(f\"{ntweets} tweets, {num_topics} topics in LDA model\")\n",
    "print(f\"shape of lda embedding is {lda_embedding.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RQEztlBiybEL"
   },
   "source": [
    "#### Visualizing LDA Topics with pyLDAvis\n",
    "\n",
    "A cool way to visualize the topics in an LDA model is using the pyLDAvis package.  To do this we use the `prepare` function in `pyLDAvis.sklearn` to create an object called `viz`.  The inputs are the model (`lda`), the tf embedding (`tf_embedding`), and the CountVectorizer (`tf_vectorizer`).  Then we create an interactive visualization of the model using the `show` function applied to `viz`.  \n",
    "Here's how to use the pyLDAvis webpage.  Each circle is a topic.  Hover over it and the bar graph lights up with the highest probabilit words in the topic.  You can slide the value of the relevance metric (lambda) to adjust how the relevance of each word is measured.  lambda = 0 means the red bar just shows the probability of the word in the topic.  lambda = 1 means the red bar shows the probability of the word in the topic divided by the probability of the word in the entire corpus of tweets.  For our purposes, lambda = 0 is fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "c2q3Ba17ypiE"
   },
   "outputs": [],
   "source": [
    "viz = pyLDAvis.lda_model.prepare(lda, tf_embedding, tf_vectorizer)\n",
    "pyLDAvis.display(viz)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sGRI8qsrO9xL"
   },
   "source": [
    "### UMAP Embedding\n",
    "\n",
    "We can use UMAP to create low-dimensional embeddings of the tweets.  This allows us to plot the tweets in two dimensions. Also, sometimes the lower dimensional embedding makes better text clusters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KBlu83sMPLio"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "umap_tf_embedding = umap.UMAP(n_components=2, metric='hellinger').fit_transform(tf_embedding)\n",
    "umap_tfidf_embedding = umap.UMAP(n_components=2, metric='hellinger').fit_transform(tfidf_embedding)\n",
    "\n",
    "#zscoring centers the vectors at zero\n",
    "umap_tf_embedding = stats.zscore(umap_tf_embedding,nan_policy='omit')\n",
    "umap_tfidf_embedding = stats.zscore(umap_tfidf_embedding,nan_policy='omit')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jAR6A1KrcuyZ"
   },
   "source": [
    "#### Add UMAP Embeddings to DataFrame\n",
    "\n",
    "Add UMAP embeddings x and y coordinates for each tweet to `df`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Cc0WQC1pcxZU"
   },
   "outputs": [],
   "source": [
    "df['tf_umap_x'] = umap_tf_embedding[:,0]\n",
    "df['tf_umap_y'] = umap_tf_embedding[:,1]\n",
    "df['tfidf_umap_x'] = umap_tfidf_embedding[:,0]\n",
    "df['tfidf_umap_y'] = umap_tfidf_embedding[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_Vdh_q704M_P"
   },
   "source": [
    "#### Visualize Embeddings\n",
    "\n",
    "We can use `scatterplot` to plot the embeddings using the UMAP x-y coordinates.  We will color the data points, which are tweets, by the screen name of their creator using the `hue` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vAz8KQ9O4afc"
   },
   "outputs": [],
   "source": [
    "xmax = 3  #range for x-axis\n",
    "ymax = 3  #range for y-axis\n",
    "s = 5  #marker size\n",
    "\n",
    "fig = plt.figure(figsize = (16,8))\n",
    "\n",
    "ax1 = plt.subplot(1,2,1)\n",
    "sns.scatterplot(data=df, x=\"tf_umap_x\", \n",
    "                y=\"tf_umap_y\", hue=\"screen_name\", s=s)\n",
    "plt.title(\"TF Embedding\")\n",
    "plt.xlim([-xmax, xmax])\n",
    "plt.ylim([-ymax,ymax])\n",
    "\n",
    "ax2 = plt.subplot(1,2,2)\n",
    "sns.scatterplot(data=df, x=\"tfidf_umap_x\", \n",
    "                y=\"tfidf_umap_y\", hue=\"screen_name\", s=s)\n",
    "plt.title(\"TF-IDF Embedding\");\n",
    "plt.xlim([-xmax, xmax])\n",
    "plt.ylim([-ymax,ymax])\n",
    "\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GmwJ6A1SJii3"
   },
   "source": [
    "# Cluster Tweets Using K-Means on Embeddings\n",
    "\n",
    "We will cluster the tf, tf-idf, and word2vec embedding vectors using the k-means algorithm.  We choose the number of clusters we want with the variable `n_clusters`.  To get the cluster label of each tweet we initiailize a `KMeans` object with the number of clusters, and then call the `fit_predict` function on the embedding array.  \n",
    "\n",
    "We create a column in `df` for each k-means cluster label.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CXTRml39LDzC"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "18kUP3fkJ9QY"
   },
   "outputs": [],
   "source": [
    "#n_clusters = len(df.screen_name.unique())\n",
    "n_clusters = 6\n",
    "\n",
    "kmeans_label = cluster.KMeans(n_clusters=n_clusters).fit_predict(tf_embedding)\n",
    "df['kmeans_label_tf'] = [str(x) for x in kmeans_label]\n",
    "\n",
    "kmeans_label = cluster.KMeans(n_clusters=n_clusters).fit_predict(tfidf_embedding)\n",
    "df['kmeans_label_tfidf'] = [str(x) for x in kmeans_label]\n",
    "\n",
    "kmeans_label = cluster.KMeans(n_clusters=n_clusters).fit_predict(lda_embedding)\n",
    "df['kmeans_label_lda'] = [str(x) for x in kmeans_label]\n",
    "\n",
    "kmeans_label = cluster.KMeans(n_clusters=n_clusters).fit_predict(np.nan_to_num(umap_tf_embedding))\n",
    "df['kmeans_label_tf_umap'] = [str(x) for x in kmeans_label]\n",
    "\n",
    "kmeans_label = cluster.KMeans(n_clusters=n_clusters).fit_predict(np.nan_to_num(umap_tfidf_embedding))\n",
    "df['kmeans_label_tfidf_umap'] = [str(x) for x in kmeans_label]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mEtPD9qA5f4-"
   },
   "source": [
    "#### Plot Embeddings with Cluster Labels\n",
    "\n",
    "We can make a scatterplot of the tweet embeddings, but this time color the data points using the cluster label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "scywKN7S5pPf"
   },
   "outputs": [],
   "source": [
    "embedding_types = ['tf_umap','tfidf_umap','lda']\n",
    "s = 5\n",
    "xmax,ymax = 3,3\n",
    "\n",
    "for embedding_type in embedding_types:\n",
    "  fig = plt.figure(figsize = (16,8))\n",
    "  ax1 = plt.subplot(1,2,1)\n",
    "  kmeans_label = f\"kmeans_label_{embedding_type}\"\n",
    "  sns.scatterplot(data=df, x=f\"tfidf_umap_x\", \n",
    "                    y=f\"tfidf_umap_y\", \n",
    "                    hue=\"screen_name\", s=s)\n",
    "  plt.title(\"True Clusters\")\n",
    "  plt.xlim([-xmax, xmax])\n",
    "  plt.ylim([-ymax,ymax])\n",
    "\n",
    "  ax2 = plt.subplot(1,2,2)\n",
    "  sns.scatterplot(data=df, x=f\"tfidf_umap_x\", \n",
    "                    y=f\"tfidf_umap_y\", \n",
    "                    hue=kmeans_label, s=s)\n",
    "  plt.title(f\"{kmeans_label} Clusters\");    \n",
    "  plt.xlim([-xmax, xmax])\n",
    "  plt.ylim([-ymax,ymax])\n",
    "  plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gzP2JRsyKXjc"
   },
   "source": [
    "### Histograms of Users and Word Clouds of Tweets in the Clusters\n",
    "\n",
    "We will take the tweets in each cluster, make a word cloud for them, and a histogram of the screen names of the users who posted the tweets.  If we have good clusters, we expect one user to dominate each cluster, or a group of users who use tweet about similar topics.\n",
    "\n",
    "We will be creating word clouds and histograms again later on, so lets write a function to do it.  The function is called `kmeans_wordcloud_userhist`.  Its inputs are the dataframe with the tweets and cluster labels, `df`, the name of the column with the cluster labels `cluster_label_column`, and a set of stopwords called `stopwords`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gG01suVBLTa6"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "t7t4VH6iKbUm"
   },
   "outputs": [],
   "source": [
    "def kmeans_wordcloud_userhist(df, cluster_label_column,stopwords):\n",
    "    print(cluster_label_column)\n",
    "    for k in np.sort(df[cluster_label_column].unique()):\n",
    "        s=df[df[cluster_label_column]==k]\n",
    "        text=' '.join(s.text_clean.tolist()).lower()\n",
    "        wordcloud = WordCloud(stopwords=stopwords,max_font_size=150, max_words=100, background_color=\"white\",width=1000, height=600)\n",
    "        wordcloud.generate(text)\n",
    "     \n",
    "        print(f\"\\n\\tCluster {k} {cluster_label_column} has {len(s)} tweets\")\n",
    "        plt.figure(figsize = (16,4))\n",
    "        plt.subplot(1,2,1)\n",
    "        ax = sns.countplot(data = s, x = 'screen_name')\n",
    "        plt.xticks(rotation=45)\n",
    "        plt.ylabel(\"Number of tweets\")\n",
    "        plt.xlabel(\"Screen name\")\n",
    "\n",
    "        plt.subplot(1,2,2)\n",
    "        plt.imshow(wordcloud, interpolation=\"bilinear\")\n",
    "        plt.axis(\"off\")\n",
    "        plt.show()\n",
    "    return 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QLPmChV6Z0YS"
   },
   "source": [
    "# Wordcloud of Clusters\n",
    "\n",
    "We can plot a word cloud for each cluster found, along with a histogram of the screen names in the cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ER6dzQi2YXuy"
   },
   "source": [
    "#### Wordcloud for TF Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LSzDaRvcKfUw"
   },
   "outputs": [],
   "source": [
    "stopwords = set(STOPWORDS)\n",
    "cluster_label_column= 'kmeans_label_tf'\n",
    "kmeans_wordcloud_userhist(df,cluster_label_column,stopwords )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RG7VACS5YdWV"
   },
   "source": [
    "#### Wordcloud for TF-IDF Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oKtCDQaZKnml"
   },
   "outputs": [],
   "source": [
    "stopwords = set(STOPWORDS)\n",
    "cluster_label_column= 'kmeans_label_tfidf'\n",
    "kmeans_wordcloud_userhist(df,cluster_label_column,stopwords )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nx_J8Z_DbX2m"
   },
   "source": [
    "#### Wordcloud for LDA UMAP Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Sjwf50xBbYBU"
   },
   "outputs": [],
   "source": [
    "stopwords = set(STOPWORDS)\n",
    "cluster_label_column= 'kmeans_label_lda'\n",
    "kmeans_wordcloud_userhist(df,cluster_label_column,stopwords )"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "kU1he2BH-1EM",
    "sNr1_aR5_LJ_",
    "izI_8uY9_-bw",
    "jZGPOfQJLfrq"
   ],
   "name": "Lecture_04_TopicModels.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}