week3day2_lab_text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fundamentals of Social Data Science \n",
    "## Week 3 Day 2. Lab. Text Processing \n",
    "\n",
    "In this lab I have generated some methods that will allow you to download posts from Reddit. It accepts a list of subreddits of arbitrary length, which are each processed independently and stored in a single `results` dictionary. The keys of the dictionary are the subreddits. Underneath each subreddit is a dictionary of sub-specific result objects, like \"vectorizer\" and \"top terms\".\n",
    "\n",
    "Please read through the code. You will need to add your username. The code is intentionally broken so you will need to add that before running this. Other than that you should not need to make any modifications to the cell with the `RedditScraper` class. \n",
    "\n",
    "In the cell below is some code to run these methods. At the top are some parameters that you should set. These are typically written in ALL CAPS. You should read the code to understand what they do. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises: \n",
    "    \n",
    "0. **Explore subreddits**. The below code uses 'ukpolitics', 'unitedkingdom', and 'uknews'. These were loosely motivated by an interest in whether uknews has become a reactionary subreddit with generally conservative opinions. By comparing it to the other two (which other work has suggested are generally quite similar), we might get a sense of this from the top keywords. Try some other subs related to a topic where you suspect there will be some interesting distinctions. Motivate the distinctions. If you aren't sure about the subs, query an LLM (they are typically trained on a _lot_ of Reddit data and will know good subs). So instead of trying for /r/men and /r/women, if you ask about subs for gender-based interests, it might suggest /r/TwoXChromosomes and /r/MensRights as interesting distinctions. \n",
    "1. **Understand the results data structure**. The `results` object returns the top 5 terms. How would you access more than 5 terms? Expand the results to see 10. Consider what way is more general and flexible. How might you change the code so that there is a `TOP_N = 10` which is then passed through the code so that the results dictionary contains ten terms in the \"top_terms\" DataFrame rather than hard coding it in the method below?\n",
    "2. **Store results**. Every time we run the code we query Reddit again. How can we store our data so that it is cached for another round? There are many approaches to this and among your group you may discover everything from 'just save the json' to 'DataFrame and then export to feather' to some who would ambitiously use MongoDB. Given this is a simple exercise for now, keep this step simple as you need it to be while still usable enough if you want to add more data.\n",
    "3. **Plot keywords over time**. Expand your results to anywhere from 250 upwards (I would here cap at 500 max and think that the api might only return last 1000 but untested). Determine the top keywords using TFIDF. Then plot the frequency of these keywords over this time period for these results.   \n",
    "4. **Table the most common URLs for stories**. Triangulate these plots with a table summarising the top news outlets for this sub in this time period. Notice the starter code to process this from the posts data that has been stored in a large `submissions` dictionary. Note, this code does not turn all the `json` into a DataFrame, but extracts only the URL column and processes that. It also uses a _regular expression_ to separate out the top level domain, which may or may not be the most robust.  \n",
    "5. **Write a summary**. Solely for reflection at this point, write some intuitions that you discover with this exploration. \n",
    "\n",
    "## Caveats for the exercise: \n",
    "- Reddit might severely limit the number of posts you download using this scraper even with your name appropriately in the username, so be judicious with your exploration (hence exercise 2 _first_). \n",
    "- While you might not have extensive experience with Reddit, I can be confident that there are subreddits on most imaginable topics that can be found with little challenge. However, these subs will have vastly different numbers of subscribers and activity, so bear that in mind with any interpretation when tempted to generalise what is found _beyond_ Reddit (i.e. generalising from /r/republicans to Republicans in the US). \n",
    "- You may be tempted out of curiosity to expand your data collection. You will find that this will lead to a trade off if you do not further process your data. If you have 1000 rows for headlines and 3000 for words, that's a big matrix that has to be multiplied by vectors. At some point the size of the matrix will be unnecessary as well as slow. You may need to consider different parameters for `MIN_DOC_FREQ` to get a balance between a big matrix and a meaningful one. \n",
    "- These results have not been cleared for publication with CUREC, but only for use within classroom and for illustrative purposes. Please do not upload raw reddit data to your own GitHub archive nor seek to publish these results.  (Notice that I have pre-emtively edited the .gitignore to include a `data/` folder where you can store results without uploading them). Seek advice from research.fac@oii.ox.ac.uk for use for a comparable project should you wish to publish this work. If you wish to produce a blog post or other informal analysis, this should be presented in such a way that it is not misconstrued that the University has endorsed this work for publication. \n",
    "\n",
    "# Where we are headed with this exercise \n",
    "\n",
    "### Today: \n",
    "Collect reddit data, make it robust and explore TF-IDF results. \n",
    "\n",
    "### Week 3 Day 3. Friday: \n",
    "We use contine the use of the TF-IDF matrix and introduce cosine distance. We show how to plot it using t-SNE. This might sound abstract but the results will be fascinating as we see words plotted in coherent clusters that seem to reveal inductive patterns. \n",
    "\n",
    "Worksheets will be uploaded to this repo. \n",
    "\n",
    "### Week 4 Day 1. Monday: \n",
    "We will use two simple forms of classification, k-means and Naive Bayes Clustering. You might also be familiar with LDA or 'topic modelling'. We will not cover this as the technique deserves some care to understand its internals even if it is easy to run out of the box. But it is not far as an extension from where we end up. \n",
    "\n",
    "In the lab we will then compare classification results to results from the t-SNE and exploration of distance from Friday. \n",
    "\n",
    "### Week 4 Day 2. Wednesday: \n",
    "We will introduce the `networkx` and `community` package and show how to both construct a network from threaded comments and users of these comments. This will involve two types of graphs: DAGs and Bipartite graphs. \n",
    "\n",
    "In the lab you will have code that shows how to do this with the Reddit data in general. You will have to apply this to your specific case. \n",
    "\n",
    "### Week 4 Day 3. Friday: \n",
    "In the walkthrough we will see how to create 'embeddings' as abstractions even further than t-SNE but as a next-step up from cosine distance. In fact we will see how you can use cosine distance on embeddings which allows you to do these same steps not with words, but with entire sentences or whole paragraphs. We feature this on Friday and assume that your presentations will not need to use embeddings. \n",
    "\n",
    "In the afternoon you we will have the second set of group presentations: \n",
    "- Take a current event or coherent topic that could be collected from reddit data using the requests API (or more abstract packages such as `praw`, but not entire archive dumps like PushShift, only a limited subset). \n",
    "- Look at three or more subreddits who might speak to that topic. Determine which two subs are the most similar and why? Be sure to consider not only common word use. You may define similarity in creative ways so long as they can result in calculable differences without use of ML models, external APIs, or mass labelling of data. If you can download a lexicon, you are welcome to use scoring.\n",
    "- Motivate this topic deductively. Where possible try to draw upon any existing literature on the topic and not simply abductively from current events. Consider DIKW: Find ways to produce transferable _knowledge_ rather than merely _information_ from _data_. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import time\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from collections import Counter\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk.corpus import stopwords\n",
    "import re\n",
    "\n",
    "class RedditScraper:\n",
    "    def __init__(self, user_agent):\n",
    "        \"\"\"\n",
    "        Initialize the scraper with a user agent string.\n",
    "        Example user agent: \"SDS_textanalysis/1.0 (by /u/your_username)\"\n",
    "        \"\"\"\n",
    "        self.headers = {'User-Agent': user_agent}\n",
    "        self.base_url = \"https://api.reddit.com\"\n",
    "        \n",
    "    def get_subreddit_posts(self, subreddit, limit=100):\n",
    "        \"\"\"\n",
    "        Collect posts from a subreddit with proper pagination and rate limiting.\n",
    "        \"\"\"\n",
    "        posts = []\n",
    "        after = None\n",
    "        \n",
    "        while len(posts) < limit:\n",
    "            url = f\"{self.base_url}/r/{subreddit}/new\"\n",
    "            params = {\n",
    "                'limit': min(100, limit - len(posts)),\n",
    "                'after': after\n",
    "            }\n",
    "            \n",
    "            response = requests.get(url, headers=self.headers, params=params)\n",
    "            \n",
    "            if response.status_code != 200:\n",
    "                print(f\"Error accessing r/{subreddit}: {response.status_code}\")\n",
    "                break\n",
    "                \n",
    "            data = response.json()\n",
    "            new_posts = data['data']['children']\n",
    "            if not new_posts:\n",
    "                break\n",
    "                \n",
    "            posts.extend([post['data'] for post in new_posts])\n",
    "            after = data['data']['after']\n",
    "            \n",
    "            if not after:\n",
    "                break\n",
    "                \n",
    "            time.sleep(2)\n",
    "            \n",
    "        return posts[:limit]\n",
    "\n",
    "def preprocess_text(text):\n",
    "    \"\"\"\n",
    "    Clean and normalize text.\n",
    "    \"\"\"\n",
    "    if pd.isna(text):\n",
    "        return \"\"\n",
    "    \n",
    "    # Convert to lowercase and remove special characters\n",
    "    text = text.lower()\n",
    "    text = re.sub(r'[^\\w\\s]', '', text)\n",
    "    text = re.sub(r'\\s+', ' ', text).strip()\n",
    "    return text\n",
    "\n",
    "def analyze_vocabulary(texts, min_freq=2):\n",
    "    \"\"\"\n",
    "    Analyze vocabulary distribution in a corpus.\n",
    "    Returns word frequencies and vocabulary statistics.\n",
    "    \"\"\"\n",
    "    # Tokenize all texts\n",
    "    words = ' '.join(texts).split()\n",
    "    \n",
    "    # Count word frequencies\n",
    "    word_freq = Counter(words)\n",
    "    \n",
    "    # Calculate vocabulary statistics\n",
    "    total_words = len(words)\n",
    "    unique_words = len(word_freq)\n",
    "    \n",
    "    # Create frequency distribution DataFrame\n",
    "    freq_df = pd.DataFrame(list(word_freq.items()), columns=['word', 'frequency'])\n",
    "    freq_df['percentage'] = freq_df['frequency'] / total_words * 100\n",
    "    freq_df = freq_df.sort_values('frequency', ascending=False)\n",
    "    \n",
    "    # Calculate cumulative coverage\n",
    "    freq_df['cumulative_percentage'] = freq_df['percentage'].cumsum()\n",
    "    \n",
    "    stats = {\n",
    "        'total_words': total_words,\n",
    "        'unique_words': unique_words,\n",
    "        'words_min_freq': sum(1 for freq in word_freq.values() if freq >= min_freq),\n",
    "        'coverage_top_1000': freq_df.iloc[:1000]['frequency'].sum() / total_words * 100 if len(freq_df) >= 1000 else 100\n",
    "    }\n",
    "    \n",
    "    return freq_df, stats\n",
    "\n",
    "def analyze_subreddit(posts, max_terms=1000, min_doc_freq=2):\n",
    "    \"\"\"\n",
    "    Analyze a single subreddit's posts independently.\n",
    "    \"\"\"\n",
    "    # Combine title and selftext\n",
    "    texts = [\n",
    "        preprocess_text(post.get('title', '')) + ' ' + \n",
    "        preprocess_text(post.get('selftext', ''))\n",
    "        for post in posts\n",
    "    ]\n",
    "    \n",
    "    # Analyze vocabulary first\n",
    "    freq_df, vocab_stats = analyze_vocabulary(texts, min_freq=min_doc_freq)\n",
    "    \n",
    "    # Initialize TF-IDF vectorizer for this subreddit\n",
    "    stop_words = list(set(stopwords.words('english')))\n",
    "    vectorizer = TfidfVectorizer(\n",
    "        stop_words=stop_words,\n",
    "        max_features=max_terms,\n",
    "        min_df=min_doc_freq\n",
    "    )\n",
    "    \n",
    "    # Compute TF-IDF\n",
    "    tfidf_matrix = vectorizer.fit_transform(texts)\n",
    "    \n",
    "    # Get average TF-IDF scores\n",
    "    mean_tfidf = np.array(tfidf_matrix.mean(axis=0)).flatten()\n",
    "    \n",
    "    # Get top terms\n",
    "    feature_names = vectorizer.get_feature_names_out()\n",
    "    top_terms = pd.DataFrame({\n",
    "        'term': feature_names,\n",
    "        'score': mean_tfidf\n",
    "    }).sort_values('score', ascending=False)\n",
    "    \n",
    "    return {\n",
    "        'vocab_stats': vocab_stats,\n",
    "        'freq_distribution': freq_df,\n",
    "        'top_terms': top_terms.head(),\n",
    "        'vectorizer': vectorizer,\n",
    "        'matrix_shape': tfidf_matrix.shape,\n",
    "        'matrix_sparsity': 100 * (1 - tfidf_matrix.nnz / (tfidf_matrix.shape[0] * tfidf_matrix.shape[1]))\n",
    "    }\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Analyzing r/unitedkingdom...\n",
      "\n",
      "Vocabulary Statistics for r/unitedkingdom:\n",
      "Total words: 590\n",
      "Unique words: 437\n",
      "Words appearing ≥2 times: 56\n",
      "Coverage by top 1000 words: 100.00%\n",
      "Matrix shape: (50, 31)\n",
      "Matrix sparsity: 95.03%\n",
      "\n",
      "Top 5 terms by TF-IDF score:\n",
      "          term     score\n",
      "29          uk  0.110462\n",
      "13       found  0.059997\n",
      "15  government  0.049421\n",
      "26        says  0.049268\n",
      "20         man  0.044528\n",
      "\n",
      "Analyzing r/ukpolitics...\n",
      "\n",
      "Vocabulary Statistics for r/ukpolitics:\n",
      "Total words: 1712\n",
      "Unique words: 827\n",
      "Words appearing ≥2 times: 250\n",
      "Coverage by top 1000 words: 100.00%\n",
      "Matrix shape: (50, 126)\n",
      "Matrix sparsity: 94.68%\n",
      "\n",
      "Top 5 terms by TF-IDF score:\n",
      "          term     score\n",
      "19      budget  0.071309\n",
      "86      rachel  0.051219\n",
      "94      reeves  0.049968\n",
      "103  southport  0.046033\n",
      "111        tax  0.045620\n",
      "\n",
      "Analyzing r/uknews...\n",
      "\n",
      "Vocabulary Statistics for r/uknews:\n",
      "Total words: 652\n",
      "Unique words: 428\n",
      "Words appearing ≥2 times: 98\n",
      "Coverage by top 1000 words: 100.00%\n",
      "Matrix shape: (50, 65)\n",
      "Matrix sparsity: 95.14%\n",
      "\n",
      "Top 5 terms by TF-IDF score:\n",
      "       term     score\n",
      "62       uk  0.070279\n",
      "13   budget  0.055636\n",
      "54  starmer  0.054424\n",
      "28   jailed  0.053580\n",
      "8      away  0.040000\n"
     ]
    }
   ],
   "source": [
    "# Example subreddits\n",
    "subreddits = ['unitedkingdom', 'ukpolitics', 'uknews']\n",
    "\n",
    "# Analysis parameters\n",
    "MAX_TERMS = 1000\n",
    "MIN_DOC_FREQ = 2\n",
    "LIMIT = 50\n",
    "USERNAME = 1/0 # Replace with your Reddit username\n",
    "\n",
    "# Initialize scraper\n",
    "scraper = RedditScraper(\n",
    "user_agent=f\"SDS_textanalysis/1.0 (by /u/{USERNAME})\"\n",
    ")\n",
    "\n",
    "# Analyze each subreddit independently\n",
    "results = {}\n",
    "submissions = {}\n",
    "\n",
    "for subreddit in subreddits:\n",
    "    print(f\"\\nAnalyzing r/{subreddit}...\")\n",
    "\n",
    "    # Collect posts\n",
    "    submissions[subreddit] = scraper.get_subreddit_posts(subreddit, limit=LIMIT)\n",
    "\n",
    "    # Analyze subreddit\n",
    "    results[subreddit] = analyze_subreddit(\n",
    "        submissions[subreddit],\n",
    "        max_terms=MAX_TERMS,   # Maximum number of terms to keep\n",
    "        min_doc_freq=MIN_DOC_FREQ   # Term must appear in at least min_doc_freq documents\n",
    "    )\n",
    "\n",
    "    # Print results for this subreddit\n",
    "    print(f\"\\nVocabulary Statistics for r/{subreddit}:\")\n",
    "    print(f\"Total words: {results[subreddit]['vocab_stats']['total_words']}\")\n",
    "    print(f\"Unique words: {results[subreddit]['vocab_stats']['unique_words']}\")\n",
    "    print(f\"Words appearing ≥{MIN_DOC_FREQ} times: {results[subreddit]['vocab_stats']['words_min_freq']}\")\n",
    "    print(f\"Coverage by top {MAX_TERMS} words: {results[subreddit]['vocab_stats']['coverage_top_1000']:.2f}%\")\n",
    "    print(f\"Matrix shape: {results[subreddit]['matrix_shape']}\")\n",
    "    print(f\"Matrix sparsity: {results[subreddit]['matrix_sparsity']:.2f}%\")\n",
    "\n",
    "    print(\"\\nTop 5 terms by TF-IDF score:\")\n",
    "    print(results[subreddit]['top_terms'][['term', 'score']].to_string())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "domain\n",
       "https://www.dailymail.co.uk              6\n",
       "https://www.thetimes.com                 5\n",
       "https://www.mirror.co.uk                 5\n",
       "https://www.reddit.com                   4\n",
       "https://www.dailystar.co.uk              4\n",
       "https://metro.co.uk                      3\n",
       "https://inews.co.uk                      3\n",
       "https://www.gbnews.com                   2\n",
       "https://thebattleground.eu               2\n",
       "https://www.gloucestershirelive.co.uk    2\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data Exploration: \n",
    "submissions['ukpolitics'][0] # Example post from the unitedkingdom subreddit\n",
    "\n",
    "url_list = [post['url'] for post in submissions['uknews']]\n",
    "url_df = pd.DataFrame(url_list, columns=['url'])\n",
    "url_df['domain'] = url_df['url'].str.extract(r'(https?://[^/]+)')\n",
    "\n",
    "url_df['domain'].value_counts().head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['vocab_stats', 'freq_distribution', 'top_terms', 'vectorizer', 'matrix_shape', 'matrix_sparsity'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results['unitedkingdom'].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AI Declaration: \n",
    "\n",
    "Claude Sonnet 3.5 New produced much of the reddit code. I was surprised at how similar it was to my past code (knowing it was trained on GitHub I have to wonder). Several tweaks had to be made such as removing a main() function, altering the results object, altering some NLTK packages, adding the submissions dictionary and the submissions dictionary code. I kept in the `preprocess_text()` function as is, but you are encouraged to consider alternative forms of pre-processing from the walkthrough including the use of standard tokenizers, lemmatisation, and stop-words. It also used anodyne programming subreddits which I changed and I reduced the limit to 50 which is just enough to get two queries illustrating that you can get N queries through this approach. \n",
    "\n",
    "The URL code was written in VS code with co-pilot. Notably the autocomplete did an excellent job of anticipating steps with minimal prompting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}