NewsClassifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# News Classifier Notebook #\n",
    "\n",
    "This file is an [ipython notebook](https://ipython.org/ipython-doc/3/notebook/notebook.html) that defines a simple machine learning classifier for news articles.\n",
    "\n",
    "## Install ##\n",
    "The easiest way to explore machine learning with python is to [install the\n",
    "Anaconda Distribution](https://www.anaconda.com/distribution/) which contains a\n",
    "variety of useful python packages in a single installation. Alternatively, you\n",
    "can [download and install python](https://www.python.org/downloads/) and then\n",
    "use [pipenv](https://pipenv.readthedocs.io/en/latest/) to install the packages\n",
    "specified in this repo's Pipfile using `pipenv update`.\n",
    "\n",
    "Once you have python and jupyter installed, run the `jupyter notebook NewsClassifier.ipynb` command. If you used pipenv, you'll want to run `pipenv run jupyter notebook NewsClassifier.ipynb`. \n",
    "\n",
    "At that point, this notebook should be running on your local computer, and you can follow along by running the code in the following cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from newspaper import Article\n",
    "import csv\n",
    "import random\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.pipeline import make_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How does this work? ##\n",
    "\n",
    "The following function loads the [TrainingData.csv](TrainingData.csv) file, which contains URLs and\n",
    "corresponding classification labels gathered from the [News Classification\n",
    "Form](https://goo.gl/forms/nLcf2ol0o5dAJxGw1). It returns a list containing a dictionary corresponding to each row in the CSV.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_csv_data(local_file):\n",
    "    labeled_data = []\n",
    "    with open(local_file) as csv_file:\n",
    "        csv_reader = csv.reader(csv_file, delimiter=',')\n",
    "        line_count = 0\n",
    "        for row in csv_reader:\n",
    "            if line_count == 0:\n",
    "                print(\"Discarding first CSV header row: {}\".format(\", \".join(row)))\n",
    "            else:\n",
    "                # for each row, append a new dictionary with \"url\" and \"label\" keys\n",
    "                labeled_data.append({\"url\": row[1].strip(), \"label\": row[2].strip()})\n",
    "            line_count += 1\n",
    "        print(\"Processed {} lines.\".format(line_count))\n",
    "    return labeled_data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function takes a URL as input and returns a [newspaper3k library article](https://newspaper.readthedocs.io/en/latest/) loaded with the URL content. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_article(url):\n",
    "    a = Article(url, fetch_images=False)\n",
    "    a.download()\n",
    "    a.parse()\n",
    "    return a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function takes the labeled data and returns two lists. \n",
    "\n",
    "The first list contains the text content of the articles. The second list contains the label corresponding to each document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_doc_and_label_lists(labeled_data):\n",
    "    # Shuffle the labeled data\n",
    "    random.shuffle(labeled_data)\n",
    "    docs = []\n",
    "    labels = []\n",
    "    num_imported = 0\n",
    "    for item in labeled_data:\n",
    "        try: \n",
    "            article = process_article(item['url'])\n",
    "        except Exception as e:\n",
    "            print(\"Exception: {} | processing url {}\".format(str(e), item['url']))\n",
    "            continue\n",
    "        docs.append(article.text)\n",
    "        labels.append(item['label'])\n",
    "        num_imported += 1\n",
    "        if 0 == num_imported % 20:\n",
    "            print(\"Imported {} articles\".format(num_imported))\n",
    "    return docs, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function uses our model to return the label for a given URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def classify(model, url):\n",
    "    art = process_article(url)\n",
    "    label = model.predict([art.text])\n",
    "    return art.title, label[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code uses the functions defined above to prepare data. It then uses code similar to the code found in the [Multinomial Naive Bayes section of the Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/05.05-naive-bayes.html#Multinomial-Naive-Bayes) to create our classifier.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Discarding first CSV header row: Timestamp, URL, Label\n",
      "Processed 510 lines.\n",
      "Scraping data from URLs...\n",
      "Imported 20 articles\n",
      "Imported 40 articles\n",
      "Imported 60 articles\n",
      "Imported 80 articles\n",
      "Imported 100 articles\n",
      "Imported 120 articles\n",
      "Imported 140 articles\n",
      "Imported 160 articles\n",
      "Imported 180 articles\n",
      "Imported 200 articles\n",
      "Imported 220 articles\n",
      "Imported 240 articles\n",
      "Imported 260 articles\n",
      "Imported 280 articles\n",
      "Imported 300 articles\n",
      "Imported 320 articles\n",
      "Imported 340 articles\n",
      "Imported 360 articles\n",
      "Imported 380 articles\n",
      "Imported 400 articles\n",
      "Imported 420 articles\n",
      "Imported 440 articles\n",
      "Imported 460 articles\n",
      "Imported 480 articles\n",
      "Imported 500 articles\n",
      "Processing / vectorizing input text for the model\n",
      "Generating the model using the training data\n",
      "The model accurately classified 73% of 102 test articles\n"
     ]
    }
   ],
   "source": [
    "labeled_data = get_csv_data('TrainingData.csv')\n",
    "print(\"Scraping data from URLs...\")\n",
    "docs, labels = get_doc_and_label_lists(labeled_data)\n",
    "\n",
    "# use 80% of the data for training, reserve 20% for testing\n",
    "split = int(len(docs) * 0.8)  \n",
    "\n",
    "print(\"Processing / vectorizing input text for the model\")\n",
    "model = make_pipeline(TfidfVectorizer(), MultinomialNB())\n",
    "\n",
    "print(\"Generating the model using the training data\")\n",
    "model.fit(docs[:split], labels[:split])\n",
    "\n",
    "# This is debug code to manually inspect results\n",
    "#for test_doc, test_label in zip(docs[split:], labels[split:]):\n",
    "#    label = model.predict([test_doc])\n",
    "#    print(label, \" | \", test_label, \" ||||| \", test_doc[:200])\n",
    "#    print(\"--------\")\n",
    "    \n",
    "# this does essentially the same thing as the loop above,\n",
    "# while keeping track of right and wrong answers, returning an accuracy score\n",
    "score = model.score(docs[split:], labels[split:])\n",
    "print(\"The model accurately classified {:.0%} of {} test articles\".format(score, len(labels[split:])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code defines a set of test URLs, classifies them, and outputs the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Arts and Entertainment: The Best Female Friendships in Music\n",
      "Politics: Medicare for All Is Divisive (in the Democratic Party)\n",
      "Sports: Cubs at spring training: A game-by-game recap\n",
      "Nature and the Environment: How much water does the West really have?\n",
      "Science and Technology: Cloud is not a virtue\n"
     ]
    }
   ],
   "source": [
    "testcases = ['https://www.pastemagazine.com/articles/2019/02/best-female-friendships-in-music.html',\n",
    "             'https://www.nytimes.com/2019/03/18/us/politics/democrats-health-care-medicare-for-all.html',\n",
    "             'https://www.chicagotribune.com/sports/baseball/cubs/ct-spt-chicago-cubs-spring-training-results-2019-story.html',\n",
    "             'https://projects.propublica.org/killing-the-colorado/story/video-groundwater-explained',\n",
    "             'https://18f.gsa.gov/2019/02/07/the-cloud-is-not-a-virtue/'\n",
    "            ]\n",
    "for turl in testcases:\n",
    "    title, label = classify(model, turl)\n",
    "    print(\"{}: {}\".format(label, title))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}