Text Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "61360e05",
   "metadata": {},
   "source": [
    "# Text Classification\n",
    "\n",
    "The data I wrote this script for is sensitive, so instead I am using a dataset from kaggle. \n",
    "https://www.kaggle.com/guiyihan/text-classification-20\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14dbdef0",
   "metadata": {},
   "source": [
    "## Import Packages and Set Up Data Prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "25dc20d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/chasebrown/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/chasebrown/opt/anaconda3/envs/deepLearning/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "non-resource variables are not supported in the long term\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "#sys.stdout = open('output.txt', 'w')\n",
    "import nltk\n",
    "nltk.download('punkt')\n",
    "from nltk.stem.lancaster import LancasterStemmer\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tflearn\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.framework import ops\n",
    "import random\n",
    "import json\n",
    "import unicodedata\n",
    "\n",
    "# a table structure to hold the different punctuation used\n",
    "tbl = dict.fromkeys(i for i in range(sys.maxunicode)\n",
    "                    if unicodedata.category(chr(i)).startswith('P'))\n",
    "\n",
    "# method to remove punctuations from sentences.\n",
    "def remove_punctuation(text):\n",
    "    return text.translate(tbl)\n",
    "\n",
    "# initialize the stemmer\n",
    "stemmer = LancasterStemmer()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90ecc021",
   "metadata": {},
   "source": [
    "## Import Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "32028ae8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   label                                               text\n",
      "0      0  Archive name atheism resources Alt atheism arc...\n",
      "1      0  Archive name atheism introduction Alt atheism ...\n",
      "2      0  In article 65974 mimsy.umd.edu mangoe cs.umd.e...\n",
      "3      0  dmn kepler.unh.edu ...until kings become philo...\n",
      "4      0  In article N4HY.93Apr5120934 harder.ccr p.ida....\n"
     ]
    }
   ],
   "source": [
    "#Read File\n",
    "df = pd.read_csv(\"data/textClassification.csv\")\n",
    "print(df.head(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e809c654",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2]\n"
     ]
    }
   ],
   "source": [
    "#Extract Categories\n",
    "categories = sorted(df['label'].unique())\n",
    "categories = categories[:3]\n",
    "print(categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4062da6",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73fad14d",
   "metadata": {},
   "source": [
    "### Clean Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b566aae0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "196626   2983\n",
      "['Archive', 'name', 'atheism', 'resources', 'Alt', 'atheism', 'archive', 'name', 'resources', 'Last']\n"
     ]
    }
   ],
   "source": [
    "words = []\n",
    "# a list of tuples with words in the sentence and category name\n",
    "docs = []\n",
    "\n",
    "for category in categories:\n",
    "    sentences = df[df['label'] == category]['text'].unique()\n",
    "    for sentence in sentences:\n",
    "        # remove any punctuation from the sentence\n",
    "        sentence = remove_punctuation(sentence)\n",
    "        sentence = sentence[:int(len(sentence)/5)]\n",
    "        # extract words from each sentence and append to the word list\n",
    "        w = nltk.word_tokenize(sentence)\n",
    "        words.extend(w)\n",
    "        docs.append((w, category))\n",
    "print(len(words),\" \",len(docs))\n",
    "print(words[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a81da56c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\x19', '0', '00', '0000', '000005102000', '0010580b0b6r49', '0010580bvma7o9', '0010580bvmcbrt', '0028', '00451']\n"
     ]
    }
   ],
   "source": [
    "# stem and lower each word and remove duplicates\n",
    "words = [stemmer.stem(w.lower()) for w in words]\n",
    "words = sorted(list(set(words)))\n",
    "print(words[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "171e39cb",
   "metadata": {},
   "source": [
    "### Create Training and Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "988cb0fe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2387   596\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/chasebrown/opt/anaconda3/envs/deepLearning/lib/python3.7/site-packages/ipykernel_launcher.py:28: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "/Users/chasebrown/opt/anaconda3/envs/deepLearning/lib/python3.7/site-packages/ipykernel_launcher.py:29: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n"
     ]
    }
   ],
   "source": [
    "data = []\n",
    "output = []\n",
    "\n",
    "# create an empty array for our output\n",
    "output_empty = [0] * len(categories)\n",
    "for doc in docs:\n",
    "    # initialize our bag of words(bow) for each document in the list\n",
    "    bow = []\n",
    "    # list of tokenized words for the pattern\n",
    "    token_words = doc[0]\n",
    "    # stem each word\n",
    "    token_words = [stemmer.stem(word.lower()) for word in token_words]\n",
    "    # create our bag of words array\n",
    "    for w in words:\n",
    "        bow.append(1) if w in token_words else bow.append(0)\n",
    "\n",
    "    output_row = list(output_empty)\n",
    "    output_row[categories.index(doc[1])] = 1\n",
    "\n",
    "    # our training set will contain a the bag of words model and the output row that tells\n",
    "    # which catefory that bow belongs to.\n",
    "    data.append([bow, output_row])\n",
    "\n",
    "# shuffle our features and turn into np.array as tensorflow  takes in numpy array\n",
    "random.shuffle(data)\n",
    "training = data[int(len(data)/5):]\n",
    "testing = data[:int(len(data)/5)]\n",
    "training = np.array(training)\n",
    "testing = np.array(testing)\n",
    "\n",
    "print(len(training), ' ', len(testing))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "255a2b8a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2387\n",
      "2387\n"
     ]
    }
   ],
   "source": [
    "# train_x contains the Bag of words and train_y contains the label/ category\n",
    "train_x = list(training[:, 0])\n",
    "train_y = list(training[:, 1])\n",
    "print(len(train_x))\n",
    "print(len(train_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6eae32d",
   "metadata": {},
   "source": [
    "## Set Up Neural Network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "89d89e45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/chasebrown/opt/anaconda3/envs/deepLearning/lib/python3.7/site-packages/tflearn/initializations.py:165: calling TruncatedNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n"
     ]
    }
   ],
   "source": [
    "ops.reset_default_graph()\n",
    "net = tflearn.input_data(shape=[None, len(train_x[0])])\n",
    "net = tflearn.fully_connected(net, 8)\n",
    "net = tflearn.fully_connected(net, 8)\n",
    "net = tflearn.fully_connected(net, len(train_y[0]), activation='softmax')\n",
    "net = tflearn.regression(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1cf1ad4",
   "metadata": {},
   "source": [
    "## Define Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "36ea3061",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tflearn.DNN(net, tensorboard_dir='tflearn_logs', tensorboard_verbose=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bc744d8",
   "metadata": {},
   "source": [
    "## Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "34839d78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Step: 5979  | total loss: \u001b[1m\u001b[32m0.27093\u001b[0m\u001b[0m | time: 2.849s\n",
      "| Adam | epoch: 020 | loss: 0.27093 - acc: 0.9754 -- iter: 2384/2387\n",
      "Training Step: 5980  | total loss: \u001b[1m\u001b[32m0.24396\u001b[0m\u001b[0m | time: 2.858s\n",
      "| Adam | epoch: 020 | loss: 0.24396 - acc: 0.9778 -- iter: 2387/2387\n",
      "--\n"
     ]
    }
   ],
   "source": [
    "model.fit(train_x, train_y, n_epoch=20, batch_size=8, show_metric=True)\n",
    "#model.save('model.tflearn')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3e67131",
   "metadata": {},
   "source": [
    "## Test Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "e5f37d59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "596\n",
      "596\n"
     ]
    }
   ],
   "source": [
    "# test_x contains the Bag of words and test_y contains the label/ category\n",
    "test_x = list(testing[:, 0])\n",
    "test_y = list(testing[:, 1])\n",
    "print(len(test_x))\n",
    "print(len(test_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6a7a5bca",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = model.predict(test_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bff66ab1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[164   2   1]\n",
      " [ 29 168  41]\n",
      " [  0  28 163]]\n",
      "495  out of  596 correct.\n",
      "83 % accurate.\n"
     ]
    }
   ],
   "source": [
    "confusion = np.array([[0,0,0], [0,0,0], [0,0,0]])\n",
    "for index in range(0,len(test_y)):\n",
    "    answer = 0\n",
    "    if test_y[index][1] == 1:\n",
    "        answer = 1\n",
    "    elif test_y[index][2] == 1:\n",
    "        answer = 2\n",
    "    confusion[np.argmax(results[index]), answer] += 1\n",
    "    \n",
    "print(confusion)\n",
    "numCorrect = confusion[0,0] + confusion[1,1] + confusion[2,2]\n",
    "print(numCorrect, \" out of \", len(test_y), \"correct.\")\n",
    "print(int(numCorrect/len(test_y)*100), \"% accurate.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a2f38c9",
   "metadata": {},
   "source": [
    "## Comments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eb6a4c2",
   "metadata": {},
   "source": [
    "When I wrote this code, the project it was for had very obvious categories, so this had a higher accuracy than it does with this more complex dataset. It is also important to note that I didn't use the whole dataset.\n",
    "\n",
    "If I was going to go back and improve this, I would include a validation dataset and checkpoints. I just watched it until it seemed to be high accuracy and stopped there for this notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "deepLearning",
   "language": "python",
   "name": "deeplearning"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}