creating_dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objective\n",
    "* __Create our own dataset that contains a random sample of reviews__\n",
    "\n",
    "## Workflow\n",
    "\n",
    "1. Read the video game review data. Take a look at the text of the reviews and the ratings. Note that our data is not pure JSON, but new line delimited JSON. To be able to read it, install and import  <span style=\"color:red\">ndjson</span>.\n",
    "2. Create a plot of the ratings of the product. Study the distribution of the five categories.\n",
    "3. Take a random sample of the reviews by selecting 1500 reviews with rating 1, 500-500-500 reviews with ratings 2, 3, 4, and 1500 reviews with rating 5. This way we get a smaller __balanced__ corpus. \n",
    "4. Take a random sample of the reviews by selecting 100,000 reviews. This way we get a bigger representative corpus.\n",
    "5. Export our corpora to two separate .csv files. Both of our tables should contain a column for the reviews and a column for the ratings. From now on we call the review text of the JSON key “reviews” and the overall key “ratings.” Name our corpora small_corpus and big_corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ndjson\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reading reviews from json file\n",
    "with open('../data/Video_Games_5.json') as f:\n",
    "    data = ndjson.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews_df = pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>overall</th>\n",
       "      <th>verified</th>\n",
       "      <th>reviewTime</th>\n",
       "      <th>reviewerID</th>\n",
       "      <th>asin</th>\n",
       "      <th>reviewerName</th>\n",
       "      <th>reviewText</th>\n",
       "      <th>summary</th>\n",
       "      <th>unixReviewTime</th>\n",
       "      <th>vote</th>\n",
       "      <th>style</th>\n",
       "      <th>image</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.0</td>\n",
       "      <td>True</td>\n",
       "      <td>10 17, 2015</td>\n",
       "      <td>A1HP7NVNPFMA4N</td>\n",
       "      <td>0700026657</td>\n",
       "      <td>Ambrosia075</td>\n",
       "      <td>This game is a bit hard to get the hang of, bu...</td>\n",
       "      <td>but when you do it's great.</td>\n",
       "      <td>1445040000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.0</td>\n",
       "      <td>False</td>\n",
       "      <td>07 27, 2015</td>\n",
       "      <td>A1JGAP0185YJI6</td>\n",
       "      <td>0700026657</td>\n",
       "      <td>travis</td>\n",
       "      <td>I played it a while but it was alright. The st...</td>\n",
       "      <td>But in spite of that it was fun, I liked it</td>\n",
       "      <td>1437955200</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3.0</td>\n",
       "      <td>True</td>\n",
       "      <td>02 23, 2015</td>\n",
       "      <td>A1YJWEXHQBWK2B</td>\n",
       "      <td>0700026657</td>\n",
       "      <td>Vincent G. Mezera</td>\n",
       "      <td>ok game.</td>\n",
       "      <td>Three Stars</td>\n",
       "      <td>1424649600</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.0</td>\n",
       "      <td>True</td>\n",
       "      <td>02 20, 2015</td>\n",
       "      <td>A2204E1TH211HT</td>\n",
       "      <td>0700026657</td>\n",
       "      <td>Grandma KR</td>\n",
       "      <td>found the game a bit too complicated, not what...</td>\n",
       "      <td>Two Stars</td>\n",
       "      <td>1424390400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>True</td>\n",
       "      <td>12 25, 2014</td>\n",
       "      <td>A2RF5B5H74JLPE</td>\n",
       "      <td>0700026657</td>\n",
       "      <td>jon</td>\n",
       "      <td>great game, I love it and have played it since...</td>\n",
       "      <td>love this game</td>\n",
       "      <td>1419465600</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   overall  verified   reviewTime      reviewerID        asin  \\\n",
       "0      5.0      True  10 17, 2015  A1HP7NVNPFMA4N  0700026657   \n",
       "1      4.0     False  07 27, 2015  A1JGAP0185YJI6  0700026657   \n",
       "2      3.0      True  02 23, 2015  A1YJWEXHQBWK2B  0700026657   \n",
       "3      2.0      True  02 20, 2015  A2204E1TH211HT  0700026657   \n",
       "4      5.0      True  12 25, 2014  A2RF5B5H74JLPE  0700026657   \n",
       "\n",
       "        reviewerName                                         reviewText  \\\n",
       "0        Ambrosia075  This game is a bit hard to get the hang of, bu...   \n",
       "1             travis  I played it a while but it was alright. The st...   \n",
       "2  Vincent G. Mezera                                           ok game.   \n",
       "3         Grandma KR  found the game a bit too complicated, not what...   \n",
       "4                jon  great game, I love it and have played it since...   \n",
       "\n",
       "                                       summary  unixReviewTime vote style  \\\n",
       "0                  but when you do it's great.      1445040000  NaN   NaN   \n",
       "1  But in spite of that it was fun, I liked it      1437955200  NaN   NaN   \n",
       "2                                  Three Stars      1424649600  NaN   NaN   \n",
       "3                                    Two Stars      1424390400  NaN   NaN   \n",
       "4                               love this game      1419465600  NaN   NaN   \n",
       "\n",
       "  image  \n",
       "0   NaN  \n",
       "1   NaN  \n",
       "2   NaN  \n",
       "3   NaN  \n",
       "4   NaN  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Dictionry\n",
    "   * __reviewerID__ - ID of the reviewer, e.g. A2SUAM1J3GNN3B\n",
    "   * __asin__ - ID of the product, e.g. 0000013714\n",
    "   * **reviewerName** - name of the reviewer\n",
    "   * **vote** - helpful votes of the review\n",
    "   * **style** - a disctionary of the product metadata, e.g., \"Format\" is \"Hardcover\"\n",
    "   * **reviewText** - text of the review\n",
    "   * **overall** - rating of the product\n",
    "   * **summary** - summary of the review\n",
    "   * **unixReviewTime** - time of the review (unix time)\n",
    "   * **reviewTime** - time of the review (raw)\n",
    "   * **image** - images that users post after they have received the product\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(497577, 12)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 497577 entries, 0 to 497576\n",
      "Data columns (total 12 columns):\n",
      " #   Column          Non-Null Count   Dtype  \n",
      "---  ------          --------------   -----  \n",
      " 0   overall         497577 non-null  float64\n",
      " 1   verified        497577 non-null  bool   \n",
      " 2   reviewTime      497577 non-null  object \n",
      " 3   reviewerID      497577 non-null  object \n",
      " 4   asin            497577 non-null  object \n",
      " 5   reviewerName    497501 non-null  object \n",
      " 6   reviewText      497419 non-null  object \n",
      " 7   summary         497468 non-null  object \n",
      " 8   unixReviewTime  497577 non-null  int64  \n",
      " 9   vote            107793 non-null  object \n",
      " 10  style           289237 non-null  object \n",
      " 11  image           3634 non-null    object \n",
      "dtypes: bool(1), float64(1), int64(1), object(9)\n",
      "memory usage: 42.2+ MB\n"
     ]
    }
   ],
   "source": [
    "reviews_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='overall', ylabel='count'>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEGCAYAAACpXNjrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWxUlEQVR4nO3df/BddX3n8efLAGqrCEqkmIQNUzO7G90aNQvZxW0tTCGw2wY76MCMkrWs6Y6ho7O2K7ozi6LM6uwqLRbZpSUlca1AUZbUiY0ZZHXrlh9BEAiU4buIkgyaSCL4Y8QJvveP+0lz+XK/X77gufd+8/0+HzNn7jnv8znnfO4ZvnlxftxzUlVIktSlF4y7A5KkucdwkSR1znCRJHXOcJEkdc5wkSR17rBxd2C2OOaYY2rp0qXj7oYkHVLuuOOO71fVwsl1w6VZunQp27dvH3c3JOmQkuTbg+qeFpMkdc5wkSR1znCRJHXOcJEkdc5wkSR1bmjhkuRFSW5L8s0kO5J8uNVPSHJrkokk1yY5otVf2KYn2vylfev6QKs/kOT0vvrqVptIcmFffeA2JEmjMcwjlyeBU6rqdcAKYHWSVcDHgUur6tXAPuD81v58YF+rX9rakWQ5cA7wGmA18OkkC5IsAC4HzgCWA+e2tkyzDUnSCAwtXKrnR23y8DYUcApwfatvBM5q42vaNG3+qUnS6tdU1ZNV9S1gAjixDRNV9VBV/Qy4BljTlplqG5KkERjqNZd2hHEXsBvYBvw/4AdVtb812QksauOLgEcA2vzHgVf01yctM1X9FdNsQ5I0AkP9hX5VPQWsSHIUcAPwT4a5vecqyTpgHcDxxx8/5t5IOtT96fv+etxdGIoLPvHbz3mZkdwtVlU/AG4G/gVwVJIDobYY2NXGdwFLANr8lwGP9dcnLTNV/bFptjG5X1dW1cqqWrlw4TMejSNJep6GebfYwnbEQpIXA78F3E8vZM5uzdYCN7bxzW2aNv8r1XsH82bgnHY32QnAMuA24HZgWbsz7Ah6F/03t2Wm2oYkaQSGeVrsOGBju6vrBcB1VfXFJPcB1yT5KHAncFVrfxXwmSQTwF56YUFV7UhyHXAfsB9Y3063keQCYCuwANhQVTvaut4/xTYkSSMwtHCpqruB1w+oP0TvTq/J9Z8Cb51iXZcAlwyobwG2zHQbkqTR8Bf6kqTOGS6SpM4ZLpKkzhkukqTOGS6SpM4ZLpKkzhkukqTOGS6SpM4ZLpKkzhkukqTOGS6SpM4ZLpKkzhkukqTOGS6SpM4ZLpKkzhkukqTOGS6SpM4ZLpKkzhkukqTOGS6SpM4ZLpKkzhkukqTOGS6SpM4ZLpKkzhkukqTOGS6SpM4NLVySLElyc5L7kuxI8p5W/1CSXUnuasOZfct8IMlEkgeSnN5XX91qE0ku7KufkOTWVr82yRGt/sI2PdHmLx3W95QkPdMwj1z2A++rquXAKmB9kuVt3qVVtaINWwDavHOA1wCrgU8nWZBkAXA5cAawHDi3bz0fb+t6NbAPOL/Vzwf2tfqlrZ0kaUSGFi5V9WhVfaON/xC4H1g0zSJrgGuq6smq+hYwAZzYhomqeqiqfgZcA6xJEuAU4Pq2/EbgrL51bWzj1wOntvaSpBEYyTWXdlrq9cCtrXRBkruTbEhydKstAh7pW2xnq01VfwXwg6raP6n+tHW1+Y+39pP7tS7J9iTb9+zZ84t9SUnSPxh6uCR5CfB54L1V9QRwBfCrwArgUeATw+7DVKrqyqpaWVUrFy5cOK5uSNKcM9RwSXI4vWD5bFV9AaCqvldVT1XVz4E/o3faC2AXsKRv8cWtNlX9MeCoJIdNqj9tXW3+y1p7SdIIDPNusQBXAfdX1Sf76sf1NXsLcG8b3wyc0+70OgFYBtwG3A4sa3eGHUHvov/mqirgZuDstvxa4Ma+da1t42cDX2ntJUkjcNizN3neTgbeAdyT5K5W+yC9u71WAAU8DPw+QFXtSHIdcB+9O83WV9VTAEkuALYCC4ANVbWjre/9wDVJPgrcSS/MaJ+fSTIB7KUXSJKkERlauFTV3wKD7tDaMs0ylwCXDKhvGbRcVT3EwdNq/fWfAm99Lv2VJHXHX+hLkjpnuEiSOme4SJI6Z7hIkjpnuEiSOme4SJI6Z7hIkjpnuEiSOme4SJI6Z7hIkjpnuEiSOme4SJI6Z7hIkjpnuEiSOme4SJI6Z7hIkjpnuEiSOme4SJI6Z7hIkjpnuEiSOme4SJI6Z7hIkjpnuEiSOme4SJI6Z7hIkjo3tHBJsiTJzUnuS7IjyXta/eVJtiV5sH0e3epJclmSiSR3J3lD37rWtvYPJlnbV39jknvaMpclyXTbkCSNxjCPXPYD76uq5cAqYH2S5cCFwE1VtQy4qU0DnAEsa8M64AroBQVwEXAScCJwUV9YXAG8q2+51a0+1TYkSSMwtHCpqker6htt/IfA/cAiYA2wsTXbCJzVxtcAm6rnFuCoJMcBpwPbqmpvVe0DtgGr27wjq+qWqipg06R1DdqGJGkERnLNJclS4PXArcCxVfVom/Vd4Ng2vgh4pG+xna02XX3ngDrTbEOSNAJDD5ckLwE+D7y3qp7on9eOOGqY259uG0nWJdmeZPuePXuG2Q1JmleGGi5JDqcXLJ+tqi+08vfaKS3a5+5W3wUs6Vt8catNV188oD7dNp6mqq6sqpVVtXLhwoXP70tKkp5hmHeLBbgKuL+qPtk3azNw4I6vtcCNffXz2l1jq4DH26mtrcBpSY5uF/JPA7a2eU8kWdW2dd6kdQ3ahiRpBA4b4rpPBt4B3JPkrlb7IPAx4Lok5wPfBt7W5m0BzgQmgJ8A7wSoqr1JPgLc3tpdXFV72/i7gauBFwNfagPTbEOSNAJDC5eq+lsgU8w+dUD7AtZPsa4NwIYB9e3AawfUHxu0DUnSaPgLfUlS5wwXSVLnDBdJUucMF0lS5wwXSVLnDBdJUucMF0lS5wwXSVLnDBdJUucMF0lS5wwXSVLnDBdJUucMF0lS52YULklumklNkiR4lkfuJ3kR8EvAMe1FXQceoX8kB99XL0nS0zzb+1x+H3gv8CrgDg6GyxPAnw6vW5KkQ9m04VJVfwL8SZI/qKpPjahPkqRD3IzeRFlVn0ryL4Gl/ctU1aYh9UuSdAibUbgk+Qzwq8BdwFOtXIDhIkl6hhmFC7ASWN7ecy9J0rRm+juXe4FfGWZHJElzx0yPXI4B7ktyG/DkgWJV/c5QeiVJOqTNNFw+NMxOSJLmlpneLfbVYXdEkjR3zPRusR/SuzsM4AjgcODHVXXksDomSTp0zfTI5aUHxpMEWAOsGlanJEmHtuf8VOTq+V/A6dO1S7Ihye4k9/bVPpRkV5K72nBm37wPJJlI8kCS0/vqq1ttIsmFffUTktza6tcmOaLVX9imJ9r8pc/1O0qSfjEzfSry7/YNZyf5GPDTZ1nsamD1gPqlVbWiDVva+pcD5wCvact8OsmCJAuAy4EzgOXAua0twMfbul4N7APOb/XzgX2tfmlrJ0kaoZkeufx233A68EN6p8amVFVfA/bOcP1rgGuq6smq+hYwAZzYhomqeqiqfgZcA6xpp+ZOAa5vy28Ezupb18Y2fj1wamsvSRqRmV5zeWeH27wgyXnAduB9VbWP3uP7b+lrs5ODj/R/ZFL9JOAVwA+qav+A9osOLFNV+5M83tp/f3JHkqwD1gEcf/zxv/g3kyQBMz8ttjjJDe0ayu4kn0+y+Hls7wp6zyhbATwKfOJ5rKMzVXVlVa2sqpULFy4cZ1ckaU6Z6WmxvwA203uvy6uAv26156SqvldVT1XVz4E/o3faC2AXsKSv6eJWm6r+GHBUksMm1Z+2rjb/Za29JGlEZhouC6vqL6pqfxuuBp7z/+onOa5v8i30nlkGveA6p93pdQKwDLgNuB1Y1u4MO4LeRf/N7QGaNwNnt+XXAjf2rWttGz8b+IoP3JSk0Zrp418eS/J24HNt+lye5WggyeeAN9N7RfJO4CLgzUlW0PtB5sP03nRJVe1Ich1wH7AfWF9VT7X1XABsBRYAG6pqR9vE+4FrknwUuBO4qtWvAj6TZILeDQXnzPA7SpI6MtNw+T3gU/Ru7S3g/wL/droFqurcAeWrBtQOtL8EuGRAfQuwZUD9IQ6eVuuv/xR463R9kyQN10zD5WJgbbuziyQvB/4bvdCRJOlpZnrN5dcOBAtAVe0FXj+cLkmSDnUzDZcXJDn6wEQ7cpnpUY8kaZ6ZaUB8Avi7JH/Vpt/KgOsjkiTBzH+hvynJdnqPXAH43aq6b3jdkiQdymZ8aquFiYEiSXpWz/mR+5IkPRvDRZLUOcNFktQ5w0WS1DnDRZLUOcNFktQ5w0WS1DnDRZLUOcNFktQ5w0WS1DnDRZLUOcNFktQ5w0WS1DnDRZLUOcNFktQ5w0WS1DnDRZLUOcNFktQ5w0WS1DnDRZLUuaGFS5INSXYnubev9vIk25I82D6PbvUkuSzJRJK7k7yhb5m1rf2DSdb21d+Y5J62zGVJMt02JEmjM8wjl6uB1ZNqFwI3VdUy4KY2DXAGsKwN64AroBcUwEXAScCJwEV9YXEF8K6+5VY/yzYkSSMytHCpqq8BeyeV1wAb2/hG4Ky++qbquQU4KslxwOnAtqraW1X7gG3A6jbvyKq6paoK2DRpXYO2IUkakVFfczm2qh5t498Fjm3ji4BH+trtbLXp6jsH1KfbxjMkWZdke5Lte/bseR5fR5I0yNgu6LcjjhrnNqrqyqpaWVUrFy5cOMyuSNK8Mupw+V47pUX73N3qu4Alfe0Wt9p09cUD6tNtQ5I0IqMOl83AgTu+1gI39tXPa3eNrQIeb6e2tgKnJTm6Xcg/Ddja5j2RZFW7S+y8SesatA1J0ogcNqwVJ/kc8GbgmCQ76d319THguiTnA98G3taabwHOBCaAnwDvBKiqvUk+Atze2l1cVQduEng3vTvSXgx8qQ1Msw1JQ/LVX/+NcXehc7/xta+OuwuHtKGFS1WdO8WsUwe0LWD9FOvZAGwYUN8OvHZA/bFB25AkjY6/0Jckdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdW4s4ZLk4ST3JLkryfZWe3mSbUkebJ9Ht3qSXJZkIsndSd7Qt561rf2DSdb21d/Y1j/Rls3ov6UkzV/jPHL5zapaUVUr2/SFwE1VtQy4qU0DnAEsa8M64ArohRFwEXAScCJw0YFAam3e1bfc6uF/HUnSAbPptNgaYGMb3wic1VffVD23AEclOQ44HdhWVXurah+wDVjd5h1ZVbdUVQGb+tYlSRqBcYVLAV9OckeSda12bFU92sa/CxzbxhcBj/Qtu7PVpqvvHFB/hiTrkmxPsn3Pnj2/yPeRJPU5bEzbfVNV7UrySmBbkr/vn1lVlaSG3YmquhK4EmDlypVD357mlpM/dfK4u9C5r//B18fdBc0RYzlyqapd7XM3cAO9aybfa6e0aJ+7W/NdwJK+xRe32nT1xQPqkqQRGXm4JPnlJC89MA6cBtwLbAYO3PG1FrixjW8Gzmt3ja0CHm+nz7YCpyU5ul3IPw3Y2uY9kWRVu0vsvL51SZJGYBynxY4Fbmh3Bx8G/GVV/U2S24HrkpwPfBt4W2u/BTgTmAB+ArwToKr2JvkIcHtrd3FV7W3j7wauBl4MfKkNkqQRGXm4VNVDwOsG1B8DTh1QL2D9FOvaAGwYUN8OvPYX7qwk6XmZTbciS5LmCMNFktQ5w0WS1DnDRZLUOcNFktQ5w0WS1DnDRZLUuXE9W+yQ8sY/2jTuLnTujv963ri7IGkO88hFktQ5w0WS1DnDRZLUOcNFktQ5L+jrOfnOxf9s3F3o3PH/+Z5xd0GaczxykSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJEkdc5wkSR1bs6GS5LVSR5IMpHkwnH3R5LmkzkZLkkWAJcDZwDLgXOTLB9vryRp/piT4QKcCExU1UNV9TPgGmDNmPskSfNGqmrcfehckrOB1VX179r0O4CTquqCSe3WAeva5D8GHhhpR5/pGOD7Y+7DbOG+OMh9cZD74qDZsi/+UVUtnFyc12+irKorgSvH3Y8DkmyvqpXj7sds4L44yH1xkPvioNm+L+bqabFdwJK+6cWtJkkagbkaLrcDy5KckOQI4Bxg85j7JEnzxpw8LVZV+5NcAGwFFgAbqmrHmLs1E7PmFN0s4L44yH1xkPvioFm9L+bkBX1J0njN1dNikqQxMlwkSZ0zXMYgyYYku5PcO8X8JLmsPbrm7iRvGHUfRyHJkiQ3J7kvyY4k7xnQZr7sixcluS3JN9u++PCANi9Mcm3bF7cmWTqGro5MkgVJ7kzyxQHz5s2+SPJwknuS3JVk+4D5s/JvxHAZj6uB1dPMPwNY1oZ1wBUj6NM47AfeV1XLgVXA+gGP6Zkv++JJ4JSqeh2wAlidZNWkNucD+6rq1cClwMdH28WRew9w/xTz5tu++M2qWjHF71pm5d+I4TIGVfU1YO80TdYAm6rnFuCoJMeNpnejU1WPVtU32vgP6f1DsmhSs/myL6qqftQmD2/D5Ltt1gAb2/j1wKlJMqIujlSSxcC/Bv58iibzZl/MwKz8GzFcZqdFwCN90zt55j+6c0o7rfF64NZJs+bNvminge4CdgPbqmrKfVFV+4HHgVeMtJOj88fAfwR+PsX8+bQvCvhykjvaI6smm5V/I4aLxi7JS4DPA++tqifG3Z9xqaqnqmoFvSdKnJjktWPu0lgk+TfA7qq6Y9x9mSXeVFVvoHf6a32SXx93h2bCcJmd5s3ja5IcTi9YPltVXxjQZN7siwOq6gfAzTzzutw/7IskhwEvAx4baedG42Tgd5I8TO+J5qck+Z+T2syXfUFV7Wqfu4Eb6D31vd+s/BsxXGanzcB57S6QVcDjVfXouDvVtXaO/Crg/qr65BTN5su+WJjkqDb+YuC3gL+f1GwzsLaNnw18pebgr6Cr6gNVtbiqltJ7dNNXqurtk5rNi32R5JeTvPTAOHAaMPku01n5NzInH/8y2yX5HPBm4JgkO4GL6F3Apar+O7AFOBOYAH4CvHM8PR26k4F3APe0aw0AHwSOh3m3L44DNrYX3b0AuK6qvpjkYmB7VW2mF8SfSTJB74aQc8bX3dGbp/viWOCGdq/CYcBfVtXfJPn3MLv/Rnz8iySpc54WkyR1znCRJHXOcJEkdc5wkSR1znCRJHXOcJHmmCT/O8nKNv5wkmPG3SfNP4aLdIhpP5bzb1ezmv+BSiOQ5D8kubcN703ysSTr++Z/KMkftvE/SnJ7ezfHh1ttaZIHkmyi9wvtJUmuSLJ9qve/SOPkL/SlIUvyRnq/mj4JCL0nP7+d3pN/L2/N3gacnuQ0eu/lOLG13dweVPidVl/bHqtOkv9UVXvbr/pvSvJrVXX36L6ZNDXDRRq+NwE3VNWPAZJ8AfhXwCuTvApYSO/FV4+k9zbO04A727IvoRcq3wG+fSBYmre1R7AfRu/xMcsBw0WzguEijc9f0Xvo4q8A17ZagP9SVf+jv2F7382P+6ZPAP4Q+OdVtS/J1cCLRtBnaUa85iIN3/8BzkryS+3Jtm9ptWvpPXDxbHpBA7AV+L32jhuSLEryygHrPJJe2Dye5Fh67/qQZg2PXKQhq6pvtCOL21rpz6vqToD2OPVdBx6RXlVfTvJPgb9rT8L9Eb3rM09NWuc3k9xJ77H8jwBfH8V3kWbKpyJLkjrnaTFJUucMF0lS5wwXSVLnDBdJUucMF0lS5wwXSVLnDBdJUuf+P6HKMtPCrn4hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(data = reviews_df, x='overall')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17408"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(reviews_df['asin'].value_counts(dropna=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Undersampling of Reviews\n",
    "Taking a random sample of the reviews by selecting 1500 reviews with rating 1, 500-500-500 reviews with ratings 2, 3, 4, and 1500 reviews with rating 5. This way you get a smaller balanced corpus, on which you will during Milestones 2-4.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_1500 = reviews_df[reviews_df['overall']==1.0].sample(n=1500)\n",
    "two_500 = reviews_df[reviews_df['overall']==2.0].sample(n=500)\n",
    "three_500 = reviews_df[reviews_df['overall']==3.0].sample(n=500)\n",
    "four_500 = reviews_df[reviews_df['overall']==4.0].sample(n=500)\n",
    "five_1500 = reviews_df[reviews_df['overall']==5.0].sample(n=1500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "undersampled_reviews = pd.concat([one_1500, two_500, three_500, four_500, five_1500], axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0    1500\n",
       "1.0    1500\n",
       "4.0     500\n",
       "3.0     500\n",
       "2.0     500\n",
       "Name: overall, dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "undersampled_reviews['overall'].value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='overall', ylabel='count'>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEGCAYAAACUzrmNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAUaklEQVR4nO3df7DddX3n8ecLoqi1GiBXxCRsmJpxy7pW8S6yS9da2YVgrWFbpDCrpEgn21l0dbG1aGeWlo6zOv1BxVp2sxIJXVdElCV1aDEDKLuOIOGH/NTlDgWSDJho+GFlrRv63j/OJ+b0cm++N3DPOTc5z8fMmfv9fj6fc877foebF99fn2+qCkmS9uagURcgSVr4DAtJUifDQpLUybCQJHUyLCRJnRaNuoBBWLJkSa1YsWLUZUjSfuW22277XlVNzNR3QIbFihUr2Lx586jLkKT9SpKHZ+vzMJQkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSp0wF5B/fevPG3Lx91CQNx2x+eNeoSpAPOn33wL0ddwkC8949/eZ/f456FJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqdPAwiLJ+iTbk9wzQ98Hk1SSJW09SS5OMpXkriTH9o1dk+SB9lozqHolSbMb5J7FZcCq6Y1JlgMnAY/0NZ8CrGyvtcAlbexhwAXAm4DjgAuSHDrAmiVJMxhYWFTVTcDOGbouAj4EVF/bauDy6rkZWJzkSOBkYFNV7ayqx4FNzBBAkqTBGuo5iySrgW1V9a1pXUuBLX3rW1vbbO2SpCEa2qyzSV4CfITeIahBfP5aeoewOOqoowbxFZI0toa5Z/EzwNHAt5I8BCwDbk/ySmAbsLxv7LLWNlv7s1TVuqqarKrJiYmJAZQvSeNraGFRVXdX1SuqakVVraB3SOnYqnoM2Aic1a6KOh54sqoeBa4DTkpyaDuxfVJrkyQN0SAvnf0c8A3gNUm2JjlnL8OvBR4EpoD/Bvx7gKraCfwBcGt7XdjaJElDNLBzFlV1Zkf/ir7lAs6dZdx6YP28FidJ2ifewS1J6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROAwuLJOuTbE9yT1/bHyb5dpK7klydZHFf34eTTCX5TpKT+9pXtbapJOcPql5J0uwGuWdxGbBqWtsm4LVV9Trg/wAfBkhyDHAG8E/ae/48ycFJDgY+BZwCHAOc2cZKkoZoYGFRVTcBO6e1faWqdrXVm4FlbXk1cEVV/V1V/Q0wBRzXXlNV9WBV/Ri4oo2VJA3RKM9ZvAf4q7a8FNjS17e1tc3W/ixJ1ibZnGTzjh07BlCuJI2vkYRFkt8FdgGfna/PrKp1VTVZVZMTExPz9bGSJGDRsL8wya8DbwdOrKpqzduA5X3DlrU29tIuSRqSoe5ZJFkFfAh4R1U93de1ETgjySFJjgZWAt8EbgVWJjk6yQvpnQTfOMyaJUkD3LNI8jngLcCSJFuBC+hd/XQIsCkJwM1V9ZtVdW+SK4H76B2eOreqnmmf817gOuBgYH1V3TuomiVJMxtYWFTVmTM0X7qX8R8FPjpD+7XAtfNYmiRpH3kHtySpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoNLCySrE+yPck9fW2HJdmU5IH289DWniQXJ5lKcleSY/ves6aNfyDJmkHVK0ma3SD3LC4DVk1rOx+4vqpWAte3dYBTgJXttRa4BHrhAlwAvAk4Drhgd8BIkoZnYGFRVTcBO6c1rwY2tOUNwKl97ZdXz83A4iRHAicDm6pqZ1U9Dmzi2QEkSRqwYZ+zOKKqHm3LjwFHtOWlwJa+cVtb22ztz5JkbZLNSTbv2LFjfquWpDE3shPcVVVAzePnrauqyaqanJiYmK+PlSQx/LD4bju8RPu5vbVvA5b3jVvW2mZrlyQN0bDDYiOw+4qmNcA1fe1ntauijgeebIerrgNOSnJoO7F9UmuTJA3RokF9cJLPAW8BliTZSu+qpo8BVyY5B3gYOL0NvxZ4GzAFPA2cDVBVO5P8AXBrG3dhVU0/aS5JGrCBhUVVnTlL14kzjC3g3Fk+Zz2wfh5LkyTtI+/gliR1MiwkSZ0MC0lSJ8NCktTJsJAkdTIsJEmdDAtJUifDQpLUybCQJHUyLCRJnQwLSVInw0KS1GlOYZHk+rm0SZIOTHuddTbJi4CX0Jtm/FAgretlzPJ4U0nSgadrivJ/B3wAeBVwG3vC4ingzwZXliRpIdlrWFTVJ4BPJHlfVX1ySDVJkhaYOT38qKo+meRfACv631NVlw+oLknSAjKnsEjyF8DPAHcCz7TmAgwLSRoDc32s6iRwTHv86fOW5D8Cv0EvcO6m98ztI4ErgMPpnR95d1X9OMkh9ELpjcD3gV+rqofmow5J0tzM9T6Le4BXzscXJlkK/AdgsqpeCxwMnAF8HLioql4NPA6c095yDvB4a7+ojZMkDdFcw2IJcF+S65Js3P16Ht+7CHhxkkX0Ls19FHgrcFXr3wCc2pZXt3Va/4lJgiRpaOZ6GOr35usLq2pbkj8CHgH+L/AVeoednqiqXW3YVvbcx7EU2NLeuyvJk/QOVX2v/3OTrAXWAhx11FHzVa4kiblfDfW1+frCdnPfauBo4AngC8Cq5/u5VbUOWAcwOTk5L+dWJEk9c53u4wdJnmqvHyV5JslTz/E7/xXwN1W1o6r+H/Al4ARgcTssBbAM2NaWtwHLWx2LgJfTO9EtSRqSOYVFVf10Vb2sql4GvBj4VeDPn+N3PgIcn+Ql7dzDicB9wI3AaW3MGuCatryxrdP6b5ivq7IkSXOzz7POVs//BE5+Ll9YVbfQO1F9O73LZg+id/jod4DzkkzROydxaXvLpcDhrf084Pzn8r2SpOdurjfl/Urf6kH07rv40XP90qq6ALhgWvODwHEzjP0R8M7n+l2SpOdvrldD/XLf8i7gIXonqSVJY2CuV0OdPehCJEkL11yvhlqW5Ook29vri0mWDbo4SdLCMNcT3J+hd1XSq9rrL1ubJGkMzDUsJqrqM1W1q70uAyYGWJckaQGZa1h8P8m7khzcXu/CG+MkaWzMNSzeA5wOPEZv0r/TgF8fUE2SpAVmrpfOXgisqarHAZIcBvwRvRCRJB3g5rpn8brdQQFQVTuBNwymJEnSQjPXsDiozRYL/GTPYq57JZKk/dxc/8H/Y+AbSb7Q1t8JfHQwJUmSFpq53sF9eZLN9J5mB/ArVXXf4MqSJC0kcz6U1MLBgJCkMbTPU5RLksaPYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSeo0krBIsjjJVUm+neT+JP88yWFJNiV5oP08tI1NkouTTCW5K8mxo6hZksbZqPYsPgH8dVX9Y+DngPuB84Hrq2olcH1bBzgFWNlea4FLhl+uJI23oYdFkpcDbwYuBaiqH1fVE8BqYEMbtgE4tS2vBi6vnpuBxUmOHGrRkjTmRrFncTSwA/hMkjuSfDrJTwFHVNWjbcxjwBFteSmwpe/9W1vbP5BkbZLNSTbv2LFjgOVL0vgZRVgsAo4FLqmqNwA/ZM8hJwCqqoDalw+tqnVVNVlVkxMTPvFVkubTKMJiK7C1qm5p61fRC4/v7j681H5ub/3bgOV971/W2iRJQzL0sKiqx4AtSV7Tmk6kN0HhRmBNa1sDXNOWNwJntauijgee7DtcJUkaglE9wOh9wGeTvBB4EDibXnBdmeQc4GF6z/wGuBZ4GzAFPN3GSpKGaCRhUVV3ApMzdJ04w9gCzh10TZKk2XkHtySpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkTqO6g1sLwCMX/tNRlzAQR/2nu/f5PSd88oQBVDJ6X3/f1/f5PV978y8MoJLR+4WbvjbqEvZr7llIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOo0sLJIcnOSOJF9u60cnuSXJVJLPt+dzk+SQtj7V+leMqmZJGlej3LN4P3B/3/rHgYuq6tXA48A5rf0c4PHWflEbJ0kaopGERZJlwC8Bn27rAd4KXNWGbABObcur2zqt/8Q2XpI0JKPas/hT4EPA37f1w4EnqmpXW98KLG3LS4EtAK3/yTZekjQkQw+LJG8HtlfVbfP8uWuTbE6yeceOHfP50ZI09kaxZ3EC8I4kDwFX0Dv89AlgcZLdU6YvA7a15W3AcoDW/3Lg+9M/tKrWVdVkVU1OTEwM9jeQpDEz9LCoqg9X1bKqWgGcAdxQVf8WuBE4rQ1bA1zTlje2dVr/DVVVQyxZksbeQrrP4neA85JM0TsncWlrvxQ4vLWfB5w/ovokaWyN9El5VfVV4Ktt+UHguBnG/Ah451ALkyT9Awtpz0KStEAZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSp09DDIsnyJDcmuS/JvUne39oPS7IpyQPt56GtPUkuTjKV5K4kxw67Zkkad6PYs9gFfLCqjgGOB85NcgxwPnB9Va0Erm/rAKcAK9trLXDJ8EuWpPE29LCoqker6va2/APgfmApsBrY0IZtAE5ty6uBy6vnZmBxkiOHW7UkjbeRnrNIsgJ4A3ALcERVPdq6HgOOaMtLgS19b9va2qZ/1tokm5Ns3rFjx+CKlqQxNLKwSPJS4IvAB6rqqf6+qiqg9uXzqmpdVU1W1eTExMQ8VipJGklYJHkBvaD4bFV9qTV/d/fhpfZze2vfBizve/uy1iZJGpJRXA0V4FLg/qr6k76ujcCatrwGuKav/ax2VdTxwJN9h6skSUOwaATfeQLwbuDuJHe2to8AHwOuTHIO8DBweuu7FngbMAU8DZw91GolScMPi6r630Bm6T5xhvEFnDvQoiRJe+Ud3JKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSepkWEiSOhkWkqROhoUkqZNhIUnqZFhIkjoZFpKkToaFJKmTYSFJ6mRYSJI6GRaSpE6GhSSpk2EhSeq034RFklVJvpNkKsn5o65HksbJfhEWSQ4GPgWcAhwDnJnkmNFWJUnjY78IC+A4YKqqHqyqHwNXAKtHXJMkjY1U1ahr6JTkNGBVVf1GW3838Kaqem/fmLXA2rb6GuA7Qy/02ZYA3xt1EQuE22IPt8Uebos9FsK2+EdVNTFTx6JhVzIoVbUOWDfqOvol2VxVk6OuYyFwW+zhttjDbbHHQt8W+8thqG3A8r71Za1NkjQE+0tY3AqsTHJ0khcCZwAbR1yTJI2N/eIwVFXtSvJe4DrgYGB9Vd074rLmYkEdFhsxt8Uebos93BZ7LOhtsV+c4JYkjdb+chhKkjRChoUkqZNh8TwlWZ9ke5J7ZulPkovbNCV3JTl22DUOS5LlSW5Mcl+Se5O8f4YxY7E9krwoyTeTfKtti9+fYcwhST7ftsUtSVaMoNShSHJwkjuSfHmGvrHZDgBJHkpyd5I7k2yeoX9B/o0YFs/fZcCqvfSfAqxsr7XAJUOoaVR2AR+sqmOA44FzZ5iWZVy2x98Bb62qnwNeD6xKcvy0MecAj1fVq4GLgI8Pt8Shej9w/yx947QddvvFqnr9LPdVLMi/EcPieaqqm4CdexmyGri8em4GFic5cjjVDVdVPVpVt7flH9D7x2HptGFjsT3a7/e3bfUF7TX9apLVwIa2fBVwYpIMqcShSbIM+CXg07MMGYvtsA8W5N+IYTF4S4EtfetbefY/oAecdijhDcAt07rGZnu0Qy93AtuBTVU167aoql3Ak8DhQy1yOP4U+BDw97P0j8t22K2AryS5rU1TNN2C/BsxLDTvkrwU+CLwgap6atT1jEpVPVNVr6c348BxSV474pKGLsnbge1Vdduoa1lAfr6qjqV3uOncJG8edUFzYVgM3lhNVZLkBfSC4rNV9aUZhozV9gCoqieAG3n2ua2fbIski4CXA98fanGDdwLwjiQP0Zst+q1J/vu0MeOwHX6iqra1n9uBq+nNqt1vQf6NGBaDtxE4q13hcDzwZFU9OuqiBqEdZ74UuL+q/mSWYWOxPZJMJFncll8M/Gvg29OGbQTWtOXTgBvqALtLtqo+XFXLqmoFvWl6bqiqd00bdsBvh92S/FSSn969DJwETL+SckH+jewX030sZEk+B7wFWJJkK3ABvZOZVNV/Aa4F3gZMAU8DZ4+m0qE4AXg3cHc7Vg/wEeAoGLvtcSSwoT246yDgyqr6cpILgc1VtZFesP5Fkil6F0mcMbpyh2uMt8MRwNXt/P0i4H9U1V8n+U1Y2H8jTvchSerkYShJUifDQpLUybCQJHUyLCRJnQwLSVInw0JawJJ8NclkW34oyZJR16TxZFhII9RuvPLvUAue/5FK+yjJeUnuaa8PJPlYknP7+n8vyW+15d9Ocmt7LsHvt7YVSb6T5HJ6d+8uT3JJks2zPftCGjXv4Jb2QZI30ruj9k1A6M2q+y56M6t+qg07HTg5yUn0nklwXBu7sU0a90hrX9OmoCbJ71bVznbH9/VJXldVdw3vN5P2zrCQ9s3PA1dX1Q8BknwJ+JfAK5K8Cpig9yCfLek9KfAk4I723pfSC4lHgId3B0VzepuuehG9qUKOAQwLLRiGhTQ/vkBvErxXAp9vbQH+c1X91/6B7VkfP+xbPxr4LeCfVdXjSS4DXjSEmqU585yFtG/+F3Bqkpe0WUP/TWv7PL0J8E6jFxwA1wHvac/3IMnSJK+Y4TNfRi88nkxyBL3nHEgLinsW0j6oqtvb//l/szV9uqruAGhTT2/bPZ10VX0lyc8C32izjP4tvfMbz0z7zG8luYPeFOZbgK8P43eR9oWzzkqSOnkYSpLUybCQJHUyLCRJnQwLSVInw0KS1MmwkCR1MiwkSZ3+Pz5ExirPkY9zAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(data=undersampled_reviews, x='overall')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Sampling of 100K Reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_100K_revs = reviews_df.sample(n=100000, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing Corpora"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "undersampled_reviews.to_csv(\"../data/small_corpus.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_100K_revs.to_csv(\"../data/big_corpus.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}