hw_sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework and bake-off: Sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2021\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Methodological note](#Methodological-note)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [Train set](#Train-set)\n",
    "1. [Dev sets](#Dev-sets)\n",
    "1. [A softmax baseline](#A-softmax-baseline)\n",
    "1. [RNNClassifier wrapper](#RNNClassifier-wrapper)\n",
    "1. [Error analysis](#Error-analysis)\n",
    "1. [Homework questions](#Homework-questions)\n",
    "  1. [Token-level differences [1 point]](#Token-level-differences-[1-point])\n",
    "  1. [Training on some of the bakeoff data [1 point]](#Training-on-some-of-the-bakeoff-data-[1-point])\n",
    "  1. [A more powerful vector-averaging baseline [2 points]](#A-more-powerful-vector-averaging-baseline-[2-points])\n",
    "  1. [BERT encoding [2 points]](#BERT-encoding-[2-points])\n",
    "  1. [Your original system [3 points]](#Your-original-system-[3-points])\n",
    "1. [Bakeoff [1 point]](#Bakeoff-[1-point])\n",
    "1. [Submission Instruction](#Submission-Instruction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This homework and associated bakeoff are devoted to supervised sentiment analysis using the ternary (positive/negative/neutral) version of the Stanford Sentiment Treebank (SST-3) as well as a new dev/test dataset drawn from restaurant reviews. Our goal in introducing the new dataset is to push you to create a system that performs well in both the movie and restaurant domains.\n",
    "\n",
    "The homework questions ask you to implement some baseline system, and the bakeoff challenge is to define a system that does well at both the SST-3 test set and the new restaurant test set. Both are ternary tasks, and our central bakeoff score is the mean of the macro-FI scores for the two datasets. This assigns equal weight to all classes and datasets regardless of size.\n",
    "\n",
    "The SST-3 test set will be used for the bakeoff evaluation. This dataset is already publicly distributed, so we are counting on people not to cheat by developing their models on the test set. You must do all your development without using the test set at all, and then evaluate exactly once on the test set and turn in the results, with no further system tuning or additional runs. __Much of the scientific integrity of our field depends on people adhering to this honor code__. \n",
    "\n",
    "One of our goals for this homework and bakeoff is to encourage you to engage in __the basic development cycle for supervised models__, in which you\n",
    "\n",
    "1. Design a new system. We recommend starting with something simple.\n",
    "1. Use `sst.experiment` to evaluate your system, using random train/test splits initially.\n",
    "1. If you have time, compare your system with others using `sst.compare_models` or `utils.mcnemar`. (For discussion, see [this notebook section](sst_02_hand_built_features.ipynb#Statistical-comparison-of-classifier-models).)\n",
    "1. Return to step 1, or stop the cycle and conduct a more rigorous evaluation with hyperparameter tuning and assessment on the `dev` set.\n",
    "\n",
    "[Error analysis](#Error-analysis) is one of the most important methods for steadily improving a system, as it facilitates a kind of human-powered hill-climbing on your ultimate objective. Often, it takes a careful human analyst just a few examples to spot a major pattern that can lead to a beneficial change to the feature representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Methodological note\n",
    "\n",
    "You don't have to use the experimental framework defined below (based on `sst`). The only constraint we need to place on your system is that it must have a `predict_one` method that can map directly from an example text to a prediction, and it must be able to make predictions without having any information beyond the text. (For example, it can't depend on knowing which task the text comes from.) See [the bakeoff section below](#Bakeoff-[1-point]) for examples of functions that conform to this specification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up\n",
    "\n",
    "See [the first notebook in this unit](sst_01_overview.ipynb#Set-up) for set-up instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import torch.nn as nn\n",
    "\n",
    "from torch_rnn_classifier import TorchRNNClassifier\n",
    "from torch_tree_nn import TorchTreeNN\n",
    "import sst\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "SST_HOME = os.path.join('data', 'sentiment')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our primary train set is the SST-3 train set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "sst_train = sst.train_reader(SST_HOME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8544"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sst_train.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the train set we will use for all the regular homework questions. You are welcome to bring in new datasets for your original system. You are also free to add `include_subtrees=True`. This is very likely to lead to better systems, but it substantially increases the overall size of the dataset (from 8,544 examples to 159,274), which will in turn substantially increase the time it takes to run experiments.\n",
    "\n",
    "See [this notebook](sst_01_overview.ipynb) for additional details of this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dev sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have two development set. SST3-dev consists of sentences from movie reviews, just like SST-3 train:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sst_dev = sst.dev_reader(SST_HOME)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our new bakeoff dev set consists of sentences from restaurant reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "bakeoff_dev = sst.bakeoff_dev_reader(SST_HOME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'example_id': 57,\n",
       "  'sentence': 'I would recommend that you make reservations in advance.',\n",
       "  'label': 'neutral',\n",
       "  'is_subtree': 0},\n",
       " {'example_id': 590,\n",
       "  'sentence': 'We were welcomed warmly.',\n",
       "  'label': 'positive',\n",
       "  'is_subtree': 0},\n",
       " {'example_id': 1968,\n",
       "  'sentence': 'We have been to Oceanaire twice in the last 6 weeks.',\n",
       "  'label': 'neutral',\n",
       "  'is_subtree': 0}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bakeoff_dev.sample(3, random_state=1).to_dict(orient='records')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the label distribution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neutral     1019\n",
       "positive     777\n",
       "negative     565\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bakeoff_dev.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The label distribution for the corresponding test set is similar to this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A softmax baseline\n",
    "\n",
    "This example is here mainly as a reminder of how to use our experimental framework with linear models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unigrams_phi(text):\n",
    "    return Counter(text.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thin wrapper around `LogisticRegression` for the sake of `sst.experiment`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_softmax_classifier(X, y):\n",
    "    mod = LogisticRegression(\n",
    "        fit_intercept=True,\n",
    "        solver='liblinear',\n",
    "        multi_class='ovr')\n",
    "    mod.fit(X, y)\n",
    "    return mod"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The experimental run with some notes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assessment dataset 1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.628     0.689     0.657       428\n",
      "     neutral      0.343     0.153     0.211       229\n",
      "    positive      0.629     0.750     0.684       444\n",
      "\n",
      "    accuracy                          0.602      1101\n",
      "   macro avg      0.533     0.531     0.518      1101\n",
      "weighted avg      0.569     0.602     0.575      1101\n",
      "\n",
      "Assessment dataset 2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.272     0.690     0.390       565\n",
      "     neutral      0.429     0.113     0.179      1019\n",
      "    positive      0.409     0.346     0.375       777\n",
      "\n",
      "    accuracy                          0.328      2361\n",
      "   macro avg      0.370     0.383     0.315      2361\n",
      "weighted avg      0.385     0.328     0.294      2361\n",
      "\n",
      "Mean of macro-F1 scores: 0.416\n"
     ]
    }
   ],
   "source": [
    "softmax_experiment = sst.experiment(\n",
    "    sst.train_reader(SST_HOME),   # Train on any data you like except SST-3 test!\n",
    "    unigrams_phi,                 # Free to write your own!\n",
    "    fit_softmax_classifier,       # Free to write your own!\n",
    "    assess_dataframes=[sst_dev, bakeoff_dev]) # Free to change this during development!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`softmax_experiment` contains a lot of information that you can use for error analysis; see [this section below](#Error-analysis) for starter code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNNClassifier wrapper\n",
    "\n",
    "This section illustrates how to use `sst.experiment` with `TorchRNNClassifier`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To featurize examples for an RNN, we can just get the words in order, letting the model take care of mapping them into an embedding space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rnn_phi(text):\n",
    "    return text.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model wrapper gets the vocabulary using `sst.get_vocab`. If you want to use pretrained word representations in here, then you can have `fit_rnn_classifier` build that space too; see [this notebook section for details](sst_03_neural_networks.ipynb#Pretrained-embeddings). See also [torch_model_base.py](torch_model_base.py) for details on the many optimization parameters that `TorchRNNClassifier` accepts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_rnn_classifier(X, y):\n",
    "    sst_glove_vocab = utils.get_vocab(X, mincount=2)\n",
    "    mod = TorchRNNClassifier(\n",
    "        sst_glove_vocab,\n",
    "        early_stopping=True)\n",
    "    mod.fit(X, y)\n",
    "    return mod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stopping after epoch 47. Validation score did not improve by tol=1e-05 for more than 10 epochs. Final error is 1.29877108335495"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assessment dataset 1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.600     0.584     0.592       428\n",
      "     neutral      0.249     0.231     0.240       229\n",
      "    positive      0.607     0.644     0.625       444\n",
      "\n",
      "    accuracy                          0.535      1101\n",
      "   macro avg      0.485     0.487     0.486      1101\n",
      "weighted avg      0.530     0.535     0.532      1101\n",
      "\n",
      "Assessment dataset 2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.248     0.724     0.370       565\n",
      "     neutral      0.444     0.137     0.210      1019\n",
      "    positive      0.412     0.211     0.279       777\n",
      "\n",
      "    accuracy                          0.302      2361\n",
      "   macro avg      0.368     0.357     0.286      2361\n",
      "weighted avg      0.387     0.302     0.271      2361\n",
      "\n",
      "Mean of macro-F1 scores: 0.386\n"
     ]
    }
   ],
   "source": [
    "rnn_experiment = sst.experiment(\n",
    "    sst.train_reader(SST_HOME),\n",
    "    rnn_phi,\n",
    "    fit_rnn_classifier,\n",
    "    vectorize=False,  # For deep learning, use `vectorize=False`.\n",
    "    assess_dataframes=[sst_dev, bakeoff_dev])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Error analysis\n",
    "\n",
    "This section begins to build an error-analysis framework using the dicts returned by `sst.experiment`. These have the following structure:\n",
    "\n",
    "```\n",
    "'model': trained model\n",
    "'phi': the feature function used\n",
    "'train_dataset':\n",
    "   'X': feature matrix\n",
    "   'y': list of labels\n",
    "   'vectorizer': DictVectorizer,\n",
    "   'raw_examples': list of raw inputs, before featurizing   \n",
    "'assess_datasets': list of datasets, each with the same structure as the value of 'train_dataset'\n",
    "'predictions': list of lists of predictions on the assessment datasets\n",
    "'metric': `score_func.__name__`, where `score_func` is an `sst.experiment` argument\n",
    "'score': the `score_func` score on the each of the assessment dataasets\n",
    "```\n",
    "The following function just finds mistakes, and returns a `pd.DataFrame` for easy subsequent processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_errors(experiment):\n",
    "    \"\"\"Find mistaken predictions.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    experiment : dict\n",
    "        As returned by `sst.experiment`.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    pd.DataFrame\n",
    "\n",
    "    \"\"\"\n",
    "    dfs = []\n",
    "    for i, dataset in enumerate(experiment['assess_datasets']):\n",
    "        df = pd.DataFrame({\n",
    "            'raw_examples': dataset['raw_examples'],\n",
    "            'predicted': experiment['predictions'][i],\n",
    "            'gold': dataset['y']})\n",
    "        df['correct'] = df['predicted'] == df['gold']\n",
    "        df['dataset'] = i\n",
    "        dfs.append(df)\n",
    "    return pd.concat(dfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax_analysis = find_errors(softmax_experiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnn_analysis = find_errors(rnn_experiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we merge the sotmax and RNN experiments into a single DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "analysis = softmax_analysis.merge(\n",
    "    rnn_analysis, left_on='raw_examples', right_on='raw_examples')\n",
    "\n",
    "analysis = analysis.drop('gold_y', axis=1).rename(columns={'gold_x': 'gold'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code collects a specific subset of examples; small modifications to its structure will give you different interesting subsets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examples where the softmax model is correct, the RNN is not,\n",
    "# and the gold label is 'positive'\n",
    "\n",
    "error_group = analysis[\n",
    "    (analysis['predicted_x'] == analysis['gold'])\n",
    "    &\n",
    "    (analysis['predicted_y'] != analysis['gold'])\n",
    "    &\n",
    "    (analysis['gold'] == 'positive')\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "236"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "error_group.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================================================================\n",
      "The restuarant knew that it was our anniversary.\n",
      "======================================================================\n",
      "We had eaten dinner at Caffe Molise and enjoyed the food, deciding to come back.\n",
      "======================================================================\n",
      "Wonderful pianist to complement the food.\n",
      "======================================================================\n",
      "Classic, old-school, great food.\n",
      "======================================================================\n",
      "The quality of the food is top shelf.\n"
     ]
    }
   ],
   "source": [
    "for ex in error_group['raw_examples'].sample(5, random_state=1):\n",
    "    print(\"=\"*70)\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework questions\n",
    "\n",
    "Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Token-level differences [1 point]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can begin to get a sense for how our two dev sets differ by considering the most frequent tokens from each. This question asks you to begin such analysis.\n",
    "\n",
    "Your task: write a function `get_token_counts` that, given a `pd.DataFrame` in the format of our datasets, tokenizes the example sentences based on whitespace and creates a count distribution over all of the tokens. The function should return a `pd.Series` sorted by frequency; if you create a count dictionary `d`, then `pd.Series(d).sort_values(ascending=False)` will give you what you need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_token_counts(df):\n",
    "    total_counts = df[\"sentence\"].apply(lambda sentence: Counter(sentence.lower().split())).sum()\n",
    "\n",
    "    return pd.Series(total_counts).sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_get_token_counts(func):\n",
    "    df = pd.DataFrame([\n",
    "        {'sentence': 'a a b'},\n",
    "        {'sentence': 'a b a'},\n",
    "        {'sentence': 'a a a b.'}])\n",
    "    result = func(df)\n",
    "    for token, expected in (('a', 7), ('b', 2), ('b.', 1)):\n",
    "        actual = result.loc[token]\n",
    "        assert actual == expected, \\\n",
    "            \"For token {}, expected {}; got {}\".format(\n",
    "            token, expected, actual)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_get_token_counts(get_token_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you develop your original system, you might review these results. The two dev sets have different vocabularies and different low-level encoding details that are sure to impact model performance, especially when one considers that the train set is like `sst_dev` in all these respects. For additional discussion, see [this notebook section](sst_01_overview.ipynb#Tokenization)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training on some of the bakeoff data [1 point]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have so far presented the bakeoff dev set as purely for evaluation. Since the train set consists entirely of SST-3 data, this makes the bakeoff split especially challenging. We might be able to reduce the challenging by adding some of the bakeoff dev set to the train set, keeping some of it for evaluation. The current question asks to begin explore the effects of such training.\n",
    "\n",
    "Your task: write a function `run_mixed_training_experiment`. The function should:\n",
    "\n",
    "1. Take as inputs (a) a model training wrapper like `fit_softmax_classifier` and (b) an integer `bakeoff_train_size` specifying the number of examples from `bakeoff_dev` that should be included in the train set.\n",
    "1. Split `bakeoff_dev` so that the first `bakeoff_train_size` examples are in the train set and the rest are used for evaluation.\n",
    "1. Use `sst.experiment` with the user-supplied model training wrapper, `unigram_phi` as defined above, and a train set that consists of SST-3 train and the train portion of `bakeoff_dev` as defined in step 2. The value of `assess_dataframes` should be a list consisting of the SST-3 dev set and the evaluation portion of `bakeoff_dev` as defined in step 2.\n",
    "1. Return the return value of `sst.experiment`.\n",
    "\n",
    "The function `test_run_mixed_training_experiment` will help you iterate to the required design."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_mixed_training_experiment(wrapper_func, bakeoff_train_size):\n",
    "    bakeoff_dev = sst.bakeoff_dev_reader(SST_HOME)\n",
    "    \n",
    "    bakeoff_train = bakeoff_dev.iloc[:bakeoff_train_size, :]\n",
    "    bakeoff_dev = bakeoff_dev.iloc[bakeoff_train_size:, :]\n",
    "\n",
    "    return sst.experiment(train_dataframes=[sst.train_reader(SST_HOME), bakeoff_train],\n",
    "                          phi=unigrams_phi,\n",
    "                          train_func=wrapper_func,\n",
    "                          assess_dataframes=[sst.dev_reader(SST_HOME), bakeoff_dev])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_run_mixed_training_experiment(func):\n",
    "    bakeoff_train_size = 1000\n",
    "    experiment = func(fit_softmax_classifier, bakeoff_train_size)\n",
    "\n",
    "    assess_size = len(experiment['assess_datasets'])\n",
    "    assert len(experiment['assess_datasets']) == 2, \\\n",
    "        (\"The evaluation should be done on two datasets: \"\n",
    "         \"SST3 and part of the bakeoff dev set. \"\n",
    "         \"You have {} datasets.\".format(assess_size))\n",
    "\n",
    "    bakeoff_test_size = bakeoff_dev.shape[0] - bakeoff_train_size\n",
    "    expected_eval_examples = bakeoff_test_size + sst_dev.shape[0]\n",
    "    eval_examples = sum(len(d['raw_examples']) for d in experiment['assess_datasets'])\n",
    "    assert expected_eval_examples == eval_examples, \\\n",
    "        \"Expected {} evaluation examples; got {}\".format(\n",
    "        expected_eval_examples, eval_examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Assessment dataset 1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.627     0.671     0.648       428\n",
      "     neutral      0.319     0.162     0.214       229\n",
      "    positive      0.638     0.757     0.692       444\n",
      "\n",
      "    accuracy                          0.599      1101\n",
      "   macro avg      0.528     0.530     0.518      1101\n",
      "weighted avg      0.567     0.599     0.576      1101\n",
      "\n",
      "Assessment dataset 2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.471     0.412     0.440       320\n",
      "     neutral      0.588     0.590     0.589       612\n",
      "    positive      0.503     0.548     0.525       429\n",
      "\n",
      "    accuracy                          0.535      1361\n",
      "   macro avg      0.521     0.517     0.518      1361\n",
      "weighted avg      0.534     0.535     0.534      1361\n",
      "\n",
      "Mean of macro-F1 scores: 0.518\n"
     ]
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_run_mixed_training_experiment(run_mixed_training_experiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A more powerful vector-averaging baseline [2 points]\n",
    "\n",
    "In [Distributed representations as features](sst_03_neural_networks.ipynb#Distributed-representations-as-features), we looked at a baseline for the ternary SST-3 problem in which each example is modeled as the mean of its GloVe representations. A `LogisticRegression` model was used for prediction. A neural network might do better with these representations, since there might be complex relationships between the input feature dimensions that a linear classifier can't learn. To address this question, we want to get set up to run the experiment with a shallow neural classifier. \n",
    "\n",
    "Your task: write and submit a model wrapper function around `TorchShallowNeuralClassifier`. This function should implement hyperparameter search according to this specification:\n",
    "\n",
    "* Set `early_stopping=True` for all experiments.\n",
    "* Using 3-fold cross-validation, exhaustively explore this set of hyperparameter combinations:\n",
    "  * The hidden dimensionality at 50, 100, and 200.\n",
    "  * The hidden activation function as `nn.Tanh()` and `nn.ReLU()`.\n",
    "* For all other parameters to `TorchShallowNeuralClassifier`, use the defaults.\n",
    "\n",
    "See [this notebook section](sst_02_hand_built_features.ipynb#Hyperparameter-search) for examples. You are not required to run a full evaluation with this function using `sst.experiment`, but we assume you will want to.\n",
    "\n",
    "We're not evaluating the quality of your model. (We've specified the protocols completely, but there will still be variation in the results.) However, the primary goal of this question is to get you thinking more about this strong baseline feature representation scheme for SST-3, so we're sort of hoping you feel compelled to try out variations on your own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch_shallow_neural_classifier import TorchShallowNeuralClassifier\n",
    "\n",
    "def fit_shallow_neural_classifier_with_hyperparameter_search(X, y):\n",
    "    basemod = TorchShallowNeuralClassifier(early_stopping=True)\n",
    "    cv = 3\n",
    "    param_grid = {'hidden_dim': [50, 100, 200],\n",
    "                  'hidden_activation': [nn.Tanh(), nn.ReLU()]}\n",
    "    \n",
    "    bestmod = utils.fit_classifier_with_hyperparameter_search(\n",
    "        X, y, basemod, cv, param_grid)    \n",
    "    \n",
    "    return bestmod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stopping after epoch 56. Validation score did not improve by tol=1e-05 for more than 10 epochs. Final error is 5.5138059258461444"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params: {'hidden_activation': ReLU(), 'hidden_dim': 200}\n",
      "Best score: 0.519\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.633     0.685     0.658       428\n",
      "     neutral      0.400     0.131     0.197       229\n",
      "    positive      0.623     0.791     0.697       444\n",
      "\n",
      "    accuracy                          0.612      1101\n",
      "   macro avg      0.552     0.535     0.517      1101\n",
      "weighted avg      0.581     0.612     0.578      1101\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    GLOVE_HOME = os.path.join('data', 'glove.6B')\n",
    "\n",
    "    glove_lookup = utils.glove2dict(os.path.join(GLOVE_HOME, 'glove.6B.300d.txt'))\n",
    "\n",
    "    def vsm_phi(text, lookup, np_func=np.mean):\n",
    "        \"\"\"Represent `text` as a combination of the vector of its words.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        text : str\n",
    "\n",
    "        lookup : dict\n",
    "            From words to vectors.\n",
    "\n",
    "        np_func : function (default: np.sum)\n",
    "            A numpy matrix operation that can be applied columnwise,\n",
    "            like `np.mean`, `np.sum`, or `np.prod`. The requirement is that\n",
    "            the function take `axis=0` as one of its arguments (to ensure\n",
    "            columnwise combination) and that it return a vector of a\n",
    "            fixed length, no matter what the size of the text is.\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "        np.array, dimension `X.shape[1]`\n",
    "\n",
    "        \"\"\"\n",
    "        allvecs = np.array([lookup[w] for w in text.split() if w in lookup])\n",
    "        if len(allvecs) == 0:\n",
    "            dim = len(next(iter(lookup.values())))\n",
    "            feats = np.zeros(dim)\n",
    "        else:\n",
    "            feats = np_func(allvecs, axis=0)\n",
    "        return feats\n",
    "\n",
    "    def glove_phi(text, np_func=np.mean):\n",
    "        return vsm_phi(text, glove_lookup, np_func=np_func)\n",
    "\n",
    "    _ = sst.experiment(sst.train_reader(SST_HOME),\n",
    "                       glove_phi,\n",
    "                       fit_shallow_neural_classifier_with_hyperparameter_search,\n",
    "                       assess_dataframes=sst.dev_reader(SST_HOME),\n",
    "                       vectorize=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BERT encoding [2 points]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We might hypothesize that encoding our examples with BERT will yield improvements over the GloVe averaging method explored in the previous question, since BERT implements a much more complex and data-driven function for this kind of combination. This question asks you to begin exploring this general hypothesis.\n",
    "\n",
    "Your task: write a function `hf_cls_phi` that uses Hugging Face functionality to encode individual examples with BERT and returns the final output representation above the [CLS] token.\n",
    "\n",
    "You are not required to evaluate this feature function, but it is easy to do so with `sst.experiment` and `vectorize=False` (since your feature function directly encodes every example as a vector). Your code should also be a natural basis for even more powerful approaches – for example, it might be even better to pool all the output states rather than using just the first output state. Another option is [fine-tuning](finetuning.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertModel, BertTokenizer\n",
    "import vsm\n",
    "\n",
    "# Instantiate a Bert model and tokenizer based on `bert_weights_name`:\n",
    "bert_weights_name = 'bert-base-uncased'\n",
    "bert_tokenizer = BertTokenizer.from_pretrained(bert_weights_name)\n",
    "bert_model = BertModel.from_pretrained(bert_weights_name)\n",
    "\n",
    "\n",
    "def hf_cls_phi(text):\n",
    "    # Get the ids. `vsm.hf_encode` will help; be sure to\n",
    "    # set `add_special_tokens=True`.\n",
    "    bert_token_ids = vsm.hf_encode(text, bert_tokenizer, add_special_tokens=True)\n",
    "\n",
    "    # Get the BERT representations. `vsm.hf_represent` will help:\n",
    "    bert_reps = vsm.hf_represent(bert_token_ids, bert_model)\n",
    "\n",
    "    # Index into `reps` to get the representation above [CLS].\n",
    "    # The shape of `reps` should be (1, n, 768), where n is the\n",
    "    # number of tokens. You need the 0th element of the 2nd dim:\n",
    "    cls_rep = bert_reps.squeeze(0)[0]\n",
    "\n",
    "    # These conversions should ensure that you can work with the\n",
    "    # representations flexibly. Feel free to change the variable\n",
    "    # name:\n",
    "    return cls_rep.cpu().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_hf_cls_phi(func):\n",
    "    rep = func(\"Just testing!\")\n",
    "\n",
    "    expected_shape = (768,)\n",
    "    result_shape = rep.shape\n",
    "    assert rep.shape == (768,), \\\n",
    "        \"Expected shape {}; got {}\".format(\n",
    "        expected_shape, result_shape)\n",
    "\n",
    "    # String conversion to avoid precision errors:\n",
    "    expected_first_val = str(0.1709)\n",
    "    result_first_val = \"{0:.04f}\".format(rep[0])\n",
    "\n",
    "    assert expected_first_val == result_first_val, \\\n",
    "        (\"Unexpected representation values. Expected the \"\n",
    "        \"first value to be {}; got {}\".format(\n",
    "            expected_first_val, result_first_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_hf_cls_phi(hf_cls_phi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stopping after epoch 37. Validation score did not improve by tol=1e-05 for more than 10 epochs. Final error is 4.9238194823265082"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params: {'hidden_activation': Tanh(), 'hidden_dim': 200}\n",
      "Best score: 0.593\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.703     0.792     0.745       428\n",
      "     neutral      0.403     0.236     0.298       229\n",
      "    positive      0.734     0.802     0.766       444\n",
      "\n",
      "    accuracy                          0.680      1101\n",
      "   macro avg      0.613     0.610     0.603      1101\n",
      "weighted avg      0.653     0.680     0.661      1101\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    _ = sst.experiment(sst.train_reader(SST_HOME),\n",
    "                       hf_cls_phi,\n",
    "                       fit_shallow_neural_classifier_with_hyperparameter_search,\n",
    "                       assess_dataframes=sst.dev_reader(SST_HOME),\n",
    "                       vectorize=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: encoding all of SST-3 train (no subtrees) takes about 11 minutes on my 2015 iMac, CPU only (32GB)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your original system [3 points]\n",
    "\n",
    "Your task is to develop an original model for the SST-3 problem and our new bakeoff dataset. There are many options. If you spend more than a few hours on this homework problem, you should consider letting it grow into your final project! Here are some relatively manageable ideas that you might try:\n",
    "\n",
    "1. We didn't systematically evaluate the `bidirectional` option to the `TorchRNNClassifier`. Similarly, that model could be tweaked to allow multiple LSTM layers (at present there is only one), and you could try adding layers to the classifier portion of the model as well.\n",
    "\n",
    "1. We've already glimpsed the power of rich initial word representations, and later in the course we'll see that smart initialization usually leads to a performance gain in NLP, so you could perhaps achieve a winning entry with a simple model that starts in a great place.\n",
    "\n",
    "1. Our [practical introduction to contextual word representations](finetuning.ipynb) covers pretrained representations and interfaces that are likely to boost the performance of any system.\n",
    "\n",
    "We want to emphasize that this needs to be an __original__ system. It doesn't suffice to download code from the Web, retrain, and submit. You can build on others' code, but you have to do something new and meaningful with it. See the course website for additional guidance on how original systems will be evaluated.\n",
    "\n",
    "In the cell below, please provide a brief technical description of your original system, so that the teaching team can gain an understanding of what it does. This will help us to understand your code and analyze all the submissions to identify patterns and strategies.  We also ask that you report the best score your system got during development (your best average of macro-F1 scores), just to help us understand how systems performed overall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: imbalanced-learn in /raid/miguelm/miniconda3/envs/nlu/lib/python3.8/site-packages (0.8.0)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /raid/miguelm/miniconda3/envs/nlu/lib/python3.8/site-packages (from imbalanced-learn) (1.20.1)\n",
      "Requirement already satisfied: scikit-learn>=0.24 in /raid/miguelm/miniconda3/envs/nlu/lib/python3.8/site-packages (from imbalanced-learn) (0.24.1)\n",
      "Requirement already satisfied: joblib>=0.11 in /raid/miguelm/miniconda3/envs/nlu/lib/python3.8/site-packages (from imbalanced-learn) (1.0.1)\n",
      "Requirement already satisfied: scipy>=0.19.1 in /raid/miguelm/miniconda3/envs/nlu/lib/python3.8/site-packages (from imbalanced-learn) (1.6.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /raid/miguelm/miniconda3/envs/nlu/lib/python3.8/site-packages (from scikit-learn>=0.24->imbalanced-learn) (2.1.0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Stopping after epoch 74. Validation score did not improve by tol=1e-05 for more than 10 epochs. Final error is 3.9114710055291653"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best params: {'hidden_activation': ReLU(), 'hidden_dim': 1400}\n",
      "Best score: 0.840\n",
      "Assessment dataset 1\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.712     0.631     0.669       428\n",
      "     neutral      0.317     0.371     0.342       229\n",
      "    positive      0.700     0.716     0.708       444\n",
      "\n",
      "    accuracy                          0.611      1101\n",
      "   macro avg      0.577     0.573     0.573      1101\n",
      "weighted avg      0.625     0.611     0.617      1101\n",
      "\n",
      "Assessment dataset 2\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative      0.588     0.513     0.548       117\n",
      "     neutral      0.590     0.758     0.664       194\n",
      "    positive      0.713     0.537     0.613       162\n",
      "\n",
      "    accuracy                          0.622       473\n",
      "   macro avg      0.631     0.603     0.608       473\n",
      "weighted avg      0.632     0.622     0.618       473\n",
      "\n",
      "Mean of macro-F1 scores: 0.591\n"
     ]
    }
   ],
   "source": [
    "# PLEASE MAKE SURE TO INCLUDE THE FOLLOWING BETWEEN THE START AND STOP COMMENTS:\n",
    "#   1) Textual description of your system.\n",
    "#   2) The code for your original system.\n",
    "#   3) The score achieved by your system in place of MY_NUMBER.\n",
    "#        With no other changes to that line.\n",
    "#        You should report your score as a decimal value <=1.0\n",
    "# PLEASE MAKE SURE NOT TO DELETE OR EDIT THE START AND STOP COMMENTS\n",
    "\n",
    "# NOTE: MODULES, CODE AND DATASETS REQUIRED FOR YOUR ORIGINAL SYSTEM\n",
    "# SHOULD BE ADDED BELOW THE 'IS_GRADESCOPE_ENV' CHECK CONDITION. DOING\n",
    "# SO ABOVE THE CHECK MAY CAUSE THE AUTOGRADER TO FAIL.\n",
    "\n",
    "# START COMMENT: Enter your system description in this cell.\n",
    "# In my system, I have tried to follow what I have learnt during these lectures, but\n",
    "# trying to incorporate a few things to make it perform better.\n",
    "# One of the things I have done is to include the subtrees of the SST dataset, removing its duplicates\n",
    "# for performance reasons.\n",
    "# I have also used part of the bakeoff_dev dataset to train my model, assumming that extra training will\n",
    "# help during the test of my model.\n",
    "# Another thing that I have done is to randomly augment the datasets, in order to balance the number of\n",
    "# examples per class. I am aware that randomly augmenty the classes might lead to overfitting, but I think\n",
    "# the benefit will be bigger than the lost.\n",
    "# In the phi function, I have added an additional preprocessing to the text, by removing any consecutive\n",
    "# duplicate character within the words. My objective is to represent with the same token multiple texts\n",
    "# with similar spellings.I am aware that this transformation is  not perfect, and which might lead that \n",
    "# some tokens that are different might finish represented in a similar way. Neverthless, I think the beneffit\n",
    "# will be bigger than the lost, therefore its usage.\n",
    "# Finally, I have performed some hyperparameter optimisation. I have noticed that nn.ReLU() is the activation\n",
    "# function that better works, once tested agains nn.LeakyRelu and nn.GELU().\n",
    "# Likewise, increasing the hidden dimensionality improves the results.\n",
    "# As the classifier, I have used a simple shadow classifier.\n",
    "# Very important: You will notice that in param_grid, each of the hyperparameters contains a single value.\n",
    "#                 This is due because the Jupyter Notebook hanged once the training had finished, and before producing\n",
    "#                 the bakeoff entry. I have re-executed the training of my system with only the best parameters:\n",
    "#                     hidden_dim=1400 and hidden_activation=nn.ReLU()\n",
    "# My peak score was: 0.840\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    !pip install imbalanced-learn\n",
    "\n",
    "    import numpy as np\n",
    "    import os\n",
    "    import pandas as pd    \n",
    "    import sst\n",
    "    import torch.nn as nn\n",
    "    import utils\n",
    "    import vsm\n",
    "\n",
    "    \n",
    "    from collections import Counter\n",
    "    from imblearn.over_sampling import RandomOverSampler\n",
    "    from sklearn.linear_model import LogisticRegression\n",
    "    from torch_rnn_classifier import TorchRNNClassifier\n",
    "    from torch_shallow_neural_classifier import TorchShallowNeuralClassifier\n",
    "    from torch_tree_nn import TorchTreeNN\n",
    "    from transformers import BertModel, BertTokenizer\n",
    "\n",
    "    def my_system():\n",
    "        # Fix random seeds for reproducibility purposes\n",
    "        utils.fix_random_seeds()\n",
    "\n",
    "        # Set sst dataset location\n",
    "        SST_HOME = os.path.join('data', 'sentiment')\n",
    "\n",
    "        # Load the sst train data, including subtrees and removing duplicates\n",
    "        sst_train = sst.train_reader(SST_HOME, include_subtrees=True, dedup=True)\n",
    "\n",
    "        # Load the sst dev data, and the bakeoff dev data too.\n",
    "        sst_dev = sst.dev_reader(SST_HOME)\n",
    "        bakeoff_dev = sst.bakeoff_dev_reader(SST_HOME)\n",
    "\n",
    "        # Use a percentage of the dev data to train our model\n",
    "        bakeoff_dev_percentage_for_training = 0.8 \n",
    "        bakeoff_train_size = int(bakeoff_dev.shape[0] * bakeoff_dev_percentage_for_training)\n",
    "        bakeoff_train = bakeoff_dev.iloc[:bakeoff_train_size, :]\n",
    "        bakeoff_dev = bakeoff_dev.iloc[bakeoff_train_size:, :]\n",
    "\n",
    "        # Randomly augment the train datasets, to avoid having imbalanced classes.\n",
    "        sst_train, _ = RandomOverSampler(random_state=0).fit_resample(sst_train, sst_train[\"label\"])\n",
    "        bakeoff_train, _ = RandomOverSampler(random_state=0).fit_resample(bakeoff_train, bakeoff_train[\"label\"])\n",
    "\n",
    "        # Instantiate a Bert model and tokenizer based on `bert_weights_name`:\n",
    "        bert_weights_name = 'bert-base-uncased'\n",
    "        bert_tokenizer = BertTokenizer.from_pretrained(bert_weights_name)\n",
    "        bert_model = BertModel.from_pretrained(bert_weights_name)\n",
    "\n",
    "        # Remove consecutive duplicated characters from 'text', trying to represent with the \n",
    "        # same token multiple texts with similar spellings.I am aware that this transformation is\n",
    "        # not perfect, and which might lead that some tokens that are different to finish represented\n",
    "        # in a similar way, but I will prioritise that some expressions could be grouped in a single token.\n",
    "        def removeConsecutiveDuplicatedCharacters(text):\n",
    "            \"\"\"\n",
    "            This method deletes any duplicated letter from 'text'.\n",
    "\n",
    "            Some examples:\n",
    "                            Yahoo!    -> Yaho!\n",
    "                            Noooo     -> No\n",
    "                            looove     -> love\n",
    "            \"\"\" \n",
    "\n",
    "            if len(text) == 0:\n",
    "                return text\n",
    "\n",
    "            result = ''\n",
    "            # loop to check for each character\n",
    "            for i in range(1, len(text)):  \n",
    "                if text[i - 1] !=  text[i]:\n",
    "                    result += text[i - 1]\n",
    "\n",
    "            result += text[-1]\n",
    "            return result\n",
    "\n",
    "        def hf_cls_phi(text):\n",
    "            # Remove consecutive duplicated characters from the text\n",
    "            text = removeConsecutiveDuplicatedCharacters(text)\n",
    "\n",
    "            # Get the ids. `vsm.hf_encode` will help; be sure to\n",
    "            # set `add_special_tokens=True`.\n",
    "            bert_token_ids = vsm.hf_encode(text, bert_tokenizer, add_special_tokens=True)\n",
    "\n",
    "            # Get the BERT representations. `vsm.hf_represent` will help:\n",
    "            bert_reps = vsm.hf_represent(bert_token_ids, bert_model)\n",
    "\n",
    "            # Index into `reps` to get the representation above [CLS].\n",
    "            # The shape of `reps` should be (1, n, 768), where n is the\n",
    "            # number of tokens. You need the 0th element of the 2nd dim:\n",
    "            cls_rep = bert_reps.squeeze(0)[0]\n",
    "\n",
    "            # These conversions should ensure that you can work with the\n",
    "            # representations flexibly. Feel free to change the variable\n",
    "            # name:\n",
    "            return cls_rep.cpu().numpy()\n",
    "\n",
    "        def fit_shallow_neural_classifier_with_hyperparameter_search(X, y):\n",
    "            basemod = TorchShallowNeuralClassifier(early_stopping=True)\n",
    "\n",
    "            cv = 3\n",
    "            param_grid = {'hidden_dim': [1400],               # [200, 500, 800, 1100, 1400] <- I tried before with these combinations\n",
    "                          'hidden_activation': [nn.ReLU()]}   # [nn.ReLU(), nn.GELU(), nn.LeakyReLU()] <- I tried before with these combinations\n",
    "\n",
    "            bestmod = utils.fit_classifier_with_hyperparameter_search(\n",
    "                X, y, basemod, cv, param_grid)    \n",
    "\n",
    "            return bestmod\n",
    "        \n",
    "        my_experiment = sst.experiment(\n",
    "            train_dataframes=[sst_train, bakeoff_train],\n",
    "            phi=hf_cls_phi,\n",
    "            train_func=fit_shallow_neural_classifier_with_hyperparameter_search,\n",
    "            assess_dataframes=[sst_dev, bakeoff_dev],\n",
    "            vectorize=False)\n",
    "        \n",
    "        return my_experiment\n",
    "    \n",
    "    my_experiment = my_system()\n",
    "# STOP COMMENT: Please do not remove this comment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bakeoff [1 point]\n",
    "\n",
    "As we said above, the bakeoff evaluation data is the official SST test-set release and a new test set derived from the same sources and labeling methods as for `bakeoff_dev`.\n",
    "\n",
    "For this bakeoff, you'll evaluate your original system from the above homework problem on these test sets. Our metric will be the mean of the macro-F1 values, which weights both datasets equally despite their differing sizes.\n",
    "\n",
    "The central requirement for your system is that you have define a `predict_one` method for it that maps a text (str) directly to a label prediction – one of 'positive', 'negative', 'neutral'. If you used `sst.experiment` with `vectorize=True`, then the following function (for `softmax_experiment`) will be easy to adapt – you probably just need to change the variable `softmax_experiment` to the variable for your experiment output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_one_softmax(text):\n",
    "    # Singleton list of feature dicts:\n",
    "    feats = [my_experiment['phi'](text)]\n",
    "    # Vectorize to get a feature matrix:\n",
    "    X = my_experiment['train_dataset']['vectorizer'].transform(feats)\n",
    "    # Standard sklearn `predict` step:\n",
    "    preds = my_experiment['model'].predict(X)\n",
    "    # Be sure to return the only member of the predictions,\n",
    "    # rather than the singleton list:\n",
    "    return preds[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you used an RNN like the one we demoed above, then featurization is a bit more straightforward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_one_rnn(text):\n",
    "    # List of tokenized examples:\n",
    "    X = [rnn_experiment['phi'](text)]\n",
    "    # Standard `predict` step on a list of lists of str:\n",
    "    preds = rnn_experiment['model'].predict(X)\n",
    "    # Be sure to return the only member of the predictions,\n",
    "    # rather than the singleton list:\n",
    "    return preds[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function is used to create the bakeoff submission file. Its arguments are your `predict_one` function and an output filename (str)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_bakeoff_submission(\n",
    "        predict_one_func,\n",
    "        output_filename='cs224u-sentiment-bakeoff-entry.csv'):\n",
    "\n",
    "    bakeoff_test = sst.bakeoff_test_reader(SST_HOME)\n",
    "    sst_test = sst.test_reader(SST_HOME)\n",
    "    bakeoff_test['dataset'] = 'bakeoff'\n",
    "    sst_test['dataset'] = 'sst3'\n",
    "    df = pd.concat((bakeoff_test, sst_test))\n",
    "\n",
    "    df['prediction'] = df['sentence'].apply(predict_one_func)\n",
    "\n",
    "    df.to_csv(output_filename, index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, for example, the following will create a bake-off entry based on `predict_one_softmax`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This check ensure that the following code only runs on the local environment only.\n",
    "# The following call will not be run on the autograder environment.\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    def predict_one(text):\n",
    "        # List of tokenized examples:\n",
    "        X = [my_experiment['phi'](text)]    \n",
    "\n",
    "        # Standard sklearn `predict` step:\n",
    "        preds = my_experiment['model'].predict(X)\n",
    "\n",
    "        # Be sure to return the only member of the predictions,\n",
    "        # rather than the singleton list:\n",
    "        return preds[0]\n",
    "    \n",
    "    create_bakeoff_submission(predict_one)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates a file `cs224u-sentiment-bakeoff-entry.csv` in the current directory. That file should be uploaded as-is. Please do not change its name.\n",
    "\n",
    "Only one upload per team is permitted, and you should do no tuning of your system based on what you see in our bakeoff prediction file – you should not study that file in anyway, beyond perhaps checking that it contains what you expected it to contain. The upload function will do some additional checking to ensure that your file is well-formed.\n",
    "\n",
    "People who enter will receive the additional homework point, and people whose systems achieve the top score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.\n",
    "\n",
    "Late entries will be accepted, but they cannot earn the extra 0.5 points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission Instruction\n",
    "\n",
    "Submit the following files to gradescope submission\n",
    "\n",
    "- Please do not change the file name as described below\n",
    "- `hw_sentiment.ipynb` (this notebook)\n",
    "- `cs224u-sentiment-bakeoff-entry.csv` (bake-off output)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nlu",
   "language": "python",
   "name": "nlu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}