labs/practical_07.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "94ce8f61",
   "metadata": {},
   "source": [
    "# Practical 09 - Recurrent Neural Networks\n",
    "\n",
    "**Disclamer**: Some material from the suggested exercises were retrieved and adapted from the following sources:\n",
    "* [Coursera Sequence Models course by DeepLearningAI](https://www.coursera.org/learn/nlp-sequence-models)\n",
    "* [Pytorch Tutorial for RNNs](https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html)\n",
    "* [Pytorch Sentiment Analysis Tutorial](https://github.com/bentrevett/pytorch-sentiment-analysis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c787d01b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d47c74d5",
   "metadata": {},
   "source": [
    "## Question 01\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c7dd504",
   "metadata": {},
   "source": [
    "Perform a forward propagation for \n",
    "\n",
    "$ x^{(1)} = \\begin{bmatrix} \\begin{pmatrix}\n",
    "4 \\\\ 0 \\\\ 0 \\\\ 0\n",
    "\\end{pmatrix}, \n",
    " \\begin{pmatrix}\n",
    "0 \\\\ 8 \\\\ 2 \\\\ 0\n",
    "\\end{pmatrix}\n",
    "\\end{bmatrix}\n",
    "$\n",
    "\n",
    "With targets:\n",
    "\n",
    "$ t^{(1)} = \\begin{bmatrix} \\begin{pmatrix}\n",
    "1 \\\\ 0\n",
    "\\end{pmatrix}, \n",
    " \\begin{pmatrix}\n",
    "0 \\\\ 1\n",
    "\\end{pmatrix}\n",
    "\\end{bmatrix}\n",
    "$\n",
    "\n",
    "Initialize all weights and biases to 0.1, using 3 units per hidden layer, initializing the hidden state to all zeros and using η = 1.0.\n",
    "\n",
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a42d2ad",
   "metadata": {},
   "source": [
    "A Recurrent neural network can be seen as the repetition of a single cell. We are first going to implement the computations for a single time-step. The following figure describes the operations for a single time-step of an RNN cell. \n",
    "\n",
    "<img src=\"forward-net.png\" style=\"width:700px;height:300px;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d1c935e",
   "metadata": {},
   "source": [
    "\n",
    "**Steps to code the RNN cell**:\n",
    "1. Compute the hidden state with tanh activation: $a^{\\langle t \\rangle} = \\tanh(W_{aa} a^{\\langle t-1 \\rangle} + W_{ax} x^{\\langle t \\rangle} + b_a)$.\n",
    "2. Using your new hidden state $a^{\\langle t \\rangle}$, compute the prediction $\\hat{y}^{\\langle t \\rangle} = softmax(W_{ya} a^{\\langle t \\rangle} + b_y)$.\n",
    "3. Return $a^{\\langle t \\rangle}$ and $y^{\\langle t \\rangle}$ \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "74b7c254",
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    e_x = np.exp(x - np.max(x))\n",
    "    return e_x / e_x.sum(axis=0)\n",
    "\n",
    "\n",
    "def rnn_cell_forward(xt, a_prev, Wax, Waa, Wya, ba, by):\n",
    "    \"\"\"\n",
    "    Implements a single forward step of the RNN-cell\n",
    "    \n",
    "    Arguments:\n",
    "        xt -- your input data at timestep \"t\", numpy array of shape (n_x).\n",
    "        a_prev -- Hidden state at timestep \"t-1\", numpy array of shape (n_a)\n",
    "        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)\n",
    "        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)\n",
    "        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)\n",
    "        ba --  Bias, numpy array of shape (n_a, 1)\n",
    "        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)\n",
    "\n",
    "    Returns:\n",
    "        a_next -- next hidden state, of shape (n_a)\n",
    "        yt_pred -- prediction at timestep \"t\", numpy array of shape (n_y)\n",
    "\n",
    "    \"\"\"\n",
    "    # YOUR CODE HERE\n",
    "\n",
    "    # compute next activation state\n",
    "    a_next = np.tanh(np.dot(Wax, xt) + np.dot(Waa, a_prev) + ba)\n",
    "\n",
    "    # compute output of the current cell\n",
    "    yt_pred = softmax(np.dot(Wya, a_next) + by)\n",
    "\n",
    "    return a_next, yt_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf3a0603",
   "metadata": {},
   "source": [
    "You can see an RNN as the repetition of the cell you've just built. If your input sequence of data is carried over 10 time steps, then you will copy the RNN cell 10 times. Each cell takes as input the hidden state from the previous cell ($a^{\\langle t-1 \\rangle}$) and the current time-step's input data ($x^{\\langle t \\rangle}$). It outputs a hidden state ($a^{\\langle t \\rangle}$) and a prediction ($y^{\\langle t \\rangle}$) for this time-step.\n",
    "\n",
    "\n",
    "<img src=\"rnn_forward.png\" style=\"width:800px;height:300px;\">\n",
    "\n",
    "\n",
    "\n",
    "Now, we are going to code the RNN forward propagation over the sequence\n",
    "\n",
    "**Steps**:\n",
    "1. Create a vector of zeros ($a$) that will store all the hidden states computed by the RNN.\n",
    "2. Initialize the \"next\" hidden state as $a_0$ (initial hidden state).\n",
    "3. Start looping over each time step, your incremental index is $t$ :\n",
    "    - Update the \"next\" hidden state and the cache by running `rnn_cell_forward`\n",
    "    - Keep a history of the parameters for the gradient update\n",
    "    - Store the \"next\" hidden state in $a$ ($t^{th}$ position) \n",
    "    - Store the prediction in y\n",
    "4. Return $a$, $y$ and history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "35d73175",
   "metadata": {},
   "outputs": [],
   "source": [
    "def rnn_forward(x, a0, Wax, Waa, Wya, ba, by):\n",
    "    \"\"\"\n",
    "    Implement the forward propagation of the recurrent neural network\n",
    "\n",
    "    Arguments:\n",
    "        x -- Input data for every time-step, of shape (n_x, m, T_x).\n",
    "        a0 -- Initial hidden state, of shape (n_a, m)\n",
    "        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)\n",
    "        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)\n",
    "        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)\n",
    "        ba --  Bias numpy array of shape (n_a, 1)\n",
    "        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)\n",
    "\n",
    "    Returns:\n",
    "        a -- Hidden states for every time-step, numpy array of shape (n_a, m, T_x)\n",
    "        y_pred -- Predictions for every time-step, numpy array of shape (n_y, m, T_x)\n",
    "        history -- tuple of values needed for the backward pass, contains (list of caches, x)\n",
    "    \"\"\"\n",
    "\n",
    "    # YOUR CODE HERE\n",
    "    # TIP: use rnn_cell_forward() inside a for loop\n",
    "\n",
    "    # Initialize \"history\" which will contain the list the parameters in each time-step\n",
    "    history = []\n",
    "\n",
    "    # Retrieve dimensions from shapes of x and Wy\n",
    "    n_x, T_x = x.shape\n",
    "    n_y, n_a = Wya.shape\n",
    "\n",
    "    # initialize \"a\" and \"y_pred\" with zeros\n",
    "    a = np.zeros((n_a, T_x))\n",
    "    y_pred = np.zeros((n_y, T_x))\n",
    "\n",
    "    # Initialize a_next\n",
    "    a_next = a0\n",
    "\n",
    "    # loop over all time-steps\n",
    "    for t in range(T_x):\n",
    "\n",
    "        # Update next hidden state, compute the prediction\n",
    "        a_prev = a_next\n",
    "        a_next, yt_pred = rnn_cell_forward(x[:,t], a_next, Wax, Waa, Wya, ba, by)\n",
    "\n",
    "        # Store the time-step in the history\n",
    "        step_parms = (a_next, a_prev, x[:,t], Wax, Waa, Wya, ba, by)\n",
    "        history.append(step_parms)\n",
    "\n",
    "        # Save the value of the new \"next\" hidden state in a (≈1 line)\n",
    "        a[:, t] = a_next\n",
    "\n",
    "        # Save the value of the prediction in y\n",
    "        y_pred[:, t] = yt_pred\n",
    "\n",
    "    return a, y_pred, history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "08d5213c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's run the rnn_forward for a sequence of 10 elements.\n",
    "inputs = np.array([[4, 0, 0, 0], [0, 8, 2, 0]]).transpose()\n",
    "\n",
    "# YOUR CODE HERE\n",
    "# what are the shapes of the weights and biases?\n",
    "a0 = .1 * np.ones(3)\n",
    "Waa = .1 * np.ones((3,3))\n",
    "Wax = .1 * np.ones((3,4))\n",
    "Wya = .1 * np.ones((2,3))\n",
    "ba = .1 * np.ones(3)\n",
    "by = .1 * np.ones(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e6cd3130",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a =  [[0.48538109 0.84704925]\n",
      " [0.48538109 0.84704925]\n",
      " [0.48538109 0.84704925]]\n",
      "a.shape =  (3, 2)\n",
      "\n",
      "y_pred = [[0.5 0.5]\n",
      " [0.5 0.5]]\n",
      "y_pred.shape =  (2, 2)\n"
     ]
    }
   ],
   "source": [
    "a, y_pred, history = rnn_forward(inputs, a0, Wax, Waa, Wya, ba, by)\n",
    "\n",
    "print(\"a = \", a)\n",
    "print(\"a.shape = \", a.shape)\n",
    "print()\n",
    "print(\"y_pred =\", y_pred)\n",
    "print(\"y_pred.shape = \", y_pred.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c69eec4a",
   "metadata": {},
   "source": [
    "## Question 02\n",
    "\n",
    "Use PyTorch to implement an RNN cell and use it to generate random person names using the file `names.txt` as training examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d2f2052f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch import optim\n",
    "\n",
    "\n",
    "# Vanilla RNN implementation\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(RNN, self).__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)\n",
    "        self.tanh= nn.Tanh()\n",
    "        self.o2o = nn.Linear(hidden_size, output_size)\n",
    "        self.initial_hidden = nn.Parameter(\n",
    "            torch.zeros(1, hidden_size)\n",
    "        )\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    def single_forward(self, input, hidden):\n",
    "        # YOUR CODE HERE\n",
    "        input_combined = torch.cat((input, hidden), 1)\n",
    "        hidden = self.tanh(self.i2h(input_combined))\n",
    "        output = self.o2o(hidden)\n",
    "        output = self.softmax(output)\n",
    "        return output, hidden\n",
    "\n",
    "    def forward(self, input_line_tensor):\n",
    "        # YOUR CODE HERE\n",
    "        outputs = []\n",
    "\n",
    "        hidden = self.initial_hidden\n",
    "\n",
    "        for x_t in input_line_tensor:\n",
    "            output, hidden = self.single_forward(\n",
    "                x_t, hidden)\n",
    "            outputs.append(output)\n",
    "\n",
    "        return outputs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7746d482",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['francisco', 'maria', 'joão', 'leonor', 'afonso', 'matilde', 'tomás', 'carolina', 'duarte', 'alice', 'gabriel', 'benedita', 'santiago', 'beatriz', 'lourenço', 'mariana', 'rodrigo', 'francisca', 'miguel', 'sofia', 'martim', 'margarida', 'vicente', 'laura', 'mateus', 'camila', 'lucas', 'clara', 'guilherme', 'ana', 'salvador', 'inês', 'pedro', 'lara', 'gonçalo', 'madalena', 'dinis', 'diana', 'rafael', 'vitória', 'gustavo', 'constança', 'bernardo', 'eva', 'josé', 'mafalda', 'diogo', 'gabriela', 'manuel', 'bianca', 'tiago', 'valentina', 'vasco', 'joana', 'henrique', 'íris', 'simão', 'sara', 'diego', 'luana', 'antónio', 'ariana', 'david', 'letícia', 'daniel', 'rita', 'leonardo', 'júlia', 'enzo', 'carlota', 'xavier', 'carminho', 'eduardo', 'luísa', 'andré', 'ema', 'luís', 'olívia', 'isaac', 'helena', 'matias', 'yara', 'valentim', 'aurora', 'artur', 'yasmin', 'samuel', 'emma', 'kevin', 'teresa', 'frederico', 'mara', 'filipe', 'catarina', 'matheus', 'caetana', 'lorenzo', 'melissa', 'noah', 'alícia', 'alexandre', 'isabel', 'benjamim', 'noa', 'davi', 'marta', 'bryan', 'victória', 'sebastião', 'iara', 'ricardo', 'alana', 'joaquim', 'amélia', 'carlos', 'pilar', 'leandro', 'daniela', 'nicole']\n"
     ]
    }
   ],
   "source": [
    "# all letters to work with the Portuguese names from the train file\n",
    "all_letters = 'abcdefghijklmnopqrstuvxwyzáàãçéêíóõôú'\n",
    "n_letters = len(all_letters) + 2 # Plus BOS and EOS marker\n",
    "\n",
    "# One-hot matrix of first to last letters (not including EOS) for input\n",
    "# Adds a BOS token as the first token\n",
    "def inputTensor(line):\n",
    "    tensor = torch.zeros(len(line)+1, 1, n_letters)\n",
    "    tensor[0][0][-2] = 1  # BOS\n",
    "    for letter_idx in range(len(line)):\n",
    "        letter = line[letter_idx]\n",
    "        tensor[letter_idx+1][0][all_letters.find(letter)] = 1\n",
    "    return tensor\n",
    "\n",
    "# LongTensor of second letter to end (EOS) for target\n",
    "def targetTensor(line):\n",
    "    letter_indexes = [all_letters.find(line[l_idx]) for l_idx in range(len(line))]\n",
    "    letter_indexes.append(n_letters - 1) # EOS\n",
    "    return torch.LongTensor(letter_indexes).unsqueeze(-1)\n",
    "\n",
    "# Read the list of names\n",
    "with open(\"names.txt\") as file:\n",
    "    names = [name.lower().strip() for name in file.readlines()]\n",
    "\n",
    "print(names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c9b02b0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1000/1000 [02:18<00:00,  7.20it/s]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "from random import shuffle, choices\n",
    "\n",
    "train_data = [ (inputTensor(name),targetTensor(name)) for name in names ]\n",
    "hidden_size = 128\n",
    "num_epochs = 1000\n",
    "criterion = nn.NLLLoss()\n",
    "learning_rate = 5e-4\n",
    "model = RNN(n_letters, hidden_size, n_letters)\n",
    "optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n",
    "\n",
    "\n",
    "def train(model, train_data, num_epochs, optimizer, criterion ):\n",
    "    all_losses = []\n",
    "\n",
    "    for epoch in tqdm(range(num_epochs)):\n",
    "\n",
    "        shuffle(train_data)\n",
    "        loss_agg = 0\n",
    "\n",
    "        for input_line_tensor, target_line_tensor in train_data:\n",
    "\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            outputs = model.forward(input_line_tensor)\n",
    "\n",
    "            loss = torch.stack(\n",
    "                [ criterion(o_i, t_i) for o_i ,t_i in zip(outputs, target_line_tensor)]\n",
    "            ).sum()\n",
    "\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            loss_agg += loss.item()/ input_line_tensor.size(0)\n",
    "\n",
    "        all_losses.append(loss_agg)\n",
    "\n",
    "    return all_losses\n",
    "\n",
    "all_losses = train(\n",
    "    model=model,\n",
    "    train_data=train_data,\n",
    "    num_epochs=num_epochs,\n",
    "    optimizer=optimizer,\n",
    "    criterion=criterion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "84af223f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x11fcf6aa0>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAGdCAYAAAA44ojeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA9PklEQVR4nO3de3iU9Z3//9ccMpNMkpmcSCaBBEEUjBxEUIhStZWCSK1WultbirT1qz9paKt2XaRrbdW1WLvfdmuviute3Wq/ldK1K7ayHoogWGtEQJGTRjlIAmSSkJCZHCdzuH9/hAwJckoyk3sCz8d1zXVl5v7MzPv+gOTl5/O5P7fFMAxDAAAAScRqdgEAAADHI6AAAICkQ0ABAABJh4ACAACSDgEFAAAkHQIKAABIOgQUAACQdAgoAAAg6djNLqA/otGoDh06pMzMTFksFrPLAQAAZ8AwDDU3N6uoqEhW66nHSIZkQDl06JCKi4vNLgMAAPRDdXW1RowYcco2QzKgZGZmSuo6QbfbbXI1AADgTAQCARUXF8d+j5/KkAwo3dM6brebgAIAwBBzJsszWCQLAACSDgEFAAAkHQIKAABIOgQUAACQdAgoAAAg6RBQAABA0iGgAACApENAAQAASYeAAgAAkg4BBQAAJB0CCgAASDoEFAAAkHSG5M0CE2XzJ41ava1G47yZuuXyErPLAQDgnMUISg+Vtc16+q1PtO7DOrNLAQDgnEZA6cF69PbPUcPkQgAAOMcRUHqwduUTRQ0SCgAAZiKg9HBsBIWAAgCAmQgoPTDFAwBAciCg9GA92htREgoAAKYioPTAFA8AAMmBgNIDAQUAgORAQOkhFlCiJhcCAMA5joDSg617DQojKAAAmIqA0oOFKR4AAJICAaWH7imeCPkEAABTEVB66J7iMRhBAQDAVASUHpjiAQAgORBQeohN8XAVDwAApiKg9NB9s0CmeAAAMBcBpQcbUzwAACSFPgWU5cuXa+LEiXK73XK73SorK9PLL78cO37NNdfIYrH0etx55529PqOqqkpz586Vy+VSfn6+7r33XoXD4ficzQBZuFkgAABJwd6XxiNGjNCjjz6qCy64QIZh6JlnntGNN96o9957TxdffLEk6fbbb9dDDz0Ue4/L5Yr9HIlENHfuXHm9Xr311luqqanRrbfeqpSUFP3kJz+J0yn1X/cUDzcLBADAXH0KKDfccEOv54888oiWL1+ut99+OxZQXC6XvF7vCd//17/+Vbt27dJrr72mgoICXXLJJXr44Ye1ZMkS/fjHP5bD4ejnacSHzcoUDwAAyaDfa1AikYhWrlyp1tZWlZWVxV5/9tlnlZeXp/Hjx2vp0qVqa2uLHauoqNCECRNUUFAQe2327NkKBALauXPnSb8rGAwqEAj0eiQCUzwAACSHPo2gSNL27dtVVlamjo4OZWRkaNWqVSotLZUkfe1rX9PIkSNVVFSkbdu2acmSJaqsrNTzzz8vSfL5fL3CiaTYc5/Pd9LvXLZsmR588MG+ltpn3VM8ERIKAACm6nNAGTt2rLZu3Sq/368//elPWrhwoTZs2KDS0lLdcccdsXYTJkxQYWGhrr32Wu3Zs0fnn39+v4tcunSp7rnnntjzQCCg4uLifn/eyXRP8XCZMQAA5urzFI/D4dCYMWM0ZcoULVu2TJMmTdIvf/nLE7adNm2aJGn37t2SJK/Xq9ra2l5tup+fbN2KJDmdztiVQ92PRLAyxQMAQFIY8D4o0WhUwWDwhMe2bt0qSSosLJQklZWVafv27aqrq4u1WbNmjdxud2yayEyW7ikeRlAAADBVn6Z4li5dqjlz5qikpETNzc1asWKF1q9fr1dffVV79uzRihUrdP311ys3N1fbtm3T3XffrauuukoTJ06UJM2aNUulpaVasGCBHnvsMfl8Pt1///0qLy+X0+lMyAn2BVM8AAAkhz4FlLq6Ot16662qqamRx+PRxIkT9eqrr+rzn/+8qqur9dprr+nf//3f1draquLiYs2bN0/3339/7P02m02rV6/WokWLVFZWpvT0dC1cuLDXvilmYooHAIDk0KeA8pvf/Oakx4qLi7Vhw4bTfsbIkSP10ksv9eVrBw1X8QAAkBy4F08PFu7FAwBAUiCg9NB9s0DyCQAA5iKg9NC9BoUpHgAAzEVA6aH7MmOmeAAAMBcBpYdjlxmbXAgAAOc4AkoPVhbJAgCQFAgoPVjZSRYAgKRAQOnB2mOKh91kAQAwDwGlh+4pHol1KAAAmImA0oP1WD5hmgcAABMRUHqw9kgoLJQFAMA8BJQemOIBACA5EFB66DXFw26yAACYhoDSQ88RFKZ4AAAwDwGlh94BxcRCAAA4xxFQeug5xRMloQAAYBoCSg9M8QAAkBwIKD30vszYxEIAADjHEVCO051RGEEBAMA8BJTjcEdjAADMR0A5Tvc0D1M8AACYh4BynNgUDwkFAADTEFCOYzs6xRMmoAAAYBoCynHSHDZJUntnxORKAAA4dxFQjhMLKKGwyZUAAHDuIqAcx5VilyS1MYICAIBpCCjHcTm7RlAIKAAAmIeAchwXa1AAADAdAeU4aUzxAABgOgLKcbpHUNo6WSQLAIBZCCjHYYoHAADzEVCO032ZcVuIgAIAgFkIKMeJTfEEmeIBAMAsBJTjZKamSJJagoygAABgFgLKcdxHA0qgI2RyJQAAnLsIKMdxp3VdZhxoJ6AAAGAWAspxjo2gsAYFAACzEFCO4047GlAYQQEAwDQElOO4U5niAQDAbASU43iOjqA0B8OKRA2TqwEA4NxEQDlOutMe+7mV7e4BADAFAeU4TrtVVkvXz2x3DwCAOQgox7FYLHI5uKMxAABmIqCcQBp3NAYAwFQElBPgjsYAAJiLgHICaSndIygEFAAAzEBAOYHYHY0JKAAAmIKAcgLdi2TbQ6xBAQDADASUE0hjBAUAAFP1KaAsX75cEydOlNvtltvtVllZmV5++eXY8Y6ODpWXlys3N1cZGRmaN2+eamtre31GVVWV5s6dK5fLpfz8fN17770Kh5NrpIJFsgAAmKtPAWXEiBF69NFHtWXLFm3evFmf+9zndOONN2rnzp2SpLvvvlsvvviinnvuOW3YsEGHDh3SzTffHHt/JBLR3Llz1dnZqbfeekvPPPOMnn76aT3wwAPxPasB6p7iaQkmV3ACAOBcYTEMY0A3nMnJydHPfvYzffnLX9awYcO0YsUKffnLX5Ykffjhh7roootUUVGh6dOn6+WXX9YXvvAFHTp0SAUFBZKkJ598UkuWLFF9fb0cDscZfWcgEJDH45Hf75fb7R5I+Sf0f/9aqV+t260F00fq4ZvGx/3zAQA4F/Xl93e/16BEIhGtXLlSra2tKisr05YtWxQKhTRz5sxYm3HjxqmkpEQVFRWSpIqKCk2YMCEWTiRp9uzZCgQCsVGYEwkGgwoEAr0eiZSf6ZQk1TV3JPR7AADAifU5oGzfvl0ZGRlyOp268847tWrVKpWWlsrn88nhcCgrK6tX+4KCAvl8PkmSz+frFU66j3cfO5lly5bJ4/HEHsXFxX0tu0+GxQJKMKHfAwAATqzPAWXs2LHaunWrNm7cqEWLFmnhwoXatWtXImqLWbp0qfx+f+xRXV2d0O8blpkqSaonoAAAYAp7X9/gcDg0ZswYSdKUKVO0adMm/fKXv9RXvvIVdXZ2qqmpqdcoSm1trbxeryTJ6/XqnXfe6fV53Vf5dLc5EafTKafT2ddS+21YRtd3NbR0Dtp3AgCAYwa8D0o0GlUwGNSUKVOUkpKitWvXxo5VVlaqqqpKZWVlkqSysjJt375ddXV1sTZr1qyR2+1WaWnpQEuJG09aiiSpPRRRZzhqcjUAAJx7+jSCsnTpUs2ZM0clJSVqbm7WihUrtH79er366qvyeDy67bbbdM899ygnJ0dut1vf+c53VFZWpunTp0uSZs2apdLSUi1YsECPPfaYfD6f7r//fpWXlw/qCMnpZKQe6xZ/eyi2JgUAAAyOPgWUuro63XrrraqpqZHH49HEiRP16quv6vOf/7wk6Re/+IWsVqvmzZunYDCo2bNn64knnoi932azafXq1Vq0aJHKysqUnp6uhQsX6qGHHorvWQ2QzWpRZqpdzR1hAgoAACYY8D4oZkj0PiiSNOOn63TgSLue//YVurQkOyHfAQDAuWRQ9kE523WvQ/G3h0yuBACAcw8B5STcqV0BJUBAAQBg0BFQToIRFAAAzENAOYnugMIICgAAg4+AchIeFyMoAACYhYByEu6je6EQUAAAGHwElJM4NsUTNrkSAADOPQSUk3CzSBYAANMQUE6Cq3gAADAPAeUksl0OSVJTG3c0BgBgsBFQTiI3oyugHG7p1BC8GwAAAEMaAeUk8jK6bhDYGYmqOchCWQAABhMB5SRSU2zKcHZdatzQwjQPAACDiYByCsemeYImVwIAwLmFgHIKueldAaWBgAIAwKAioJxC9zqUw0zxAAAwqAgop5AbCyiMoAAAMJgIKKeQl9E9xcMICgAAg4mAcgrdUzwNrYygAAAwmAgopxC7iqeZERQAAAYTAeUUctOPrkFhBAUAgEFFQDmFYZmsQQEAwAwElFPoHkHxt4fUGY6aXA0AAOcOAsopeNJSZLNaJEmNrYyiAAAwWAgop2C1WpSTznb3AAAMNgLKaWSlpUiSAu0hkysBAODcQUA5DU93QOkgoAAAMFgIKKfhPhpQ/IygAAAwaAgopxEbQWkPm1wJAADnDgLKabhT7ZIYQQEAYDARUE6DNSgAAAw+AsppdK9BaWojoAAAMFgIKKcxLLNrN9n6ZvZBAQBgsBBQTqM7oNQ1d5hcCQAA5w4CymnkZ6ZKkuoYQQEAYNAQUE4j3901gtLcEVZ7Z8TkagAAODcQUE4j02lXakpXNzHNAwDA4CCgnIbFYmGaBwCAQUZAOQP53QtlAwQUAAAGAwHlDHSvQ2GKBwCAwUFAOQPdUzy1jKAAADAoCChngBEUAAAGFwHlDBQcHUFhN1kAAAYHAeUMdI+g1AYYQQEAYDAQUM5AgZvLjAEAGEwElDPQfZlxU1tIHSF2kwUAINEIKGfAk5Yih72rq1iHAgBA4hFQzoDFYlEBV/IAADBoCChnKLbdPXuhAACQcH0KKMuWLdNll12mzMxM5efn66abblJlZWWvNtdcc40sFkuvx5133tmrTVVVlebOnSuXy6X8/Hzde++9CofDAz+bBCrgSh4AAAaNvS+NN2zYoPLycl122WUKh8P6wQ9+oFmzZmnXrl1KT0+Ptbv99tv10EMPxZ67XK7Yz5FIRHPnzpXX69Vbb72lmpoa3XrrrUpJSdFPfvKTOJxSYuSkOyRJja2dJlcCAMDZr08B5ZVXXun1/Omnn1Z+fr62bNmiq666Kva6y+WS1+s94Wf89a9/1a5du/Taa6+poKBAl1xyiR5++GEtWbJEP/7xj+VwOPpxGomX7eqq60hbyORKAAA4+w1oDYrf75ck5eTk9Hr92WefVV5ensaPH6+lS5eqra0tdqyiokITJkxQQUFB7LXZs2crEAho586dAyknobJiAYURFAAAEq1PIyg9RaNR3XXXXbryyis1fvz42Otf+9rXNHLkSBUVFWnbtm1asmSJKisr9fzzz0uSfD5fr3AiKfbc5/Od8LuCwaCCwWOLUwOBQH/L7rdsV4qkrr1QAABAYvU7oJSXl2vHjh168803e71+xx13xH6eMGGCCgsLde2112rPnj06//zz+/Vdy5Yt04MPPtjfUuMimxEUAAAGTb+meBYvXqzVq1fr9ddf14gRI07Zdtq0aZKk3bt3S5K8Xq9qa2t7tel+frJ1K0uXLpXf7489qqur+1P2gGQdHUE5wiJZAAASrk8BxTAMLV68WKtWrdK6des0atSo075n69atkqTCwkJJUllZmbZv3666urpYmzVr1sjtdqu0tPSEn+F0OuV2u3s9BtvwrDRJki/QwXb3AAAkWJ8CSnl5uX7/+99rxYoVyszMlM/nk8/nU3t7uyRpz549evjhh7VlyxZ98skn+stf/qJbb71VV111lSZOnChJmjVrlkpLS7VgwQK9//77evXVV3X//fervLxcTqcz/mcYJ8MynfKkpShqSHvqW8wuBwCAs1qfAsry5cvl9/t1zTXXqLCwMPb44x//KElyOBx67bXXNGvWLI0bN07f//73NW/ePL344ouxz7DZbFq9erVsNpvKysr09a9/XbfeemuvfVOSkcVi0YUFGZKk3XUEFAAAEqlPi2QNwzjl8eLiYm3YsOG0nzNy5Ei99NJLffnqpFDg7tru/nAL61AAAEgk7sXTB3kZXVNQDS3cjwcAgEQioPRBLtvdAwAwKAgofZCT0RVQGggoAAAkFAGlD3LTu6Z4DjPFAwBAQhFQ+qB7L5TqxrbTtAQAAANBQOmD8/PTJXVdxcOOsgAAJA4BpQ9cDntsFIXN2gAASBwCSh8Verr2QqlrZh0KAACJQkDpo+x0ruQBACDRCCh9FNsLhd1kAQBIGAJKH+XENmtjigcAgEQhoPRRDlM8AAAkHAGlj4Zldm3WxiJZAAASh4DSR0VHLzOu8bebXAkAAGcvAkofde+D4vN3KBI1TK4GAICzEwGlj/IznbJZLQpFDNUzzQMAQEIQUPrIbrMq7+hdjQkoAAAkBgGlH7JdXQGlqZ0reQAASAQCSj9kuVIkSUfaQiZXAgDA2YmA0g+xEZQ2RlAAAEgEAko/xEZQWhlBAQAgEQgo/ZDFGhQAABKKgNIPBUd3k61ubDO5EgAAzk4ElH4YP9wjSdp+0G9yJQAAnJ0IKP1QWuSWJNUGgiyUBQAgAQgo/eBy2JWZapckNXJXYwAA4o6A0k856V0LZQkoAADEHwGln7r3QiGgAAAQfwSUfuoeQTnCGhQAAOKOgNJPx0ZQ2KwNAIB4I6D0U15mV0CpDXSYXAkAAGcfAko/leS4JLFZGwAAiUBA6afugFJFQAEAIO4IKP1UnH10BOVImwzDMLkaAADOLgSUfsp3d92PpyMUVVtnxORqAAA4uxBQ+snlsCs1pav7Glq41BgAgHgioAxAbnrXKEpDa9DkSgAAOLsQUAYgN6PrUmNGUAAAiC8CygDkHt1NlhEUAADii4AyAF5PmiTpwJF2kysBAODsQkAZgPOHpUuS9ta3mlwJAABnFwLKAIw+GlD21LeYXAkAAGcXAsoAdG/WdrCJKR4AAOKJgDIAXk+qJKm5I6y2zrDJ1QAAcPYgoAxAZmqK0h02SZLPz12NAQCIFwLKABUcHUXxBQgoAADECwFlgLzuroBSS0ABACBuCCgD1B1QapjiAQAgbggoA9Q9xVNLQAEAIG4IKAPUPYLCGhQAAOKnTwFl2bJluuyyy5SZman8/HzddNNNqqys7NWmo6ND5eXlys3NVUZGhubNm6fa2tpebaqqqjR37ly5XC7l5+fr3nvvVTg8NC/TLYgFFO7HAwBAvPQpoGzYsEHl5eV6++23tWbNGoVCIc2aNUutrce2er/77rv14osv6rnnntOGDRt06NAh3XzzzbHjkUhEc+fOVWdnp9566y0988wzevrpp/XAAw/E76wGkZcpHgAA4s5iGIbR3zfX19crPz9fGzZs0FVXXSW/369hw4ZpxYoV+vKXvyxJ+vDDD3XRRRepoqJC06dP18svv6wvfOELOnTokAoKCiRJTz75pJYsWaL6+no5HI7Tfm8gEJDH45Hf75fb7e5v+XHh83do+rK1slkt+uhf58hmtZhaDwAAyaovv78HtAbF7/dLknJyciRJW7ZsUSgU0syZM2Ntxo0bp5KSElVUVEiSKioqNGHChFg4kaTZs2crEAho586dJ/yeYDCoQCDQ65Es8jIcslqkSNRQQwvTPAAAxEO/A0o0GtVdd92lK6+8UuPHj5ck+Xw+ORwOZWVl9WpbUFAgn88Xa9MznHQf7z52IsuWLZPH44k9iouL+1t23NltVg3LdEriUmMAAOKl3wGlvLxcO3bs0MqVK+NZzwktXbpUfr8/9qiurk74d/bFeblddzXeXcddjQEAiId+BZTFixdr9erVev311zVixIjY616vV52dnWpqaurVvra2Vl6vN9bm+Kt6up93tzme0+mU2+3u9Ugm47yZkqTK2maTKwEA4OzQp4BiGIYWL16sVatWad26dRo1alSv41OmTFFKSorWrl0be62yslJVVVUqKyuTJJWVlWn79u2qq6uLtVmzZo3cbrdKS0sHci6mGVPQFVD2MIICAEBc2PvSuLy8XCtWrNCf//xnZWZmxtaMeDwepaWlyePx6LbbbtM999yjnJwcud1ufec731FZWZmmT58uSZo1a5ZKS0u1YMECPfbYY/L5fLr//vtVXl4up9MZ/zMcBEXcMBAAgLjqU0BZvny5JOmaa67p9fpvf/tbfeMb35Ak/eIXv5DVatW8efMUDAY1e/ZsPfHEE7G2NptNq1ev1qJFi1RWVqb09HQtXLhQDz300MDOxEQF3DAQAIC4GtA+KGZJpn1QJOlwS1BT//U1SdJH/zpHDjt3EAAA4HiDtg8KuuS4HEqxdW3QVtfMKAoAAANFQIkDq9Wi/EymeQAAiBcCSpx035PH52c3WQAABoqAEideN1fyAAAQLwSUOOm+kqemqd3kSgAAGPoIKHFSWtS1Grlib4PJlQAAMPQRUOLk6guHSZJ2HgqorTNscjUAAAxtBJQ4yctwKDWlqzvrm1koCwDAQBBQ4sRisSgvo2ur/sMtBBQAAAaCgBJH3QGlvrnT5EoAABjaCChxxAgKAADxQUCJo3x3V0BhN1kAAAaGgBJH5+W6JEn7DreaXAkAAEMbASWORudlSCKgAAAwUASUOBo1LF1SV0AxDMPkagAAGLoIKHFUnO2SzWpRW2dEtQEWygIA0F8ElDhy2K0qyelah7L3cIvJ1QAAMHQRUOKMhbIAAAwcASXORuZ2rUOpbuSuxgAA9BcBJc5GZKdJkqqPtJlcCQAAQxcBJc6Kj65BOdBIQAEAoL8IKHHWvUi2ioACAEC/EVDirHsE5UhbSC3BsMnVAAAwNBFQ4izDaVe2K0WSVM0oCgAA/UJASQCmeQAAGBgCSgKMOBpQGEEBAKB/CCgJUJx99EqeI+yFAgBAfxBQEoApHgAABoaAkgDFOUc3ayOgAADQLwSUBOie4qk+0ibDMEyuBgCAoYeAkgBFWWmyWqSOUFT1LUGzywEAYMghoCSAw26Nbdi281DA5GoAABh6CCgJUjY6V5L05seHTa4EAIChh4CSIJNLsiRJu+tazC0EAIAhiICSIMf2QuFKHgAA+oqAkiAjemzWxpU8AAD0DQElQQqzUmW3WhQMR7W/gVEUAAD6goCSICk2q6aNzpEkvbrTZ3I1AAAMLQSUBLrmwnxJ0rtVR0yuBACAoYWAkkDjh3skSTsOshcKAAB9QUBJoIsKMyVJB5va1d4ZMbkaAACGDgJKAnnSUuRy2CRJvkCHydUAADB0EFASyGKxqNCTKkmqaWo3uRoAAIYOAkqCFXrSJEk1fkZQAAA4UwSUBCvO6QooH9U2m1wJAABDBwElwS4f1bUXSsXeBpMrAQBg6CCgJNglxdmSukZQ2PIeAIAzQ0BJsKKsVFksUkcoqobWTrPLAQBgSCCgJJjTblNBZteVPAePcCUPAABnos8B5Y033tANN9ygoqIiWSwWvfDCC72Of+Mb35DFYun1uO6663q1aWxs1Pz58+V2u5WVlaXbbrtNLS0tAzqRZDY8u2uh7P5GbhoIAMCZ6HNAaW1t1aRJk/TrX//6pG2uu+461dTUxB5/+MMfeh2fP3++du7cqTVr1mj16tV64403dMcdd/S9+iFinLdrR9kdB/0mVwIAwNBg7+sb5syZozlz5pyyjdPplNfrPeGxDz74QK+88oo2bdqkqVOnSpJ+9atf6frrr9e//du/qaioqK8lJb1LirP07MYqba1qMrsUAACGhISsQVm/fr3y8/M1duxYLVq0SA0Nxy6xraioUFZWViycSNLMmTNltVq1cePGE35eMBhUIBDo9RhKJpdkSZK2H/QrHImaWwwAAENA3APKddddp9/97ndau3atfvrTn2rDhg2aM2eOIpGum+X5fD7l5+f3eo/dbldOTo58Pt8JP3PZsmXyeDyxR3FxcbzLTqjReRnKdNrVHoqokg3bAAA4rbgHlFtuuUVf/OIXNWHCBN10001avXq1Nm3apPXr1/f7M5cuXSq/3x97VFdXx6/gQWC1WnTJ0VGUd/cfMbcYAACGgIRfZjx69Gjl5eVp9+7dkiSv16u6urpebcLhsBobG0+6bsXpdMrtdvd6DDVTRnZt2PbnrYfYsA0AgNNIeEA5cOCAGhoaVFhYKEkqKytTU1OTtmzZEmuzbt06RaNRTZs2LdHlmOYLE4tksUib9x9hmgcAgNPoc0BpaWnR1q1btXXrVknSvn37tHXrVlVVVamlpUX33nuv3n77bX3yySdau3atbrzxRo0ZM0azZ8+WJF100UW67rrrdPvtt+udd97R3//+dy1evFi33HLLWXkFT7cx+Rm6tKR72/uzd88XAADioc8BZfPmzZo8ebImT54sSbrnnns0efJkPfDAA7LZbNq2bZu++MUv6sILL9Rtt92mKVOm6G9/+5ucTmfsM5599lmNGzdO1157ra6//nrNmDFDTz31VPzOKkmNGZYhSdrNCAoAAKdkMYbggohAICCPxyO/3z+k1qP8v7f364cv7ND5w9L12j1Xy2KxmF0SAACDpi+/v7kXzyC66ZIi2a0W7alvlS/QYXY5AAAkLQLKIMpMTTl2X54G7ssDAMDJEFAGWUmOS5JURUABAOCkCCiDbFReuiTpQx8LZQEAOBkCyiC7fFSOJOm//r5Pe+u53BgAgBMhoAyyz1wwLPbzX3fVmlgJAADJi4AyyDxpKfr69BJJUk1Tu8nVAACQnAgoJhhbkClJOuTnUmMAAE6EgGKCQk/XpcYHjzCCAgDAiRBQTDDW2zWC8oEvwOXGAACcAAHFBMU5LpWNzpVhSK9X1pldDgAASYeAYpLpo3MlSSs3VWsI3g4JAICEIqCYZPb4AknSBzUBVextMLkaAACSCwHFJOO8bn12bNeeKNsO+E2uBgCA5EJAMdHlo7qmed7awwgKAAA9EVBM9Llx+ZKkNz6q1/9sOWByNQAAJA8CionGejN1cZFbkvTC1oMmVwMAQPIgoJjsZ1+eJEl6d/8RhSNRk6sBACA5EFBMNtabqcxUu1o7I/qgptnscgAASAoEFJPZrBZddl6OJGnjPhbLAgAgEVCSQndA2fRJo8mVAACQHAgoSeDyUV0B5e29jQqGIyZXAwCA+QgoSeCS4ix53anyt4e09gPuzQMAAAElCdisFs2bMlyS9O1n3+UGggCAcx4BJUl89fKS2M/3r9qhSJQbCAIAzl0ElCQxItulP5dfKUk62NSud6uOmFwRAADmIaAkkUnFWfrCxEJJXdvfAwBwriKgJJkrx+RJkjZ/wggKAODcRUBJMpcUZ0mSKvY26HcVn8gwWIsCADj3EFCSzAX5GXLau/5YHvjzTv3Pu9xEEABw7iGgJBm7zaqfzpsYe/7E67tNrAYAAHMQUJLQ3ImFynTaJUn+9hC7ywIAzjkElCSUYrPq3Qc+rwK3Uw2tnfqHJysUZV8UAMA5hICSpFJsVt1adp4kadsBv17e4TO3IAAABhEBJYktvOK82M//8+4B8woBAGCQEVCSWIbTrrXfv1qStOGjelU3tplcEQAAg4OAkuTOH5ahaaNyFIkauunXf5fP32F2SQAAJBwBZQh48MaLleG0q6G1U1f/7HV1hLiqBwBwdiOgDAHjvG49tWCKJCkYjurzv9ig3XUtJlcFAEDiEFCGiCvG5OmBL5RKkqob2zXz5xu0tbrJ3KIAAEgQAsoQ8q0Zo/TZscNiz3/26ocmVgMAQOIQUIaYf//KZP2fGaMkSX/f3aDXdtWaXBEAAPFHQBliPK4U3f+FUk0Y7pEk/Z/fbdbDq3cpHImaXBkAAPFjN7sA9M+Xp4zQ9oN+SdJv3tynD2oCSkux6bvXXqBJxVnmFgcAwAARUIaor08fKa8nVb95c5/e2deot/Y0SJLerTqi9x6YZXJ1AAAMDFM8Q5TNatHsi7367/+vTD//x0mx14+0hbR62yETKwMAYOAIKGeBmy8doZV3TFdaik2StHjFezrvvv/Vz9d8ZHJlAAD0DwHlLDF9dK52PDhbM8bkxV57fO3H8reFTKwKAID+IaCcRWxWix7/6mRdfeGxvVImPfRXnXff/+q5zdUmVgYAQN/0OaC88cYbuuGGG1RUVCSLxaIXXnih13HDMPTAAw+osLBQaWlpmjlzpj7++ONebRobGzV//ny53W5lZWXptttuU0sLW7fHQ066Q89863L973dnKN1hi71+75+26Sv/UaHXdtVqf0OriRUCAHB6fQ4ora2tmjRpkn7961+f8Phjjz2mxx9/XE8++aQ2btyo9PR0zZ49Wx0dx+7CO3/+fO3cuVNr1qzR6tWr9cYbb+iOO+7o/1ngUy4u8uhPi67QdRd7Y69t3Neo//O7zbr6Z+u5lw8AIKlZDMMw+v1mi0WrVq3STTfdJKlr9KSoqEjf//739U//9E+SJL/fr4KCAj399NO65ZZb9MEHH6i0tFSbNm3S1KlTJUmvvPKKrr/+eh04cEBFRUWn/d5AICCPxyO/3y+3293f8s8pf956UN9bubXXaxcVuvXdz43Rsxur9J3PjdG00bnmFAcAOCf05fd3XNeg7Nu3Tz6fTzNnzoy95vF4NG3aNFVUVEiSKioqlJWVFQsnkjRz5kxZrVZt3LjxhJ8bDAYVCAR6PdA3N14yXO//aJYu6bGJ2wc1AS169l29ufuwvvLU23p49S59VNusNz6q16r3DphXLADgnBfXjdp8Pp8kqaCgoNfrBQUFsWM+n0/5+fm9i7DblZOTE2tzvGXLlunBBx+MZ6nnJE9ail4ov1KtwbDue367Xny/934pv3lzn37z5r7Y85G56bq0JHuwywQAYGhcxbN06VL5/f7Yo7qaK1IGIt1p1y+/com2/XiWHvnS+JO2W/zsu7r9d5v1/LsH9NQbe7TrUEBHWjsHsVIAwLkqriMoXm/Xgsza2loVFhbGXq+trdUll1wSa1NXV9frfeFwWI2NjbH3H8/pdMrpdMaz1HOe1WqROzVF86eN1KxSr17Z6dNv39wnjytFH9Y0qz0U0SF/hw75O7QmdsfkDyVJi645X/84tVij8tJV42+X025TTrrDvJMBAJx14hpQRo0aJa/Xq7Vr18YCSSAQ0MaNG7Vo0SJJUllZmZqamrRlyxZNmTJFkrRu3TpFo1FNmzYtnuXgDA3LdGrB9JFaMH1k7LW99S267/ntemdf46faL1+/R8vX7+n12sQRHs0Yk6ebLx2u84dlqKktJJfTJqfd9qn3AwBwOn2+iqelpUW7d++WJE2ePFk///nP9dnPflY5OTkqKSnRT3/6Uz366KN65plnNGrUKP3whz/Utm3btGvXLqWmpkqS5syZo9raWj355JMKhUL65je/qalTp2rFihVnVANX8QwuwzC0u65Fq7fV6In1uxWKnNlfmdx0h741Y5T87SHNn1aij2tbNNabqeIcl8KRqOy2ITHDCACIk778/u5zQFm/fr0++9nPfur1hQsX6umnn5ZhGPrRj36kp556Sk1NTZoxY4aeeOIJXXjhhbG2jY2NWrx4sV588UVZrVbNmzdPjz/+uDIyMuJ+goiv2kCHavwduu9/tulDX7NSbJYzDiySVOB2qrTQrdcr6/VPsy5UgTtVrcGwIoY0f1qJUlMYcQGAs1VCA0oyIKAkD8Mw9Pu39ys73aGXtteoMxzVax/Unf6NJzE6L117D7fq0pIsjSt0a+ZF+RrrdWt4VprqmjskQ8p3pyoYjqg1GFFDS1B2m1Wj8tLjeFYAgEQgoMBU0aghX6BDneGoHl/3sfbUtej6CYXKdzv1py0H9PfdDf3+bJvVoosKM/WRr0WdkWjs9asvHKYZY/JU1dimiGEoKy1F35oxSm9+fFhWq0WXnZetSl+zLh+VI5cjrkuvAABniICCpPbfm6oli3RpSbZ217Xoj5uq1BqMyBfo0Dhvpv4au2ooMfIynMpypai6sU2SdGvZSKU77Wps7dSBI+1qCYb19ekj5UlLUX1zUBlOmz5zwTBtO+DX/oZWtQTD2lPfqqsvzNOk4iwZhpSb4ZDP36GRuemKRg0dbgkqL8OptlBEGc6TB6Jo1FDUMFiPA+CcQEDBkOZvCyliGMp2pag5GNZbuw/rr7tq5bTb9LXLS/RBTUB1zR1y2m1a9d5BSdLHdc19WgszmIo8qfrSpcP1X29+ovZQRF53qmaW5uu2GaN1/wvbtWnfES29fpy2H/QrGjX02XH5urQkW8U5LkWjho60dWrHoYCGZ6VpTH7XOq0PagLa39CqWaVeWa2W2Hf520OqDXRo+wG/5kzw9hot+uRwqzJT7crN4JJ9AOYgoOCc0xGKyN8e0rAMp3659mO9tL1Go4ela2HZearxdygUiSoUNRSNGsrLcKq1M6z7/mebokf/9o/zZqrAnaoNH9WbeyI9ZLtS1NwRVjg6sP9EJxVnKS3Fqrf3NirTadfM0gJtO9AkQ9KYYRnKy3TKYbNq16GAmoNhXVzUteYnN8OhvfWtCoajslst+qShVR/UBLRg+nmaPb5AH9e2aJw3U9VH2hSOGKpvCSrQHtYF+RkKdIR07bgC7Tncold2+LS3vkV3XHW+Lh+VI0kKR6KyWS3ad7hVB5vadXGR56R76XSEIgpFospMTfnUseOvBjMMQxaL5VPtACQHAgpwBiJRQ22d4U/94otEDf1992F9XNeiqy8cFgsKNqtFHleKqhratL+hTXabRRv3Nqo4J00tHWH9cXO1Jo7wqLTQrWA4qtXbarTvcKsk6bNjh+n1yno5bFZZrVJHKHqikk4qL8Opwy3BuJ27WU52HhlOu0Zkp+mThlZ1hKKyWKS0FJuGZTp14Ei7JGl8kVutnRFJkv1ouAmGj/VjTrpDhmHowoJMOexWbTvgl789pAsLMjS5OFs2m0UXFbr11u7Dqm8OasIIjy4qdKstGFZrZ0TBUESpDpsqfc3aW9+q1s6w/nn2WPnbQ2oJdn3v7roWtXWGde1FBfqgJqDReelKc9j0zr5GZaba1d4ZVY2/XVdfOEyNbZ3aXduidKddV47J0/6GVg3LdOriIo+OtHWqJMelqGGoqS2kcNRQMBTRxcM9ctqtauuM6MOagMZ53cpMtcvfHtKBI+1KTbEqL8Op6iNtagmGNWG4R5mpKTIMQ1FD6h5Ma2jtVEcoohHZLkm9g1w4EpXVYlGgIyRPWkos0PnbQnKn2fsd8MKRqMJR47RX4kWjhgx1rSfr1v1ryGKxxH5uD0VYL3YWIqAASaKqoU0FHmevDeuiUUOBjpDe3tugMfmZGpGdpnerjmhUXrqGZThV4++Qvz2knYf8ihrSrNIC5WY49cdNVXrwxV2aPjpX44d7NDwrVb+r2K/2UERTSrJ1yN+uGyYWyWqx6KHVu9QSDGtScZber26SJH37mvNV4+/QkbZOFWe79G7VEdmsFh080q5JxVkqyXHpzd2HtbuuRZKUn+lUXfOxMJGb7lDDKW51kJfh0OEWboUQD1aLYoHDYbeeNtDarBZFTjDSlum0KxSNqiMUVVqKTcFwRMc386SlyN8eij0fnpUmq1WyW6061NSucd5MvX/A3+s9TrtVwXBUJTkuleS4tKsmoMajfze87lRluVI0Y0yeDrcEVePvUEcoohSbVYdbgqoNBGWzWnTF+blKsVu146Bf+xu61oPdPHm49tS3xL6vbHSustNTZLVYlGKzxqYw0512GYah6sZ2zSzNV6EnTZW+Zv3l/UOaNipHXk+qfP4ODc9Ok9ViUYbTrs5IVFZL16jh3sOtqvF3aEx+hmwWi9IcNn1c26w0h11XXdBV997DrXrjo3qNH+7RxUVu1TcHVZSVpo9qm9UZNjQs06kvTR6ujlBEe+pbZBhdf1apKTalpdi093CLNlTWa2t1k5x2q/7vP16ijlBEu2oCOtTUrtpAUA67RV+cVKRgOKpsl0OBjpA6w1F5Paly2m1aX1mng0fadfXYYUp32NXQ2rW2bXh2mvY3tMkwDAU6wirJcSkYjqrQk6qOUEQNLZ0qyXWpoaVThZ5U5aQ79LePDys3w6EJwz1qaO2U025VusOuGn+7/O0hTRyRpWD4WCjcsr9RlxRn9wqS8UBAAdBLJGr06R+aaNSQxdL1vo9qW3RRYWbs/6xfr6zT/6vYrx/fcLFSU6wKdIRja2M6w1H9fc9hXViQqbf3NOiascOUmZqi2kCHXt5RoxHZXSEo0B7SRYVujfNm6orz87Tuwzq9utOntBSb8t1O5WU4NWVktlqDYbnTUtTWGdaGynplpNqV4UzRW3sOa/W2GjlsVkUNQyOy03TlmDw57Fb97eOukJWX4ZAnLUWdkahyXA75Ah0qznbJUNcv4cbWTlU1tqmxtTMW5kZkpWnf4VbVtwRV3yOcXXXhMNks0v6GNu09Oio2PCtNB5vaT9h/dqvlpFNzJwsTZ8Jhs/a6eg2IlxP9vbzn8xfqu9deENfvIaAAQB+caO1Kpa9ZdptFVovlU/vsRKOGrFaLfP4OvbXnsMZ53bqoMFN1zUEVuLt2zH6/ukkpNmvXZfG1Lcp2pSg73RELLy/v8Gl4VqrGet3KcNr1XtURtYciGp2XIUOGvO5U1TUHVRcIqiTXJXeqXZGooY5wVOkOm1o7I9r8SaPqAkHlZTp0sKlD26qbVOBOVWmRW5NLsvTJ4a6RicbWTvkCHbqk2KMt+4+oMxxVXkbX9FmWK0UF7lRt/qRRl43KUU66Q3Zr12jHlv1HVOhJlWFIKTaLWoJh+du7pqTsVosslq7XLszPUNSQPqptVmtnROs/rNNlo3JUkuOSvz2kz1yQp6ghHTjSphSbVQeb2mW1dN24dE9di177oE5XjslVkSdNoUhUrZ0R/e3jet04abgihqHG1k4VuJ0amZuuxtZOvfZBrWqaOuSwW1U2OleGDH3oa1ZrMKLPXJCnzfsb1d4Z0RcmFsmTlqL3qpvUEYro4JF2RaKGaps7ZLce22TS5bBpTH6GqhvbZLdZVd8cjPWxJJ0/LF176ltV4HaqNtB7ijIvw6FgKKqMVLsKPanyt4dkGNLIXJdq/B360Nccazsy1yWb1aK99V0h1261KMvl+NS0Z066IzYidSZG5aVr3+FWWSxSz9/oxz8/me4Ru+N944rz9KMbSuO6rouAAgDASXSEIkpNsamuuUNZaQ457McWWkejhiKGoZSTXPrfEgzLYbMenXqLnNHu18bRtUbZJ1kI3h14j/+ejKNTWZK0p75Vo/PStb+xrStIWS060hbSnPFdV/J1hCJqCYaVmWpXQ0unPGkpSnd2hdqWjrAyUu0KR6NqaguprTMil8Mmf3vXOqQsV4p8/g65HHb5/B1q6wzLbrNoysicM+3SM0ZAAQAASacvv7/ZHQoAACQdAgoAAEg6BBQAAJB0CCgAACDpEFAAAEDSIaAAAICkQ0ABAABJh4ACAACSDgEFAAAkHQIKAABIOgQUAACQdAgoAAAg6RBQAABA0rGbXUB/dN+AORAImFwJAAA4U92/t7t/j5/KkAwozc3NkqTi4mKTKwEAAH3V3Nwsj8dzyjYW40xiTJKJRqM6dOiQMjMzZbFY4vrZgUBAxcXFqq6ultvtjutn4xj6eXDQz4ODfh489PXgSFQ/G4ah5uZmFRUVyWo99SqTITmCYrVaNWLEiIR+h9vt5i//IKCfBwf9PDjo58FDXw+ORPTz6UZOurFIFgAAJB0CCgAASDoElOM4nU796Ec/ktPpNLuUsxr9PDjo58FBPw8e+npwJEM/D8lFsgAA4OzGCAoAAEg6BBQAAJB0CCgAACDpEFAAAEDSIaD08Otf/1rnnXeeUlNTNW3aNL3zzjtmlzSkLFu2TJdddpkyMzOVn5+vm266SZWVlb3adHR0qLy8XLm5ucrIyNC8efNUW1vbq01VVZXmzp0rl8ul/Px83XvvvQqHw4N5KkPKo48+KovForvuuiv2Gv0cHwcPHtTXv/515ebmKi0tTRMmTNDmzZtjxw3D0AMPPKDCwkKlpaVp5syZ+vjjj3t9RmNjo+bPny+3262srCzddtttamlpGexTSWqRSEQ//OEPNWrUKKWlpen888/Xww8/3Ot+LfR1373xxhu64YYbVFRUJIvFohdeeKHX8Xj16bZt2/SZz3xGqampKi4u1mOPPRafEzBgGIZhrFy50nA4HMZ//dd/GTt37jRuv/12Iysry6itrTW7tCFj9uzZxm9/+1tjx44dxtatW43rr7/eKCkpMVpaWmJt7rzzTqO4uNhYu3atsXnzZmP69OnGFVdcETseDoeN8ePHGzNnzjTee+8946WXXjLy8vKMpUuXmnFKSe+dd94xzjvvPGPixInG9773vdjr9PPANTY2GiNHjjS+8Y1vGBs3bjT27t1rvPrqq8bu3btjbR599FHD4/EYL7zwgvH+++8bX/ziF41Ro0YZ7e3tsTbXXXedMWnSJOPtt982/va3vxljxowxvvrVr5pxSknrkUceMXJzc43Vq1cb+/btM5577jkjIyPD+OUvfxlrQ1/33UsvvWT8y7/8i/H8888bkoxVq1b1Oh6PPvX7/UZBQYExf/58Y8eOHcYf/vAHIy0tzfiP//iPAddPQDnq8ssvN8rLy2PPI5GIUVRUZCxbtszEqoa2uro6Q5KxYcMGwzAMo6mpyUhJSTGee+65WJsPPvjAkGRUVFQYhtH1H5TVajV8Pl+szfLlyw23220Eg8HBPYEk19zcbFxwwQXGmjVrjKuvvjoWUOjn+FiyZIkxY8aMkx6PRqOG1+s1fvazn8Vea2pqMpxOp/GHP/zBMAzD2LVrlyHJ2LRpU6zNyy+/bFgsFuPgwYOJK36ImTt3rvGtb32r12s333yzMX/+fMMw6Ot4OD6gxKtPn3jiCSM7O7vXvxtLliwxxo4dO+CameKR1NnZqS1btmjmzJmx16xWq2bOnKmKigoTKxva/H6/JCknJ0eStGXLFoVCoV79PG7cOJWUlMT6uaKiQhMmTFBBQUGszezZsxUIBLRz585BrD75lZeXa+7cub36U6Kf4+Uvf/mLpk6dqn/4h39Qfn6+Jk+erP/8z/+MHd+3b598Pl+vfvZ4PJo2bVqvfs7KytLUqVNjbWbOnCmr1aqNGzcO3skkuSuuuEJr167VRx99JEl6//339eabb2rOnDmS6OtEiFefVlRU6KqrrpLD4Yi1mT17tiorK3XkyJEB1TgkbxYYb4cPH1YkEun1j7UkFRQU6MMPPzSpqqEtGo3qrrvu0pVXXqnx48dLknw+nxwOh7Kysnq1LSgokM/ni7U50Z9D9zF0Wblypd59911t2rTpU8fo5/jYu3evli9frnvuuUc/+MEPtGnTJn33u9+Vw+HQwoULY/10on7s2c/5+fm9jtvtduXk5NDPPdx3330KBAIaN26cbDabIpGIHnnkEc2fP1+S6OsEiFef+nw+jRo16lOf0X0sOzu73zUSUJAQ5eXl2rFjh958802zSznrVFdX63vf+57WrFmj1NRUs8s5a0WjUU2dOlU/+clPJEmTJ0/Wjh079OSTT2rhwoUmV3d2+e///m89++yzWrFihS6++GJt3bpVd911l4qKiujrcxhTPJLy8vJks9k+dZVDbW2tvF6vSVUNXYsXL9bq1av1+uuva8SIEbHXvV6vOjs71dTU1Kt9z372er0n/HPoPoauKZy6ujpdeumlstvtstvt2rBhgx5//HHZ7XYVFBTQz3FQWFio0tLSXq9ddNFFqqqqknSsn07174bX61VdXV2v4+FwWI2NjfRzD/fee6/uu+8+3XLLLZowYYIWLFigu+++W8uWLZNEXydCvPo0kf+WEFAkORwOTZkyRWvXro29Fo1GtXbtWpWVlZlY2dBiGIYWL16sVatWad26dZ8a9psyZYpSUlJ69XNlZaWqqqpi/VxWVqbt27f3+o9izZo1crvdn/plca669tprtX37dm3dujX2mDp1qubPnx/7mX4euCuvvPJTl8l/9NFHGjlypCRp1KhR8nq9vfo5EAho48aNvfq5qalJW7ZsibVZt26dotGopk2bNghnMTS0tbXJau3968hmsykajUqirxMhXn1aVlamN954Q6FQKNZmzZo1Gjt27ICmdyRxmXG3lStXGk6n03j66aeNXbt2GXfccYeRlZXV6yoHnNqiRYsMj8djrF+/3qipqYk92traYm3uvPNOo6SkxFi3bp2xefNmo6yszCgrK4sd7778ddasWcbWrVuNV155xRg2bBiXv55Gz6t4DIN+jod33nnHsNvtxiOPPGJ8/PHHxrPPPmu4XC7j97//fazNo48+amRlZRl//vOfjW3bthk33njjCS/TnDx5srFx40bjzTffNC644IJz+tLXE1m4cKExfPjw2GXGzz//vJGXl2f88z//c6wNfd13zc3NxnvvvWe89957hiTj5z//ufHee+8Z+/fvNwwjPn3a1NRkFBQUGAsWLDB27NhhrFy50nC5XFxmHG+/+tWvjJKSEsPhcBiXX3658fbbb5td0pAi6YSP3/72t7E27e3txre//W0jOzvbcLlcxpe+9CWjpqam1+d88sknxpw5c4y0tDQjLy/P+P73v2+EQqFBPpuh5fiAQj/Hx4svvmiMHz/ecDqdxrhx44ynnnqq1/FoNGr88Ic/NAoKCgyn02lce+21RmVlZa82DQ0Nxle/+lUjIyPDcLvdxje/+U2jubl5ME8j6QUCAeN73/ueUVJSYqSmphqjR482/uVf/qXXpav0dd+9/vrrJ/w3eeHChYZhxK9P33//fWPGjBmG0+k0hg8fbjz66KNxqd9iGD226gMAAEgCrEEBAABJh4ACAACSDgEFAAAkHQIKAABIOgQUAACQdAgoAAAg6RBQAABA0iGgAACApENAAQAASYeAAgAAkg4BBQAAJB0CCgAASDr/P6QPIigp0hdkAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Let's inpsect the training loss\n",
    "plt.figure()\n",
    "plt.plot(all_losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "655eb366",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--random sampling--\n",
      "letícia\n",
      "constança\n",
      "matias\n",
      "\n",
      "--top-k sampling--\n",
      "matheledanzozo\n",
      "miguiss\n",
      "luasau\n"
     ]
    }
   ],
   "source": [
    "# Sample from a category and starting letter\n",
    "def sampling_names(start_string='', method=\"sample\", kk=1, max_length=20, vocabulary=all_letters):\n",
    "    with torch.no_grad():  # no need to track history in sampling\n",
    "        hidden = model.initial_hidden\n",
    "\n",
    "        # process each letter of start_string, including BOS\n",
    "        output_name = start_string\n",
    "        input = inputTensor(start_string)\n",
    "        if input.shape[0] > 1:\n",
    "            for input_t in input[:-1].split(1):\n",
    "                _, hidden = model.single_forward(input_t[0], hidden)\n",
    "\n",
    "        # start sampling from the last letter of start_string\n",
    "        for _ in range(max_length):\n",
    "            output, hidden = model.single_forward(input[-1], hidden)\n",
    "            # top-k sampling\n",
    "            if method == \"top_k\":\n",
    "                # YOUR CODE HERE\n",
    "                _, topi = output.topk(kk)\n",
    "                topi = choices(topi[0])[0]\n",
    "            # random sampling\n",
    "            elif method == \"sample\":\n",
    "                # YOUR CODE HERE\n",
    "                topi = torch.multinomial(output.exp(), 1)[0]\n",
    "\n",
    "            # stop sampling if EOS\n",
    "            if topi == n_letters - 1:\n",
    "                break\n",
    "            # ignore BOS\n",
    "            elif topi == n_letters - 2:\n",
    "                continue\n",
    "            # add letter to output_name\n",
    "            else:\n",
    "                letter = vocabulary[topi]\n",
    "                output_name += letter\n",
    "            # use the sampled letter as the next input\n",
    "            input = inputTensor(letter)\n",
    "\n",
    "        return output_name\n",
    "\n",
    "starting_string = ''\n",
    "print(\"--random sampling--\")\n",
    "# generate 3 random names with random sampling\n",
    "# note how it is only copying the names from the training set\n",
    "for _ in range(10):\n",
    "    print(sampling_names(starting_string, method=\"sample\"))\n",
    "print(\"\\n--top-k sampling--\")\n",
    "# generate 3 random names with random sampling\n",
    "for _ in range(10):\n",
    "    print(sampling_names(starting_string, method=\"top_k\", kk=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de9f503",
   "metadata": {},
   "source": [
    "## Question 03"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6d3482bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: datasets in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (2.7.1)\n",
      "Requirement already satisfied: numpy>=1.17 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (1.23.3)\n",
      "Requirement already satisfied: pyarrow>=6.0.0 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (10.0.1)\n",
      "Requirement already satisfied: dill<0.3.7 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (0.3.6)\n",
      "Requirement already satisfied: pandas in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (1.5.0)\n",
      "Requirement already satisfied: requests>=2.19.0 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (2.28.1)\n",
      "Requirement already satisfied: tqdm>=4.62.1 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (4.62.3)\n",
      "Requirement already satisfied: xxhash in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (3.1.0)\n",
      "Requirement already satisfied: multiprocess in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (0.70.14)\n",
      "Requirement already satisfied: fsspec[http]>=2021.11.1 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (2022.11.0)\n",
      "Requirement already satisfied: aiohttp in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (3.8.3)\n",
      "Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (0.11.1)\n",
      "Requirement already satisfied: packaging in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (21.3)\n",
      "Requirement already satisfied: responses<0.19 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (0.18.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from datasets) (6.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from aiohttp->datasets) (22.1.0)\n",
      "Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from aiohttp->datasets) (2.1.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from aiohttp->datasets) (6.0.3)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from aiohttp->datasets) (4.0.2)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from aiohttp->datasets) (1.8.2)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from aiohttp->datasets) (1.3.3)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from aiohttp->datasets) (1.3.1)\n",
      "Requirement already satisfied: filelock in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.8.2)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (4.4.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from packaging->datasets) (3.0.9)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from requests>=2.19.0->datasets) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from requests>=2.19.0->datasets) (1.26.13)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from requests>=2.19.0->datasets) (2022.12.7)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from pandas->datasets) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from pandas->datasets) (2022.2.1)\n",
      "Requirement already satisfied: six>=1.5 in /Users/gmc/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e9548318",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using the latest cached version of the module from /Users/gmc/.cache/huggingface/modules/datasets_modules/datasets/imdb/d613c88cf8fa3bab83b4ded3713f1f74830d1100e171db75bbddb80b3345c9c0 (last modified on Sun Oct 29 20:25:52 2023) since it couldn't be found locally at imdb., or remotely on the Hugging Face Hub.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'text': Value(dtype='string', id=None),\n",
       " 'label': ClassLabel(names=['neg', 'pos'], id=None)}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset_builder\n",
    "from datasets import load_dataset\n",
    "\n",
    "\n",
    "ds_builder = load_dataset_builder(\"imdb\")\n",
    "ds_builder.info.features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d40ac1de",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using the latest cached version of the module from /Users/gmc/.cache/huggingface/modules/datasets_modules/datasets/imdb/d613c88cf8fa3bab83b4ded3713f1f74830d1100e171db75bbddb80b3345c9c0 (last modified on Sun Oct 29 20:25:52 2023) since it couldn't be found locally at imdb., or remotely on the Hugging Face Hub.\n",
      "Found cached dataset imdb (/Users/gmc/.cache/huggingface/datasets/imdb/plain_text/1.0.0/d613c88cf8fa3bab83b4ded3713f1f74830d1100e171db75bbddb80b3345c9c0)\n",
      "100%|██████████| 3/3 [00:00<00:00, 507.44it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: 25000\n",
      "Number of test examples: 25000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "dataset = load_dataset(\"imdb\")\n",
    "\n",
    "\n",
    "print(f'Number of training examples: {len(dataset[\"train\"])}')\n",
    "print(f'Number of test examples: {len(dataset[\"test\"])}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "142e4ab8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Error loading punkt: <urlopen error [Errno 8] nodename nor\n",
      "[nltk_data]     servname provided, or not known>\n"
     ]
    }
   ],
   "source": [
    "# get a vocabulary of the top K words in the dataset, without stopwords\n",
    "from collections import Counter\n",
    "import nltk\n",
    "nltk.download('punkt')\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "def get_vocab(dataset, vocab_size=1024):\n",
    "    counter = Counter()\n",
    "    for example in dataset:\n",
    "        counter.update([word for word in word_tokenize(example['text'].lower())])\n",
    "    vocab = counter.most_common(vocab_size)\n",
    "    return [word for word, _ in vocab]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b2cbbc1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a pytorch dataset\n",
    "import torch\n",
    "from torch.utils.data import Dataset\n",
    "from collections import defaultdict\n",
    "\n",
    "class IMDBDataset(Dataset):\n",
    "    def __init__(self, dataset, vocab, split=\"train\", first_n_words=128):\n",
    "        self.dataset = dataset\n",
    "        self.vocab = vocab\n",
    "        self.unk_idx = 0\n",
    "        self.pad_idx = 1\n",
    "        self.first_n_words = first_n_words\n",
    "        # use a special dict that maps words to indices and if the word is not\n",
    "        # in the vocab, the special dict automatically returns the index of the\n",
    "        # unknown token\n",
    "        self.word2idx = defaultdict(lambda: self.unk_idx)\n",
    "        self.word2idx['<pad>'] = self.pad_idx\n",
    "        # add the words to the dict, starting from index 2 (0 is for the unknown token,\n",
    "        # 1 is for the padding token)\n",
    "        self.word2idx.update({word: idx + 2 for idx, word in enumerate(self.vocab)})\n",
    "        self.idx2word = {idx: word for word, idx in self.word2idx.items()}\n",
    "        \n",
    "        self.split = split\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.dataset)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        example = self.dataset[idx]\n",
    "        text = word_tokenize(example['text'])\n",
    "        text = [self.word2idx[word] for word in text][:self.first_n_words]\n",
    "        return torch.LongTensor(text), example['label']\n",
    "    \n",
    "    def collate_fn(self, batch):\n",
    "        \"\"\"\n",
    "        This function takes a list of tokens, already as the index of a vocabulary\n",
    "        and turns it into a tensor, with padding where appropriate.\n",
    "        The padding index is stored in self.pad_idx\n",
    "\n",
    "        Args:\n",
    "            batch (list): a list of tuples,\n",
    "                where each tuple has the tokens for one sequence in position 0\n",
    "                and the label for that sequence in position 1.\n",
    "\n",
    "        Returns:\n",
    "            torch.tensor: return the padded tensor and the labels in tensor form.\n",
    "                The padded tensor is of size [batch_size, max_len],\n",
    "                the labels tensor is [batch_size]\n",
    "        \"\"\"\n",
    "        # YOUR CODE HERE\n",
    "        # get the max length of the sentences in the batch\n",
    "        max_len = max([len(text_tensor) for text_tensor, _ in batch])\n",
    "        # pad all the sentences to the max length\n",
    "        padded_tensors = []\n",
    "        labels = []\n",
    "        for text_tensor, label in batch:\n",
    "            # 1 is the index of the padding token\n",
    "            padded_tensor = torch.ones(max_len, dtype=torch.long) * self.pad_idx\n",
    "            padded_tensor[:len(text_tensor)] = text_tensor\n",
    "            padded_tensors.append(padded_tensor)\n",
    "            labels.append(label)\n",
    "        # convert the lists to tensors\n",
    "        padded_tensors = torch.stack(padded_tensors)\n",
    "        return padded_tensors, torch.LongTensor(labels)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "a7bfa34d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, dropout=0.3):\n",
    "\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.rnn = nn.LSTM(embedding_dim, hidden_dim, batch_first=True)\n",
    "        self.fc = nn.Linear(hidden_dim, output_dim)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, text):\n",
    "        # YOUR CODE HERE\n",
    "\n",
    "        # pass the tokens through the embedding layer\n",
    "        emb = self.embedding(text)\n",
    "        # apply dropout to the embeddings\n",
    "        emb = self.dropout(emb)\n",
    "\n",
    "        # pass it simply through the LSTM initialized above\n",
    "        # (you don't need a for loop, nn.LSTM takes care of that for you)\n",
    "        enc_output, _ = self.rnn(emb)\n",
    "\n",
    "        # apply dropout to the output\n",
    "        enc_output = self.dropout(enc_output)\n",
    "\n",
    "        # Take the output of the last token of the sequence\n",
    "        hidden = enc_output[:, -1, :]\n",
    "\n",
    "        # pass it through the fully connected\n",
    "        return self.fc(hidden)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "170a444f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def binary_accuracy(preds, y):\n",
    "    \"\"\"\n",
    "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
    "    \"\"\"\n",
    "\n",
    "    #round predictions to the closest integer\n",
    "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
    "\n",
    "    correct = (rounded_preds == y).float() #convert into float for division \n",
    "\n",
    "    acc = correct.sum() / len(correct)\n",
    "    return acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c768bb2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "def train(model, train_daloader, optimizer, criterion):\n",
    "\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "\n",
    "    model.train()\n",
    "\n",
    "    for text_tensors, targets in tqdm(train_daloader):\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        predictions = model(text_tensors).squeeze(1)\n",
    "\n",
    "        loss = criterion(predictions, targets.float())\n",
    "\n",
    "        acc = binary_accuracy(predictions, targets)\n",
    "\n",
    "        loss.backward()\n",
    "\n",
    "        optimizer.step()\n",
    "\n",
    "        epoch_loss += loss.item()\n",
    "        epoch_acc += acc.item()\n",
    "\n",
    "    return epoch_loss / len(train_daloader), epoch_acc / len(train_daloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "39297ade",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, eval_dataloader, criterion):\n",
    "\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "\n",
    "    model.eval()\n",
    "\n",
    "    with torch.no_grad():\n",
    "\n",
    "        for text_tensors, targets in tqdm(eval_dataloader):\n",
    "\n",
    "            predictions = model(text_tensors).squeeze(1)\n",
    "\n",
    "            loss = criterion(predictions, targets.float())\n",
    "\n",
    "            acc = binary_accuracy(predictions, targets)\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_acc += acc.item()\n",
    "\n",
    "    return epoch_loss / len(eval_dataloader), epoch_acc / len(eval_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "c5aaef9b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:38<00:00,  3.19it/s]\n",
      "100%|██████████| 79/79 [00:16<00:00,  4.83it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01\n",
      "\tTrain Loss: 0.692 | Train Acc: 52.03%\n",
      "\t Val. Loss: 0.689 |  Val. Acc: 54.53%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:38<00:00,  3.18it/s]\n",
      "100%|██████████| 79/79 [00:16<00:00,  4.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02\n",
      "\tTrain Loss: 0.687 | Train Acc: 55.16%\n",
      "\t Val. Loss: 0.679 |  Val. Acc: 56.96%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:38<00:00,  3.18it/s]\n",
      "100%|██████████| 79/79 [00:16<00:00,  4.80it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03\n",
      "\tTrain Loss: 0.688 | Train Acc: 54.76%\n",
      "\t Val. Loss: 0.685 |  Val. Acc: 58.78%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:38<00:00,  3.17it/s]\n",
      "100%|██████████| 79/79 [00:16<00:00,  4.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04\n",
      "\tTrain Loss: 0.681 | Train Acc: 56.32%\n",
      "\t Val. Loss: 0.669 |  Val. Acc: 60.15%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:38<00:00,  3.17it/s]\n",
      "100%|██████████| 79/79 [00:16<00:00,  4.78it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05\n",
      "\tTrain Loss: 0.663 | Train Acc: 60.83%\n",
      "\t Val. Loss: 0.660 |  Val. Acc: 61.63%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "from torch import optim\n",
    "\n",
    "# set the seed for reproducibility\n",
    "torch.manual_seed(42)\n",
    "\n",
    "VOCAB_SIZE = 1024\n",
    "FIRST_N_WORDS = 128\n",
    "BATCH_SIZE = 64\n",
    "INPUT_DIM = VOCAB_SIZE + 2  # +2 for the unknown and padding tokens\n",
    "EMBEDDING_DIM = 128\n",
    "HIDDEN_DIM = 256\n",
    "OUTPUT_DIM = 1\n",
    "DROPOUT_RATE = 0.2\n",
    "LR_RATE = 1e-4\n",
    "\n",
    "vocab = get_vocab(dataset['train'], vocab_size=1024)\n",
    "# use 80% of the training set for training and 20% for validation\n",
    "train_size = int(0.8 * len(dataset['train']))\n",
    "valid_size = len(dataset['train']) - train_size\n",
    "# split the training set\n",
    "train_dataset, valid_dataset = torch.utils.data.random_split(dataset['train'], [train_size, valid_size])\n",
    "# create the pytorch datasets\n",
    "train_dataset = IMDBDataset(train_dataset, vocab, split=\"train\", first_n_words=FIRST_N_WORDS)\n",
    "valid_dataset = IMDBDataset(valid_dataset, vocab, split=\"valid\")\n",
    "test_dataset = IMDBDataset(dataset['test'], vocab, split=\"test\")\n",
    "\n",
    "train_loader = DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, collate_fn=train_dataset.collate_fn)\n",
    "valid_loader = DataLoader(valid_dataset, batch_size=BATCH_SIZE, collate_fn=valid_dataset.collate_fn)\n",
    "test_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, collate_fn=test_dataset.collate_fn)\n",
    "\n",
    "model = RNN(INPUT_DIM,\n",
    "            EMBEDDING_DIM,\n",
    "            HIDDEN_DIM,\n",
    "            OUTPUT_DIM,\n",
    "            dropout=DROPOUT_RATE)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=LR_RATE)\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "\n",
    "# Train for 5 epochs\n",
    "for epoch in range(5):\n",
    "\n",
    "    train_loss, train_acc = train(model, train_loader, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_loader, criterion)\n",
    "\n",
    "    print(f'Epoch: {epoch+1:02}')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "2248ea09",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 391/391 [01:21<00:00,  4.81it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t Test Loss: 0.663 |  Test Acc: 61.53%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = evaluate(model, test_loader, criterion)\n",
    "print(f'\\t Test Loss: {test_loss:.3f} |  Test Acc: {test_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb39e81b",
   "metadata": {},
   "source": [
    "Let's do some changes to use a bidirectional LSTM and implement some other improvements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "7662d89f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers, bidirectional, dropout):\n",
    "\n",
    "        super().__init__()\n",
    "\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.bidirectional = bidirectional\n",
    "\n",
    "        self.rnn = nn.LSTM(embedding_dim,\n",
    "                            hidden_dim//2 if bidirectional else hidden_dim,\n",
    "                            num_layers=n_layers,\n",
    "                            bidirectional=bidirectional,\n",
    "                            batch_first=True)\n",
    "\n",
    "        self.fc = nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, text):\n",
    "        # YOUR CODE HERE\n",
    "\n",
    "        #text = [batch size, sent len]\n",
    "        #embedded = [batch size, sent len, emb dim]\n",
    "        embedded = self.dropout(self.embedding(text))\n",
    "        embedded = self.dropout(embedded)\n",
    "\n",
    "        # output = [batch size, sent len, hid dim * num directions]\n",
    "        # hidden = [num layers * num directions, batch size, hid dim]\n",
    "        # cell = [num layers * num directions, batch size, hid dim]\n",
    "        enc_output, (hidden, _) = self.rnn(embedded)\n",
    "\n",
    "        #hidden = [batch size, hid dim * num directions]\n",
    "        if self.bidirectional:\n",
    "            #concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers\n",
    "            hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1))\n",
    "        else:\n",
    "            hidden = self.dropout(hidden[-1,:,:])\n",
    "                \n",
    "        return self.fc(hidden)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f37cf56c",
   "metadata": {},
   "outputs": [],
   "source": [
    "VOCAB_SIZE = 1024\n",
    "FIRST_N_WORDS = 256\n",
    "BATCH_SIZE = 64\n",
    "INPUT_DIM = VOCAB_SIZE + 2  # +2 for the unknown and padding tokens\n",
    "EMBEDDING_DIM = 128\n",
    "HIDDEN_DIM = 256\n",
    "OUTPUT_DIM = 1\n",
    "N_LAYERS = 1\n",
    "BIDIRECTIONAL = True\n",
    "DROPOUT_RATE = 0.2\n",
    "LR_RATE = 1e-4\n",
    "EMBEDDING_DIM = 128\n",
    "\n",
    "model = RNN(INPUT_DIM,\n",
    "            EMBEDDING_DIM,\n",
    "            HIDDEN_DIM,\n",
    "            OUTPUT_DIM,\n",
    "            N_LAYERS,\n",
    "            BIDIRECTIONAL,\n",
    "            DROPOUT_RATE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "7ea38e5b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:36<00:00,  3.26it/s]\n",
      "100%|██████████| 79/79 [00:15<00:00,  5.23it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01\n",
      "\tTrain Loss: 0.692 | Train Acc: 51.88%\n",
      "\t Val. Loss: 0.687 |  Val. Acc: 55.46%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:36<00:00,  3.24it/s]\n",
      "100%|██████████| 79/79 [00:15<00:00,  5.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 02\n",
      "\tTrain Loss: 0.687 | Train Acc: 54.96%\n",
      "\t Val. Loss: 0.682 |  Val. Acc: 56.41%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:36<00:00,  3.23it/s]\n",
      "100%|██████████| 79/79 [00:15<00:00,  5.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 03\n",
      "\tTrain Loss: 0.677 | Train Acc: 57.29%\n",
      "\t Val. Loss: 0.665 |  Val. Acc: 60.09%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:36<00:00,  3.24it/s]\n",
      "100%|██████████| 79/79 [00:15<00:00,  5.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 04\n",
      "\tTrain Loss: 0.658 | Train Acc: 61.00%\n",
      "\t Val. Loss: 0.722 |  Val. Acc: 60.78%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 313/313 [01:36<00:00,  3.24it/s]\n",
      "100%|██████████| 79/79 [00:15<00:00,  5.02it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 05\n",
      "\tTrain Loss: 0.628 | Train Acc: 65.24%\n",
      "\t Val. Loss: 0.667 |  Val. Acc: 65.61%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "optimizer = optim.Adam(model.parameters(), lr=LR_RATE)\n",
    "\n",
    "# Train for 5 epochs\n",
    "for epoch in range(5):\n",
    "\n",
    "    train_loss, train_acc = train(model, train_loader, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_loader, criterion)\n",
    "\n",
    "    print(f'Epoch: {epoch+1:02}')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "e6adb7b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 391/391 [01:16<00:00,  5.13it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t Test Loss: 0.659 |  Test Acc: 65.82%\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = evaluate(model, test_loader, criterion)\n",
    "print(f'\\t Test Loss: {test_loss:.3f} |  Test Acc: {test_acc*100:.2f}%')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "DSLabPreprocess",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "8a96707b1aefed7c05ab8288ccb93c4e2b95de2651b5fe43ac3cd3e321955411"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}