minGPT-Fastai_Play_Char.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Quick Demo of Andrej Karpathy's minGPT Play Char Demo \n",
    "\n",
    "**Our Goal**: Generate Shakesperean text\n",
    "- You can find the original Play Char demo in the minGPT repo here: https://github.com/karpathy/minGPT\n",
    "- This notebook is partially based on the fastai Transformers tutorial: http://docs.fast.ai/tutorial.transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For Colab\n",
    "# !pip install fastai --upgrade\n",
    "# !git clone https://github.com/karpathy/minGPT\n",
    "# !pip install snakeviz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text.all import *\n",
    "from minGPT.mingpt.model import GPT, GPTConfig, GPT1Config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the raw text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read it in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'First Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_text = open('input.txt', 'r').read()\n",
    "raw_text[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loaders\n",
    "\n",
    "We re-implement Karpathy's `CharDataset` as a fastai Transform, which will then be passed to a dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CharTransform(DisplayedTransform):\n",
    "    def __init__(self, data, block_size):\n",
    "        self.cat = Categorize()\n",
    "        self.block_size = block_size + 1\n",
    "        self.data = data\n",
    "        \n",
    "    def setups(self, items=None):\n",
    "        self.data_len = len(self.data)\n",
    "        self.cat.setup(L(*self.data))\n",
    "        self.itos = self.cat.vocab\n",
    "        self.stoi = self.itos.items.val2idx()\n",
    "        print(f'data has {self.data_len} characters, {len(self.itos)} unique.')\n",
    "        self.n_sequences = math.ceil(self.data_len / (self.block_size))\n",
    "        self.idxs = L(np.random.randint(0, self.data_len - (self.block_size), self.n_sequences).tolist())\n",
    "\n",
    "    def encodes(self, o):\n",
    "        chunk = self.data[self.idxs[o]:self.idxs[o]+self.block_size]\n",
    "        return tensor([self.stoi[s] for s in chunk])\n",
    "\n",
    "    def decodes(self, o): return TitledStr(''.join([self.itos[s.item()] for s in o]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define out block_size and n_samples (which is the number of batches in a single epoch)\n",
    "\n",
    "**Note**:\n",
    "- Note `block_size` in Karpathy's code is equivalent to `Sequence Length` in fastai\n",
    "- We do not specify a validation set here as Karpathy does not in their notebook. Therefore we set `split_idx=0` in `TfmdLists`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8647"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "block_size=128\n",
    "n_sequences = math.ceil(len(raw_text) / (block_size+1)); n_sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we test our transform quickly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data has 1115394 characters, 65 unique.\n"
     ]
    }
   ],
   "source": [
    "t = CharTransform(data=raw_text, block_size=block_size)\n",
    "t.setups()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we add our Transform to a `Dataset`, read more about fastai's `Dataset` [here](https://docs.fast.ai/data.core#Datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data has 1115394 characters, 65 unique.\n"
     ]
    }
   ],
   "source": [
    "dset = Datasets(L(range(n_sequences)), tfms=[CharTransform(raw_text, block_size)], dl_type=LMDataLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use an `LMDataLoader` since we have a language modeling problem, meaning it is designed to be used when trying to predict the next character/token.\n",
    "\n",
    "Lets have a look at the first item in our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([53, 58, 46,  1, 46, 43, 11,  0, 13, 52, 42,  6,  1, 54, 56, 43, 58, 58,\n",
       "         63,  1, 44, 53, 53, 50,  6,  1, 47, 58,  1, 57, 58, 47, 52, 58, 43, 42,\n",
       "          1, 39, 52, 42,  1, 57, 39, 47, 42,  1,  5, 13, 63,  8,  5,  0,  0, 24,\n",
       "         13, 16, 37,  1, 15, 13, 28, 33, 24, 17, 32, 10,  0, 17, 52, 53, 59, 45,\n",
       "         46,  1, 53, 44,  1, 58, 46, 47, 57, 11,  1, 21,  1, 54, 56, 39, 63,  1,\n",
       "         58, 46, 43, 43,  6,  1, 46, 53, 50, 42,  1, 58, 46, 63,  1, 54, 43, 39,\n",
       "         41, 43,  8,  0,  0, 26, 59, 56, 57, 43, 10,  0, 37, 43, 57,  6,  1, 51,\n",
       "         39, 42, 39]),)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And what the original text for this first item was:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "oth he;\n",
      "And, pretty fool, it stinted and said 'Ay.'\n",
      "\n",
      "LADY CAPULET:\n",
      "Enough of this; I pray thee, hold thy peace.\n",
      "\n",
      "Nurse:\n",
      "Yes, mada\n"
     ]
    }
   ],
   "source": [
    "show_at(dset.train, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fastai library expects the data to be assembled in a `DataLoaders` object (something that has a training and validation dataloader). We can get one by using the `dataloaders` method. We just have to specify a batch size and a sequence length. \n",
    "\n",
    "To further optimize the `LMDataLoader` initialisation, we pass a list of item lengths. As each item is already `block_size + 1` digits long we don't need to re-calculate the actual length for each item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 256\n",
    "lens = [block_size+1 for i in range(len(dset.train))]\n",
    "dls = dset.dataloaders(bs=bs, seq_len=block_size, lens=lens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can inspect a single batch by uncommenting the below line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#o = dls.one_batch(); len(o), o[0].size(), o[1].size(), o"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can also inspect what a single batch looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>to't.\\n\\nISABELLA:\\nThis gentleman told somewhat of my tale,--\\n\\nLUCIO:\\nRight.\\n\\nDUKE VINCENTIO:\\nIt may be right; but you are i' the</td>\n",
       "      <td>o't.\\n\\nISABELLA:\\nThis gentleman told somewhat of my tale,--\\n\\nLUCIO:\\nRight.\\n\\nDUKE VINCENTIO:\\nIt may be right; but you are i' the w</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ou meddle with my master?\\n\\nCOmost likely 'tis for you:\\nConsider of it.'\\n\\nFirst Senator:\\nOur army's in the field\\nWe never yet mad</td>\n",
       "      <td>u meddle with my master?\\n\\nCOmost likely 'tis for you:\\nConsider of it.'\\n\\nFirst Senator:\\nOur army's in the field\\nWe never yet made</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callback to Grab First Output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only need the logits from the model, however Karpathy's GPT model actually also calculates and returns the loss in the forward pass, `(logits, loss)`.  So we need to write the event `after_pred` and replace `self.learn.pred` (which contains the predictions that will be passed to the loss function) by just its first element. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DropLoss(Callback):\n",
    "    def after_pred(self): self.learn.pred = self.pred[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GPT Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I had to use 6 layers instead of 8 as my 2080 GPU has 13GB of ram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "mconf = GPTConfig(vocab_size=len(dls.itos), block_size=block_size, n_layer=6, n_head=8, n_embd=512)\n",
    "model = GPT(mconf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We pass our dataloader and model to a `Learner` and adjust the Adam's hyperparameters to match Karpathy's. We'll use `to_native_fp16` to use [PyTorch's automatic mixed precision](https://pytorch.org/docs/stable/notes/amp_examples.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), \n",
    "                opt_func=partial(Adam, sqr_mom=0.95, wd=0.1), cbs=[DropLoss]).to_native_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start a training loop\n",
    "\n",
    "*(Scroll past learner output to see an example of the output text the model can generate)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/morgan/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/fastprogress/fastprogress.py:74: UserWarning: Your generator is empty.\n",
      "  warn(\"Your generator is empty.\")\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.017378008365631102, lr_steep=0.14454397559165955)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhU5d3/8fc3kz2EsCTsS9hBQUAiCFaEai36WBXrghtQFepuH2tbbX+tj9a2VmvrVmsRUbGK1qV136rizhIEBBHZd8jGloXs9++PGWyMQRLIyZnl87quuZg5c2bO92Yy+eScc5/7NuccIiISu+L8LkBERPylIBARiXEKAhGRGKcgEBGJcQoCEZEYpyAQEYlx8X4X0FSZmZkuOzvb7zJERCLKokWLCp1zWQ09F3FBkJ2dTW5urt9liIhEFDPbeKDndGhIRCTGKQhERGKcgkBEJMYpCEREYpyCQEQkxikIRERiXMR1Hz1U6wtLWZtfQmpSgFZJ8aQlxZMQF0dZVTWlFTWUVVZTXeNITQyQmhhPalKAtMR4UhIDpCYGSAgoM0UkOsVMELy+fAd/fH3lIb8+MRBHWlKAVsnxpCclkJ4cT7u0RDplJNM5I5mOrZPJSk+ibWoibVITaJOSSEpioBlbICLijZgJgnNyunFc3/aUVFRTVlFDaWU1ldW1pIX2DtISAwTijH2VNZRWBvcQ9u8p7F9WWlFNSUU1xeVV7C2vZnV+Ce+vKqC0sqbBbbZKiv8qKDq1TqZb21SyM1PJbp9Gdvs0MlITWvh/QUTkm2ImCDJbJZHZKsmT9y4ur2L7nnIKSyrYU1bF7n1V7CqrpKC4gh17ytm+p5xVeQXk7a342uvapibQs30avTKDwdCnQxp9O7SiV2YaSfHamxCRlhEzQeCl9OQE0pMT6N8x/VvXK6+qYdPOMjYUlrKhqJQNRcH789cV8a/FW79aL86gZ/s0jujcmqO6ZTCkWwaDu2bQOll7ECLS/BQELSg5IUD/jukNBkZ5VQ3rCkpZU1DCmrxiVuWVsHTLbl5Zth0AMxjcJYMT+mdxwoAshndvQ7xOYItIM7BIm7w+JyfHxdKgcztLK1m2dQ9LNu3mwzUFfLppNzW1jtbJ8Ywb0IHvH9mJEwZk0SpJmS4iB2Zmi5xzOQ0+pyCILHv2VfHRmkLeXZnP2yvz2VlaSWJ8HN/pm8mpQzpz8pEddQhJRL5BQRClamoduRt28uaKPF5fvoOtu/eRGIhj3IAsThvahe8N6qgurCICKAhignOOJZt389LS7byybBt5eytISwzw/cGdmDi8K2P6ZBKIM7/LFBGfKAhiTG2tY/76nbywZCuvLNtOcXk1WelJXDSqJ5NH96RtWqLfJYpIC1MQxLDyqhrmfpnPUws3M/fLApIT4jg3pzuXfac3Pdqn+l2eiLQQBYEAsCqvmIfeX8e/l2ylptYxcXg3rjuxnwJBJAYoCORr8vaW89D763h83kZqah3n5HTnmu/2pUubFL9LExGPKAikQXl7y/nru2uYs2ATZsblY3tz5fi+JCeop5FItPm2INClqTGsY+tkbj1jMO/eMI4JR3bi3nfWcNKf3+M/K/L8Lk1EWpCCQOjWNpV7zx/OnGnHkpIQ4LLZuVz66EK27d7nd2ki0gIUBPKV0X3a8+p1x/OrUwfxyboiTv7L+zy1YBORdvhQRJpGQSBfkxCIY9rY3rx+3ViGdM3gxueXMXnWArZq70AkaikIpEE92qfyxGWjuO3MwXy6cRff/8v7PLdoi/YORKKQgkAOKC7OuOjYnrz+k7Ec0bk1P31mKdfMWcyefVV+lyYizUhBIAfVvV0qc6Yfy8++P4DXl+/g1Hs+YP66Ir/LEpFmoiCQRgnEGVeN78uzV4whIWCc/9A8/v7eWh0qEokCCgJpkmHd2/DytcczYXAn/vDaSq6es5iyymq/yxKRw6AgkCZrlRTPXy84mhtPGchry7Yz8a8fs6Gw1O+yROQQKQjkkJgZl5/Qh8cuGUlecTmn3/8hCzfs9LssETkECgI5LMf3y+LFq75DZqskLpw5n9eX7/C7JBFpIgWBHLYe7VN59ooxHNG5NVc+sYjH5230uyQRaQIFgTSLdmmJPDltFOMHdODX/17OXW9+qR5FIhFCQSDNJjUxnr9fPILzcrpz3ztruPXlFQoDkQgQ7/UGzCwA5AJbnXOn1XsuCZgNjACKgPOccxu8rkm8Ex+I4/YfDiEtKZ5ZH62norqW284YTFyc+V2aiByA50EAXAd8AbRu4LlLgV3Oub5mNgn4I3BeC9QkHjIzfn3aIJIT4nhg7lrKq2q444dHER/QDqhIOPL0m2lm3YD/AWYeYJUzgMdC958FTjQz/ekYBcyMn08YyE+/15/nP93KT55eQnVNrd9liUgDvN4juBv4OZB+gOe7ApsBnHPVZrYHaA8U1l3JzKYD0wF69OjhWbHS/K45sR+J8XH84bWVmBl/OXeo9gxEwoxn30gzOw3Id84t+rbVGlj2jbOLzrkZzrkc51xOVlZWs9UoLePHJ/ThFxMG8tLSbfzs2c+oqdUJZJFw4uUewXHA6WZ2KpAMtDazfzjnLqqzzhagO7DFzOKBDECXp0ahK8b1oaa2lj+9uYo4M+48+yidQBYJE54FgXPuJuAmADMbB9xQLwQAXgSmAJ8AZwPvOPU3jFpXf7cf1bWOu/+zmoSA8YezhqBTQiL+a4leQ19jZrcCuc65F4GHgcfNbA3BPYFJLV2PtKzrTuxHVU0tf313LenJ8fzy1EEKAxGftUgQOOfmAnND939TZ3k5cE5L1CDhwcy44eQBlJRX89AH62mTmshV4/v6XZZITGvxPQIRM+PmHxzJnn1V3PnGl7ROSeDiY3v6XZZIzFIQiC/i4ow7zxlKSUU1v3lhOa2T4zljWFe/yxKJSerQLb5JCMRx/wVHMzK7HTc8s5SP1xYe/EUi0uwUBOKr5IQAMybnkN0+jR8/vojVecV+lyQScxQE4ruMlAQe+dExJCcEmPrIQvKLy/0uSSSmKAgkLHRrm8qsKcews7SSSx/NpbSi2u+SRGKGgkDCxpBuGdx/wXA+37aH655arKEoRFqIgkDCyomDOnLzD47kP1/kc+cbX/pdjkhMUPdRCTuTR/fky7xiHnxvLQM6tWLi8G5+lyQS1bRHIGHHzLjl9CM5tnc7fvHcMhZv2uV3SSJRTUEgYSkhEMcDF46gY+skpj++iO179vldkkjUUhBI2GqXlsjDU46hrKKa6bMXsa+yxu+SRKKSgkDCWv+O6dx7/nCWb9vDDc8spVY9iUSanYJAwt6Jgzpy0ykDeWXZdu55e7Xf5YhEHfUakogw7fjerM4r4Z63V9O3Qyt+MLSL3yWJRA3tEUhEMDNumziYY7LbcsMzS1m6ebffJYlEDQWBRIyk+AAPXjSCrPQkLpudy7bd6kkk0hwUBBJR2rdKYtbUYyivrOGSRxdSojGJRA6bgkAiTv+O6fz1wqNZnV/CNU9+SnVNrd8liUQ0BYFEpLH9s7j1jCN598sCbnvlC7/LEYlo6jUkEevCUT1ZX1DKzA/X0yszjSljsv0uSSQiKQgkot106iA2FJVxy0uf06NdKuMHdvC7JJGIo0NDEtECccY9k4YxqHNrrn7yU77YvtfvkkQijoJAIl5aUjwPTzmG9OQELn10Ifl7NdWlSFMoCCQqdMpIZuaUHHbvq+LSx3Ipq1S3UpHGUhBI1BjcNYN7JwWnuvzfp5dogDqRRlIQSFQ56YiO/Op/juCNz/O4Q1NdijSKeg1J1LnkuGzWFZTw4Htr6Z2Vxrk53f0uSSSsKQgk6pgZ/3f6kWzaWcYvn19G97apjO7T3u+yRMKWDg1JVEoIxHH/BUeTnZnG5f9YxLqCEr9LEglbCgKJWhkpCcyacgyBOONHjy6kqKTC75JEwpKCQKJaj/apPDQ5hx17ypk2O5fyKs17LFKfgkCi3oiebbn7vGEs3rxb3UpFGqAgkJhwypDO/PKUQby2fAe3v77S73JEwop6DUnMuOz4XmzeVcaM99fRIT2Jy47v7XdJImFBQSAxw8y4+QdHUlhS8dUcBgoDEQWBxJjgaKXDgcXc9soXOAfTxioMJLYpCCTmJATiuGfScIwl/O7VL6h1jh+f0MfvskR8oyCQmBQMg2Fg8IfXVpIUH8fU43r5XZaILzwLAjNLBt4HkkLbedY5d3O9dXoAjwFtgABwo3PuVa9qEqkrPhDHPecNo6q6llteXkGnjGQmDO7sd1kiLc7L7qMVwHedc0OBYcAEMzu23jr/D/inc244MAl4wMN6RL4hPhDHvecPZ1j3Nlz31BIWbdzpd0kiLc6zIHBB+wd4SQjd6l/J44DWofsZwDav6hE5kOSEAA9POYYubVK49LFc1mpcIokxnl5QZmYBM1sC5ANvOefm11vl/4CLzGwL8CpwzQHeZ7qZ5ZpZbkFBgZclS4xql5bIoz86hoAZUx9ZQH6xpruU2OFpEDjnapxzw4BuwEgzG1xvlfOBR51z3YBTgcfN7Bs1OedmOOdynHM5WVlZXpYsMaxn+zQennoMhcWVXDxzATtLK/0uSaRFtMgQE8653cBcYEK9py4F/hla5xMgGchsiZpEGjKsexsenpLDhqJSLpo5n91lCgOJfp4FgZllmVmb0P0U4CSg/iAvm4ATQ+sMIhgEOvYjvhrTN5MZk3NYk1/C5FkL2Fte5XdJIp7yco+gM/CumX0GLCR4juBlM7vVzE4PrfNTYJqZLQXmAFOdcxoaUnx3Qv8sHrjwaFZs28vUWQsoqaj2uyQRz1ik/d7Nyclxubm5fpchMeL15du56snFjOjZlsd+NJKUxIDfJYkcEjNb5JzLaeg5DUMt8i0mDO7Mn88dysINO5n+uCa2keikIBA5iDOGdeWOHx7FB6sLufrJT6msrvW7JJFmpSAQaYRzcrrz2zMH858v8vnJ04uprlEYSPTQoHMijXTxsT2pqKrhtle+ID5uKX8+dyjxAf0tJZFPQSDSBJcd35uqGscfQ9NdKgwkGjQqCMysD7DFOVdhZuOAo4DZoQvFRGLKFeOCcxf88fWVmMFd5ygMJLI19qf3OaDGzPoCDwO9gCc9q0okzF0xrg8/nzCAF5Zs46fPLNU5A4lojT00VOucqzazicDdzrn7zGyxl4WJhLsrx/UF4I7Xv8SAu84dRiDO/C1K5BA0NgiqzOx8YArwg9CyBG9KEokcV47ri3Nw5xtfEmfGnecMVRhIxGlsEPwIuBz4nXNuvZn1Av7hXVkikeOq8cE9gzvf+BIM7jxbYSDNr7bWEefRz1WjzhE451Y45651zs0xs7ZAunPudk8qEolAV43vy0+/15/nP93KL577jJrayBq6RcLbnn1VDPrN6zy1YJMn79/YXkNzgdND6y8BCszsPefc9Z5UJRKBrjmxH7UO/vKfVVRW13LXuUNJUG8iaQYbi0qpqK6lTWqiJ+/f2ENDGc65vWZ2GfCIc+7m0KiiIlLHdSf1IykhjttfW0lJRTUPXHg0yQkaqE4Oz/rCUgB6ZaZ58v6N/XMl3sw6A+cCL3tSiUiUuPyEPvx+4hDe/TKfybMWUKz5DOQwbSgsA6BHu1RP3r+xQXAr8Aaw1jm30Mx6A6s9qUgkClwwqgf3TBrOpxt3ccFD8ykqqfC7JIlgG4pK6ZyR7Nkw6I09WfyMc+4o59wVocfrnHM/9KQikShx+tAuzJg8glV5xZz790/Ytnuf3yVJhFpfWEp2e28OC0Ejg8DMupnZv8ws38zyzOw5M+vmWVUiUeK7Azsy+5KR5O+t4JwHP2FdQYnfJUkE2lhUSrZH5weg8YeGHgFeBLoAXYGXQstE5CBG9W7PnOnHUl5VwzkPfsLyrXv8LkkiyJ6yKnaVVdEr05vzA9D4IMhyzj3inKsO3R4FsjyrSiTKDO6awTOXjyY5IcD5M+bx0ZpCv0uSCLG+KNhjyPdDQ0ChmV1kZoHQ7SKgyLOqRKJQ76xWPHvFaDq3SWbqIwt4/tMtfpckEWBDqOtoOBwauoRg19EdwHbgbILDTohIE3TOSOGZy8eQ07Md1/9zKfe/sxrndBWyHNiGolLMvOs6Co3vNbTJOXe6cy7LOdfBOXcmcJZnVYlEsYyUBB67ZCRnDuvCn95cxS//tUzDWMsBbSgspUtGiqcXJh7O9e8aXkLkECXGx/GX84Zx5bg+zFmwmWmzcymtqPa7LAlD64vKyPbwRDEcXhBoeEWRw2Bm/HzCQG47czDvrSpg0ox5FBTrwjP5ug0eX0MAhxcEOrAp0gwuOrYnMy7OYU1+CWf97SPW6loDCdlVWsmefVX+BoGZFZvZ3gZuxQSvKRCRZnDSER15avqxlFXU8MO/fUzuhp1+lyRhYEOR9z2G4CBB4JxLd861buCW7pxr7MilItIIQ7u34fkrx9A2NZELZs7ntWXb/S5JfLY/CLy8mAwO79CQiDSznu3TeO6KMQzu0porn/yUWR+u97sk8dH6wjLiDLp72HUUFAQiYaddWiJPTjuWk4/oyK0vr+C2l1dQqxnPYtKGwlK6tEkhKd7bOS0UBCJhKDkhwAMXjmDqmGxmfriea55aTHlVjd9lSQvbWFTq2WQ0dSkIRMJUIM64+QdH8MtTB/LKZ9uZPGsBe8o0yU2scM6xvrCUnu29PSwECgKRsGZmTB/bh3smDWPxpl2c/eDHbNW8BjFhV1kVe8urPe86CgoCkYhwxrCuPHbJSHbsLeesBz7iyx3FfpckHvN6nuK6FAQiEWJMn0yeuXw0AOc8+DELda1BVGuJUUf3UxCIRJCBnVrz7OVjyGyVxEUz5/OfFXl+lyQe2VhUGuw62lbnCESknu7tUnnm8tEM6JTOj/+xiGdyN/tdknhgfVEZ3dqmkhjv/a9pBYFIBGrfKoknpx3LmD7t+dmznzFnwSa/S5JmtqGFegyBgkAkYrVKiuehyTmMG5DFTc8vUxhEEeccGwpb5hoCUBCIRLTkhAAPXjRCYRBlCkoqKK6ojvwgMLNkM1tgZkvN7HMzu+UA651rZitC6zzpVT0i0ap+GDwxf6PfJclhWp0XHIq8f8f0Ftmel3sEFcB3nXNDgWHABDM7tu4KZtYPuAk4zjl3JPATD+sRiVr7w2D8gCx+9a/l3PXml5oLOYKtygteJ9KvY6sW2Z5nQeCC9s+wkRC61f/JnAb81Tm3K/SafK/qEYl2yQkBZkzO4byc7tz3zhp+8vQSKqo1PlEkWpVXTNvUBLJaJbXI9jw9R2BmATNbAuQDbznn5tdbpT/Q38w+MrN5ZjbBy3pEol1CII7bfziEn31/AC8s2cbFMxewq7TS77KkiVblldCvYzpmLTMjsKdB4Jyrcc4NA7oBI81scL1V4oF+wDjgfGCmmbWp/z5mNt3Mcs0st6CgwMuSRSKemXHV+L7cM2kYSzbvZuIDH7E6T0NSRArnHKvyiunfQoeFoIV6DTnndgNzgfp/8W8BXnDOVTnn1gNfEgyG+q+f4ZzLcc7lZGVleV6vSDQ4Y1hXnpw2ipKKGiY+8LGuQo4QO/aWU1xezYAWOlEM3vYaytr/172ZpQAnASvrrfZvYHxonUyCh4rWeVWTSKzJyW7Hi1cfR6/MNKY9nsv976zWSeQwtyrUY6hfNAQB0Bl418w+AxYSPEfwspndamanh9Z5AygysxXAu8DPnHNFHtYkEnO6tEnhmctHc+awrvzpzVVc/eRi9lXqJHK4WhUaWbaluo5C8Bi9J5xznwHDG1j+mzr3HXB96CYiHklOCPDnc4cyqHM6f3htJRuKSnlocg5d2qT4XZrUsyqvmMxWSbRLS2yxberKYpEYsX+Sm1lTjmFjURmn3/8Rn27a5XdZUk9LnygGBYFIzBk/sAP/unIMqYkBJs2Yxz81emnYqK11rM4vadHDQqAgEIlJ/Tqm88JVx5HTsy0/f/Yz/vfpJZRUVPtdVszbunsfZZU1CgIRaRlt0xJ5/NJR/O9J/XlhyVZOu/cDlm3Z43dZMW3/0BIDOunQkIi0kECccd1J/Xhq+mgqqms5628f8fg8DVrnl/1dR/t20B6BiLSwkb3a8eq1x3N8vyx+/e/l3PH6Sl1v4INVecV0ap1MRkpCi25XQSAiQPBQ0YyLR3D+yB48MHctNzzzGVU1tX6XFVNW5RXTv1PL7g2Ah9cRiEjkiQ/E8fuJg+mckcyf31pFQUkFf7vwaNKS9KvCazW1jjX5JYzu3b7Ft609AhH5GjPj2hP7cftZQ/hoTSGTZy1Qj6IWsGlnGRXVtS3eYwgUBCJyAJNG9uD+84ezZPNuLnlkIWWVCgMv7e8x5MehIQWBiBzQKUM6c/d5w8jduJNLH83VGEUe2j/GUL8OLdt1FBQEInIQPxjahT+fO4x564uYNjuX8iqFgRdW5ZfQrW2KL+djFAQiclBnDu/KnWcP5aO1hfz48UWaAtMDq3YU+3J+ABQEItJIZ4/oxu1nDeG9VQVc9cRiKqvVtbS57CqtZF1hy48xtJ+CQEQa7bxjevDbM47kP1/kcd1Ti6nWdQbN4rFPNlBV45g4vKsv21cQiEiTXDw6m1+fdgSvLd/B9f9cSk2trkA+HKUV1Tz68QZOGtSBAT70GAJdUCYih+DS7/SiqqaW219bSUpCgNt/OAQz87usiDRnwSZ2l1Vxxbi+vtWgIBCRQ3L5CX0oq6jm3nfW0DYtkRtPGeh3SRGnsrqWmR+sZ1Svdozo2da3OhQEInLI/vd7/dlZVsmD762lXVoC08f28bukiPLvxVvZsbecP559lK91KAhE5JCZGbecPpjdZVX8/tWVtE1N5Jyc7n6XFRFqah0PvreWI7u0Zmy/TF9rURCIyGEJxBl/PncYe/ZVcePzy8hISeDkIzv5XVbYe+PzHawrLOWvFxzt+/kV9RoSkcOWGB/HgxeNYEjXDK6es5iP1xb6XVJYc87xt7lr6Z2ZxoTB/oemgkBEmkVaUjyPTD2G7PapTHssl8+27Pa7pLD14ZpClm3dw/SxvQnE+d/bSkEgIs1m/zzIbdMSmTJrAWvyi/0uKSw98O5aOrZOYuLR/lxAVp+CQESaVcfWyTxx2SjiA3FcNHMBW3aV+V1SWFm8aRefrCti2vG9SYoP+F0OoCAQEQ/0bJ/G7EtGUlpZzZRZC9hVWul3SWHjb3PXkpGSwKSRPfwu5SsKAhHxxKDOrZk5OYfNO/dxmYavBmB1XjFvrshjyphsWoXR9J8KAhHxzKje7bl70jA+3bSLa+csjvlxiR58bx0pCQGmjsn2u5SvURCIiKdOHdKZm087gjdX5HHzi8txLjbDYOvufbywZCuTRnanXVqi3+V8Tfjsm4hI1Jp6XC+27y3n7++to0N6Mtee2M/vklrcQ++vA2Da8b19ruSbFAQi0iJ+8f2BFOyt4M9vrSIrPYnzw+hkqdf27Kvi6YWbOWNYV7q0SfG7nG9QEIhIi4iLM/549lHsLKvkV/9aRvu0xJgZiuLZRVvYV1XDj47L9ruUBukcgYi0mIRAHA9ceDRDurXhmjmLWbB+p98lea621vGPeRs5ukcbBnfN8LucBikIRKRFpSYGh6Lo2jaFyx5byMode/0uyVMfrilkfWEpk0dn+13KASkIRKTFtUtLZPYlI0lJDDD54QVs3hm9Vx/P/mQj7dMSOWVI+B4GUxCIiC+6tU1l9iWjqKiu5eKH51NYUuF3Sc1u884y3l6Zx/kje4TNcBINURCIiG8GdEpn1tQcduwtZ+ojCygur/K7pGb1xPxNGHDBqPDuIaUgEBFfjejZjr9dNIKV24uZPntR1AxFUV5Vw9MLN/G9IzqGZZfRuhQEIuK78QM68KdzhvLJuiKu/+eSqBiK4pXPtrOrrCqsTxLv51kQmFmymS0ws6Vm9rmZ3fIt655tZs7McryqR0TC25nDu/L//mcQry7bwS0vfR7xQ1H8Y/5G+mSlMaZPe79LOSgvLyirAL7rnCsxswTgQzN7zTk3r+5KZpYOXAvM97AWEYkAlx3fm/ziCma8v44O6Ulc/d3IHIpiy64yFm/azS8mDPR9PuLG8GyPwAWVhB4mhG4NRfxvgTuAcq9qEZHIceOEgUwc3pU/vbmKpxdu8rucQ/L68h0AnBrGXUbr8vQcgZkFzGwJkA+85ZybX+/54UB359zLXtYhIpEjLs644+yjGNs/i5ueX8abn+/wu6Qme3XZdo7s0pqe7dP8LqVRPA0C51yNc24Y0A0YaWaD9z9nZnHAX4CfHux9zGy6meWaWW5BQYF3BYtIWEgIxPG30FAUV89ZzPx1RX6X1Gjb9+zj0027OXVIZ79LabQW6TXknNsNzAUm1FmcDgwG5prZBuBY4MWGThg752Y453KcczlZWVktULGI+C0tKTgURfe2KVz2WC4rtkXGUBT7DwudMjgyDguBt72GssysTeh+CnASsHL/8865Pc65TOdctnMuG5gHnO6cy/WqJhGJLO3SEpl96ShaJcczedYCNhaV+l3SQb22bAcDO6XTO6uV36U0mpd7BJ2Bd83sM2AhwXMEL5vZrWZ2uofbFZEo0rVNCo9fOpLq2loufngB+XvDt19J/t5yFm7cySmDI+ewEHjba+gz59xw59xRzrnBzrlbQ8t/45x7sYH1x2lvQEQa0rdDOo9MPYbCkgomz1rA7rJKv0tq0Buf78C5yOkttJ+uLBaRiDC8R1sempzDuoJSfvToQkorqv0u6RteXbaDvh1a0a9jut+lNImCQEQixnF9M7n3/OEs3byb6Y/nUlEdPuMSFZZUMH99UUSdJN5PQSAiEWXC4E7ccfZQPlpTxLVzFlNdU+t3SQC8+XketY6IOz8ACgIRiUBnj+jGb047gjc+z+O6p5aERRi8sGQr2e1TGdQ5sg4LgSavF5EIdcl3elFdW8vvXw32Sr970jASAv78bfvuynzmr9/Jr04dFBFjC9WnIBCRiDV9bB8M43evfgH4EwaV1bX89pUV9MpMY8qY7BbddnNREIhIRJs2tjcAv3v1CxyOu88bTmJ8y4XB7E82sK6glFlTc1p0u81JQSAiEW/a2N6YwW2vfMG+ylweuHAEKYnezxFcWFLBPW+v5oT+WYwf0MHz7XklMuNLRKSey47vze8mDiu3TyUAAAlDSURBVGbuqgKmtND8x3e9uYp9lTX8+rTIPDewn4JARKLGhaN6cs+k4Xy6cRfnPzSPopIKz7b1+bY9PLVwE5NHZ9O3Q+T1FKpLQSAiUeX0oV14aHIOq/NKOOfvn7CuoOTgL2qirbv3cf3TS2mbmsh1J0bmLGp1KQhEJOqMH9iBf1w2it1lVZzx1494Z2Ves733wg07Of2+D9m2Zx/3ThpORmpCs723XxQEIhKVjslux4tXH0ePdqlc+lgu9729mtrahmbLbbw5CzZxwUPzyEhJ4N9XHcd3+mU2U7X+Uq8hEYla3dqm8twVY7jp+WXc9dYqFmzYycXH9uSEAVkkxTeuV9HGolI+WF3IOyvzeWdlPmP7Z3FflOwJ7GfOHV5CtrScnByXm6vRqkWk8ZxzPPrxBu59ezW7yqpIT4rne0d25OQjOnJklwy6tU35qtfP3vIqPl5TyHurCvlwTQGbd+4DoEtGMmeP6Ma1J/Yj3qcrmA+HmS1yzn1jBkhQEIhIDKmqqeXjtUW8vHQbb3y+g73lwaGs05PiGdg5nVoHSzbvpqbW0SopnmN7t+f4fpl8p18mvTPTIrqLqIJARKSeiuoaVmzbyxfbi/li+16+2L6X6lrHd/pmMrZ/FsN7tPFt7CIvfFsQ6ByBiMSkpPgAw3u0ZXiPtn6X4rvoiTsRETkkCgIRkRinIBARiXEKAhGRGKcgEBGJcQoCEZEYpyAQEYlxCgIRkRgXcVcWm9keYHWdRRnAngM83n9//7+ZQOEhbrr+dpqyTkPLG1N33ft1l3nZDi/bUPd+rH8Wfreh7v1w+Sya8ljf7cbXuF9P51xWg8845yLqBsxo7OP99+v8m9tc223KOg0tb0zdDbXB63Z42QZ9FuHThnD8LPTd9r4dB7pF4qGhl5rw+KUDrNMc223KOg0tb0zdde83Rxsa8z5etqEx22+MaPgs/G5DY2s4mOZsh77bh+eQ3yfiDg0dDjPLdQcYdCmSREM7oqENEB3tUBvCh1/tiMQ9gsMxw+8Cmkk0tCMa2gDR0Q61IXz40o6Y2iMQEZFvirU9AhERqUdBICIS4xQEIiIxTkEQYmbHm9mDZjbTzD72u55DYWZxZvY7M7vPzKb4Xc+hMrNxZvZB6PMY53c9h8rM0sxskZmd5ncth8rMBoU+h2fN7Aq/6zkUZnammT1kZi+Y2cl+13OozKy3mT1sZs8293tHRRCY2Swzyzez5fWWTzCzL81sjZnd+G3v4Zz7wDl3OfAy8JiX9TakOdoAnAF0BaqALV7V+m2aqR0OKAGS8aEdzdQGgF8A//SmyoNrpu/FF6HvxblAi3drbKY2/Ns5Nw2YCpznYbkH1EztWOecu9STAg/1SrRwugFjgaOB5XWWBYC1QG8gEVgKHAEMIfjLvu6tQ53X/RNoHYltAG4Efhx67bOR+lkAcaHXdQSeiNA2nARMIvjL57RI/SxCrzkd+Bi4IFLbEHrdXcDRkfxZhF7X7N/tqJi83jn3vpll11s8EljjnFsHYGZPAWc45/4ANLirbmY9gD3Oub0eltug5miDmW0BKkMPa7yr9sCa67MI2QUkeVHnt2mmz2I8kEbwi73PzF51ztV6Wng9zfVZOOdeBF40s1eAJ72ruMFtN8dnYcDtwGvOuU+9rbhhzfy9aHZREQQH0BXYXOfxFmDUQV5zKfCIZxU1XVPb8Dxwn5kdD7zvZWFN1KR2mNlZwPeBNsD93pbWaE1qg3PuVwBmNhUobOkQ+BZN/SzGAWcRDORXPa2s8Zr6vbiG4B5ahpn1dc496GVxTdDUz6I98DtguJndFAqMZhHNQWANLPvWq+ecczd7VMuhalIbnHNlBMMs3DS1Hc8TDLVw0uSfJwDn3KPNX8phaepnMReY61Uxh6ipbbgXuNe7cg5ZU9tRBFzuRSFRcbL4ALYA3es87gZs86mWQxUNbYDoaEc0tAGiox3R0AYIo3ZEcxAsBPqZWS8zSyR44u5Fn2tqqmhoA0RHO6KhDRAd7YiGNkA4tcOPM+genJGfA2znv90mLw0tPxVYRfDM/K/8rjPa2xAt7YiGNkRLO6KhDZHQDg06JyIS46L50JCIiDSCgkBEJMYpCEREYpyCQEQkxikIRERinIJARCTGKQgk4plZSQtvb6aZHdFM71VjZkvMbLmZvWRmbQ6yfhszu7I5ti2yn64jkIhnZiXOuVbN+H7xzrnq5nq/g2zrq9rN7DFglXPud9+yfjbwsnNucEvUJ7FBewQSlcwsy8yeM7OFodtxoeUjzexjM1sc+ndAaPlUM3vGzF4C3rTgLGlzLTgz10ozeyI0nDGh5Tmh+yUWnBVuqZnNM7OOoeV9Qo8Xmtmtjdxr+YTgiJSYWSsze9vMPjWzZWZ2Rmid24E+ob2IO0Pr/iy0nc/M7JZm/G+UGKEgkGh1D/AX59wxwA+BmaHlK4GxzrnhwG+A39d5zWhginPuu6HHw4GfEJxToDdwXAPbSQPmOeeGEhz6e1qd7d8T2v5BBxIzswBwIv8da6YcmOicOxoYD9wVCqIbgbXOuWHOuZ9ZcOrFfgTHth8GjDCzsQfbnkhd0TwMtcS2k4AjQn/EA7Q2s3QgA3jMzPoRHPI3oc5r3nLO7azzeIFzbguAmS0BsoEP622nkuAMUgCLgO+F7o8GzgzdfxL40wHqTKnz3ouAt0LLDfh96Jd6LcE9hY4NvP7k0G1x6HErgsEQTvNRSJhTEEi0igNGO+f21V1oZvcB7zrnJoaOt8+t83RpvfeoqHO/hoa/L1XuvyfaDrTOt9nnnBtmZhkEA+UqgmPnXwhkASOcc1VmtoHgHM71GfAH59zfm7hdka/o0JBEqzeBq/c/MLNhobsZwNbQ/akebn8ewUNSEBxe+Fs55/YA1wI3mFkCwTrzQyEwHugZWrUYSK/z0jeAS8xs/wnnrmbWoZnaIDFCQSDRINXMttS5XU/wl2pO6ATqCv47s9MdwB/M7COCk4d75SfA9Wa2AOgM7DnYC5xziwlOYD4JeIJg/bkE9w5WhtYpAj4KdTe90zn3JsFDT5+Y2TLgWb4eFCIHpe6jIh4ws1SCh32cmU0CznfOnXGw14n4QecIRLwxArg/1NNnN3CJz/WIHJD2CEREYpzOEYiIxDgFgYhIjFMQiIjEOAWBiEiMUxCIiMQ4BYGISIz7/14AXIdFSTo1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.317739</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.033413</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.807518</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.665807</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.572921</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.509148</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.463208</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.428268</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.394268</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.353075</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.300225</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.243087</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.176459</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.103145</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.034520</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>1.967749</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>1.909871</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>1.858214</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>1.808828</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>1.766990</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>1.726754</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>1.692783</td>\n",
       "      <td>None</td>\n",
       "      <td>00:11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>1.657690</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>1.624187</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>1.594954</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>1.566474</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>1.541964</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>1.514784</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>1.492658</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>1.470435</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>1.450287</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>31</td>\n",
       "      <td>1.429739</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>32</td>\n",
       "      <td>1.409604</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33</td>\n",
       "      <td>1.390563</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>34</td>\n",
       "      <td>1.372044</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>35</td>\n",
       "      <td>1.352105</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>36</td>\n",
       "      <td>1.333806</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>37</td>\n",
       "      <td>1.314617</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>38</td>\n",
       "      <td>1.295365</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>39</td>\n",
       "      <td>1.275487</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>40</td>\n",
       "      <td>1.256758</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>41</td>\n",
       "      <td>1.236971</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>42</td>\n",
       "      <td>1.216748</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>43</td>\n",
       "      <td>1.194249</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>44</td>\n",
       "      <td>1.172974</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>45</td>\n",
       "      <td>1.150926</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>46</td>\n",
       "      <td>1.130048</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>47</td>\n",
       "      <td>1.107042</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>48</td>\n",
       "      <td>1.084572</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>49</td>\n",
       "      <td>1.062451</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>1.041875</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>51</td>\n",
       "      <td>1.020664</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>52</td>\n",
       "      <td>0.997576</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>53</td>\n",
       "      <td>0.976921</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>54</td>\n",
       "      <td>0.956086</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>55</td>\n",
       "      <td>0.936446</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>56</td>\n",
       "      <td>0.917717</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>57</td>\n",
       "      <td>0.899041</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>58</td>\n",
       "      <td>0.880910</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>59</td>\n",
       "      <td>0.862721</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60</td>\n",
       "      <td>0.844015</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>61</td>\n",
       "      <td>0.828155</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>62</td>\n",
       "      <td>0.812719</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>63</td>\n",
       "      <td>0.797472</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>64</td>\n",
       "      <td>0.783071</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>65</td>\n",
       "      <td>0.769556</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>66</td>\n",
       "      <td>0.755334</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>67</td>\n",
       "      <td>0.742516</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>68</td>\n",
       "      <td>0.729470</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>69</td>\n",
       "      <td>0.719807</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>70</td>\n",
       "      <td>0.707666</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>71</td>\n",
       "      <td>0.697987</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>72</td>\n",
       "      <td>0.687649</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>73</td>\n",
       "      <td>0.675826</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>74</td>\n",
       "      <td>0.666926</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>75</td>\n",
       "      <td>0.657704</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>76</td>\n",
       "      <td>0.648814</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>77</td>\n",
       "      <td>0.640211</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>78</td>\n",
       "      <td>0.633288</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>79</td>\n",
       "      <td>0.625470</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>80</td>\n",
       "      <td>0.617117</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>81</td>\n",
       "      <td>0.609131</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>82</td>\n",
       "      <td>0.603800</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>83</td>\n",
       "      <td>0.596508</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>84</td>\n",
       "      <td>0.592238</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>85</td>\n",
       "      <td>0.585604</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>86</td>\n",
       "      <td>0.579585</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>87</td>\n",
       "      <td>0.574960</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>88</td>\n",
       "      <td>0.569906</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>89</td>\n",
       "      <td>0.565836</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>90</td>\n",
       "      <td>0.561620</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>91</td>\n",
       "      <td>0.557717</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>92</td>\n",
       "      <td>0.555219</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>93</td>\n",
       "      <td>0.552747</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>94</td>\n",
       "      <td>0.550309</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>95</td>\n",
       "      <td>0.547639</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>96</td>\n",
       "      <td>0.544183</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>97</td>\n",
       "      <td>0.541255</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>98</td>\n",
       "      <td>0.538740</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>99</td>\n",
       "      <td>0.537343</td>\n",
       "      <td>None</td>\n",
       "      <td>00:10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/morgan/anaconda3/envs/fastai2_me/lib/python3.7/site-packages/fastprogress/fastprogress.py:74: UserWarning: Your generator is empty.\n",
      "  warn(\"Your generator is empty.\")\n"
     ]
    }
   ],
   "source": [
    "# FP16 NATIVE\n",
    "learn.fit_one_cycle(100, 6e-4, div_final=10)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note there is no validation loss or validation perplexity as we we didn't specify a validation set (as per Karpathy's notebook)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhU9d3+8fcnyWQjCQlJgJAAAUWQfQkIbsWlFbUurVipe/tUKtbHpbUuv25q+1Rtn7aPa61rXSiKuFuxiqC4ABowQNh3CBCysCQhe/L9/TFDRAgkQJKTmdyv68rFzJmTmXvONd6efM+Z7zHnHCIiEvzCvA4gIiItQ4UuIhIiVOgiIiFChS4iEiJU6CIiISLCqxdOSUlxmZmZXr28iEhQWrhwYZFzLrWxxzwr9MzMTLKzs716eRGRoGRmmw71mIZcRERChApdRCREqNBFREKEZ2PoIiJHqqamhry8PCorK72O0uqio6PJyMjA5/M1+3dU6CISNPLy8oiPjyczMxMz8zpOq3HOUVxcTF5eHn369Gn272nIRUSCRmVlJcnJySFd5gBmRnJy8hH/JaJCF5GgEuplvs/RvM+gK/RV+aX85f1VFJdVeR1FRKRdCbpCX19YxsOz11JQqkIXkba1e/duHnvssSP+vfPOO4/du3e3QqJvCrpCj44MB6Cyps7jJCLS0Ryq0OvqDt9H7777LomJia0Vq0HQneUSHeEv9AoVuoi0sTvvvJN169YxfPhwfD4fcXFxpKWlkZOTw/Lly7n44ovZsmULlZWV3HzzzUyePBn4eqqTsrIyzj33XE499VQ+//xz0tPTefPNN4mJiWmRfEFX6DHaQxcR4J63l7F8W0mLPufAHgn87oJBh3z8/vvvJzc3l5ycHD766CPOP/98cnNzG04tfOaZZ+jSpQsVFRWMHj2aSy65hOTk5G88x5o1a5g2bRpPPvkkP/jBD3j11Ve58sorWyR/8BW6b1+h13ucREQ6ujFjxnzjPPGHHnqI119/HYAtW7awZs2agwq9T58+DB8+HIBRo0axcePGFssTdIUe7fMP+1dUaw9dpCM73J50W+nUqVPD7Y8++ohZs2Yxb948YmNjGT9+fKPnkUdFRTXcDg8Pp6KiosXyBN1B0X176BpDF5G2Fh8fT2lpaaOP7dmzh6SkJGJjY1m5ciXz589v43TBuIeuMXQR8UhycjKnnHIKgwcPJiYmhm7dujU8NmHCBB5//HGGDh1K//79GTt2bJvnC7pC/3oMXYUuIm3vX//6V6PLo6KimDlzZqOP7RsnT0lJITc3t2H5bbfd1qLZgm7IxRcehi/c2KsxdBGRbwi6QgfoHBPJ7vIar2OIiLQrQVnoXTr52LW32usYIiLtSlAWelJsJDvLVegiIvtrstDNLNrMvjCzxWa2zMzuaWSda82s0MxyAj8/aZ24fkmxkdpDFxE5QHPOcqkCznTOlZmZD/jUzGY65w48yfJl59yNLR/xYEmdItm1SWPoIiL7a3IP3fmVBe76Aj+uVVM1oUsnH7vKq3HO0xgiIocVFxcHwLZt25g4cWKj64wfP57s7OwWeb1mjaGbWbiZ5QAFwAfOuQWNrHaJmS0xsxlm1vMQzzPZzLLNLLuwsPCoQ6fGRVFX7yjWsIuIBIEePXowY8aMVn+dZhW6c67OOTccyADGmNngA1Z5G8h0zg0FZgHPHeJ5nnDOZTnnslJTU486dK/kWAA2FZcf9XOIiBypO+644xvzod99993cc889nHXWWYwcOZIhQ4bw5ptvHvR7GzduZPBgf21WVFQwadIkhg4dymWXXdaic7kc0TdFnXO7zewjYAKQu9/y4v1WexJ4oEXSHUKvLv4JcTbv3Muo3kmt+VIi0l7NvBPyl7bsc3YfAufef8iHJ02axC233MINN9wAwPTp03nvvfe49dZbSUhIoKioiLFjx3LhhRce8pqgf//734mNjWXJkiUsWbKEkSNHtlj8JgvdzFKBmkCZxwBnc0Bhm1mac2574O6FwIoWS9iIjKQYzLSHLiJta8SIERQUFLBt2zYKCwtJSkoiLS2NW2+9lblz5xIWFsbWrVvZsWMH3bt3b/Q55s6dy0033QTA0KFDGTp0aIvla84eehrwnJmF4x+ime6ce8fM7gWynXNvATeZ2YVALbATuLbFEjYi2hdOWkI0G4v2tubLiEh7dpg96dY0ceJEZsyYQX5+PpMmTWLq1KkUFhaycOFCfD4fmZmZjU6bu79D7b0fqyYL3Tm3BBjRyPLf7nf7LuCulo12eP27x7Myv/FpLEVEWsukSZO47rrrKCoq4uOPP2b69Ol07doVn8/HnDlz2LRp02F///TTT2fq1KmcccYZ5ObmsmTJkhbLFpTfFAUY1KMzawrKNOuiiLSpQYMGUVpaSnp6OmlpaVxxxRVkZ2eTlZXF1KlTGTBgwGF/f8qUKZSVlTF06FD+9Kc/MWbMmBbLFnTT5+4zqEcCdfWOVfmlDOvZ+lfTFhHZZ+nSrw/GpqSkMG/evEbXKyvzf4UnMzOzYdrcmJgYXnrppVbJFbR76IPTOwOwZOsej5OIiLQPQVvoGUkxJHeKJGfzbq+jiIi0C0Fb6GbG8J6J5GzZ5XUUEWlDHWXKj6N5n0Fb6AAjeiWyrnAveyo0UZdIRxAdHU1xcXHIl7pzjuLiYqKjo4/o94L2oCh8PY6+YnsJY/sme5xGRFpbRkYGeXl5HMtcUMEiOjqajIyMI/qdoC70E9MSAFiVX6pCF+kAfD4fffr08TpGuxXUQy5d46NIivWxMr/E6ygiIp4L6kI3M/p3j2fFdn1jVEQkqAsdYED3BFbvKKW+PrQPkoiINCXoC/3EtHjKq+vYskszL4pIxxb0hT6gu//AqIZdRKSjC/pCP6FbPGbowKiIdHhBX+gxkeFkJndixXYVuoh0bEFf6OAfR9fc6CLS0YVGoXdPYFNxOWVVtV5HERHxTEgU+oCGb4xq2EVEOq6QKPQT0+IBWK4zXUSkAwuJQk9PjCEhOkIHRkWkQwuJQjczBqQlqNBFpEMLiUIHGJiWwKp8TQEgIh1Xk4VuZtFm9oWZLTazZWZ2TyPrRJnZy2a21swWmFlma4Q9nH1TAGzaqSkARKRjas4eehVwpnNuGDAcmGBmYw9Y57+AXc6544G/AQ+0bMym7Zsbffk2DbuISMfUZKE7v7LAXV/g58BxjYuA5wK3ZwBnmZm1WMpm6N89Hl+4sXTrnrZ8WRGRdqNZY+hmFm5mOUAB8IFzbsEBq6QDWwCcc7XAHuCgSwiZ2WQzyzaz7Ja+hFRURDgDuiewJG93iz6viEiwaFahO+fqnHPDgQxgjJkNPmCVxvbGDzo66Zx7wjmX5ZzLSk1NPfK0TRia0ZmleXt0YFREOqQjOsvFObcb+AiYcMBDeUBPADOLADoDO1sg3xEZmtGZ0qpaNhTvbeuXFhHxXHPOckk1s8TA7RjgbGDlAau9BVwTuD0RmO2ca/Pd5KEZiQAadhGRDqk5e+hpwBwzWwJ8iX8M/R0zu9fMLgys8zSQbGZrgZ8Dd7ZO3MPr1zWOaF8YS/J0YFREOp6IplZwzi0BRjSy/Lf73a4ELm3ZaEcuIjyMwT06s3iL9tBFpOMJmW+K7jOqdxJLt+6horrO6ygiIm0q5Ap9bN9kauocizbv8jqKiEibCrlCz8pMIjzMmLeu2OsoIiJtKuQKPT7ax+D0zsxfr0IXkY4l5AodYFzfZHK27KakssbrKCIibSYkC/2M/qnU1js+XVPkdRQRkTYTkoU+qncSnWN8fLiiwOsoIiJtJiQLPSI8jPH9U/loVQF1mtdFRDqIkCx0gLNO7Ebx3mqyN7b5lDIiIp4I2UI/+8SuxEaG8/pXW72OIiLSJkK20GMjIzh3cBr/XrKdyhp9a1REQl/IFjrAJSPTKa2q5f3lO7yOIiLS6kK60Mf2TaZH52hmLMzzOoqISKsL6UIPCzN+MLonc1cXsragrOlfEBEJYiFd6ABXje1NVEQYT32y3usoIiKtKuQLPTkuiomjMnht0VYKSiq9jiMi0mpCvtABJp/el3rnePDDNV5HERFpNR2i0Hsnd+KKk3rx0pdbNJYuIiGrQxQ6wE1n9SPGF87dby3Dg+tXi4i0ug5T6MlxUdwxoT+fri3ilWydxigioafDFDrAFSf15qQ+XfjtW7nkbt3jdRwRkRbVZKGbWU8zm2NmK8xsmZnd3Mg6481sj5nlBH5+2zpxj01YmPHI5SNJio3kpy8spKisyutIIiItpjl76LXAL5xzJwJjgZ+Z2cBG1vvEOTc88HNvi6ZsQanxUTxxVRZFZVXc8OIizfMiIiGjyUJ3zm13zi0K3C4FVgDprR2sNQ3J6MyfLx3GFxt38rOpi6iurfc6kojIMTuiMXQzywRGAAsaeXicmS02s5lmNugQvz/ZzLLNLLuwsPCIw7akC4f14A8XD+bDlQX85PlsSnX9UREJcs0udDOLA14FbnHOlRzw8CKgt3NuGPAw8EZjz+Gce8I5l+Wcy0pNTT3azC3myrG9eeCSIXy2tojvPfY5G4r2eh1JROSoNavQzcyHv8ynOudeO/Bx51yJc64scPtdwGdmKS2atJVcNroXL/zXGIrLqrjwkU95e/E2ryOJiByV5pzlYsDTwArn3F8PsU73wHqY2ZjA8xa3ZNDWdPJxKbx146kc3zWO/572FT+fnqMhGBEJOhHNWOcU4CpgqZnlBJb9P6AXgHPucWAiMMXMaoEKYJILsq9j9uwSy/SfjuPh2Wt5ZPYavty4k7/9YDhZmV28jiYi0izmVe9mZWW57OxsT167Kdkbd3Lr9By27qrg+m8dxy1nn0BkRIf6DpaItFNmttA5l9XYY2qpRmRldmHmzadz6aiePPbROi5+9DM2FeuAqYi0byr0Q4iLiuCBiUN58uostu2p4OJHP+PLjTu9jiUickgq9CZ8e2A33rjhFBJjI7niqQW8l5vvdSQRkUap0JshM6UTr005mUE9ErjxX4v4cMUOryOJiBxEhd5MSZ0ief7HYxjYI4Ebpi4iW8MvItLOqNCPQHy0j2evHU16Ygw//ueXrMov9TqSiEgDFfoRSo6L4rkfjyEmMpyrn1nA9j0VXkcSEQFU6EelZ5dYnvvxGMoqa7n15Rzq6oPqO1QiEqJU6EdpQPcE7r5wEPPX7+ShD9d4HUdERIV+LCaOyuD8IWk8+OEa3l+m0xlFxFsq9GNgZtx3yRCifWHc/uoS8vdUeh1JRDowFfoxSoj28e5Np1FVU89dry0hyOYkE5EQokJvAX1T47h9Qn/mrCrk1UVbvY4jIh2UCr2FXDMuk9GZSdz79jJ2lGjoRUTangq9hYSFGX+aOIyq2nruem2phl5EpM2p0FtQn5RO3DFhALNXFvBGjoZeRKRtqdBb2LUnZzK8ZyJ/fHelLmMnIm1Khd7CwsKMey4cRFFZFY/MXut1HBHpQFTorWBYz0QuHNaDf8xdz8yl272OIyIdhAq9lfzmuwMBuOmlr9hcXO5xGhHpCFTorSQlLorP7jyTqIhwfvNmrs56EZFW12Shm1lPM5tjZivMbJmZ3dzIOmZmD5nZWjNbYmYjWyducElPjOHn3z6Bj1cX6tJ1ItLqmrOHXgv8wjl3IjAW+JmZDTxgnXOBfoGfycDfWzRlELt6XG8GdI9nytRFrMwv8TqOiISwJgvdObfdObcocLsUWAGkH7DaRcDzzm8+kGhmaS2eNghFhIdx/yVDAfj+Y5+zvrDM40QiEqqOaAzdzDKBEcCCAx5KB7bsdz+Pg0sfM5tsZtlmll1YWHhkSYPY8J6JzPr5t4iKCOPn0xfrghgi0iqaXehmFge8CtzinDtw7MAa+ZWDWss594RzLss5l5WamnpkSYPc8V3j+N0Fg8jZspsX5m30Oo6IhKBmFbqZ+fCX+VTn3GuNrJIH9Nzvfgaw7djjhZaLhvfgtH4p3P32cl0QQ0RaXHPOcjHgaWCFc+6vh1jtLeDqwNkuY4E9zjl9o+YAZsZ93x8CwOQXFpK7dY/HiUQklDRnD/0U4CrgTDPLCfycZ2bXm9n1gXXeBdYDa4EngRtaJ27wy0iK5YtfnUWPztHc+K9F7K2q9TqSiIQI8+oLL1lZWS47O9uT124P5q8v5odPzueM/l15/MpRREboO14i0jQzW+icy2rsMbWIR8b2TWby6X2ZvbKAM/73I/ZUaGZGETk2KnQP/fI7/RnXN5mtuyuY8uJCnc4oIsdEhe6hiPAwpk0ey58mDuXzdcU8PHuN15FEJIip0NuBS0dl8P0R6fzfrDVc8dR8yqt1oFREjpwKvR0wM/7wvcH0TenEZ2uL+dnURRp+EZEjpkJvJ2IjI5h923j+cPFg5qwq5N63l1FbV+91LBEJIir0dubKsb25amxvnpu3idH/M4v8PZVeRxKRIKFCb4d+e8FALhreg13lNVzx1Hx27q32OpKIBAEVejvkCw/jwUkjeHnyWPJ2VTDy9x8wY2Ee9RpXF5HDUKG3Yyf1TebRy/0Xf7rtlcX86o2lupSdiBySCr2dO3tgN9675TSuGtubaV9s4ffvrNBpjSLSKBV6EBjQPYF7LxrEpNE9eeazDQz87X946YvNXscSkXYmwusA0jxmxh8uHoxz8HL2Fu56fSlmcNHwdKJ94V7HE5F2QLMtBqHKmjp+9OyXzFtfDMCd5w7gp6f3xT91vYiEMs22GGKifeE8+6PRfG+E/7Kt989cye0zluj0RpEOToUepKJ94fztsuFsuO88rj05k1cW5jHy9x8wdcEmr6OJiEdU6EHOzPjdBQOZMv44AH71ei6/mL6YT9cUaT4YkQ5GY+ghpK7ecfdby3hhvn8vfVTvJJ64ahTJcVEeJxORlnK4MXSd5RJCwsOMey8axJCMzsxZWcDslQWM+sMszh+axp0TBtCzS6zXEUWkFWkPPYR9uXEnlz4+D4C4qAgev3IUp/ZL8TiViByLw+2hq9BDXEFJJVt2lfOr13NZmV8KwN+vGMm5Q9I8TiYiR0OnLXZgXROiGdW7Cy9PHseYPl0AmDJ1EbfPWMzHqws14ZdICGlyD93MngG+CxQ45wY38vh44E1gQ2DRa865e5t6Ye2he6Oqto7fv7OcF+f7pw4YltGZRy4fqfF1kSBxrAdF/wk8Ajx/mHU+cc599yiySRuLigjnDxcPYXRmF6bO38yK/BJO+9Mc4qIiePjyEZzRv6vXEUXkKDU55OKcmwvsbIMs0oYuGp7O9OvH8faNpxLtC6OsqpYfPfslv3kjl8/WFmkoRiQItdRpi+PMbDGwDbjNObessZXMbDIwGaBXr14t9NJyLDJTOrH07nPYVV7No7PX8ty8TbwwfxPDMjrz50uHkZEUQ2ykzm4VCQbNOsvFzDKBdw4xhp4A1DvnyszsPOBB51y/pp5TY+jt03u52/nL+6vZubea4sDcMBcN78Gvzx9Iary+oCTitWM+bfFwhd7IuhuBLOdc0eHWU6G3b8VlVdz99nLeXryN8DBrmEbgoR+O4IKhaZrZUcQjrVroZtYd2OGcc2Y2BpgB9HZNPLEKPXisyi/l6mcWsKOkCoCRvRI5qW8yl2X1JDOlk8fpRDqWYyp0M5sGjAdSgB3A7wAfgHPucTO7EZgC1AIVwM+dc583FUqFHlxq6+rZVV7DrBU7uOu1pQ3Lf3lOf0b0TGR4r0SNtYu0AX1TVFrUhqK9vLt0O8u3lfDvpdsBCDN46posuifEcHzXOCIj9J01kdagQpdW8+7S7dw3cwVhZmwqLgcgITqCZ64dzbCeifjCVewiLUmFLq2uqraOhz9cyyNz1uILN2rq/J+rk/p04cFJI+jeOdrjhCKhQYUubWpvVS33z1zZMC97VEQYVbX1TD69Lzef1Y9OURprFzlaKnTxzIaivUx5cWHDTI89u8SQ3CmKyaf35TzN+ChyxFTo4innHOuL9lJUWsWUqYsaLmZ98fAenDskjRO7J9ArWZODiTSHCl3ajYrqOmbmbmdtQRmPfbSuYfnvLhjIaf1S6JYQTXy0z8OEIu2bCl3apdyte/iff69gy65y8nZVNCyf+pOTOPm4ZH0bVaQRKnRp15xzvJGzlVtfXvyN5aN6J/HYFSPplqAzZET2UaFL0KioruP+mSt4bp7/DJnEWP/wy8SRGfxyQn+iIsK9jCfiORW6BJ2isip27q3m9hlLyNmyG4BBPRI4b0gax6XGMWFwd48TinhDhS5Bq77eMXtlAdtLKvnr+6vYVV4DwFVje3PdaX3xRRhpnWM8TinSdlToEhIKSip5aPYasjfuajivHeDZa0czvn8q9Q7Cw3QgVUKbCl1CzpyVBfzon19+Y1l4mPHezafRr1u8R6lEWp8KXUJWZU0dj8z2zyEDEBsZzhn9u5KeFMMvz+mvycEk5Byu0DWphgS1aF84t53Tn8tG96SksoZfv5HbMKXv0rw93D6hPw4Y2SvJ26AibUB76BJSqmrreObTjSzctJPP1hZTUVMHwL0XDeKqsb2prqvXqY8S1DTkIh3ShqK9/PSFbFbvKPvG8hnXjyMrs4tHqUSOjQpdOrS6escD763kibnrAYiMCOOacb1xDm47pz/RPu2xS/DQGLp0aOFhxv8770QuGZnBrvJq/vyfVTz5yQYAlm8v4YFLhrKrvJqhGYkeJxU5NtpDlw6npq6ef3y8jpe+3EJhaRVVtfUA3H3BQK49pQ/19Y4wnc8u7ZSGXEQOYfm2EiY+/jnl1f6Dp90SothRUsW068Yy7rhkj9OJHOxwhd7kSbpm9oyZFZhZ7iEeNzN7yMzWmtkSMxt5rIFF2srAHgksvfsclt1zDtd/6zh2lFQBcOXTC3j4wzX8YvrihgtyiLR3Te6hm9npQBnwvHNucCOPnwf8N3AecBLwoHPupKZeWHvo0h4tWF/M8u0l/HvJdrI37QL8e+3TrhvLsm0lnDu4OxH6spJ46JiHXMwsE3jnEIX+D+Aj59y0wP1VwHjn3PbDPacKXdqz2rp6nvxkA/+Yu47q2vqGIZnRmUlM/clYivdWaVIw8cQxDbk0QzqwZb/7eYFljQWZbGbZZpZdWFjYAi8t0joiwsOYMv44cn77Hf5902mkJ/rL+8uNuzjh1zMZd99s7n5rGV4dgxJpTEucttjY6QCNfsqdc08AT4B/D70FXluk1fVJ6cQnt59BRU0dn6wp5PoXFwHwz883UlhaxZ6KGm6f0F+nPYrnWqLQ84Ce+93PALa1wPOKtBthYUanqAgmDE5j3l1nsnDTLnK3lvD4x/4LXX/6SBFPX5PFiu0lTBzVk+6dddk8aXstMYZ+PnAjXx8Ufcg5N6ap59QYuoSC95fl88Tc9RSUVrF5Z3nD8k9uP4NNxeWM7J1IbKS+vyct55gOiprZNGA8kALsAH4H+ACcc4+b/9LsjwATgHLgR865JptahS6hpLKmjrvfWsZLX275xvJoXxgL7jqbgtJKMlM6aTpfOWb6YpFIG3HOsXlnOTdMXcSybSUApMRFUlRWTXpiDB/+4lsAREWE4d8XEjkyKnSRNuacY1NxOfkllVz3XDalVbUADOgez8r8Usb1Teb5/xqjPXY5Yip0EQ+VV9cyd3URANe/uLBh+fj+qWzbXcFp/VL59fknAmivXZqk2RZFPBQbGcGEwd0B/8HS6dlbiAgL42+zVgOwekcZ5dV1fLV5F+mJMTx+1SjtuctR0R66iEeW5u3h2c82sLuihtkrCxqWn9YvhZq6etITY7nv+0OIjFC5y9e0hy7SDg3J6MxfLxsOwOfrinhx/ibio3y8kbM1MKXvTrbsKicizKh3jqeuGU1cVATOOQ3NSKO0hy7SzizfVsKDH66mU1QE7yzZTnVgvvb+3eJJjPVRWVvPS9eNJTxQ9LriUseig6IiQWrF9hKemLueLp0ieSV7CyWV/rNlBvVIYHd5DVt3VzD/rrOIjQqnuraelLgojxNLa1Ohi4SAzcXlPD53HfX1jlkrCigq88/dHh8dQWmg6N/42Sn0TIphd0UNx6XGeRlXWokKXSTE7Kmo4alP1tM5xsesFTuYv34n4L8A9r4hmr9dNozhPZNYsd0/j7vG3UODCl0khDnneGfJdiIjwnglewuzVhQctM7V43pzzqDuzMzdzq/PH0i0L5y6eke4rp0adFToIh2Ec473l+8gIdrHhyt28NSnGw5aZ0xmF8b06cIjc9by6pSTGZrRma27KshM6eRBYjlSKnSRDmpT8V6cg4WbdnHX60uprq0nITqi4eCqL9zo0imSHSVV3DFhAJdmZfDO4m1cflJvnf/eTqnQRYSaunoM//j7/76/moKSSuKiI3gz5+DLFwzN6Mzt5wzgLx+s4ncXDGJQjwQ+X1fMqcenaJjGYyp0ETmkD1fsYPueSmrq6rnn7eWNrjNhUHfeW5bP4PQEHr18JHe8uoRbzj6BsX2TWZlfwgld4wlT0bcJFbqINMvW3RXsKKmkc4yPX0xfTM6W3YzolchXm3c3uv61J2fyz883MiazC8/9eAwPvLeSH47pRf/u8VTX1mvYphWo0EXkqDjncA7eX57P6h1lZGUmcdO0HIrKqjgutRPrCvc2rLtv3neABycN55evLOHMAV15+PIRzFiYx9i+yfRJ6aSpC46RCl1EWkxdvaOgtJLuCdG8uzSfj1cXML5/Vx6ctYZVO0oPWr93ciybiv2X53v22tHc+85yju8ax8M/HMF7ufn0SIxhTJ8ulFXVEuML1xh9E1ToItLq6uodq/JL6dctjqnzNzE9O4/zh6bx4Kw1VNfVH7T+4PQEcrf6r+r0+4sG8dSnG9i5t5r3bjmdaQs2E+0LY8r449m8s5zIiDDSE2Ooq3c454jowNMLq9BFxDMllTV8vKqQ009I5YV5G3k5ewvfPrE7b+ZspXhv9UHrx0dFNFzh6bR+KXyyxn9xkId/OIL3luUzd1Uh7958GhuK9pK/p5JLszIo3ltNSUUNfTvAdAcqdBFpd8qra/l4VSFj+ybz2boipn2xmQmD0/hoZQEfriwgJS6KvVW1VNTUHfS7sZHhlFf7l180vAebd5bz1ebd/Pr8E4nyhfOvBZv54/cGU1Vbz3+W5XPbd/pTUlnDwk27OH9IGs7BrvJqkoNwMqlFmHwAAAjBSURBVDMVuogElW27K+jSKZJd5dVMnb+ZU/ulkLergv+btZpxfZPZW13Lu0vzAYgIM2rrD+6xMIN9i/umdKJzrI+vNu/mOwO70a9bHI/OWcdNZx7PyN5J3PnqUu77/hCO7xrHo3PWcvPZ/YiLiuCV7DwmZmXQKTKCBRuKGZPZhYjwMPaU19A51teWm6TBMRe6mU0AHgTCgaecc/cf8Pi1wJ+BrYFFjzjnnjrcc6rQReRYrNheQucYH3X1jufnbaR/9wRq6+r53/dXM7BHAl3jo5ixMA/gG2fkRPvCqKw5eEwfIDI8rGG8f3z/VD5aVYgv3Lj+W8fx8Oy1ZCTF8JvvDuSnLyzkWyek8sfvD+H6Fxby/ZHpXH5SL37zRi4XDOvBqcen8MB7q/jWCamMOy6Zl7/czLCeiQzonsDTn25gSHpnxvTpclTv+5gK3czCgdXAt4E84Evgh8655futcy2Q5Zy7sbmhVOgi0to2FO3FgIykGN7M2YYDTjk+mb++v5rymjrOG5zGrdNzCDdj4qgMXpi/CfBfTGTfGTvhYUZdI38BHCguKoKywNj/WQO68mHgsoJXj+vN8/P8z/vfZx7Pw7PX8rMzjuOX5ww4qvd0rJegGwOsdc6tDzzZS8BFQONfKRMRaSf67Dfh2CWjMhpu//nSYQ23zxnUjaraejpFRfCT0/pQUFpFVu8kPlxRwNrCMn50SiYvfbGFnC27uX1Cf575dAMfry7krnNP5I/vrmBNQRlTxh/H04GJ0Pp3i28o89jI8IYyB3h49loAOkW1ztU/m7OHPhGY4Jz7SeD+VcBJ+++NB/bQ7wMK8e/N3+qc23K459UeuogEO+cceypqSIyNpLKmjtytexjVO4mC0io+XlXIBcN6kLernP8sy2fSmF7k76lkxsI8rjk58xv/szkSx7qH3thZ/gf+X+BtYJpzrsrMrgeeA85sJMhkYDJAr169mvHSIiLtl5mRGBsJQLQvnKxM/7h4t4RofjC6JwD9usXTr1s8AClxUQxO79xqeZpzdn4e0HO/+xnAN6Znc84VO+eqAnefBEY19kTOuSecc1nOuazU1NSjySsiIofQnEL/EuhnZn3MLBKYBLy1/wpmlrbf3QuBFS0XUUREmqPJIRfnXK2Z3Qj8B/9pi88455aZ2b1AtnPuLeAmM7sQqAV2Ate2YmYREWmEvlgkIhJEDndQtOPOcCMiEmJU6CIiIUKFLiISIlToIiIhwrODomZWCGxqcsXGpQBFLRintQVT3mDKCsGVN5iyQnDlDaascGx5ezvnGv0ij2eFfizMLPtQR3nbo2DKG0xZIbjyBlNWCK68wZQVWi+vhlxEREKECl1EJEQEa6E/4XWAIxRMeYMpKwRX3mDKCsGVN5iyQivlDcoxdBEROViw7qGLiMgBVOgiIiEiqArdzCaY2SozW2tmd3qdZx8z22hmS80sx8yyA8u6mNkHZrYm8G9SYLmZ2UOB97DEzEa2Qb5nzKzAzHL3W3bE+czsmsD6a8zsmjbMereZbQ1s3xwzO2+/x+4KZF1lZufst7zVPytm1tPM5pjZCjNbZmY3B5a31217qLztdftGm9kXZrY4kPeewPI+ZrYgsK1eDkzrjZlFBe6vDTye2dT7aIOs/zSzDftt2+GB5a3zWXDOBcUP/ql71wF9gUhgMTDQ61yBbBuBlAOW/Qm4M3D7TuCBwO3zgJn4rwQ1FljQBvlOB0YCuUebD+gCrA/8mxS4ndRGWe8Gbmtk3YGBz0EU0Cfw+Qhvq88KkAaMDNyOx3/5xYHteNseKm973b4GxAVu+4AFge02HZgUWP44MCVw+wbg8cDtScDLh3sfbZT1n8DERtZvlc9CMO2hN1ys2jlXDey7WHV7dRH+S/ER+Pfi/ZY/7/zmA4n2zQuEtDjn3Fz889QfS75zgA+cczudc7uAD4AJbZT1UC4CXnLOVTnnNgBr8X9O2uSz4pzb7pxbFLhdiv/CLum03217qLyH4vX2dc65ssBdX+DH4b+85YzA8gO3777tPgM4y8zsMO+jLbIeSqt8FoKp0NOB/S88ncfhP4xtyQHvm9lC8183FaCbc247+P9DAroGlreX93Gk+bzOfWPgT9Nn9g1hHCZTm2cN/Hk/Av+eWbvftgfkhXa6fc0s3MxygAL85bYO2O2cq23ktRtyBR7fAyS3Vd4Dszrn9m3b/wls27+ZWdSBWQ/IdExZg6nQm3Oxaq+c4pwbCZwL/MzMTj/Muu35fcCh83mZ++/AccBwYDvwl8DydpHVzOKAV4FbnHMlh1u1kWXtIW+73b7OuTrn3HD81zIeA5x4mNf2NO+BWc1sMHAXMAAYjX8Y5Y7WzBpMhd7kxaq94pzbFvi3AHgd/wdvx76hlMC/BYHV28v7ONJ8nuV2zu0I/MdSj/8i5Pv+XPY8q5n58JfjVOfca4HF7XbbNpa3PW/ffZxzu4GP8I83J5rZvstn7v/aDbkCj3fGP3zXpnn3yzohMMzlnHNVwLO08rYNpkJv8mLVXjCzTmYWv+828B0gF3+2fUeorwHeDNx+C7g6cJR7LLBn35/nbexI8/0H+I6ZJQX+JP9OYFmrO+AYw/fwb999WScFzm7oA/QDvqCNPiuB8dmngRXOub/u91C73LaHytuOt2+qmSUGbscAZ+Mf958DTAysduD23bfdJwKznf9I46HeR2tnXbnf/9gN/1j//tu25T8LR3tU14sf/EeGV+MfR/uV13kCmfriP4K+GFi2Lxf+sbsPgTWBf7u4r4+GPxp4D0uBrDbIOA3/n9I1+PcA/uto8gE/xn9AaS3wozbM+kIgy5LAfwhp+63/q0DWVcC5bflZAU7F/+fwEiAn8HNeO962h8rbXrfvUOCrQK5c4Lf7/Tf3RWBbvQJEBZZHB+6vDTzet6n30QZZZwe2bS7wIl+fCdMqnwV99V9EJEQE05CLiIgchgpdRCREqNBFREKECl1EJESo0EVEQoQKXUQkRKjQRURCxP8H8bLEaGiBl6kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuMAAAD4CAYAAABG8+jVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3xV9f3H8dcnO2RvIAlJICFsAgmooIijIIriAsG9qj+F2rpaba1WW1v31taFdaCouGjdG0FkJECYgZCQEGZIQva+398fudiUBhIgN+eOz/PxyKM3555zeJ8iN598z+d8v2KMQSmllFJKKdXzvKwOoJRSSimllKfSYlwppZRSSimLaDGulFJKKaWURbQYV0oppZRSyiJajCullFJKKWURH6sDWCk6OtokJydbHUMppY5Ydnb2PmNMjNU5epJ+ZiulXNXhPrM9uhhPTk5m5cqVVsdQSqkjJiJFVmfoafqZrZRyVYf7zNY2FaWUUkoppSyixbhSSimllFIW0WJcKaWUUkopi2gxrpRSSimllEW0GFdKKaWUUsoiDi3GReQMEckTkXwRuaOD9/1F5G37+8tEJLnde3fat+eJyOTOzilt7heRzSKyUURucuS1KaWUUkopdawcNrWhiHgDzwK/AEqAFSKy0Bizod1u1wAVxphUEZkJPAhcJCJDgJnAUKAv8JWIDLQfc6hzXgkkAoOMMTYRiXXUtSmllFJKKdUdHDnP+Fgg3xhTACAi84FpQPtifBrwJ/vrBcAzIiL27fONMY1AoYjk28/HYc55A3CxMcYGYIzZ68BrU0BNYwuLt5SytbSWVpshwNeLAF9vQgJ8iAsJIC4sgD5hAfTy8+jp7JVSFnrphwJsxjCsbxhjUyLx8dbuTKVU5yrrmympqGNHRT3765upbWyhtrGFphYbZ43oS3rvkG77sxxZJcUD29t9XwIcd6h9jDEtIlIJRNm3/3TQsfH214c65wDaRtXPA0qBm4wxWw4OJSLXAdcB9OvX78ivStHcauP577fyj+8LqGls6XT/+PBA0nuHkN47hFGJ4YxJjiQiyK8HkiqlPN28ZcUU7qsFIDEykN9PGcyU4X0sTqWUcia7KxtYVljGuh2VrNtRxYZdVVTWN3e4rwikxYW4TDEuHWwzXdznUNs7GtI4cE5/oMEYkyUi5wNzgZP+Z2djXgBeAMjKyjo4j+pEeW0T17++khXbKpg0JI6rT0xhZEI4vt5CQ4uNhuZWquqb2V3VwJ6qBnZU1LN5Tw2b91Tzw5ZSmlvb/i9PjwthfGo0vxgSx5jkCB2tUko5xDe3nkxVfQuL8/fxzLf53DAvh8tPSOKes4fi7dXRjxqllLtrabWxtKCMrzfuZXH+PvL31gDg5+PF4N4hnDm8DynRvUiI6EV8eCCRQX4E+/sQ5O+Dn0/31yuOLMZLaOvhPiAB2HmIfUpExAcIA8o7OfZQ20uA9+yvPwBeOcb86iAVtU1c/OJPFO6r5cmZGUzLiP+v94O9vQj29yE62J/+McH/c3xjSyu5JZUsLyznp4Iy3lhWxNwlhYT38uX0wXFcMDqB41Ii8dIfkEqpbiIihPXy5awRfZg0NI4HP93ES4sLqWlo4ZHpI/XzRikPYYxhZVEFH67awWfrdlNW20SArxdjU6KYkZXAuAHRpPcOwdeCwUFHFuMrgDQRSQF20PZA5sUH7bMQuAJYClwIfGOMMSKyEHhTRB6j7QHONGA5bSPmhzrnh8CptI2InwxsduC1eZzmVhvXv5FN4b5aXroii5PSYo74HP4+3oxJjmRMciSzT0mltrGFRZtL+XLDHj5ft5sF2SUkRgYyPTORmWMSiQ0NcMCVKKU8la+3F3dNHUJYoC+PfrmZ+IhAbp2UbnUspZQDVTc088GqHbzxUxGb99QQ6OvNaYNjmTqiLxPTYwjw9bY6ouOKcXsP+Bzgc8AbmGuMWS8i9wErjTELgZeB1+0PaJbTVlxj3+8d2h7MbAFmG2NaATo6p/2PfACYJyI3AzXAtY66Nk/04KebWF5YzhMXZRxVId6RIH8fpgzvw5ThfahvauWz9bt4d2UJj325mae/2cK0jHh+eVL/bu3LUkqpOaemsr2ijqe/yWd4fBiThva2OpJSqpuVVjfy0uIC3lhaRG1TK8Pjw3joghFMHdnH6SaWEGM8t206KyvLrFy50uoYTm/p1jJmvfgTl5+QxH3Thjn8z9u2r5ZXlhTyzsoS6ptbmZgew82nD2RkYrjD/2ylXIWIZBtjsqzO0ZO68zO7saWVC/7+I7v2N/DlLScTqQ+VK+UW9lQ18PfvtvLW8mKaW9tmPrn2xBTLa4jDfWZrMa7F+GE1NLdyxhOLsBn4/DcTCPTruds5++uamLesmJd+KKCirplJQ+K4dVK6jpQrhRbj3WHjrirOeWYxZw7vw5MzR3XbeZVSPa+2sYXnFxXw4qICmlttnD86nhsmppISHWR1NODwn9nONU6vnM4/vt/KtrI65l17XI8W4gDhvfyYfUoqV4xLZu7iQl5cVMAZTy7iwtEJ/G7KIKKD/Xs0j1LKvQzuE8oNE1N56ustXHJcEmNTIq2OpJQ6Qjab4d3s7TzyxWZKqxs5a0Qffjd5EP2ielkdrct0Pjl1SKXVjbywqIAzh/dmfGq0ZTmC/X246bQ0fvjdKfzypP58uHoHpzzyHXMXF9LcarMsl1LK9d1w8gB6hwZw/8cbsNk8906xUq5o855qZjy/lN+9t5bEiEDeu2Ecz1482qUKcdBiXB3G099sobHFxm1OMttAeC8/fn/mYD77zQQyEsO5798bmPrUYlYVV1gdTSnlogL9vLltcjprSir599pdVsdRSnVBQ3Mrj36Rx1lP/UB+aQ0PXziC924YR2ZShNXRjooW46pD28vreHNZMTPHJHY4Z7iVBsQE89rVY3n+skyqGpq54O8/8tdPNtLQ3Gp1NKWUCzp/VDwD44J55pstOjqulJNbv7OSqU8v5ulv8pk6oi9f33Iy07MSEXHdNQO0GFcdenlxIQCzT0m1OEnHRITJQ3vzxc0TuGhMP15YVMCUJ39g5bZyq6MppVyMl5dw48RUNu+p4cuNe6yOo5TqgM1meP77rZz77BIq65v551VjePyiDKLc4PkxLcbV/6iobeLtFds5J6MvfcMDrY5zWCEBvvzt/OG8cc1xNLXYmPH8Up74ajOtOrqllDoCU0f0oV9kL579Nh9PnmVMKWe0u7KBS15axt8+3cSpg2L5/DcTmJgea3WsbqPFuPofr/9URH1zK9dN6G91lC47MS2az2+ewLSMeJ74aguzXvyJnfvrrY6llHIRPt5e/N/JA8gtqWRZod5hU8pZLN1axtSnf2BNyX4evGA4/7g00+3WBdBiXP2XphYbry3dxsT0GAb1DrU6zhEJ9vfh8YsyeHT6SNbtqOTMp37gqw16y1kp1TXnjYonLNCX15cWWR1FKY9njOGFRVu59OVlhAb68tHs8Vw0pp9L94Yfihbj6r98vn43+2qauHJcstVRjtoFmQn8+1cnkhARyLWvreTxLzfrQ1lKqU4F+nkzIyuBz9fvZndlg9VxlPJYtY0tzH4zh79+solJQ+L4aPZ40uLcd8E/LcbVf3lzWTEJEYFMSIuxOsox6R8TzIL/G8cFoxN48ustXPd6NtUNzVbHUko5uUuPT6LVGN5cXmx1FKU80q7Keqb/YymfrdvNH84czHOXjCYkwNfqWA6lxbj6WUFpDUsLypg1th9eXq5/GyjA15tHpo/gT2cP4du8vUx7dglbS2usjqWUWxGRM0QkT0TyReSODt5PEpGvRSRXRL4TkYSD3g8VkR0i8kzPpT60pKggJg6MYf7yYlp0UTGletS6HZWc++wSisvrmHvlGH45ob9btqUcTItx9bP5K7bj4yVMz0rofGcXISJcOT6FedceR2VdM+c9u4SlW8usjqWUWxARb+BZYAowBJglIkMO2u0R4DVjzAjgPuBvB73/Z+B7R2c9EjOyEtlb3cji/H1WR1HKY3y1YQ/T/7EUHy8vFtxwglvNltIZLcYVAC2tNt7PKeG0wbHEhgRYHafbHd8/io/mjCcuNIDL5y7jg1UlVkdSyh2MBfKNMQXGmCZgPjDtoH2GAF/bX3/b/n0RyQTigC96IGuXnTo4lvBevryXs8PqKEp5hHnLivjl6ytJiwvmg9njXG4CiWOlxbgCYMnWMvbVNHHeKPcZFT9YQkQvFtwwjqykSG5+ew1Pfb1F5xNW6tjEA9vbfV9i39beGuAC++vzgBARiRIRL+BR4PbD/QEicp2IrBSRlaWlpd0U+/D8fbw5Z2Rfvli/myp91kQphzHG8Oy3+fzhg3Wckh7L29ed4JYDgp3RYlwB8NGqHYQG+HDKINd+cLMzYYG+vHr1WM4fFc9jX27md+/lal+oUkevo2bOg3/DvQ04WURWAScDO4AW4EbgE2PMdg7DGPOCMSbLGJMVE9Nzn08XjE6gscXGx7m7euzPVMqTGGP426ebePjzPM7N6Mvzl2US6OdtdSxL+FgdQFmvvqmVz9fv5uyRffH3cf9/CH4+Xjw6YyQJEYE89U0+VfUtPDkrwyOuXaluVgIktvs+AdjZfgdjzE7gfAARCQYuMMZUisgJwEkiciMQDPiJSI0x5n8eArXCiIQwUmODeT+nhFlj+1kdRym30moz/P79tby9cjuXn5DEn84e6hYTRxwtHRlXfLVxD7VNrZyT0dfqKD1GRLhlUjp3Tx3CZ+t3c+2rK6lrarE6llKuZgWQJiIpIuIHzAQWtt9BRKLtLSkAdwJzAYwxlxhj+hljkmkbPX/NWQpxaPuMOGdkX1YWVbCnSuccV6q7tLTa+M3bq3l75XZuOjWVe8/x7EIctBhXwEerd9A7NIDjU6KsjtLjrj4xhYcvHMGS/H1c9vJyKuu1P1SprjLGtABzgM+BjcA7xpj1InKfiJxj320ikCcim2l7WPN+S8IehTOH98EY+HSttqoo1R1aWm3c8s4a/rVmJ787YxC3TEr3iKkLO6PFuIeramjm+82lTB3Rx2N/M52elcizF48mt2Q/M1/4ibKaRqsjKeUyjDGfGGMGGmMGGGPut2+72xiz0P56gTEmzb7PtcaY//kHZoz5pzFmTk9n70xqbDDpcSF8rMW4UsespdXGre+uYaG9EL9h4gCrIzkNLcY93Leb9tLcapgyvLfVUSw1ZXgfXrpiDIX7arjkpWVakCulADhrRB9tVVHqGLXaDLe+u4aPVu/kt2ekayF+EC3GPdzn63cTE+LPqMQIq6NY7uSBMbx8xRgK99VyyUvLKK9tsjqSUspi2qqi1LGx2Qy32Qvx2yenc+PEVKsjOR0txj1YQ3Mr3+WVMmlInMe2qBxsfGr0zwX5pS8to0ILcqU82oFWlU/W7rY6ilIuxxjDPQvX88GqHdz6i4HMPkUL8Y5oMe7BFm/ZR11TK5OHenaLysFOTIvmxcuzyC+t4dKXl7G/TgtypTzZ5KFxrCwq11/OlTpCj3+5mdd/KuL6Cf2Zc6oW4oeixbgH+2z9bkICfDi+v+fNotKZCQNjeOGyTLbsqeGyl5dTravwKeWxTh8Sh83At3l7rY6ilMt4eXEhT32Tz0VZidwxZZDOmnIYDi3GReQMEckTkXwR+Z/5Y0XEX0Tetr+/TESS2713p317nohM7uycIvJPESkUkdX2rwxHXpura2m18fXGPZw2KBY/H/2drCMT02P5x2Wj2birimtfXUlDc6vVkZRSFhjWN4zYEH++2rjH6ihKuYQF2SX8+d8bmDKsN389f7gW4p1wWBUmIt7As8AUYAgwS0SGHLTbNUCFMSYVeBx40H7sENoWjxgKnAE8JyLeXTjn7caYDPvXakddmztYWVRBRV0zk7RF5bBOHRTHozNGsnxbObPn5dDcarM6klKqh3l5CacNjmXR5n00tehngFKH8+WGPfzuvVxOTI3miZkZeOszaZ1y5JDoWCDfGFNgjGkC5gPTDtpnGvCq/fUC4DRp+/VpGjDfGNNojCkE8u3n68o5VRd8m7cXX2/hpLRoq6M4vWkZ8fx52jC+3rSX299dg81mrI6klOphpw2Ko6axhWWFZVZHUcpprd6+n1+9lcOw+DCevywTfx9vqyO5BEcW4/HA9nbfl9i3dbiPfSW3SiDqMMd2ds77RSRXRB4XEf/uuAh39X1eKVlJkYQE+FodxSVcenwSt09O58PVO7n3X+sxRgtypTzJ+NRoAny9+GqDtqoo1ZHt5XVc++oKYkL8efmKLIL8fayO5DIcWYx3dF/i4ArmUPsc6XaAO4FBwBggEvhdh6FErhORlSKysrS0tKNd3N7O/fVs2l3NxPQYq6O4lBsnDuC6Cf15dWkRT3y1xeo4SqkeFOjnzYmp0Xy1ca/+Mq7UQfbXNXHFK8tpbjX886qxRAfreOiRcGQxXgIktvs+Adh5qH1ExAcIA8oPc+whz2mM2WXaNAKv0NbS8j+MMS8YY7KMMVkxMZ5ZjH6/ue2XkFMGxVqcxLWICHdOGcT0zASe/HoL85cXWx1JKdWDThscx4799WzZW2N1FKWcRmNLK9e9nk1JeT0vXJbJgJhgqyO5HEcW4yuANBFJERE/2h7IXHjQPguBK+yvLwS+MW1DDguBmfbZVlKANGD54c4pIn3s/yvAucA6B16bS/suby99wwJIi9V/MEdKRPjr+cOZMDCGP3y4Tqc6U8qDTBjYNoCzaLNn3lVV6mA2m+G3C3JZXljOw9NHcJxOlXxUHFaM23vA5wCfAxuBd4wx60XkPhE5x77by0CUiOQDtwB32I9dD7wDbAA+A2YbY1oPdU77ueaJyFpgLRAN/MVR1+bKmlpsLMkvY+KgWJ1q6Cj5envx3CWjGdQ7hNnzcli3o9LqSEqpHhAfHsiAmCAWbdlndRSlnMITX2/5eZn7aRkHPxaousqh3fXGmE+ATw7adne71w3A9EMcez9wf1fOad9+6rHm9QQri8qpaWxh4kDPbNHpLsH+Prxy5RjOe+5HrvrnCj64cRwJEb2sjqWUcrAJA2N4c1kxDc2tBPjqTBHKc32cu4unvt7C9MwEbpw4wOo4Lk1Xe/Ew3+eV4ustjEvVKQ2PVWxoAP+8agwNza1c+coKKut0lU6l3N2EgTE0tthYsa3c6ihKWWb9zkpue3cNo/uF85fzhumd9mOkxbiH+X5zKWOSIwnWKYe6RVpcCC9clkVxWR3Xvb5SFwRRys0dlxKJn7eX9o0rj7WvppHrXssmvJcv/9C5xLuFFuMeZG91A5t2V3NSmraodKcTBkTx0IUjWFZYzt0frdNpz5RyY738fBiTEsGizdo3rjxPU4uNG97Ipqy2kRcvzyI2JMDqSG5Bi3EPsnRr28px41P1aefudu6oeGafMoD5K7Yzd8k2q+MopRxoQloMeXuq2VPVYHUUpXqMMYa7P1rHim0VPHzhSIbFh1kdyW1oMe5BluTvIzTAh6F99R+QI9z6i3QmD43j/o836JSHSrmxA3cXtVVFeZLXlhYxf8V2Zp8ygLNH9rU6jlvRYtxDGGNYkl/GCQOi8PbSBy0cwctLeGxGBum9Q7npzVXk7622OpJSygEG9wkhJsSfH3SKQ+UhVm4r58//3sDpg2O59RfpVsdxO1qMe4ji8jp27K9nvM6i4lBB/j68dEUW/r7eXPPqSipqm6yOpJTqZiLCCf2jWFpQps+IKLdXWt3I7DdzSIgI5LGLMvDSAb1up8W4h1iS39YvPm6AFuOOFh8eyAuXZ7KrsoEb5mXrDCtKuaFxA6IorW5ka2mt1VGUcpiWVhu/eiuHyvpm/n5pJqEBvlZHcktajHuIJVv30Ts0gAExQVZH8Qij+0Xw0AUj+Kmg7daeUsq9nDCg7UH4pVu1VUW5r4c/z+OngnL+et5wBvcJtTqO29Ji3APYbIalW8sYlxqlE/P3oHNHxXPdhP68/lMR767cbnUcpVQ36hfZi75hASwtKLM6ilIO8dm6XTy/qIBLj+/H+aMTrI7j1rQY9wCbdldTXtvEeG1R6XG/nZzO+NQo/vDhOnJL9lsdRynVTUSE4wdE8VNBOTab9o0r97K1tIbb3s1lZGI4f5w6xOo4bk+LcQ/wo/02qj682fN8vL14etZoYoL9+b/XsymrabQ6klKqm4wbEE15bRN5e3TmJOU+6ppauOGNbHy9hb9fMlpX2OwBWox7gCX5++gfE0TvMF0pywqRQX7849JM9tU2MefNVbS06gOdSrmD//SNa6uKch93fbiOLXtreGrWKPqGB1odxyNoMe7mWm2GFdsqOL6/rrpppeEJYfztvOEsLSjjgU83WR1HKdUN4sMD6RfZS/vGldtYkF3C+zk7uOnUtJ8Xt1KOp8W4m9u4q4qaxhaOS4m0OorHuyAzgStOSOKlxYV8tHqH1XGUUt3ghP5R/FRQRqv2jSsXl7+3hj9+uI7j+0dy02lpVsfxKFqMu7llheUAHJeiI+PO4K6pQxiTHMHv3stl464qq+MopY7RuNQoqhta2LBT/z0r19XQ3MqcN3MI9PPmyZmjdKXuHqbFuJtbVlBGUlQv7Rd3Er7eXjx7yWhCA3y5cV4O1Q3NVkdS6qiJyBkikici+SJyRwfvJ4nI1yKSKyLfiUiCfXuGiCwVkfX29y7q+fTd4wR7C+DSAp1vXLmuP/97A5t2V/PYjJHEhWq90NO0GHdjNpthxbZyxiZri4oziQ0J4OlZoygur+OO99bqctrKJYmIN/AsMAUYAswSkYPnQHsEeM0YMwK4D/ibfXsdcLkxZihwBvCEiIT3TPLuFRsaQP/oIJYVlFsdRamj8nHuLuYtK+b6k/szMT3W6jgeSYtxN5ZfWkNFXTNjtV/c6RzXP4rbJqXz8dpdvPrjNqvjKHU0xgL5xpgCY0wTMB+YdtA+Q4Cv7a+/PfC+MWazMWaL/fVOYC/gsk+LjUmOZGVRhc43rlxOcVkdd7yXy6h+4dw2Kd3qOB5Li3E3pv3izu36Cf05bVAs93+ykVXFFVbHUepIxQPtl5YtsW9rbw1wgf31eUCIiPzXB5KIjAX8gK0OyulwY1IiqaxvZvNenW9cuY6mFhtz3spBBJ6aOQpfby0JraL/z7ux5YXl9A4NIDFS5wl1Rl5ewqMzRhIbEsCcN1exv67J6khKHYmOnvA6eGj4NuBkEVkFnAzsAFp+PoFIH+B14CpjTIcT8IvIdSKyUkRWlpaWdk/ybnagFXBFobaqKNfx0GebyC2p5KELR5AY2cvqOB5Ni3E3ZYxhWUEZY1MiEdGnop1VeC8/nrtkNKXVjdzyzhq9za1cSQmQ2O77BGBn+x2MMTuNMecbY0YBf7BvqwQQkVDgY+AuY8xPh/pDjDEvGGOyjDFZMTHO2cmSGBlIXKg/y7fpHS7lGhZtLuWlxYVcdnwSZwzrY3Ucj6fFuJsqKqtjb3Wj9ou7gJGJ4dw1dTDfbNrLPxa57J165XlWAGkikiIifsBMYGH7HUQkWkQO/Jy5E5hr3+4HfEDbw53v9mBmhxARxiRHsqKwXB/IVk6vvLaJ295dQ1psMH84a7DVcRRajLut5T/3i2sx7gouOz6Js0f25ZHP8/hJV/NTLsAY0wLMAT4HNgLvGGPWi8h9InKOfbeJQJ6IbAbigPvt22cAE4ArRWS1/SujZ6+ge41NiWR3VQMlFfVWR1HqkIwx3PFeLvvrmnliZgYBvt5WR1KAj9UBlGMsKywnMsiP1Nhgq6OoLhAR/nb+cNbvrORXb63i45tOJDZE53pVzs0Y8wnwyUHb7m73egGwoIPj3gDecHjAHjTG3je+vLBc+2+V03p7xXa+2LCHu84azNC+YVbHUXY6Mu6mlm8rY2yy9ou7kmB/H/5+SSbVDc3cqv3jSrmU9LgQQgN8WLFNH+JUzqmgtIZ7/7WBE1OjuXp8itVxVDsOLca7sDqbv4i8bX9/mYgkt3vvTvv2PBGZfATnfFpEahx1Ta5g5/56tpfXa7+4C0rvHcI9Zw/lhy37eH5RgdVxlFJd5OUlZCVHslyLceWEmltt/Obt1fj7evHI9JF46XL3TsVhxXgXV2e7BqgwxqQCjwMP2o8dQtvDQAdWZ3tORLw7O6eIZAEuuYpbdzowMqPFuGuaOSaRs0b04ZEv8sgu0tkZlHIVY5IjKSitZV9No9VRlPovT3y1mdySSh44fzi9w7QF0tk4cmS8K6uzTQNetb9eAJwmbX0V04D5xphGY0whkG8/3yHPaS/UHwZ+68BrcgnZRRUE+XkzqHeI1VHUUTjQP94nLICb3lpFZX2z1ZGUUl0wNiUCgJU6Oq6cyLKCMp77bisXZSXqNIZOypHFeFdWZ/t5H/uT+ZVA1GGOPdw55wALjTG7DhfKFRaQOFbZRRVk9AvHR1fTclmhAb48PWsUe6oauOO9XJ0uTSkXMDw+HH8fL5YX6h0t5Rwq65u55Z01JEX24u6zD25OUM7CkdVaV1ZnO9Q+R7RdRPoC04GnOwvlCgtIHIvaxhY27qois1+E1VHUMRrVL4LbJ6fz6brdzFtWbHUcpVQn/Hy8yEgM14c4ldO4+6N17K5q4ImZowjy1wn0nJUji/FOV2drv4+I+ABhQPlhjj3U9lFAKpAvItuAXiKS310X4krWbN+PzcDoJC3G3cEvT+rPhIEx3PfvDWzcVWV1HKVUJ8amRLJ+ZyW1jS1WR1EebuGanXy0eic3n55GRqLHP07n1BxZjHe6Opv9+yvsry8EvjFt9+MXAjPts62kAGnA8kOd0xjzsTGmtzEm2RiTDNTZHwr1ODnFbbdHRyVqMe4OvLyEx2aMJCzQl1+9tYq6Jv0Br5Qzy0yKwGbaBkaUssre6gbu/mgdo/qF838nD7A6juqEw4rxLq7O9jIQZR/FvgW4w37seuAdYAPwGTDbGNN6qHM66hpcUXZRBWmxwYT18rU6iuom0cH+PHFRBltLa7h34Qar4yilDmOUvUVQZ0JSVjHG8Pv311Lf1Moj00fq82MuwKENRF1Yna2Btl7vjo69n/8snXzYc3awj0cuO2mzGXKK9zNlWG+ro6huNj41mtkTU3nm23zGpUYxLePgZ6GVUs4gLNCXgXHBZBdrMa6s8V7ODr7auJc/Th3CgBiPLIdcjv665B8pI3wAACAASURBVEYK9tVQWd+s/eJu6jenp5GVFMEfPljHtn21VsdRSh1CZlIEOUUVuoqu6nG7Kuu591/rGZscyVXjkq2Oo7pIi3E3cuC2aKYW427Jx9uLJ2eNwkvg1/NX0dxqszqSUqoDmUmRVDW0sLXUoxeDVj3MGMNvF+TS0mp4ePoIXWXThWgx7kayiyoI7+VL/+ggq6MoB4kPD+SBC0awpqSSJ77abHUcpVQHDgyIaN+46knzV2znhy37+P2Zg0iK0jrAlWgx7kayiyrI7BdB2yKmyl2dObwPM7ISeO67rSzdWmZ1HKXUQZKjehEZ5KfFuOox28vr+Mu/NzA+NYpLjkuyOo46QlqMu4mK2ia2ltZqv7iHuOfsoSRHBXHLO6vZX9dkdRylVDsiwuh+EfoQp+oRNltbe4qI8NCFI7U9xQVpMe4mVm3XfnFPEuTvw5MzMyitbuTO99fSNj2/UspZZCZFUFBaS3mt/rKsHOv1n4pYWlDGH6cOJj480Oo46ihoMe4msosq8PYSRiboKlueYkRCOLdNTufTdbt5Z+V2q+Mopdo5MDCySkfHlQNt21fLA59u4pT0GGZkJXZ+gHJKWoy7ieyiCob0CSXQz9vqKKoHXXdSf8YNiOJPCzfozA1KOZERCWH4eIn2jSuHabUZbnt3Db7ewgMXjNDnxVyYFuNuoKXVxprtldqi4oG8vITHZmTg7+vFr+evoqlFpztUyhkE+HozND5Mi3HlMHMXF7KyqIJ7pw0lLjTA6jjqGGgx7gY27a6mvrlVH970UL3DAnjwghGs21HFo1/kWR1HKWWX2S+CNSX7dU0A1e3y91bz8Bd5/GJIHOfqiswuT4txN6CL/ajJQ3tz8XH9eH5RAYu37LM6jlKKts/khmYbG3dVWR1FuZGWVhu3vrOGID9v/nrecG1PcQNajLuB7KIKeocG0DdMb1N5sj+eNYQBMW3THeoMDkpZb3RS2wP12qqiutPziwpYU1LJn88dRkyIv9VxVDfQYtwNZBdVkJmki/14ukA/b56aNYr9dc38dkGuTneolMX6hAUSHx6oxbjqNpt2V/HEV5s5a0Qfpo7oa3Uc1U20GHdxuysb2LG/XvvFFQBD+4bx2zPS+WrjHuYtK7Y6jlIeb3RSBDlajKtu0GxvTwkL9OXP04ZZHUd1Iy3GXVxOsfaLq/929fgUJgyM4c//3sCWPdVWx1HKo2X2C2dnZQM799dbHUW5uGe+yWf9zir+et5wIoP8rI6jupEW4y4uu6gCfx8vhvQJtTqKchJeXsIj00cQ7O/DTfNX09DcanUk5QJExFtEzhGRm0TklgNfVudydZlJkcB/Bk6UOhrrdlTy7Lf5nD8qnklDe1sdR3UzLcZdXHZRBSMTwvHz0b9K9R+xIQE8PH0EG3dV8dBnOt2h6pJ/AVcCUUBIuy91DAb1CSHQ11v7xtVRa2xp5ZZ3VhMV7Mc9Zw+1Oo5yAB+rA6ij19DcyvqdlVx9YorVUZQTOnVQHFeckMTcJYVMGBjNxPRYqyMp55ZgjBlhdQh34+vtxchEXfxHHb0nvtrC5j01vHLVGMJ6+VodRzmADqe6sLU7KmluNWT2035x1bE7zxxMelwIt727htLqRqvjKOf2qYhMsjqEO8pMimDDzirqm7RlTB2ZnOIKnv9+KxdlJXKKDqi4LS3GXdiBJ/R1JhV1KAG+bdMdVjW0cPuCNTrdoTqcn4APRKReRKpEpFpEdLWabpCZFEGLzZBbst/qKMqFNDS3ctu7a+gTFshdUwdbHUc5kBbjLiy7qILkqF5EB+uk/+rQ0nuHcNdZg/kur5RXlmyzOo5yXo8CJwC9jDGhxpgQY4w+Gd4NRiW2DZhk60Oc6gg8/HkeBaW1PHThCEICtD3FnWkx7qKMMeQUV+iouOqSy45P4vTBsTzw6SY27NTBTtWhLcA6c4S3T0TkDBHJE5F8Ebmjg/eTRORrEckVke9EJKHde1eIyBb71xXdcA1OKSLIjwExQTrfuOqy5YXlzF1SyGXHJzE+NdrqOMrBtBh3UcXldeyradL5xVWXiAgPXTiS8F6+3DR/lfauqo7sAr4TkTu7OrWhiHgDzwJTgCHALBEZctBujwCv2R8OvQ/4m/3YSOAe4DhgLHCPiLjtB1pmUgTZRRXaKqY6VdfUwm3vriExohd3TBlkdRzVAzotxkXES0TW9UQY1XUHnszXYlx1VWSQH4/NyCB/bw1/+XiD1XGU8ykEvgb86PrUhmOBfGNMgTGmCZgPTDtonyH28wJ82+79ycCXxphyY0wF8CVwxjFfhZPKTIqgoq6Zwn21VkdRTu6BTzexvaKOR6aPJMhfJ73zBJ3+LRtjbCKyRkT6GWN0fW0nkV1UQYi/D2mxOg2w6roT06K5fkJ/nl9UwElpMZwxTBePUG2MMfcCiEhI27empguHxQPb231fQttId3trgAuAJ4HzgBARiTrEsfEH/wEich1wHUC/fv26dC3OaLR91qvsogr6xwRbnEY5qyX5+3htaRHXnJjC2JRIq+OoHtLVNpU+wHp739/CA1+dHdSFXkJ/EXnb/v4yEUlu996d9u15IjK5s3OKyMv2XxpyRWSBiLj1p112UQUZ/cLx9hKroygXc+ukdIbHh3HH+7nsqtQlulUbERkmIquAdbR93meLSGcrjHT0AXRwH8ZtwMn2c58M7ABaungsxpgXjDFZxpismJiYTq/DWQ2ICSY0wEdX4lSHVN3QzG8X5NI/OojbJ6dbHUf1oK4W4/cCU2nr93u03dchdbGX8BqgwhiTCjwOPGg/dggwExhK223L5+xLNR/unDcbY0ba+xKLgTldvDaXU93QTN6eam1RUUfFz8eLJ2dm0NRi45a319Bq0x5WBcALwC3GmCRjTBJwK/BiJ8eUAIntvk8AdrbfwRiz0xhzvjFmFPAH+7bKrhzrTry8hNH2vnGlOnL/xxvZVVnPIzNGEuDrbXUc1YO6VIwbY77v6KuTw7rSSzgNeNX+egFwmoiIfft8Y0yjMaYQyLef75DnNMZUAdiPD6SDERZ3sXr7foz5z21PpY5U/5hg/nTOUJYWlPH8oq1Wx1HOIcgY8+2Bb4wx3wFBnRyzAkgTkRQR8aNtEOW/7pqKSLSIHPhZcycw1/76c2CSiETYH9ycZN/mtjL7RbB5Tw2V9c1WR1FO5tu8vcxfsZ3rJgzQn+0e6LDF+IFFHzr46spiEF3pB/x5H2NMC1AJHK6X8LDnFJFXgN3AIODpQ1zTdSKyUkRWlpaWdnIJzim7qAIRyOgXbnUU5cKmZyZw1og+PPbFZlZv18VIFAUi8kcRSbZ/3UXbQ52HZP/cnkNbEb0ReMcYs15E7hORc+y7TQTyRGQzEAfcbz+2HPgzbQX9CuA++za3deBupv57U+1V1jVzx3u5DIwL5uZfpFkdR1ngsMX4gUUfOvjqymIQXekHPNQ+R7r9QN6rgL60/VC4qKNQ7tB/mF1UQXpcCKG6CIA6BiLCX88dTlxoADe9tYqaxharIylrXQ3EAO8B7wPRwJWdHWSM+cQYM9AYM8AYc6DQvtsYs9D+eoExJs2+z7XGmMZ2x841xqTav15xxEU5k5GJ4XgJ2qqi/su9/1rPvpomHp2egb+Ptqd4IkfOM96VfsCf9xERHyAMKD/MsV3pT2wF3qbt6X2302ozrC7ezyi9jaW6QVgvX56YmUFJRR13f6QzmHq4AbR9vnoBvsBpwCJLE7mZIH8fBvcJ1cV/1M++WL+b91ftYPYpqQxPCLM6jrKII4vxTnsJ7d8fWHXtQuAb++pvC4GZ9tlWUoA0YPmhziltUuHnnvGzgU0OvDbLbNlbTXVjC1n68KbqJmOSI/nVqWm8n7ODj1bvsDqOss482vq5z6ftgf2ptH2Wqm6UmRTBquIKfXBaUV7bxO8/WMuQPqHMOSXV6jjKQg4rxrvYS/gyECUi+cAtwB32Y9cD7wAbgM+A2caY1kOdk7b2lVdFZC2wlrapGO9z1LVZSRf7UY7wq1NTyUyK4K4P1rG9vM7qOMoapcaYfxljCo0xRQe+rA7lbkb3i6C2qZW83dVWR1EWu/ujdVTWN/PYRSPx89EF0T2ZQ5d2MsZ8Anxy0La7271uAKYf4tj7sT/o04Vz2oDx3RDZ6WUXVRAV5EdSVC+royg34uPtxRMXZXDmkz/w6/mreOf6E/Dx1h8OHuYeEXmJttUy2/d1v29dJPdzYCAlu7iCIX07e/RKuauPc3fx79xd3D45nUG99b8DT6c/bV1MTlEFo5MiaOvGUar7JEb24v7zh5NTvJ+nvt5idRzV864CMmhb2+Fs+9dUSxO5oYSIQGJC/LVv3IPtrW7grg/XMjIhjOsn9Lc6jnICDh0ZV91rX00j28rqmDXWdZeEVs7tnJF9WbS5lGe+zWd8ajTH9Y+yOpLqOSONMcOtDuHuRITMfrr4j6cyxvD799dR19TKozMy9A6kAnRk3KXkaL+46gF/Omco/SJ7cfPbq6ms08VJPMhPHaySrBwgMymC4vI69lY3WB1F9bD3cnbw1cY93D45ndTYYKvjKCehxbgLyS6uwNdbGBav0x8pxwn29+HJmaPYW93InR/k0jbBkfIAJwKrRSRPRHJFZK2I5Fodyh2Ntg+o5BTp4j+eZOf+eu7913rGJkdy1fgUq+MoJ6LFuAvJKapgWHwYAb66KIByrJGJ4dw6KZ1P1u7m7RXbOz9AuYMzaJtGdhL/6RfXqQ0dYFh8KH7eXqwq1lYVT2GM4Xfv5dJqMzw8fQTeXvrcl/oPLcZdRFOLjTUllWTqYj+qh1w/oT/jU6P407/Ws3mPTsPm7tpPZ6hTGzqWv483wxPCtG/cg8xbVswPW/bx+zMHkxQVZHUc5WS0GHcR63dW0tRi035x1WO8vITHL8og2N+H2fNyqGtqsTqSUm4jMymC3B2VNLa0Wh1FOVhRWS1//WQjJ6VFc8lxOgGD+l9ajLuIAyMoo7UYVz0oNiSAJy4aRX5pDfd8tN7qOEq5jdH9wmlqsbF+Z5XVUZQDtdoMt7+bi7eX8OAFI3RaYtUhLcZdRE5xBQkRgcSFBlgdRXmYE9OimXNKKu9ml/B+TonVcZRyC6P7HXiIU1tV3NkrSwpZvq2ce84eSt/wQKvjKCelxbgLMMaQXVShLSrKMr8+LY2xKZHc9eE68vfWWB1HKZcXGxpAYmSg9o27sfy91Tz0eR6nD47jgtHxVsdRTkyLcRewY389e6oatRhXlvHx9uKpmaMI8PVmzps5NDRrn6tSxyqzXwQriyp0+lA31NJq45Z31hDk581fzx+m7SnqsLQYdwE/94vrTCrKQr3DAnhsxkg27a7m3n9tsDqOUi4vMymC0upGSirqrY6iutnfv9tKbkklfzl3OLEh2l6qDk+LcReQU1RBLz9vBvUOsTqK8nAT02P5v5MH8NbyYhau2Wl1HKVc2s+L/+h8425l/c5Knvx6C2eP7MtZI/pYHUe5AC3GXUB2cQUZieH4eOtfl7LerZMGkpkUwZ3v5VK4r9bqOEq5rPS4EIL8vLVv3I00trRy6ztriAjy475zhlodR7kIre6cXG1jCxt3VWu/uHIavt5ePD1rFL4+Xsyep/3jSh0tH28vMvqFazHuRh79YjObdlfzwPnDiQjyszqOchFajDu5NSX7abUZnV9cOZW+4YE8On0kG3ZV8ZePtX9cqaOV2S+CTburqW3URbVc3Y9b9/HiDwVcfFw/ThscZ3Uc5UK0GHdyOfrwpnJSpw2O47oJ/Xnjp2I+XLXD6jhKuaRRSRG02gxrSvZbHUUdg8r6Zm57Zw3JUUHcddZgq+MoF6PFuJNbWVRBWmwwYYG+VkdR6n/8dnI6Y1MiueP9XDbt1pUElTpSoxN18R93cPdH69hT3cjjF2XQy8/H6jjKxWgx7sRabYbsbRVkJUdaHUWpDvl4e/HMrFGEBPhywxs5VDU0Wx1JKZcS1suXtNhg7Rt3YQvX7OSj1Tu56dQ0MhLDrY6jXJAW405s0+4qqhtbGJuiLSrKecWGBvDsxaMpLq/j9nfX6AImSh2hzKQIcor3Y7Ppvx1Xs3N/PXd9sJZR/cKZfcoAq+MoF6XFuBNbUVgOwBgdGVdObmxKJHdOGcTn6/fwwqICq+Mo5VJGJ0VQWd9Mwb4aq6OoI2CzGW57dw0tNsPjMzJ0+mF11PS/HCe2YlsFfcMCSIjoZXUUpTp1zYkpnDm8Nw9+tomlW8usjqOUyzgwde3Kbdqq4krmLinkx61l3D11CMnRQVbHUS5Mi3EnZYxh+bZyxqToqLhyDSLCQxeOJDk6iF+9lcPuygarIynlEvpHBxEV5MfybeVWR1FdtGl3FQ99lsfpg+O4aEyi1XGUi9Ni3EkVldVRWt2oLSrKpQT7+/D8pZnUNbUy+80cmlttVkdSyumJCGNTIlleqMW4K2hsaeU381cTGujDAxcMR0SsjqRcnBbjTurACMlYHRlXLiYtLoQHLhhBdlEFf/63LgikVFeMTYmkpKKeHfvrrY6iOvHQZ3ls2l3NgxeMIDrY3+o4yg04tBgXkTNEJE9E8kXkjg7e9xeRt+3vLxOR5Hbv3Wnfnicikzs7p4jMs29fJyJzRcSlJ+ZeUVhORC9fUmOCrY6i1BE7Z2Rfrj0xhdeWFjF/ebHVcZRyegcGXpYX6vMWzuzbTXt5eXEhV5yQpKtsqm7jsGJcRLyBZ4EpwBBglogMOWi3a4AKY0wq8DjwoP3YIcBMYChwBvCciHh3cs55wCBgOBAIXOuoa+sJK7aVk5UciZeX3v5SrumOKYM4KS2aP360juwivf3urrow6NJPRL4VkVUikisiZ9q3+4rIqyKyVkQ2isidPZ/eeQzqHUpIgI+2qjixvVUN3PruGgb1DuHOM3WVTdV9HDkyPhbIN8YUGGOagPnAtIP2mQa8an+9ADhN2pqvpgHzjTGNxphCIN9+vkOe0xjzibEDlgMJDrw2h9pb3cC2sjrGar+4cmFtCwKNJj48kOtfz2FXpd5+dzddHHS5C3jHGDOKtkGW5+zbpwP+xpjhQCZwffu7o57G20sYmxzJMi3GnZLNZrj5ndXUNbXwzMWjCPD1tjqSciOOLMbjge3tvi+xb+twH2NMC1AJRB3m2E7PaW9PuQz47JivwCIrCtumt9KZVJSrC+vly4uXZ1Hf1ML1r2fT0NxqdSTVvboy6GKAUPvrMGBnu+1BIuJD293MJqDK8ZGd19iUSApKaymtbrQ6ijrI84sKWJJfxp/OHkpqbIjVcZSbcWQx3lF/xcHLix1qnyPd3t5zwCJjzA8dhhK5TkRWisjK0tLSjnax3Ipt5QT6ejO0b2jnOyvl5NLiQnhi5ihySyq58/21ukKne+nKoMufgEtFpAT4BPiVffsCoBbYBRQDjxhjPHpY+EDf+Aqd4tCprCqu4NEv8jhreB+dxlA5hCOL8RKg/X+1CfxnROR/9rGPjoQB5Yc59rDnFJF7gBjglkOFMsa8YIzJMsZkxcTEHOEl9YzlheWMTgrHV1fzUm7iF0PiuPUXA/lg1Q5e+qHQ6jiq+3RlgGQW8E9jTAJwJvC6iHjRNqreCvQFUoBbRaT///wBLjCA0l2GxYcR6OutfeNOpKqhmZvmryIuNIC/nq/TGCrHcGS1twJIE5EUEfGjrVdw4UH7LASusL++EPjG3vO9EJhpn20lBUijrQ/8kOcUkWuBycAsY4zLTm5c1dDMxt1VOr+4cjtzTk3lrOF9+NunG/k2b6/VcVT36MqgyzXAOwDGmKVAABANXAx8ZoxpNsbsBZYAWQf/Aa4wgNJdfL29yEyK0L5xJ2GM4a4P1rFzfwNPzcogLNClJ2lTTsxhxbi9B3wO8DmwkbYHeNaLyH0ico59t5eBKBHJp200+w77setp+/DeQFvv92xjTOuhzmk/1z+AOGCpiKwWkbsddW2OtKKwHGPQhzeV2xERHp4+gkG9Q/nVm6vYtNuj24PdRVcGXYqB0wBEZDBtxXipffup0iYIOB7Y1GPJndTYlEg27a6isq7Z6igeb0F2CQvX7OTm09PITNKfycpxfBx5cmPMJ7T1CLbfdne71w20PVHf0bH3A/d35Zz27Q69lp6ydGsZfj5ejE6KsDqKUt2ul58PL1+ZxbnPLuHqV1bw4ezxxIYGWB1LHSVjTIuIHBgg8QbmHhh0AVYaYxYCtwIvisjNtLWwXGmMMSLyLPAKsI62dpdXjDG51lyJ8zguJRJj2vrGTx+i81hbZfOeau7+aD3H94/khompVsdRbs4tClh3srSgjNH9wnXaJOW2+oQF8vIVY5jx/FKufW0l8687nl5++lHkqrow6LIBGN/BcTUcYjDGk41MDMfP24vlWoxbpraxhRveyCbI34enZo7CW9f7UA6mTwg6kf11TWzYVcUJ/aOtjqKUQw2LD+PpWaNYt6OS38xfTatNZ1hRCiDA15uMxHDtG7eIMYbff7CWwn21PDUrQ+/cqR6hxbgT+amgrV98XGqU1VGUcrjTBsfxx6lD+GLDHh74dKPVcZRyGsf1j2TdjkqqG7RvvKe9ubyYj1bv5ObTBzJugA6MqZ6hxbgT+amgjEBfb0YmhFsdRakecdX4FK4cl8yLPxTy+k9FVsdRyimcMCCKVpvRKQ572Lodldy7cAMTBsYw+xTtE1c9R4txJ7J0axlZyRH4+ehfi/Icf5w6hNMGxXLPR+v4Yv1uq+MoZbnR/SLw9/Ficf4+q6N4jMr6Zm6cl0NUsB9PXJSBl/aJqx6kVZ+T2FfTSN6eao7vry0qyrN4ewlPzRrF8IRwfvXWKh0NVB4vwNebMcmR/JhfZnUUj2CM4fZ317Bzfz3PXDyKyCA/qyMpD6PFuJP4qaDtQ3fcAC3GlecJ8vfhlSvHEB8RyDWvrtA5yJXHG5caRd6eakqrG62O4vZe/KGALzbs4Y4pg3Q+cWUJLcadxNKtZQT7+zA8PszqKEpZIjLIj9euHksvP28uf3k528vrrI6klGXG2x8e/HGrtqo40uIt+3jg001MGdaba05MsTqO8lBajDuJpQVljEmOwMdb/0qU50qI6MVrVx9HQ3MrV8xdTlmNjgoqzzQsPozQAB9tVXGg7eV1zHkrh9TYYB6ZPhIR7RNX1tDKzwnsrmygoLSWE7RFRSnSe4fw0hVj2LG/nqv+uUKnd1MeydtLOL5/FEt0ZNwh6ptauf71bGw2wwuXZRHkrwuPKetoMe4EfthSCsBJaTEWJ1HKOYxNieSZi0ezYWcVV/9zBXVNLVZHUqrHnZgWTUlFPcVl2rLVnYwx3PF+Lht3V/HkrFEkRwdZHUl5OC3GncCiLfuICfFnUO8Qq6Mo5TR+MSSOJ2ZmkF1UwbWvrqShudXqSEr1qAOLzujoePd6eXEhH63eyW2T0jklPdbqOEppMW61Vpth8ZZSTkqL1n41pQ4ydURfHp0xkqUFZVz/ejaNLVqQK88xICaIuFB/luh8493mx/x9/O3TTZwxtDc3ThxgdRylAC3GLbd+ZyUVdc2cPFBbVJTqyHmjEvjbecP5fnMpc95cRXOrzepISvUIEWH8gGh+3FqGzWasjuPyCvfVcuObOfSPDuKRGfrApnIeWoxbbNHmtn7x8anRFidRynnNHNuP+6YN5csNe/j1fC3Ilec4OT2G8tomcndUWh3FpVXWNXPNP1cgwMtXjCFYH9hUTkSLcYst2ryPYfGhRAf7Wx1FKad2+QnJ3HXWYD5Zu5sb5+Voy4ryCBPSYhCB7/L2Wh3FZTW32rhhXjYlFfW8cHkW/aJ6WR1Jqf+ixbiFqhuaySmu0FlUlOqia0/q//MI+S9fy6a+SQty5d4igvzISAzn27xSq6O4JGMMd3+0nh+3lvG384czJllX2FTOR4txCy3dWkaLzTBBi3GluuzyE5J56IIR/LCllKv+uZzaRp32ULm3iQNjyS3Zr4tgHYWXFxfy1vJibpw4gAsyE6yOo1SHtBi30KItpfTy8yYzKcLqKEq5lBljEnl8RgYrtlVw+dzlVNbrwkDKfU1Mj8EY+GGLzqpyJL7asIf7P9nIGUN7c9ukdKvjKHVIWoxbxBjDNxv3Mj41Gj8f/WtQ6kidOyqeZ2aNIrdkPxc9v5TdlQ1WR1LKIYbHhxEV5Me32jfeZauKK5jzVg7D+obx2EUj8fLSmVOU89Iq0CIbdlWxs7KBXwyOszqKUi5ryvA+vHLlWLaX13H+c0vYsqfa6khKdTsvL+HkgTEs2lxKq05x2KmC0hqu/ucK4kIDmHvlGHr56cwpyrlpMW6RrzfuRQROGaSrfyl1LE5Mi+bt60+gqdVw4T+WsmJbudWRlOp2J6fHUFHXzJqS/VZHcWp7qxu4fO5yvER49aqxxIToTGXK+WkxbpGvN+5hZEK4flAo1Q2GxYfxwY3jiAry49KXlvHp2l1WR1KqW01Ii8FL4NtN2qpyKNUNzVw5dwVlNU3MvXIMydFBVkdSqku0GLfA3qoG1pRUcvpgHRVXqrskRvZiwQ3jGNI3lBvm5fDU11swRm/pK/cQEeRHVnIkX6zfY3UUp9TUYuOGN3LI21PNc5eOZmRiuNWRlOoyLcYt8I19ZOM07RdXqltFBvnx1i+P57xR8Tz25WbmvLlK5yJXbmPy0N7k7alm275aq6M4lZZWG7+ev4rF+ft44PzhnJKuA13KtWgxboGvNu4lPjyQQb1DrI6ilNsJ8PXmsRkjuXPKID5Zt4sL//EjO/fXWx1LqWM2eWjbAM7n63dbnMR52GyG2xfk8um63fxx6hCmZyVaHUmpI+bQYlxEzhCRPBHJF5E7OnjfX0Tetr+/TESS2713p317nohM7uycIjLHvs2ISLQjr+tY1Da2sDi/lNMHxyKiUy0p5QgiwvUnD2DuFWMoLqvj7KcXs2izrmCoXFtCRC+GxYdqMW5njOEPH67jg1U7uG3SQK45McXqSEodFYcV4yLiDTwLTAGGALNEZMhBu10DsIcwZAAAFRVJREFUVBhjUoHHgQftxw4BZgJDgTOA50TEu5NzLgFOB4ocdU3d4ZtNe2lotjFleB+royjl9k4ZFMsHs8cTHezPFa8s55HP82hptVkdS6mjNnlIb3KK97OnyrPn1TfG8JePN/68uuacU9OsjqTUUXPkyPhYIN8YU2CMaQLmA9MO2mca8Kr99QLgNGkbLp4GzDfGNBpjCoF8+/kOeU5jzCpjzDYHXk+3+GTtLqKD/RmTHGl1FKU8QmpsMB/OHs+MzESe+TafS15a5vGFjHJdk4f1BuCLDZ77IKcxhke+yOPlxYVcOS6Z2yfr6prKtTmyGI8Htrf7vsS+rcN9jDEtQCUQdZhju3LOwxKR60RkpYisLC3t2dvWtY0tfJu3lzOH98ZbVwNTqscE+nnz4IUjeHT6SHJLKjnjiUV8nKvTHyrXkxYbTEp0EJ//f3t3Hl5Vda9x/PvLzBhmCDIFCCAgCqQgBVEBEfBa5ArPRax6K15ai1ZtvRarT9tL5bb2Wm29da7WEYFSFaoIchXBpyAQ5jBDpBAGwxhQCGRY94+zo2lMQhKys3eS9/M858k+6+zhPeucs1jsYe30unmqinOO37y/jaeW7OamAe35xfU9dcqn1Hh+dsZL+nUUH2estHkqWl5uzrnnnXOpzrnUli1bVmTRC1Z4isoYnaIiEogb+7fjb3cPoX2z+kyduZa731zH8S/PBR1LpNzMjOsuSWL57iNknapbR3icc0x/dwvPLcvglss7MuOGS9QRl1rBz854JlD0suZ2wIHS5jGzGCAROFbGsuVZZ2jpFBWR4HVt1ZC37vw2P7mmG+9vOsjI3y/jA10QJzXI2MvaUuDg3Q115+hOQYHj4XfS+fPf9zB5SDLTx/YiSkeYpZbwszO+Gkgxs2QziyNyQeb8YvPMB27zpscDH7nIXTrmAxO90VaSgRRgVTnXGUqncnL5aFsWo3vrFBWRoMVER3H38BTm3TWY5g3imPLaGu54ZTX7jp0OOlqNU45RszqY2RIzW2dmG81sTJHX+pjZCjPbbGabzCyhetPXTCmtG9EzqTHzNtSYfVEXJC+/gGlvbeSNlXu586ouPHzdxdojLrWKb51x7xzwu4BFwFZgjnNus5lNN7PveLO9CDQ3s13Aj4Fp3rKbgTnAFmAhMNU5l1/aOgHM7Edmlklkb/lGM/uTX++tMhZsOsjZvALG9avQKe4i4qNebRP5291D+NmYHizffZQRjy/ljx/t5GyebhRUHuUcNethIm11XyI7UJ72lo0BXgd+4JzrBVwF5FZT9Brvhr5t2bDvBJ/V8hsA5eTm84PX1zInLZN7hqfwwLXd1RGXWifGz5U75xYAC4qV/bzIdA4woZRlZwAzyrNOr/xJ4MkLjOybv67ZT+cWDeirW/SKhEpsdBRThnbh+kvb8qt3t/DYBzt4c9U+7rumG+P6XqQjWWX7aoQrADMrHOFqS5F5HNDYm07k61MLRwIbnXMbAJxzR6slcS1x/aVt+fX725i3fj/3jugWdBxfnDh9jsmvpLF273F+NbYXtwzqFHQkEV/oDpzVYO/R06zac4wb+7fT/+hFQiopsR5P39yfmXcMpFmDOO7/ywbG/OET/m/L50TOnpMSlGeEq18C3/WOXC4A7vbKuwHOzBaZ2Voze6CkDQQ5AlaYJSXWY2ByM95Zt79Wfj8PnDjDhGdXsCkzm6cm9VNHXGo1dcarwV/XZmIG4/rqFBWRsPt21xbMmzqYP07qy9m8fO54NY3Rf/iEd9bt1w2Dvqk8I1zdBLzsnGsHjAFeM7MoIkdmhwA3e3/Hmdnwb6wswBGwwm58//bsOXqaTzOOBR2lSm3KzOZfn17OoewcXrl9gEYgk1pPnXGf5Rc45q7JZHCXFrRtUi/oOCJSDlFRxr/0acviH1/JYxMuJb/Ace/s9Vz5Px/zwrIMjn5xNuiIYVGeEa4mE7kGCOfcCiABaOEtu9Q5d8Q5d5rIXvN+vieuRa67JIlGCTG8uWpv0FGqzHsbDzLhueVERxlzfjCIQV2aBx1JxHfqjPvs4+1Z7D9xhpsGdAg6iohUUGx0FOP7t2PRvUP5062pXNSkHjMWbOXyX3/I1JlrWbbjcF3fW16eEa72AsMBzOxiIp3xw0QuxO9jZvW9izmv5J/PNZfzqBcXzY392rEw/RDHavh4+QUFjscX72DqzLX0bpvIvLsGc3FS4/MvKFIL+HoBp8CrK/5B68bxjOzVOugoIlJJUVHGiJ6tGdGzNTs+P8WsVft4a10m7208SNP6sVzTszWjeycxqEtzEmKjg45bbZxzeWZWOMJVNPBS4ahZQJpzbj7wE+AFM7uPyCks/+4NYXvczB4n0qF3wALn3HvBvJOaa9LADry8fA9z1+xjytAuQceplJM5uTzwl40s3HyICf3b8ci43sTH1J3fkYjVxgs/yis1NdWlpaX5tv7PjnzJ1Y99zH0junHPiBTftiMi1S8nN5+Ptx9mYfpBPtyaxamzecTFRNG3fRMGdWlOasdm9GzbmGYN4nzZvpmtcc6l+rLykPK7za6pxj+znKxTZ1ly/1U1bvSf9P3ZTJ25lszjZ3hwdA8mD0nWQAdSK5XVZmvPuI9eXbGH2GjjpoHtzzuviNQsCbHRjOrdhlG923A2L5/lu4+yfNcRVmQc5Q8f7qRwP0frxvF0b9OYdk3r0TYxgaTEejSuF0uDuGjqx8cQE2V0a92IuBidNSiVM3lIMne+sZZFmw/VmIsdnXO8sXIv0/+2hWYN4pg95XJSdXdqqaPUGffJsS/PMWvVPq7v05ZWjXRTOZHaLD4mmqu7t+Lq7q0AyD6dy6b92Ww9eJKtB0+yI+sU6fuzSz2vd+XPhtO6sdoJqZyRvdrQqXl9nlu6m9G924R+z/LRL87y8DvpvJ9+iCu7teSJf7vMtyNIIjWBOuM++fPfPyMnL58fXl0zz+ETkcpLrB/LkJQWDElp8U/lObn5HMrO4VROHqfP5fHluTxy8x2J9WIDSiq1QXSU8R9DO/PQ2+l8mnEs1COQLN7yOQ++tZGTZ/KYNroHU67oTFQNO7VGpKqpM+6Dkzm5vLx8D6N6taFrq0ZBxxGRkEiIjaZTiwZBx5Ba6MZ+7Xhi8Q6e/HAnl3duFrq94ydOn+OR97Yyd00mPZMa8/odl9KjjUZLEQENbeiL55dmcConj6lXdw06ioiI1AEJsdHcdXVXVmQcZdnOI0HH+UpBgWNO2j6G/W4pb6/bz93DuvLO1MHqiIsUoT3jVezAiTO88EkG37m0Lb0vSgw6joiI1BGTBnbkxb9/xm/e38YVXVsEfvrHlgMn+cX8dFbvOU5qx6Y8Mq63OuEiJdCe8Sr26MJtOOCBUd2DjiIiInVIXEwU94/sztaDJ5mdti+wHPuOnea+2eu57n8/YVfWF/x2fB/mfH+QOuIipdCe8Sq0ZHsW89Yf4O5hXWnXtH7QcUREpI65vk9bZq7cy38v2MqwHq2qdZSeg9lneG5pBjNX7sUMvj+0C3de2YXE+rpAWaQs6oxXkewzufzsrU10a92Qu4bpXHEREal+UVHGb27sw6jfL+Ohtzfxwq2pvl/MuSvrFM8tzeCd9fspcDC+XzvuvSaFpMR6vm5XpLZQZ7wK5Bc47pm1jiNfnOXZ7/bXbXxFRCQwyS0a8NNRPZj+7haeWbqbH15V9TuIzuUV8MGWQ8xevY9Pdh4hITaKSQM6cMcVnWnfTEeGRSpCnfEL5JzjV+9u4ePth3nkht5c2r5J0JFERKSO+97gTqzde5zHFm2nY7MGXNfnwu/MmV/gWL3nGAvTDzF/wwGOfXmOi5rU48fXdOPmgR1o3jC+CpKL1D3qjF+Ac3kFTH93M69/upfbBydz88AOQUcSERHBzPjt+D4cys7hR7PWcfpcHhNS21d4PYeyc/g04yjLdx/ho21ZHPniHHExUQzr3oqJA9pzRUpLonXTHpELos54JeQXOJbvPsKjC7eRvv8k3x/amWmje4TuJgsiIlJ31Y+L4eXbBzDl1TT+c+5GPt5xmB8NS6Fb64b/9O9VQYHj2OlzHMrOIfP4GbYfOsW2QyfZfOAke4+dBqBxQgxDu7VkdO8krurekgbx6j6IVBX9miog62QOt7y4igMnznDqbB5JiQk8c3M/Rl9y4Yf/REREqlrD+BhevX0ATy3ZzbNLd/PexoO0aBhPo4QYcnLzOZObz5dn88jNd18tYwadmjegZ1Jjbh3Ukcs7N+fipMbaAy7iE3XGK6BxvVg6NK/PgORmfCu5GSN7tiYhVhdriohIeMVER3HPiBQmDezAwvSDpO8/yencfBJioqgXF039uBiSEhNok5hAUmICXVs1pH6cugci1UW/tgpIiI3mhVtTg44hIiJSYS0bxXPLoE5BxxCRYnQHThERERGRgKgzLiIiIiISEHXGRUREREQC4mtn3MxGmdl2M9tlZtNKeD3ezGZ7r680s05FXnvQK99uZteeb51mluytY6e3zjg/35uIiIiIyIXyrTNuZtHAU8BooCdwk5n1LDbbZOC4c64r8ATwqLdsT2Ai0AsYBTxtZtHnWeejwBPOuRTguLduEREREZHQ8nPP+ABgl3Muwzl3DpgFjC02z1jgFW96LjDcInciGAvMcs6ddc59Buzy1lfiOr1lhnnrwFvnDT6+NxERERGRC+ZnZ/wiYF+R55leWYnzOOfygGygeRnLllbeHDjhraO0bQFgZlPMLM3M0g4fPlyJtyUiIiIiUjX87IyXdKsuV855qqr8m4XOPe+cS3XOpbZs2bKkWUREREREqoWfN/3JBNoXed4OOFDKPJlmFgMkAsfOs2xJ5UeAJmYW4+0dL2lb37BmzZojZvaPcr+jr7XwthlGYc4G4c4X5mwQ7nxhzgbhzlfZbB2rOkjYqc0OhPJVXpizQbjzhTkbVC5fqW22n53x1UCKmSUD+4lckDmp2DzzgduAFcB44CPnnDOz+cBMM3scaAukAKuI7AH/xjq9ZZZ465jlrXPe+QI65yq1a9zM0pxzobwVZ5izQbjzhTkbhDtfmLNBuPOFOVvYqM2ufspXeWHOBuHOF+ZsUPX5fOuMO+fyzOwuYBEQDbzknNtsZtOBNOfcfOBF4DUz20Vkj/hEb9nNZjYH2ALkAVOdc/kAJa3T2+RPgVlm9giwzlu3iIiIiEho+blnHOfcAmBBsbKfF5nOASaUsuwMYEZ51umVZxAZbUVEREREpEbQHTgr5/mgA5QhzNkg3PnCnA3CnS/M2SDc+cKcrbYIcx2HORso34UIczYId74wZ4MqzmfOlTjoiIiIiIiI+Ex7xkVEREREAqLOuIiIiIhIQNQZrwAzG2Vm281sl5lNCzDHHjPbZGbrzSzNK2tmZovNbKf3t6lXbmb2pJd5o5n1q+IsL5lZlpmlFymrcBYzu82bf6eZ3eZzvl+a2X6v/tab2Zgirz3o5dtuZtcWKa/yz97M2pvZEjPbamabzewerzzw+isjW1jqLsHMVpnZBi/ff3nlyWa20quH2WYW55XHe893ea93Ol9un/K9bGafFam/y7zyav9t1AV+fPcqmSM0bba3jdC226VkC0u7E9o2+zz5Aq+/MtrEZFObDc45PcrxIDKU4m6gMxAHbAB6BpRlD9CiWNlvgWne9DTgUW96DPA+kTHaLwdWVnGWoUA/IL2yWYBmQIb3t6k33dTHfL8E7i9h3p7e5xoPJHufd7Rfnz2QBPTzphsBO7wMgddfGdnCUncGNPSmY4GVXp3MASZ65c8Cd3rTPwSe9aYnArPLyu1jvpeB8SXMX+2/jdr+8Ou7V8ksewhJm+1tI7TtdinZwtLuhLbNPk++wOsPtdllbl97xstvALDLOZfhnDtH5OZCYwPOVNRY4BVv+hXghiLlr7qIT4ncqTSpqjbqnFtGZIz4C8lyLbDYOXfMOXccWAyM8jFfacYCs5xzZ51znwG7iHzuvnz2zrmDzrm13vQpYCtwESGovzKylaa66845577wnsZ6DwcMA+Z65cXrrrBO5wLDzczKyO1XvtJU+2+jDlCbXYowt9tqs33JV5pqqz+12WVTZ7z8LgL2FXmeSdlfcj854AMzW2NmU7yy1s65gxD5QQKtvPIgclc0SxAZ7/IOLb1UeEgxyHzeIbi+RP43Hqr6K5YNQlJ3ZhZtZuuBLCIN3m7ghHMur4RtfZXDez0baF6d+ZxzhfU3w6u/J8wsvni+YjnC1O7UNGGqu7C32ZXJU905Q9HuFApzm11CPghB/anNLp064+VnJZQFNS7kYOdcP2A0MNXMhpYxb5hyl5alujM+A3QBLgMOAr/zygPJZ2YNgb8C9zrnTpY1ayk5fMtXQrbQ1J1zLt85dxnQjsiekYvL2Fbg+cysN/Ag0AP4FpHDmD8NKl8dEKa6q6ltNoTjuxmadgfC3WZDeNtttdmlU2e8/DKB9kWetwMOBBHEOXfA+5sFvE3kS/154aFM72+WN3sQuSuapVozOuc+9350BcALfH2Iq9rzmVkskUbzDefcW15xKOqvpGxhqrtCzrkTwMdEzttrYmaFdxYuuq2vcnivJxI5FF6d+UZ5h5Gdc+4s8GdCUH+1WGjqrga02VQiT7XlDFO7E+Y2u7R8Yao/L4/a7GLUGS+/1UCKd+VvHJELCuZXdwgza2BmjQqngZFAupel8Krd24B53vR84Fbvyt/LgezCw2k+qmiWRcBIM2vqHT4b6ZX5otj5l+OI1F9hvoneVdzJQAqwCp8+e+/8txeBrc65x4u8FHj9lZYtRHXX0syaeNP1gBFEzo9cAoz3Zited4V1Oh74yDnnysjtR75tRf7BNiLnRhatv8B/G7WM2uyKCbzdKU2I2p3Qttll5QtD/anNPg93gVeg1qUHkatndxA5z+mhgDJ0JnIl8QZgc2EOIudSfQjs9P42c19fIfyUl3kTkFrFed4kctgrl8j/CCdXJgtwO5ELMXYB3/M532ve9jd6P6ikIvM/5OXbDoz287MHhhA5fLURWO89xoSh/srIFpa66wOs83KkAz8v8vtY5dXDX4B4rzzBe77Le73z+XL7lO8jr/7Sgdf5+ur9av9t1IWHH9+9SmQIVZvtbSO07XYp2cLS7oS2zT5PvsDrD7XZZT7MW1BERERERKqZTlMREREREQmIOuMiIiIiIgFRZ1xEREREJCDqjIuIiIiIBESdcRERERGRgKgzLiIiIiISEHXGRUREREQC8v94vltMy5yDOgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_sched()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now lets generate some Shakespeare!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "O God, O God! if my deping  power.\n",
      "\n",
      "RIVERS:\n",
      "Come, Grey, come, Vaughan, let us all embrace:\n",
      "A to the beastless dearth\n",
      "That ever enter'd in a drowsy head,\n",
      "Have I lu die with my lords o' the field? sweet flower now,\n",
      "Lest that thy love prove likewise variable.\n",
      "\n",
      "ROMEO:\n",
      "What shall I swear by? I long as I shall shake, them thus:\n",
      "Give me some breath, some little pause, my lord\n",
      "Before I positively herein:\n",
      "I will resolve your grace immediately.\n",
      "\n",
      "CATESBY:\n",
      "\n",
      "KING RICHARD III:\n",
      "Should dying men flatter with those that live?\n",
      "\n",
      "JOHN OF GAUNT:\n",
      "What, talking with a priest, lord chamberlain?\n",
      "Your friends at Pomfret, they do need the priest;\n",
      "You must hear me, are you mad?\n",
      "\n",
      "POMPEY:\n",
      "Proof?\n",
      "\n",
      "ABHORSON:\n",
      "Every true: all the lord protector of the king,\n",
      "With powerful policy strengthen themselves,\n",
      "And trust not simple Henry nor his oaths.\n",
      "\n",
      "MONTAGUE:\n",
      "Brother, I go; I'll win it out of doors.\n",
      "\n",
      "ANGELO:\n",
      "Now, what's the matter. Provost?\n",
      "\n",
      "Provost:\n",
      "Is it your will Claudio?\n",
      "\n",
      "ISABELLA:\n",
      "I am a woeful suitor to your honour!\n",
      "\n",
      "LUCIO:\n",
      "\n",
      "ISABELLA:\n",
      "Must he needs die?\n",
      "\n",
      "CLARENCE:\n",
      "That's he in his great affairs; but I will attend you awhile.\n",
      "\n",
      "DUKE VINCENTIO:\n",
      "\n",
      "Provost:\n",
      "I told you. Lord Angelo, belike thinking me remiss\n",
      "in mine office, awakens me with this unwonted\n",
      "puttting-on; methinks strangely, for he hath not used it before.\n",
      "\n",
      "DUKE VINCENTIO:\n",
      "Peace be with you!\n",
      "He who the sword of heaven will bear\n",
      "Should be as holy as severe;\n",
      "Pattern in her reputation through the world,\n",
      "Is now leased out, I die pronouncing it,\n",
      "Like to a tenement or pelting farm:\n",
      "England, bound in with the true shepherd's shame.\n",
      "\n",
      "DUKE OF AUMERLE:\n",
      "To see, madam, go comfort, good father; if\n",
      "not, he's not meet her.\n",
      "\n",
      "BRUTUS:\n",
      "Wherein our companions, no replying, yielded\n",
      "To the death--all the voices the storm,\n",
      "Which we have noted in your brother dubb'd them gentlewomen.\n",
      "Are mighty gossips in this monarchy.\n",
      "\n",
      "BRAKENBURY:\n",
      "I beseech your graces both to pardon me;\n",
      "His majesty hath straitly given in charge\n",
      "That no man shall have private conference,\n",
      "Of what degree soever, wit\n"
     ]
    }
   ],
   "source": [
    "from minGPT.mingpt.utils import sample\n",
    "\n",
    "context = \"O God, O God!\"\n",
    "x = torch.tensor([dls.char_transform.stoi[s] for s in context], dtype=torch.long)[None,...].to(dls.device)\n",
    "y = sample(model, x, 2000, temperature=0.9, sample=True, top_k=5)[0]\n",
    "completion = ''.join([dls.char_transform.itos[int(i)] for i in y])\n",
    "print(completion)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}